This post is about the engineering: how the app uses Meta's Device Access Toolkit (DAT), and how I got a hands-free "Live Scan" mode working without sending every camera frame to the cloud.

What DAT gives you

Meta's Wearables Device Access Toolkit lets an Android app open a session with supported glasses and get a camera stream. Per the docs, the stream comes in three sizes (360x640, 504x896, 720x1280) at 2 to 30 fps. The link is Bluetooth, so when bandwidth is short the SDK lowers resolution first and frame rate second. Frames are compressed, so they can look soft. You also get device state: battery, worn/not worn, hinge, and a thermal level.

Two practical limits shaped the design: one session per device, and a session ends when the hinges close or the glasses come off. The app has to start a new one.

The button flow

The first version is simple. Press the glasses' capture button, grab a photo, optionally add a spoken note, identify the item, look up prices, and read the answer through the glasses' speakers. It works, but it needs a press per item.

Live Scan: no button

Live Scan keeps the stream open at 504x896 / 15 fps and analyzes frames on the phone at around 4 to 5 fps. A frame only becomes a lookup if it passes a few gates:

1. Motion gate: skip frames while your head is moving.
2. Object gate: an on-device object detector (ML Kit) has to find an object of reasonable size near the center of the frame.
3. Dwell: the same object has to stay put for about 0.7 seconds.
4. Best frame: within that window, keep the sharpest frame.
5. Dedupe: a perceptual hash stops the app from researching the same item twice in a few minutes.

Then the lookups run cheapest first: barcode, then label text, then a paid cloud image search only if nothing else works. Hourly and daily caps keep the cloud calls bounded.

Moving quickly through a rack creates a different problem: you put an item down before its result arrives. So lookups go into a small queue, each verdict names its item ("Buy, Nike shorts, about $45 sold"), and stale results are dropped with a soft tone instead of being read out late.

Things I learned the hard way

• My first object gate was too strict for a hand-held item in a portrait frame, and the detector was running in single-object mode. Zero locks until I added per-gate counters to the log and looked at which stage was failing.
• Streams drop. The app needs one owner for the camera session, retry with backoff on a real drop, and a visible error instead of a toggle that quietly flips back off.
• Battery and heat aren't documented for continuous streaming, so the app reads the thermal level and battery and turns Live Scan off before the glasses get hot or low.

Privacy

The LED stays on while streaming and the app never tries to hide it. Raw frames stay in memory and are never saved. Only the chosen crop of an item is kept in the user's scan history, and nothing here identifies people.

Status

GlassesFlip is in early access on Android. If you resell and want to try it, you can sign up at https://glassesflip.com, and the changelog is at https://glassesflip.com/releases. I'd love to hear what a good voice verdict should say, and how you'd tune the gates.

GlassesFlip is an independent app and is not affiliated with, endorsed by, or sponsored by Meta Platforms, Inc., EssilorLuxottica, or eBay Inc. Ray-Ban is a registered trademark of Luxottica Group S.p.A.; Meta and eBay are trademarks of their respective owners.