Project · 2026 · runs in your browser
BlindAssist: a camera that says what matters
For a blind user, a wrong warning costs more than silence and ten warnings hide the one that matters. BlindAssist runs fully offline on a phone: two detectors find what is in the room, and a decision layer picks the single sentence worth saying. Below, the same models and the same rules run in your browser.
- Direction accuracy, reviewed announcements
- 100%
- Both models, RTX 3050
- 21 ms/frame
- CPU only, same laptop
- 5.8 FPS
- Phantom announcements, 7 clips
- 0
Try it
Loading two detectors, 24 MB, once…
Speak. The sentence the phone would say. A name the models are unsure of becomes 'obstacle'; when something is very close the sentence adds which way to step.
Pick a photo, upload your own or turn on your camera, then click through the steps under the picture to see each stage on that frame. Switch to Find to locate one object, or Describe for the whole scene. The decision rules are a line-for-line port of the app's Python, checked against it on random scenes.
Results
| Metric | Value | Source |
|---|---|---|
| Direction (left / ahead / right, top / bottom) correct, reviewed announcement keyframes, 7 clips | 100% | EVALUATION.md §2 |
| Proximity bucket plausible, same keyframes | 100% | EVALUATION.md §2 |
| Announcements from one-frame misdetections | 0 | EVALUATION.md §2 |
| Both models per frame, RTX 3050 / CPU only | 21 ms / 172 ms | README, EVALUATION.md §2 |
| From first sighting to speech | ≈0.35–0.5 s | EVALUATION.md §2 |
| Bedroom walkthrough: announcements over 471 frames | 8 | EVALUATION.md §2 |
The weak point is names, not warnings. COCO has no dustbin or wardrobe, so it said “toilet” and “refrigerator”; the warning was still right every time, but a wrong noun costs trust. That is why the door and dustbin specialist exists, and why an unsure name is spoken as “obstacle”. The full evaluation, failures included, is in EVALUATION.md.
What the phone app adds
Timing. On a live camera, an object must persist for two frames before it is announced, the same sentence is not repeated within a few seconds, and an obstacle that gets closer overrides the cooldown at once. Speech runs on its own thread, so the camera never waits for it.
Voice control in two tiers. A closed keyword grammar routes commands like “find bottle” in about 5 µs with nothing downloaded; a small local language model handles paraphrases a grammar cannot hear, choosing among fixed capabilities rather than writing the guidance itself.
Related. The design is the subject of a paper under revision at ICMLDE, and the wearable version is a published patent application.
Code
The Android app, the Python pipeline, the evaluation harness and the browser port are on GitHub. Team project.