Scene Companion gives blind and low-vision people real-time awareness of the space around them, using just a phone.

The phone streams its camera to an AI server, which talks back through the phone’s speaker, tones, and vibration.

What it does

• Proximity alerts: a depth model plus a floor-plane fit catches walls, chairs, and floor obstacles, then warns with rising tones and haptics as you get closer.
• Guidance to a goal: object detection (YOLOv8n) finds chairs, an EXIT-sign detector finds the exit, and a path planner merges both with depth so the app can say, out loud, how to walk to the exit around obstacles.
• “Tell me what’s in front of me”: a hands-free voice command. Gemini vision describes the scene, and Nimble web search can look up what it sees.
• Voice in, voice out: speech-to-text runs locally with faster-whisper, the mic is on by default, and replies come back as text-to-speech on the phone.

How it works

• Android app (Kotlin, CameraX) and a mobile web app (so it works on iPhone too) send camera frames over WebSockets to a Python FastAPI server.
• The server runs the perception pipeline (depth, detection, floor fit, Gemini) and sends speech and alerts back to the phone.
• It works remotely over Tailscale, with HTTPS for iPhone browsers.
• A replay harness scores the pipeline against labeled test scenes, so every change gets checked before it reaches the phone.

Sponsor tech

• Sentry: error monitoring on both the Android app and the server.
• Snowflake (Cortex) and TigerData: optional logging of scene events, kept off the live path, so past events can be searched and recalled later.

What’s next
Walk-through video tests that score danger level over time, better speech recognition, and production hosting instead of a laptop.