💡 Inspiration

Developers rely heavily on AI coding agents, but granting agents visual perception currently requires piping full-resolution screenshots through proprietary cloud APIs. This introduces substantial token costs, latency, and serious privacy/compliance risks when handling internal codebases and sensitive user interfaces. Text-only models running locally remain completely blind to visual layout and rendering bugs.

⚙️ What it Does

Local Eyes is a Model Context Protocol (MCP) server that equips text-only agents and coding tools (like Claude Code) with private, on-device vision.

• Translates UI screenshots into structured Pydantic JSON schemas detailing interactive components, validation errors, and visual anomalies.
• Offers three core tools over stdio: read_ui, extract_text, and describe_image.
• Runs locally and privately through Ollama using Gemma, with a seamless cloud fallback to the Gemini API.
• Features response caching to enable rapid execution and zero-latency replays even in offline environments.

🛠️ How We Built It

• MCP Interface: Built with FastMCP over standard I/O for direct plug-and-play compatibility with Claude Code and other MCP clients.
• Vision & Orchestration: Integrated local inference via Ollama (gemma4:e4b) and cloud processing via Gemini (gemma-4-31b-it).
• Validation & Security: Enforced strict Pydantic output schemas with self-healing retry prompts, accompanied by an image-loader security harness with dimension/file-size caps and private URL safeguards.

🏆 Accomplishments that We’re Proud Of

• Successfully built an end-to-end agentic workflow where a text-only agent debugs and resolves UI layout defects with zero image tokens sent to the cloud.
• Implemented standardized inference parameters across sessions (keep_alive, context window management) to completely prevent model reload thrashing in local VRAM.

🚀 What’s Next for Local Eyes

• Automated bounding-box image visualizer (annotate_image) for debugging agent inspection traces.
• Native screen-capture tools for hands-free live UI monitoring and visual regression testing in CI/CD pipelines.