Skip to main content

EgoCheck

Robots learn from what they watch. EgoCheck makes sure they only watch good clips.

About the project

Robotics companies are collecting a lot of first-person video of people doing tasks, so robots can learn from it. The problem is that much of this footage is not good enough. It is blurry, too dark or shaky, the hands go out of frame, the object is hidden, or the task never gets finished. Most of it goes straight into training without anyone checking it.

EgoCheck is the check that happens before training.

For each clip, it does three things. First, it runs quick checks on the laptop for blur, brightness, shake and length. Second, Gemini watches the clip and writes a short fact sheet: what the task is, how often the hands and the object are visible, whether the task was finished, whether any faces or screens show up, and the steps the person took with timestamps. Third, TypeSafe’s System One model (Jev) answers seven yes/no questions about the clip — is it blurry, did the task fail, is a screen or face visible — each with a real probability. Simple code turns those into keep, fix or discard with a confidence score. If it is less than 80% sure, the clip goes to a person to review.

Every clip gets its own data card, and the whole batch gets a short report showing how many clips passed, why the others failed, and how many usable minutes are left.

Why a System One model instead of just a big LLM? Datasets can have thousands of clips. Running a large model on every one is slow and costly, and it cannot give a real confidence number. A System One model answers quickly and gives a proper probability, so teams can set a clear rule for what goes to a human.

Built with Python, OpenCV, Gemini 4 Hydrogen (Vertex AI), TypeSafe System One (Jev), Sentry, MuJoCo and HyperFrames.

Built with

gemini

AI tools

Coded withClaude CodeGemini 4 HydrogenClaude Opus 5.5

Prize Categories Entered

  • Gemini Prize Category Q1 2025 - Q1 2026

    Best Use of Gemini API

    Submitted

    Gemini 4 Hydrogen on Vertex AI (google-genai SDK) watches each clip. The video is sent inline and Gemini returns a strict JSON fact sheet: the task, how often the hands and object are visible, whether the task was finished, whether it's one task, whether faces or screens appear, plus timestamped steps (move to, grasp, lift, place, release). We validate it against a schema with one retry. The fact sheet feeds the System One decision, and the steps drive the MuJoCo robot replay of kept clips. About 8.5 s per clip. Results are cached by file hash, so a clip is never paid for twice.

  • Sentry

    Sentry Raffle

    Submitted

    The sentry-sdk covers the whole pipeline. Each batch run is one transaction, with a span per stage (local checks, Gemini, System One, decide), so we can see where the time goes. The Google GenAI integration traces every Gemini call, including tokens. Failed API calls and bad JSON are captured with the clip's name, logs are forwarded, and metrics count the verdicts (keep/fix/discard/review) per run.

  • HyperFrames

    Best Demo Video Using HyperFrames

    Submitted

    The whole 66-second demo film is a HyperFrames project: 13 scenes written as HTML + GSAP compositions. A hand-drawn stick-figure story (SVG with a "boiling" line filter) leads into real product footage: our dashboard recording, real rejected clips, and MuJoCo robot renders, including one where a hand "trained on bad data" throws the apple at the camera. Narration (ElevenLabs), sound effects and music (Google Lyria) sit on separate audio tracks. We used the CLI loop of check, snapshot and render, and a script renders it and sets the pacing.

  • Event prize

    Best Overall Hack

    Submitted

    Bad first-person data is a real and growing problem in robot learning. EgoCheck takes real phone clips, decides which ones are good enough, and writes down why, so robotics teams only train on footage worth learning from, and replays the good ones on a simulated robot hand.

  • Event prize

    Best Use of System One Models

    Submitted

    The System One model makes the decision; Gemini only describes the clip. Jev answers seven yes/no questions on three keyframes plus the fact sheet, and each answer is a real probability, not a guess. Code turns those into keep, fix or discard with a confidence score, and anything under 80% goes to a person. It answers in under half a second for about a hundredth of a cent, so it scales to thousands of clips where a big model would be slow and costly.

  • Event prize

    Cleanest Use of a Software Factory

    Submitted

    We wrote a clear spec first, with a fixed data format and simple tests, then had coding agents build it one file at a time. Each outside service lives in its own small file, and every model result is cached by the clip's file hash, so runs are repeatable and nothing is paid for twice. The product does the same job for data: it keeps the slop out before it reaches a robot.