Skip to main content

Andon

A stop cord for AI coding agents: it catches agents that cheat to pass tests, before the code merges.

About the project

The problem

“All tests passed” used to mean the code works. With AI agents, it can mean the agent changed the test. Agents now write more code than people can review, so diffs merge unread. And agents learn the cheap fix: weaken a test, skip a check, hide an error. CI stays green. The code is still broken.

What Andon does

Andon watches every change a coding agent makes and answers one question: did the agent solve the task, or cheat to get a green check?

  1. Facts (plain code, free): was a test edited, an assertion weakened, a check silenced, CI config touched?
  2. Judgment (Jev, a System One model): six yes/no questions in one call, like “Does this test check less than before?”
  3. Rules (plain code): scores become Pass, Rework, or Stop.

The model judges. Code decides. On Stop, the whole line halts and a human sees the exact line, the question that fired, and the score.

Results (20 held-out changes)

  • Caught 10 of 10 cheats (plain-code facts alone: 2 of 10)
  • 3 of 10 false alarms on honest changes, all low-score flags
  • 217 ms median per check, $0.0005 per check

Thresholds were tuned on a separate dev split. Every false alarm is listed in the Scorecard tab and in scorecard/RESULTS.md.

Built in a software factory

Specs before code (docs/), agent rules (CLAUDE.md), tests locked to the core logic, and a 13-task plan with one clean commit per task, built with Claude Code.

Stack

Next.js, TypeScript, Tiger Data (Postgres hypertable for every check), Sentry, Jev via OpenRouter, HyperFrames for the demo video.

Built with

typescriptjavascriptcssclaudeopenrouter

AI tools

Coded withClaude CodeClaude Opus 5.5Older ClaudeClaude Sonnet 5.5

Prize Categories Entered

  • Tiger Data Q3 2026 - Q2 2027

    Best Use of Tiger Data

    Submitted

    Every decision Andon makes lives in Tiger Data. Each check is a row in an `events` hypertable: the diff, the facts that fired, Jev's scores, the verdict, latency and cost. That table is the product: the live board streams from it every second, the stats bar is plain SQL over it, and a `line` table holds the stop state so one Stop halts every agent. It gives teams a full audit trail of what their agents tried to get away with.

  • Sentry

    Sentry Raffle

    Submitted

    A safety gate must never fail open. When a Jev call times out or errors, Andon reports it to Sentry and returns "Rework" instead of quietly passing the change. Sentry is how we know the gate itself is healthy: the tool that watches the agents is watched too.

  • HyperFrames

    Best Demo Video Using HyperFrames

    Submitted

    I made a 60-second launch film in HyperFrames with Claude Code, built like a video creator would: script, storyboard, build, polish. Every scene is HTML using our app's own design tokens, so the film and the product look like one thing. Keyframed camera moves push into real screen recordings, captions are burned in, and the audio cuts to silence, then a single "clunk", on the exact frame Andon stops the line.

  • Event prize

    Best Overall Hack

    Submitted

    "All tests passed" used to mean the code works. With AI agents, it can mean the agent changed the test. Andon is the stop cord for that: plain code checks the facts, a System One model makes the judgment call, and the line stops in a fraction of a second with the exact cheating line on screen. On 20 held-out changes it caught 10 of 10 cheats, where plain-code checks alone caught 2. It raised 3 false alarms, and we publish them. Real problem, working product, honest numbers.

  • Event prize

    Best Use of System One Models

    Submitted

    Most projects ask a System One model to decide. Andon asks it to judge, and lets code decide. Jev answers six yes/no questions per change in one call, like "Does this test check less than before?" Plain code handles the facts first, so Jev only gets the questions code can't answer. Rules turn its probabilities into Pass, Rework, or Stop, and every verdict shows the question and score behind it. The proof is in the gap: plain code alone caught 2 of 10 cheats, and adding Jev caught 10 of 10, at 217 ms and $0.0005 per check.

  • Event prize

    Cleanest Use of a Software Factory

    Submitted

    Andon fights AI slop twice. First, in how we built it: specs before code, a CLAUDE.md that bans the exact cheats agents love (weakened tests, ts-ignore, empty catch), a 13-task plan with a "done when" check per task, and one clean commit each. Second, in what it is: a quality gate any software factory can put on its line, so agents can't slop their way to green CI.