AI Engineer NYC 2026
Join AI Engineer and Major League Hacking (MLH) on Sunday, October 11, for a full day of building, experimenting, and meeting fellow AI engineers.
Citation Checker
Checks whether an AI answer's citations actually support its claims, and tests whether the AI judge doing the checking can be trusted.
AI answer engines attach citations to their answers, but a citation is only useful if the page really says what the answer claims. Citation Checker audits that.
How it works
- A Nimble Web Search Agent answers a question with
[n]citations and its own confidence grades, and Nimble Extract fetches every cited page live. - Gemini splits the answer into atomic claims and checks each one against its cited page (supported / partial / contradicted / not found), quoting the evidence. Code confirms every quote really exists on the page, and unreadable pages are marked “unverifiable” instead of blaming the source.
- Claude, a model from a different family, validates Gemini’s verdicts: when a label doesn’t match Gemini’s own reasoning, it flags it for human review.
What I found
- On a live Nimble answer, 10 of 15 verifiable claims were supported by their citation. In another run, a correct answer (330 m) cited a stale page that still says 324 m.
- Against my own labels, the Gemini judge was right on 12 of 15 claims, and every error was the same kind: it was too strict.
- A Gemini checker raised a false flag; the Claude checker caught the same real errors with none, so checking a judge with a different model family worked better.
Unlike fact-checkers that search for new evidence, this checks whether the citations an AI already gave you hold up, and whether the judge checking them can be trusted. Full findings and caveats are in the README.
Built with
AI tools
Prize Categories Entered
- Submitted
Best Use of Gemini API
Gemini (via the google-genai SDK) does the core judging: 1. Claim splitter: breaks an AI answer into atomic claims with their citation numbers, using JSON mode. 2. Judge: checks each claim against its cited page (supported/partial/contradicted/not found) using only the source, and quotes its evidence; code verifies the quote exists on the page. 3. Stability tests: I compared gemini-3.5-flash-lite (fast default) with gemini-3.8-flash across prompt wordings. The fast judge was perfectly stable, but only the stronger model caught an ambiguous claim: a stable judge isn't necessarily accurate. Against my own labels, the Gemini judge was right on 12 of 15 claims, and every error was being too strict. I handled rate limits and 503/504 errors with retries, timeouts and caching.
- Event prizeSubmitted
Best Overall
It tackles a real, growing problem: AI answers come with citations that people trust but rarely check. Citation Checker audits them end to end with three tools: a Nimble agent answers and Nimble Extract fetches every cited page live, Gemini judges each claim against its source with verified quotes, and Claude, from a different model family, validates Gemini's verdicts. It goes beyond a demo by evaluating itself: I measured the judge's stability across prompts and models, then labeled the claims by hand. The judge was right on 12 of 15, every error was the same kind (too strict), and the cross-family checker removed a false flag the same-family one raised. It's honest about its limits: unreadable pages are "unverifiable", flags mean "needs review", and the README states every caveat. Built solo in one day, with working code, a live app and documented findings on GitHub.
- Event prizeSubmitted
Best Use of Nimble
It uses Nimble to turn live web context into something you can trust, and uses Nimble to check Nimble. - Web Search Agents (effort="low", ~15s) produce a cited answer with per-claim confidence grades. - Extract fetches every cited page live with JavaScript rendering, so each claim is checked against the real current page. - Search is the natural next step to suggest a better source when a citation fails. The result is an independent audit of the agent's own trust report. In live runs it found a correct answer (330 m) citing a stale page that still says 324 m, with the same confidence as everything else; sources all labeled "official", including a travel site; and the same question giving different sources and confidence across runs. Unlike Nimble's fact-check cookbook, which searches for new evidence, this verifies the citations an agent already gave, which makes Nimble agents more trustworthy in production.