Decision Overload
A benchmark harness that measures whether a System 1 decision model (Jev, SubQ Snap, Gemini Decision) in front of Gemini makes agent tool us
About the project
Agents get slower and sloppier as their tool list grows. System 1 decision models like Jev promise to fix that by picking the right tools before the big LLM thinks. We built a harness to test that claim fairly, on third-party benchmarks with their own graders.
Built with
AI tools
Prize Categories Entered
- Submitted

Best Use of Gemini API
Gemini API with streaming function calling powers both agents under test (Gemini 4 Carbon and Hydrogen, thinking_level low). We timestamp the first streamed chunk and the first function-call chunk to measure time to first tool call. Gemini Decision 1 is one of the three decision models, ranking tools for each request. Gemini 3.8 Flash runs τ-bench's simulated customer and its natural-language judges. We also fixed τ's judge so it parses Gemini's fenced JSON output.
- Submitted

Best Use of DigitalOcean
The whole stack (FHIR server, FastAPI dashboard, benchmark runner) runs on one SFO3 droplet in Docker, provisioned with scripts and cloud-init. A probe measures the network floor to each provider so latency results are honest. The DigitalOcean MCP server is wired in for managing it from Claude Code.
- Submitted

Best Use of Tiger Data
Tiger Cloud stores runs, episodes and an events hypertable. A continuous aggregate (call_latency_1m) gives the live dashboard pre-computed latency stats while the benchmark runs.
- Submitted
Sentry Raffle
Each benchmark episode is a Sentry transaction, with spans for every decision-model call, Gemini call and tool call, so you can see where an agent stalled. The Sentry MCP server let Claude Code inspect traces while we built it.
- Submitted
Best Demo Video Using HyperFrames
MCP to create demo video
- Event prizeSubmitted
Best Overall Hack
Everyone is shipping agents, but nobody can say whether their latest add-on actually helps. Decision Overload answers that with evidence instead of a demo. In one day we built a reproducible A/B harness that runs 2 Gemini models × 4 conditions across three outside benchmarks (Sierra's τ-bench, Stanford's MedAgentBench, MCPToolBench++), graded by their own answer keys, not ours. It's built to be fair: identical prompts and tools, a clock that excludes tool execution and simulators, and deltas paired per task so nothing wins by failing fast. It's also a real system: live dashboard on Tiger Data, a Sentry trace for every episode, one-command deploy to DigitalOcean, budget guard and resume. Along the way we found and fixed a real bug in τ-bench's judge (it crashed on Gemini's fenced JSON). The result is a clear, useful finding: a decision model in front of Gemini raised success from 60% to 80%, nearly halved the slowest-case time to the first correct tool call, and cut tokens and cost by
- Event prizeSubmitted
Best Use of System One Models
System 1 models are the whole project. Instead of using Jev as one feature, we put it in front of a Gemini agent and measured exactly what it buys, head to head against two other decision models (SubQ Snap and Gemini Decision) under identical conditions. Jev answers one fast "which tools does this need?" question over every available tool, and only that shortlist (95% of the probability mass, 3–8 tools, plus an escape hatch) reaches the slower System 2 LLM. That's the System 1 / System 2 split, applied to tool use. Results from 40 live episodes: - With Jev, success went from 60% to 80%. - Slowest-case (p95) time to the first correct tool call nearly halved: 19.3s to 10.2s. - Input tokens and cost per episode dropped about 30%. - Jev added only 0.16s per decision, the lowest of the three (SubQ 0.39s, Gemini Decision 0.69s). - It kept the correct tool in 100% of shortlists while cutting up to 100 tools to about 3.