Skip to main content

Decision Overload

A benchmark harness that measures whether a System 1 decision model (Jev, SubQ Snap, Gemini Decision) in front of Gemini makes agent tool us

About the project

Agents get slower and sloppier as their tool list grows. System 1 decision models like Jev promise to fix that by picking the right tools before the big LLM thinks. We built a harness to test that claim fairly, on third-party benchmarks with their own graders.

Built with

pythongeminigoogle-cloudfastapipostgresqldockerdigitaloceantimescaledbclaude

AI tools

Coded withClaude CodeGoogle AI StudioClaude Opus 5.5Claude Sonnet 5.5

Prize Categories Entered

  • Gemini Prize Category Q1 2025 - Q1 2026

    Best Use of Gemini API

    Submitted

    Gemini API with streaming function calling powers both agents under test (Gemini 4 Carbon and Hydrogen, thinking_level low). We timestamp the first streamed chunk and the first function-call chunk to measure time to first tool call. Gemini Decision 1 is one of the three decision models, ranking tools for each request. Gemini 3.8 Flash runs τ-bench's simulated customer and its natural-language judges. We also fixed τ's judge so it parses Gemini's fenced JSON output.

  • DigitalOcean

    Best Use of DigitalOcean

    Submitted

    The whole stack (FHIR server, FastAPI dashboard, benchmark runner) runs on one SFO3 droplet in Docker, provisioned with scripts and cloud-init. A probe measures the network floor to each provider so latency results are honest. The DigitalOcean MCP server is wired in for managing it from Claude Code.

  • Tiger Data Q3 2026 - Q2 2027

    Best Use of Tiger Data

    Submitted

    Tiger Cloud stores runs, episodes and an events hypertable. A continuous aggregate (call_latency_1m) gives the live dashboard pre-computed latency stats while the benchmark runs.

  • Sentry

    Sentry Raffle

    Submitted

    Each benchmark episode is a Sentry transaction, with spans for every decision-model call, Gemini call and tool call, so you can see where an agent stalled. The Sentry MCP server let Claude Code inspect traces while we built it.

  • HyperFrames

    Best Demo Video Using HyperFrames

    Submitted

    MCP to create demo video

  • Event prize

    Best Overall Hack

    Submitted

    Everyone is shipping agents, but nobody can say whether their latest add-on actually helps. Decision Overload answers that with evidence instead of a demo. In one day we built a reproducible A/B harness that runs 2 Gemini models × 4 conditions across three outside benchmarks (Sierra's τ-bench, Stanford's MedAgentBench, MCPToolBench++), graded by their own answer keys, not ours. It's built to be fair: identical prompts and tools, a clock that excludes tool execution and simulators, and deltas paired per task so nothing wins by failing fast. It's also a real system: live dashboard on Tiger Data, a Sentry trace for every episode, one-command deploy to DigitalOcean, budget guard and resume. Along the way we found and fixed a real bug in τ-bench's judge (it crashed on Gemini's fenced JSON). The result is a clear, useful finding: a decision model in front of Gemini raised success from 60% to 80%, nearly halved the slowest-case time to the first correct tool call, and cut tokens and cost by

  • Event prize

    Best Use of System One Models

    Submitted

    System 1 models are the whole project. Instead of using Jev as one feature, we put it in front of a Gemini agent and measured exactly what it buys, head to head against two other decision models (SubQ Snap and Gemini Decision) under identical conditions. Jev answers one fast "which tools does this need?" question over every available tool, and only that shortlist (95% of the probability mass, 3–8 tools, plus an escape hatch) reaches the slower System 2 LLM. That's the System 1 / System 2 split, applied to tool use. Results from 40 live episodes: - With Jev, success went from 60% to 80%. - Slowest-case (p95) time to the first correct tool call nearly halved: 19.3s to 10.2s. - Input tokens and cost per episode dropped about 30%. - Jev added only 0.16s per decision, the lowest of the three (SubQ 0.39s, Gemini Decision 0.69s). - It kept the correct tool in 100% of shortlists while cutting up to 100 tools to about 3.