Skip to main content

About the project

The problem. Small open models (around 4B parameters) can run on a laptop, but they fail as coding agents. They aren’t too dumb; they fail at process. In our baseline, every failure was a process failure: the model opened the right file, kept reading others, and never committed to an edit. We measured zero knowledge failures and zero tool-format failures.

What Manifest does. We don’t give the small model a bigger prompt. We grow the process around it as code:

  1. Student: Qwen3.5-4B runs locally through Ollama and handles only the semantic steps: diagnose a bug, write a patch.
  2. Teacher: DeepSeek V4.1 Flash (MIT open weights) reads the student’s failed traces on practice tasks and writes one new control routine per round, as real Python.
  3. Gate: a routine is kept only if it improves results on separate gate tasks with no regressions.
  4. Warden: every teacher-written routine is untrusted code.
    • Warden statically scans it, has the teacher explain what it does, and issues a permission manifest (read the repo, write src/ only, run pytest only, no network).
    • The manifest is enforced at runtime by the macOS sandbox. A rigged third-party “marketplace” skill that tries to read ~/.ssh and phone home gets blocked.
  5. Agent Skills: accepted routines are exported as Agent Skills, so they work in other agents too.
  6. Offline: after growth, the 4B model plus the grown harness runs with Wi-Fi off. The teacher only ever sees synthetic practice tasks, never your code.

Results (real run, raw logs in the repo):

  • The teacher grew triage-fix: run the tests, follow the traceback to the innermost frame of your own code, show the student only that code, apply one patch. Gate score went from 0/3 to 2/3 with no regressions; Warden approved it with no network access and writes to src/ only.
  • On held-out tasks the teacher never saw, model calls per task fell from 7.3 to 4.3 (41% fewer).
  • The live-demo task went from timing out after 8 calls and 240 s to passing with 1 call in about 8 s.
  • Total teacher cost: $0.014. Student cost: $0 (local).
  • Warden also caught the small model itself trying shell pipes and paths outside the task folder, 6 times.

Built to be honest.

  • 26 generated debugging tasks across 5 realistic Python services.
  • Each claimed bug type is mechanically verified (for example, a “regression trap” must have a tempting wrong fix that provably breaks another test).
  • The runner judges pass/fail itself and fails any run that edits or deletes tests.
  • Held-out tasks, including a whole domain, are never shown to the teacher; every teacher prompt is checked for leaks before it’s sent.
  • Limitations are stated in RESULTS.md: a small sample (3/3/3 tasks), a 240 s per-task budget, and one lost round.

Built with: Python, Ollama, Qwen3.5-4B, DeepSeek V4.1 Flash (OpenAI-compatible API, model id deepseek-flash), pytest, pydantic, macOS sandbox-exec, React + Vite + TypeScript, the Agent Skills open standard.

Prior work: builds on “Grow the Harness, Not the Context” (arXiv 2609.26760). Our additions are:

  • the coding domain with verified bug shapes;
  • routines exported as Agent Skills;
  • Warden in the gate and at runtime;
  • fully offline deployment.

Open source: MIT, github.com/Arshgill01/manifest

Built with

pythonjavascriptdeep-learningtypescriptsandbox-execqwen-4Bdeepseekv4.1flashagentskills

Prize Categories Entered

  • DigitalOcean

    Best Open-Source AI Project

    Submitted