Keel
About the project
Keel stops open models from inventing file paths.
Ask a small open model “which files should I change to fix this issue?” in a repo it hasn’t seen, and it confidently names files that don’t exist. In our benchmark, 96% of the files Gemma 4 cited were made up when it wasn’t shown the repo. For a coding agent or a first-time contributor, a plausible wrong path is the most expensive kind of wrong.
What it does
-
keel plan <repo> "<issue>"builds a Graphify code graph of the repo (no LLM needed; 1,320 nodes for psf/requests in about 3 seconds), pulls the relevant subgraph withgraphify query, asks Gemma 4 for a change plan, then checks every cited file against the graph. A made-up path goes back to the model with “did you mean…?” and the list of real files, until the plan only cites code that exists. It also estimates the ripple: other files that call or inherit from what you are changing. -
keel triage <repo>ranks a repo’s “good first issue” tickets by how small and safe the change looks. Run live on python-poetry/poetry. - Underneath is a reliability harness (extract → validate → verify → repair) that works with any OpenAI-compatible server and needs no native function-calling.
Results (Gemma 4 26B, 10 issues on Keel’s own repo, Graphify graph as ground truth)
- Model not shown the repo’s files: plain calls gave 1/10 valid, correct plans and 25 of 26 cited files did not exist. With Keel: 8/10, and 0 of 22 invented, using about 2 model calls instead of 1.
- Model given the file list: no gain (9/10 both).
- JSON extraction and tool use: Gemma 4 was already at 100% without Keel, so no gain there.
How Graphify is used
- Maps the repo into a graph.
- Retrieves the subgraph relevant to an issue.
- Verifies every file the model cites.
- Estimates ripple from call, inherit and use edges.
- Is connected to Claude Code in the repo itself (
graphify install --project), with Keel’s own graph committed, including an interactivegraph.html.
Open-source AI
- Runs on Gemma 4 (open weights) through the Gemini API free tier, and on any OpenAI-compatible endpoint (DigitalOcean Gradient, Ollama, vLLM) by changing one environment variable.
- MIT licensed.
- Two Agent Skills (
keel-structured-output,keel-plan-change), both validated with the officialskills-reftool.
Honest limits
- Small benchmark: 10 tasks, one model, one run per cell.
- A real file is not necessarily the right file.
- Ripple is a static estimate.
- We changed one metric after seeing the first run (symbols that were really parameters were counted as errors), and say so in the README.
- The benchmark also caught two bugs in our own code, which are fixed with regression tests.
- Not deployed on DigitalOcean (no credits were available).
Built with: Python, Gemma 4 via the Gemini API, Graphify, httpx, jsonschema.
Built with
claudegemini
AI tools
Coded withClaude CodeClaude Sonnet 5.5Gemini 3.8 FlashAntigravity
Prize Categories Entered
- Submitted
Best Use of Gemma 4
- Submitted

Best Open-Source AI Project
- Event prizeSubmitted
Best Use of Graphify