Skip to main content

About the project

Keel stops open models from inventing file paths.

Ask a small open model “which files should I change to fix this issue?” in a repo it hasn’t seen, and it confidently names files that don’t exist. In our benchmark, 96% of the files Gemma 4 cited were made up when it wasn’t shown the repo. For a coding agent or a first-time contributor, a plausible wrong path is the most expensive kind of wrong.

What it does

  • keel plan <repo> "<issue>" builds a Graphify code graph of the repo (no LLM needed; 1,320 nodes for psf/requests in about 3 seconds), pulls the relevant subgraph with graphify query, asks Gemma 4 for a change plan, then checks every cited file against the graph. A made-up path goes back to the model with “did you mean…?” and the list of real files, until the plan only cites code that exists. It also estimates the ripple: other files that call or inherit from what you are changing.
  • keel triage <repo> ranks a repo’s “good first issue” tickets by how small and safe the change looks. Run live on python-poetry/poetry.
  • Underneath is a reliability harness (extract → validate → verify → repair) that works with any OpenAI-compatible server and needs no native function-calling.

Results (Gemma 4 26B, 10 issues on Keel’s own repo, Graphify graph as ground truth)

  1. Model not shown the repo’s files: plain calls gave 1/10 valid, correct plans and 25 of 26 cited files did not exist. With Keel: 8/10, and 0 of 22 invented, using about 2 model calls instead of 1.
  2. Model given the file list: no gain (9/10 both).
  3. JSON extraction and tool use: Gemma 4 was already at 100% without Keel, so no gain there.

How Graphify is used

  1. Maps the repo into a graph.
  2. Retrieves the subgraph relevant to an issue.
  3. Verifies every file the model cites.
  4. Estimates ripple from call, inherit and use edges.
  5. Is connected to Claude Code in the repo itself (graphify install --project), with Keel’s own graph committed, including an interactive graph.html.

Open-source AI

  • Runs on Gemma 4 (open weights) through the Gemini API free tier, and on any OpenAI-compatible endpoint (DigitalOcean Gradient, Ollama, vLLM) by changing one environment variable.
  • MIT licensed.
  • Two Agent Skills (keel-structured-output, keel-plan-change), both validated with the official skills-ref tool.

Honest limits

  • Small benchmark: 10 tasks, one model, one run per cell.
  • A real file is not necessarily the right file.
  • Ripple is a static estimate.
  • We changed one metric after seeing the first run (symbols that were really parameters were counted as errors), and say so in the README.
  • The benchmark also caught two bugs in our own code, which are fixed with regression tests.
  • Not deployed on DigitalOcean (no credits were available).

Built with: Python, Gemma 4 via the Gemini API, Graphify, httpx, jsonschema.

Repo: https://github.com/KanavCode/Keel

Built with

claudegemini

AI tools

Coded withClaude CodeClaude Sonnet 5.5Gemini 3.8 FlashAntigravity

Prize Categories Entered

  • Deepmind Google

    Best Use of Gemma 4

    Submitted
  • DigitalOcean

    Best Open-Source AI Project

    Submitted
  • Event prize

    Best Use of Graphify

    Submitted