AI Engineer NYC 2026
Join AI Engineer and Major League Hacking (MLH) on Sunday, October 11, for a full day of building, experimenting, and meeting fellow AI engineers.
Somo
Snap a math problem you're stuck on: Somo circles the exact step you got wrong and teaches it in Swahili, Spanish or English, then gives you
About the project
Why
I grew up in Tanzania, where students switch from Swahili to English for secondary school and sit NECTA national exams in a language they’re still learning. In New York public schools, English learners face the same gap: they understand the math but get stuck on the words.
What it does
- Snap a photo of a problem and your handwritten working.
- Somo circles the first wrong line in red pen, with a note in your language.
- You confirm what Somo read (human in the loop) before it teaches.
- A lesson streams in English, Swahili or Spanish: the idea, where it went wrong, a local analogy, a worked example with different numbers, and bilingual key terms.
- A check question: wrong → simpler re-explanation; right → practice like the real exam with a real NECTA or NY Regents past-paper question.
How we used Nimble (Best Use of Nimble)
- When the lesson starts, Nimble searches the live web for past exam papers on the student’s concept: official NECTA sites first (necta.go.tz, maktaba.tetea.org), Regents question banks (jmap.org, nysedregents.org) for US students, then the open web.
- Search returns short summaries for PDFs, so Somo downloads the papers Nimble finds and reads their full text, then Gemini picks one real question on the same topic.
- Every question is labeled honestly: official past paper (with exam stamp and source link), practice question from the web, or written by Somo when nothing suitable was found.
- Result: a student who just fixed their mistake practises on a real NECTA CSEE 2023 question.
How we used Sentry (Best Use of Sentry)
- Every Gemini, Gemma, Claude and Nimble call is a traced span, and background jobs (verification, question prep, past-paper search) are their own transactions.
- When Gemini is overloaded, Somo fails over to Claude, and Sentry raises a warning issue for every failover.
- Sentry found a real bug: the trace waterfall showed one NECTA server taking 3.6 minutes to send a PDF (httpx’s timeout is per chunk, so a slow trickle never tripped it). We added a hard 12-second deadline per download.
- No student photos or prompt text are sent to Sentry.
How we used Gemini (Best Use of Gemini API)
- Gemini reads handwriting and diagnoses the mistake with structured output, then streams lessons.
- A second, independent Gemini solve verifies the answer before teaching; disagreement shows a low-confidence warning.
- Gemma 4 grades check answers quickly and cheaply.
AI engineering
- Multi-provider routing with a circuit breaker (Gemini → Claude): diagnosis went from ~116s to 9–12s during Gemini overloads.
- Parallel background work (verification, check question, past-paper search) hides latency.
- Disk cache with fuzzy topic matching for practice questions.
What’s next
Offline mode on Gemma, full NECTA and Regents syllabus coverage, the sciences, and pilots in Dar es Salaam and New York.
Built with
AI tools
Prize Categories Entered
- Submitted
Best Use of Gemini API
Gemini (gemini-flash-latest via the google-genai SDK) is Somo's core. It reads a photo of handwritten math working and returns structured JSON (Pydantic response_schema): the transcription, the concept, the first wrong line, and why it's wrong, so the UI can circle that exact line in red pen. A second, independent Gemini call solves the problem blind to verify the answer before Somo teaches; if they disagree, the student sees a low-confidence warning. Gemini then streams the lesson in Swahili, Spanish or English with local analogies and bilingual key terms, writes adaptive check questions, and picks a real past exam question from the NECTA papers Nimble finds. Gemma 4, served through the Gemini API, grades answers quickly and cheaply. When Gemini is overloaded, a circuit breaker fails over to Claude so students are never stuck.
- Event prizeSubmitted
Best Use of Nimble
Nimble turns Somo from a tutor into exam prep. As soon as a lesson starts, Nimble searches the live web for real past exam papers on the student's exact concept: official NECTA sites first (necta.go.tz, maktaba.tetea.org) for Tanzanian students, Regents question banks (jmap.org, nysedregents.org) for US students, then the open web. Search returns short summaries for PDFs, so Somo downloads the papers Nimble finds and reads their full text, and Gemini picks one real question on the same topic at the right level. Every question is labeled honestly: official past paper (with an exam stamp like "NECTA CSEE 2023" and a source link), practice question from the web, or written by Somo. A student who just fixed their mistake immediately practises on the real question they'll face, which no static question bank or plain LLM can provide.
- Event prizeSubmitted
Sentry Raffle
Somo is a multi-model agent (Gemini, Gemma, Claude, and Nimble web search), and Sentry traces all of it. Every model and web call is a span with provider, model and attempt number; background jobs (answer verification, question prep, past-paper search) are their own transactions; and every Gemini → Claude failover raises a Sentry warning, so we can see exactly when and why the agent switched providers. Sentry also found a real bug: the trace waterfall showed one NECTA government server taking 3.6 minutes to send a PDF. httpx's timeout applies per chunk, so a slow trickle never tripped it. We added a hard 12-second deadline per download, cutting the worst case from about 4 minutes to about 30 seconds. No student photos or prompt text are sent to Sentry.