Self-Healing Scraper Factory
A factory that builds web scrapers from a URL and repairs them when sites change, for teams that depend on scraped data.
About the project
Scrapers break every time a site changes. I have built a factory that builds them and heals them, and that you can trust to ship without a human because every model judgement is checked by something that isn’t that model.
Create. POST /scrapers { name, url }: the factory fetches the site through Bright Data’s Web Unlocker, has an agent write the parser and tests, runs it against a fresh fetch of the live site, checks the output in code (items, unique ids, titles, prices), has a second model from a different lab read the page and compare every item, and asks Gemini Decision 1 three calibrated questions (correct? complete? fails loudly?). Only then it merges the PR and deploys the scraper as its own DigitalOcean App Platform app, writing to Snowflake.
Heal. The scraper reports parser failures to Sentry with the exact page attached. The monitor polls Sentry, screens the evidence with Apify‘s prompt guard, has Gemini explain it and Gemini Decision 1 rate severity and fixability, sets the Sentry priority, and pauses the scraper (no bad data reaches Snowflake). The factory then writes a test that fails the way production did (checked by running it), fixes the parser with the test locked, runs an adversarial review, enforces limits in code (only parser files, ≤60 lines, no dependencies), asks Gemini’s gate, merges, redeploys on DigitalOcean, resumes the scraper and resolves the Sentry issue.
The demo site is a live auction house for (fictional) GPUs and accelerators: state in Tiger Data (a TimescaleDB hypertable of bids), a simulation bidding, selling and listing every 2 s, server-rendered and updated over a WebSocket, product photos generated with Gemini image generation. Its design switches on the fly: a redesign, new currency formats, a login wall, a CAPTCHA, a prompt-injection lot.
What we saw today: create mode built the scraper from the live site, the verifier matched every lot field by field, Gemini cleared it, and it deployed as its own DigitalOcean app; Snowflake’s dynamic table now turns every scrape into one live price per lot with its growth, and Tiger keeps each lot’s price per minute. A full heal ran on its own earlier (PR #2: failing test, fix, review, merge, redeploy, resolved). In the final run the site switched to its redesign, Sentry caught the break, Gemini triaged it critical (98%), the scraper was paused, the factory wrote a failing test from the exact page and fixed it, and the reviewer could not break the fix. Gemini Decision 1 then rated it only 66% low-risk, below our 80% bar, so the factory opened PR #5 for a human instead of merging; one approval later it redeployed and resumed. Along the way the safeguards also stopped a fix that bent the parser toward wrong data and a fix over the line limit. It fails safe, and asks when it is not sure.
Agents run on DigitalOcean Serverless Inference (Kimi K3.1 Pro fixes, GLM 5.3 reviews: two labs, different blind spots) through the Claude Agent SDK, and optionally on DigitalOcean Managed Agents sandboxes (AGENT_RUNTIME=do-managed).
Built with
AI tools
Prize Categories Entered
- Submitted

Best Use of Gemini API
Decision 1 triages each Sentry error: severity, fixable Decision 1 gates every new scraper and every repair Gemini 4 Hydrogen explains what broke Image generation made the product photos and the logo Gemini 4 Carbon built an example Hacker News scraper
- Submitted

Best Use of DigitalOcean
App Platform runs the live auction site The factory deploys every scraper as its own App Platform app via the API, and redeploys after each repair Serverless Inference runs the coding agents (Kimi fixes, GLM reviews) Managed Agents as an optional agent sandbox
- Submitted

Best Use of Tiger Data
The live auction’s state: lots and a hypertable of every bid A continuous aggregate keeps each lot’s price per minute
- Submitted

Best Use of Snowflake
Every scrape lands in SCRAPED_ITEMS A dynamic table (LOT_PRICES) refreshes every minute, one row per lot: latest price growth in percent dollars per minute
- Submitted
Sentry Raffle
Scrapers report parser failures with the exact HTML page attached The monitor polls the Sentry API The attached page becomes the repair’s test fixture Sets the issue’s priority, and resolves it once the fix ships
- Submitted
Best Use of Bright Data
Web Unlocker fetches every page The factory investigates and verifies sites through it The scrapers poll through it with retries
- Submitted
Best Demo Video Using HyperFrames
The whole demo video was made with claude and the hyperframes and the related skills
- Event prizeSubmitted
Best Overall Hack
Builds a scraper from just a URL, verifies it on the live site, and deploys it When the site changes: Sentry catches it, the scraper pauses, the factory repairs it and redeploys Ran end to end today on live infrastructure: DigitalOcean, Sentry, Bright Data, Tiger Data, Snowflake Fails safe: it stopped bad fixes and asked a human when unsure
- Event prizeSubmitted
Best Use of System One Models
Gemini Decision 1 makes every go/no-go with calibrated probabilities Code applies the thresholds At 66% sure, it asked a human instead of merging
- Event prizeSubmitted
Cleanest Use of a Software Factory
Writes a test that fails on the exact page that broke production, and proves it fails Fixes the parser with that test locked A model from another lab reviews the fix and tries to break it Limits enforced in code: parser and test files only, at most 120 lines Gemini Decision 1 scores the fix; code decides Today it stopped a wrong-data fix, and asked a human when only 66% sure