Skip to main content

AI Engineer NYC 2026

​Join AI Engineer and Major League Hacking (MLH) on Sunday, October 11, for a full day of building, experimenting, and meeting fellow AI engineers.

Custom Detections as Code

Nimble finds new threat reports. Claude writes and attacks a Defender detection for each. Gemini and GPT review it. A detection engineer app

Setup GuideCCCS write-up it follows:

About the project

Microsoft Defender XDR custom detections as code, with an AI pipeline that researches, writes, attacks, reviews and lab-tests every rule. A detection engineer approves or denies each one, and Sentry traces every run.

It follows the detection engineering loop the Canadian Centre for Cyber Security (CCCS) published from its Frontier AI Lab in July 2026. I rebuilt it on Defender XDR and Azure DevOps and added two things CCCS didn’t have: live web research through Nimble, and a jury of other model families that checks every rule Claude writes.

How it works

  1. Research. Every weekday, Nimble searches 18 trusted security research sites for threat reports from the past week and returns each page’s full text. RSS feeds and analyst write-ups are added too.
  2. Triage. Gemini 3.8 Flash reads each article and decides if it describes endpoint behavior Defender’s tables record.
  3. Draft. Claude Opus 5.5 writes the KQL rule, a hunt query and a Sigma version, and quotes the article for every point. Code checks each quote word for word and compiles the query in advanced hunting.
  4. Attack. Claude Fable 5.1 attacks the rule first, and Claude Opus 4.8 takes over when Fable’s cyber safeguards decline. It lists ways around the rule by CCCS tier: baseline, cosmetic, structural and behavioral. Opus 5.5 hardens the rule, up to 3 rounds.
  5. Jury. Gemini and GPT-6.1 Sol review the finished rule on their own. If either asks for changes, Opus revises once and they vote again.
  6. Pull request. Each rule becomes a pull request with the tier table and every vote. A required check runs it over 3 days of production data, and the bar is zero hits, like CCCS’s.
  7. Lab. After a person approves the fire plan, a Windows VM with Defender for Endpoint runs pinned Atomic Red Team tests. The pipeline waits 10 minutes for telemetry and counts the rule’s hits in each test’s window.
  8. Ship. The rule deploys turned off. A daily shadow run counts what it would have caught, and after 7 days with zero hits it opens a promotion pull request. Completing that turns the rule on.

The first live run

The first rule through the pipeline today, “Defender exclusion added from the command line,” compiled in advanced hunting and had 0 hits over 3 days of real data. Fable 5.1 declined the red-side prompt, and Opus 4.8 answered with 12 bypass variants: 2 caught, 10 missed. Gemini approved it, and GPT-6.1 Sol asked for a revision because the rule also matches reads and removals of exclusions. It deployed to Defender turned off in 41 seconds. The red side and the jury agreed on the real gaps, and that’s the spec for version 1.1.

The run also caught two bugs. Opus 5.5 and Fable 5.1 reject forced tool choice with a 400, so I switched to auto with one retry and added a test that fails if forced tool choice comes back. And when Fable’s safeguards declined, I moved the red-side fallback to Opus 4.8.

Guardrails

  • The AI never deploys anything and never sees hunting results. Checks only ask advanced hunting for counts and column names.
  • No AI-written attack code. The lab only runs Atomic Red Team tests a person allowlisted, pinned to one commit with a SHA-256 per file.
  • AI pull requests can only add files under detections/ai/ and sigma/ai/, and rules can’t carry response actions like isolating a device.
  • Every detonation waits for a person’s approval, and only one runs at a time.
  • Azure sign-ins use workload identity federation, so there are no Azure secrets. The only secrets are the API keys.

What’s in the repo

  • 5 Azure Pipelines: rule checks, drafting, lab detonation, deploy and shadow run
  • Sentry on every run: each model call is an AI agent span with its role, model and token counts, and errors are tagged with the pipeline and pull request
  • 88 offline tests with fake Graph, Azure DevOps, Claude, Gemini, OpenAI and Nimble, plus a capturing Sentry transport
  • A 15-step setup guide with 18 copy-paste blocks for Azure Cloud Shell
  • 7 Claude Code skills matching the ones CCCS described, like /purple-loop

How I built it

I built this at AI Engineer NYC on October 11, 2026. Claude Code on Claude Opus 5.5 wrote most of the code from my requirements. I planned the idea and built a prototype before the event. None of the prototype’s code is in this repo. The full commit history is in my private Azure DevOps repo, and the GitHub repo is a snapshot of main.

Built with

geminiopenaiclaude

AI tools

Coded withClaude CodeClaude Sonnet 5.5Claude Opus 5.5

Prize Categories Entered

  • Gemini Prize Category Q1 2025 - Q1 2026

    Best Use of Gemini API

    Submitted

    Gemini 3.8 Flash does two jobs in the pipeline. First, it triages every article the research step finds and decides if it describes endpoint behavior that Defender's advanced hunting tables record. Only articles it passes get drafted. Second, it sits on a two-model jury with GPT-6.1 Sol and reviews every finished rule on its own: does it match the source, is the KQL right, how noisy will it be, and what variants does it miss. Its answers come back as JSON locked to a schema through responseJsonSchema. On today's first live run, Gemini approved the rule and flagged that encoded PowerShell and direct registry writes bypass it. If Gemini asks for changes, Claude Opus revises the rule once and the jury votes again. Every vote is posted on the pull request.

  • Event prize

    Best Overall

    Submitted

    It takes the AI detection engineering loop the Canadian Centre for Cyber Security published in July 2026 and makes it open source for any Microsoft Defender XDR team. An AI-drafted rule has to earn its way to production. Claude Opus drafts it and quotes the source for every point, and code checks each quote. Claude Fable 5.1 attacks it first, and Claude Opus 4.8 takes over when Fable's cyber safeguards decline. Gemini and GPT review it on their own. Advanced hunting runs it over 3 days of real data, where the bar is zero hits. A lab VM runs pinned Atomic Red Team tests after a person approves the fire plan. It deploys turned off and goes live only after 7 quiet days and a second approval. It ran end to end today on a real tenant: the first rule compiled, had 0 hits, got a 12-variant red-side review and two jury votes, and deployed to Defender in 41 seconds. Three model families check each other, and a person makes every call.

  • Event prize

    Best Use of Nimble

    Submitted

    Nimble is how the pipeline finds new attacks to write detections for. Every weekday, Nimble Search runs 4 queries for threat reports from the past week, limited to 18 trusted research sites like The DFIR Report, Red Canary, CISA and the Canadian Centre for Cyber Security, and returns each page's full text as Markdown. Code checks every result's URL against the same list before any model sees it. Nimble's main-content extraction also drops hidden page content, which matters when every article is a possible prompt injection. When an analyst pastes a single article URL into the pipeline, Nimble Extract pulls that article's main content. Without Nimble, the drafter only reads 4 RSS feeds.

  • Event prize

    Sentry Raffle

    Submitted

    The pipeline is a chain of AI agents, and Sentry is how I see what they did. Each pipeline step is a Sentry transaction. Inside a drafting run, each article gets a span and each role gets a gen_ai.invoke_agent span: triage, Claude on the blue side, Claude on the red side, and the Gemini and GPT jury. Every model call under them is a gen_ai.chat span with the provider, model, input and output tokens, time and finish reason, and gen_ai.handoff spans mark where the purple loop passes the rule between blue and red. Errors from all 5 pipelines land in Sentry tagged with the pipeline, build and pull request. It's a security tool, so only metadata leaves: no prompts, articles, model answers, hunting results or keys. Local variables and default PII are off, and every event is scrubbed of key values before it's sent. Tests check all of it against sentry-sdk 2.71.0.