Garrett Cobb

Program · AI code verification

Dev Instincts

A verification program for AI-written code. Every defect found becomes a reusable check, ranked by what it actually cost, so each review looks first where the expensive bugs are.

Overview

Dev Instincts holds 63 approved instincts, ranked by real cost from a ledger of 627 findings across 3 projects (my private procurement platform, built with AI; Ferrite, an open-source editor written by AI; and Django, as a human-written reference) and 57 review rounds.

Of those findings, 496 set the check order and 131 are reference data from human-written code. The costliest instinct so far, “A rule at one door is a rule at every door,” covers 48 findings. I own the program and approve every proposed instinct before it joins the standard.

The business problem

Passing test suites and AI reviewers kept missing the bugs that cost the most: double submits, races, partial saves and stale screens. Each one got fixed once and then showed up again somewhere else.

Requirements I set

  • Every finding from every project goes into one ledger with its severity, area and the instincts that should have caught it.
  • Checks run in order of real cost, not of how often a problem comes up.
  • No instinct joins the standard without evidence and my approval, and instinct numbers are never reused.
  • The agent that writes the code never reviews it.

Solution design decisions

  • Separate roles instead of more agents. I turned down a manager, lead and junior chain. Writing code isn't the slow part. Checking it is, and every handoff drops detail.
  • Outside data. I mined Ferrite, an open-source editor of 133,000 lines written entirely by AI. It gave 154 bugs from that project's own records, 14 hidden bugs nobody had recorded, and 9 new instincts. I filed 6 of those bugs with the project, each with code lines and a probe.
  • Graded sources. I collected 35 published sources on code written by AI and graded each from strong to anecdotal. A check backed only by weak sources stays a proposal until the ledger supports it.
  • Stop rule. If defects don't roughly halve each pass, or regressions go above about 10% of fixes, patching stops and the code gets restructured or rebuilt from the baseline.
  • A round that got away, kept on record. In one fix round the architect dropped out after planning and the builder graded its own work: 15 of 34 fixes caused regressions. With the architect back and tests written before the code, the close had none. Both halves are in the public ledger.

How the project was run

  • A stats tool ranks instincts by cost. Data loss and security weigh 8, cosmetic issues weigh 1, and regressions count double. Changes get checked in that order.
  • New instincts start as proposals with evidence. I approve them before they get a number.
  • Three agents work separately. An architect agent writes the plan and the acceptance tests with expected values worked by hand. A builder session writes the code. A reviewer agent that didn't write it checks it.
  • A senior coder agent, still in calibration, reads the code and data model before a rule is chosen and writes down where each option breaks.
  • Documents produced: the instinct standard, findings ledger and stats, convergence tracker, audit spec, review and verify templates, agent definitions, two case studies.

Results

MeasureResult
Approved instincts63, numbers never reused
Ledger627 findings across 3 projects and 57 review rounds
Check order496 findings set it; 131 are reference data from human written code
Costliest instinct“A rule at one door is a rule at every door,” 48 findings
Outside calibrationFerrite: 154 recorded bugs, 14 unrecorded bugs found, 9 new instincts
Upstream6 bugs filed on Ferrite (issues #195 to #200)
Publicgithub.com/garrettcobb/dev-instincts, MIT

What I learned

Green checks prove what the code was meant to do, not what it does. Ranking checks by what defects actually cost tells every review where to look first. And the gate has to sit before the rule is chosen: when the planner dropped out, fixes started breaking other fixes.

How AI was used

What AI did: Three Claude Code agents with separate jobs: an architect writes the plan and hand worked acceptance tests, a builder writes the code, and a reviewer that didn't write it checks it.

What I did: I approve every instinct before it gets a number, set the cost weights and the stop rule, and decide when patching stops.

How quality was controlled

  • Check order ranked by real cost from the ledger
  • Approval gate before an instinct joins the standard
  • Convergence tracked on every verification pass
  • No sign off without the real pass count and coverage line

Open source under MIT. The method, agents, ledger and Ferrite case study are public; findings from my private platform are published without file paths or report links.