Closed loop · prototype 0.1 · 29 October 2026
How to use AI for games
Our internal training manual at TNT Tools, adapted for Heavy Iron. The method is the closed loop: requirements first, tested by AI, so nothing is a guess.
An AI that tests the requirements before it writes the code: testing raw. The slides are the highlights. This is the whole story in plain words: what each part does, why it is there, and for every number on the slides, the test that produced it and what the number means.
Every slide has a link to the section here that backs it up. Each number carries a tag:
Contents
The key is accuracyRequirements firstAI as a tester firstWhere it comes fromTesting rawThe idea in one minuteRequirements make requirementsWhy use itWhen it fitsWhy a thin model misleadsSeed it first, then it fliesHow to run itThe ten pointsOne pass, end to endThe modelBreaking code on purposeWhy it is fast enoughThe right test must noticeBreaks that change nothingClaims and locksThe knowledge graphThe 18 checksMust and mayThe GNNFresh reviewersArgument chainsBuzzsaw testingWho, what, why: where askedShaping the graph for the GNNThe test subjectsOne way flattersThe tool on itselfOn public codeSelf-improvement, on by defaultTelemetryHow the stats chooseWhat it can learnThe loop on its own improvementHow new is itAmong other toolsWhat it is not, yetThe two targetsInstallOther languagesYour harnessWords used hereThe key is accuracy
Every value an AI guesses is one somebody has to check later. A value read from a source, with the source named, is one they can check.
The gap between what AI guesses and what is true is what people deem AI slop. At TNT Tools the standard is no slop: every value is either read from a source, with a note of where, or marked absent with the reason. There is no third kind.
The numbers on code are in three trials on code you can read for yourself, and the target for agents writing code using the loop is still a target.
Back to 1999: requirements first
This is how we review our code: we pour the effort into requirements first, review them before any code, then iterate on milestones like everything else. It is time to learn systems analysis and design again.
The reason is mechanical. The more requirements you have, the more of the loop closes, so that nothing is left to a guess. That is what gets you slopless code.
Where this comes from
Measured On Unreal Engine’s SemVer parser the requirements were written first, from the public Semantic Versioning 2.0.0 spec: 16 requirements, every numbered rule mapped to one or excluded with a reason. Writing them against the code found 4 faults that fail when run, before any break was made, and 7 of the 16 have no test about them. See on code you can read for yourself. The requirements and the reading found those faults; the breaks add the count of what no test holds.
There is no such thing as "we don't use AI"
AI is a first-class testing tool first, and a code writer second. Write every line by hand if you like: it will still tell you which of your tests cannot fail, and which code no test notices. When it does write code, it is held to the same requirements and the same tests as everyone else.
There is no reason to throw away really well done tests. That is the baby with the bathwater. Your tests become the judge of what AI does.
Where it comes from: a mutation of an ARC3 self-improvement loop
This toolkit is a mutation of the self-improvement loop from our ARC3 harness. It was first built against one of our own game features, which is internal and cannot be extracted, so what is shown here is run on public code instead: Unreal Engine’s SemVer parser and JSON object, osu!’s scoring rules, and the toolkit’s own source. What came across is the part that survived: propose, then verify; abstain rather than guess; and never trust a number you could game.
What the ARC3 work measured
Measured A weak proposer and a strong verifier beat either alone. On a frozen set of 63 games, at a budget of 150 steps, the unified agent reached 31 where random reached 14. At 250 steps it reached 35 against random's 29, reproduced identically twice.
Measured The failure that shaped this tool. An autonomous self-improvement run went 16 days and 330 commits and found nothing that generalized. A counter that could be gamed rose from 147 to 168 while the honest number stayed flat. That is why here every stat must have a reader, a control and a fresh reviewer, and why the loop audits its own stats.
Measured The honest baseline. The one-shot solver covered all 249 games and reached 106 of them, but blind transfer to games it had not seen was about 1 of 81. We report that number because it is the real one.
If this harness is useful to you and working together does not happen, perhaps the harnesses can merge: it is self-improving, so a merged one keeps improving for both sides.
The point of all of it
Testing raw: AI that tests the requirements
We test code. Requirements get reviewed by people, now and then. What AI adds is requirements testing done by a machine, all the time, at the scale of the code. This loop is built for exactly that.
The requirements are the code behind the code. When an agent writes the program, it writes it from the requirements, so in its rawest form the requirements are the code. A mistake in a requirement becomes a mistake in every line written from it. Testing before that stage, on the requirements themselves, is what we call testing raw.
It makes new requirements by finding failures in the ones it has. The loop walks each use case step by step and asks what every step needs. When a step needs something no requirement provides, that gap is a failure of the requirements, and it becomes a new requirement: marked inferred, and put to the owner as a question.
Measured
Measured On SemVer the model holds 16 requirements taken from the spec and 3 marked inferred: what the spec leaves unsaid and the class needs, each put to the owner as a question, for example “Parse throws on exactly the strings TryParse refuses”. On osu!’s scoring rules all 30 were written from the code’s own doc comments and none was inferred.
Open How many requirements a rough epic grows into, and whether code written from them is more accurate, has not been measured: see the two targets.
The code still gets tested. Breaking the code on purpose (how) is how the loop checks that each requirement really reached the code. But the requirements are tested first, because that is where a mistake is cheapest to catch.
Very little work has been done on AI requirements testing yet. As agents write more of the code, the part people write is the specification, and the work moves back toward writing it first, closer to waterfall. Testing that specification becomes the main job, and that is where this loop puts its effort.
The idea in one minute
When an AI agent writes code, it usually finishes by saying it is done. Most of the time you cannot check that without reading everything it did. The closed loop makes “done” something a machine can check.
Think of a building inspector with the plans in hand. The plans are your requirements, written down as data instead of as a paragraph. The inspector never takes the builder’s word. It checks every item on the plans against the building, and for every part of the building it asks which item on the plans that part is for.
What keeps the inspection honest is that the inspector also breaks things on purpose. It takes a piece of working code, changes it slightly so it is wrong, and checks that at least one test notices. If no test notices, nothing is really checking that code, whatever the test report says.
At the end you get a count of three things: breaks no test noticed, code no requirement asked for, and questions only the owner can answer. “Done” means that count is zero.
Why a count and not a feeling
“Did I think of every requirement?” has no answer: what you forgot leaves no trace. “Is the list of open questions empty?” does have one, because the list is generated from the plans and the code, not from memory. So the loop never tries to be complete by writing more. It walks the plans, and every gap it meets becomes a question with a file and a line.
On the slides: slide 8.
It extracts new requirements from the ones you have
The tool automatically extracts requirements from existing requirements. You do not list them by hand. It walks each use case step by step: every step needs something to be true, and when no requirement provides that, the chain has inferred one nobody wrote. It is marked inferred and goes to the owner as a question, because only the owner can decide it. See testing raw for what that found on SemVer.
Complete in, accurate out
The more complete the requirements, the more accurate the prompt built from them: an agent acting on them has less to guess. Which ones you write first matters as well, so start with the ones that run through the most code (seed it first).
Where two features meet
A large new feature often joins features you already have. We expected the walk to find the gaps where two such features meet, because each side’s requirements are already tested. We tried it once, on two features of our own game, and it does not: joined, the walk asked no new question, and an agent reading both models named places the graph missed and found real faults neither model states. No count is given because the pair is not public.
Why the graph misses them: only sets and branches make an edge to a name, so two requirements that call or read the same thing do not meet, and neither model says what it offers the other. It would take three things, none of them built: calls and reads as edges, each model declaring what it offers and what it takes, and a check that compares the two lists.
Open One pair, and not blind: the second feature was built as a clone of the first, both documents name code classes, and the agent had read one model first, so the reading number is an upper bound. Whether to remix rather than invent was not measured, and remixing with zero errors was not shown. That more complete requirements give more accurate prompts is our judgment from building with it; no run has measured it directly.
Why use it
Requirements written as prose cannot fail
A paragraph saying “the player can buy an item if they have enough money” never turns red when the code drifts away from it. The loop keeps each requirement as a record with an id, its source, and the code it points at, so a requirement that no longer matches the code shows up as a failure.
Tests can pass for the wrong reason
A test that checks the price shown on screen passes even if buying never takes the money out of the wallet. Every test is green and the rule is broken. Breaking the code on purpose finds this: take the line that subtracts the money out, and see whether any test fails. If none does, the rule has no test, however many tests there are.
Long runs forget
An agent’s working memory (its context window) fills up during a long job, and early details drop out. Later steps then guess at what was settled before. The loop writes everything it learns to a graph on disk, so a later step looks the answer up instead of guessing it again. Every entry says where it came from.
On the slides: slide 6.
When it fits, and when it does not
It is built for large, complicated systems. To build something simple, do not use it. It has not been measured on large systems yet.
Use it for larger systems: code that many changes will touch, where a rule must still be true after the agent’s tenth edit as well as its first, and where you want to know which line exists because of which requirement. It also helps most when agents run for hours, across many context windows.
Skip it for a one-off prompt, a quick script, or exploring an idea. The loop costs setup and run time, and it pays that back only when the same requirements are held across many changes. It is also no help with how something looks or feels: visuals, timing and feel still need a person.
On the slides: slide 15.
Why a thin model can be worse than nothing
The loop can only check what is written in the model. Suppose the model holds three requirements and two tests. The loop may honestly report everything closed, because everything it knows about is checked. But a green report reads as “the work is complete”, when all it proves is that three small things are right. That false confidence is worse than having no report at all.
How large a model must be before its report means something is only partly known. The first measurements are in Seed it first, then it flies. Until there are more, treat a green loop over a small model as “these few things hold”, never as “it is done”.
Open How many requirements, tests and graph nodes are “enough” for the whole report is not known yet; for the part that learns there is a first answer.
On the slides: slide 15.
Seed it first, then it flies
The loop learns from what is already written down. Give it a model with a handful of requirements and it can check those few, but it has little to reason from when new code arrives. Give it many, with their code claimed, and it starts to see the patterns: which requirement a new branch most likely belongs to, which questions follow from which. So the order of work is: seed the model with as many sourced requirements as you can, then let the loop carry it.
How full is full enough
Measured while the network ran, before it was switched off on 3 November because a plain rule beat it on the tool's own code. Whether the same order of writing helps the owner head that replaced it has not been measured.
We measured this for the part that learns, the graph neural network (GNN). For each graph it was allowed to see only a share of the known links between code and the requirements that own it (a tenth, a quarter, half, three quarters, all of them), always tested on methods it had never seen, and compared with a plain rule that does not learn: “suggest the requirements whose effects already sit in this method”.
Where these numbers come from
Measured The two kit examples (4 requirements each): the plain rule wins at every fullness, 0.90 and 0.86 against the graph neural network (GNN)’s best of 0.78 and 0.85.
Measured The tool’s own model (20 requirements): the graph neural network (GNN) wins only at half and falls back after, so it has no dependable crossover yet.
Measured The largest graph measured, a game feature of ours that is not public: the GNN lost below half the links and won from half on, against the same plain rule. No count is given because the feature is not public, and the curve has not been drawn on the public targets here.
Two seeds each, five folds by method, 29 October. That is thin: read it as a first shape, not a law.
So on a small model the GNN stayed quiet and the plain rule did the work. Where it has earned its place, it did so only on the largest graph, and only once about half of its code was claimed.
Some requirements are pillars
Which requirements are written matters as much as how many. A few carry most of the code. On osu!’s scoring rules, 13 of its 30 requirements own 80% of the branches a requirement owns, and the top three own 28%; on SemVer 11 of 19 own 80% and the top three 37%; on JsonObject 9 of 27 own 80% and the top three 46%. On the tool’s own model, 7 of the 12 requirements that own any code own 80% of it, and one of them runs through 16 methods. Seed those first: a rule that runs through many methods gives the rest of the graph something to attach to.
The same curve was then seeded by requirement instead of by method: only some requirements' claims were seen, added in three orders.
Where these numbers come from
Measured On the largest graph, the game feature of ours: the GNN beat the plain rule in random order only once every requirement was seeded. Seeded pillars first (the requirements owning the most code) it won from about a quarter of the requirements, and seeded cross-cutting first (the requirements running through the most methods) it won from the first three. No counts are given because the feature is not public.
Measured On the tool's own model (20 requirements) no order makes the GNN beat the plain rule.
One seed each, five folds by method, 29 October. Thin, but the gap between the orders was large.
So which requirements are written first matters more than how many. On the one graph it won on, the few that run through the most methods gave the learning part what a random start needed nearly all of them for. Write the cross-cutting rules first.
The smallest seed that grows into the whole loop
The loop already grows requirements. It walks each step, asks what the step needs, and every need nobody wrote becomes a new requirement for the owner to settle. On SemVer, 3 of the 19 requirements are marked inferred that way, on top of the 16 that cite the spec. So the open question is how small the written set can be and still grow into the full model. A small, well-chosen seed would mean less to write up front, and a structure that is easier to keep consistent.
Open The experiment for it: give an agent only the problem story and a few pillar requirements, let it grow the model using the loop, and count how many of the full model’s requirements it arrives at, and at what cost.
On the slides: slide 24.
How to run it
- Install.
python install.pyputs the skill where Claude Code finds it and the tools beside it, then runs every selftest and the whole loop on a small example project. The example must pass (“hold”), or the install says so. See Install. - Write the model of your system. First the problem story in your own words, each sentence with its source. Then the requirements, each pointing at the code it is about. Then the use cases: who wants what, step by step. Then a short block that names your code folders and your test files.
python -X utf8 usecase_model.py <model>checks the 19 rules (see The model). - Run the one command.
It breaks your code every way it knows, runs your tests against every break, builds the graph, applies the 18 checks and writes a report.CLOSED_LOOP_ROOT=<your repo> python -X utf8 closed_loop.py run <model> --out REPORT.md --pin-claims "<why>" - Close what is yours. The report sorts what is open into two kinds. Yours: a missing test, a line no requirement claims, a break that changes nothing and needs explaining. The owner’s: questions about what the system should do, which only the person who owns the requirements can answer.
- Take the lock. When nothing of yours is left,
--accept "<why>"records the verified state. A later change to that code makes the affected requirements stale until they are verified again. The lock refuses while any hole of yours stands, so it can never be used to turn a red report green.
On the slides: slide 9.
The ten points, each with its section
- What it is: one command that holds an agent’s requirements to the code, the tests and the agent’s own reasoning. More
- How it tests: it breaks every decision in the code on purpose, every way it knows, and each break must make a test fail, the test of the requirement that owns that code. More, the right test
- Speed: every break is built in one compile, then each runs alone. More
- The graph is the record: every piece is a node, and the checks read the graph. More
- Three trials on code you can read: breaks caught and bugs run. More
- Equivalents: a break that changes nothing is proven or explained, never waved away. More
- The test subjects: a published spec, a JSON object and a shipped game’s scoring rules, each chosen for a reason. More
- The GNN: a learning model that suggested which requirement owns new code, switched off on 3 November, because a plain rule beat it. More
- A fresh reviewer each pass: a new agent hunts for ways a check could pass wrongly. More
On the slides: slide 5.
Part 2 · The pieces
One pass, end to end
- The model is read and checked against its 19 rules.
- The code is broken: every statement the requirements cover, every way the tool knows, all built at once.
- The tests run against each break separately. Each break should make some test fail.
- Everything goes into the graph: requirements, steps, code places, breaks, tests, and which test caught which break.
- The 18 checks read the graph. Each one is named after a mistake it stops.
- The verdict is either HOLDS, or OPEN with a list of questions and whose each one is.
You close your questions and run again. That repeat is the loop.
On the slides: slide 9.
The model: your requirements as data
The model is a few small text files (YAML) beside your code:
- The problem story: one paragraph on what the system is for, each sentence with where it came from (a person, a design document, a date).
- Requirements, each with an id, its text, how to check it, its source, and the code it points at. A requirement is authored (someone wrote it) or inferred (the loop found that a step needs it and nobody wrote it). Inferred ones are questions for the owner, never facts.
- Use cases: who wants what, as 3 to 11 numbered steps, plus what happens when a step cannot complete. Each step names the requirement it relies on.
- The loop block: which code folders and test files the loop works on.
The 19 rules refuse a model that is malformed before anything runs. Some examples of what they refuse, in their own words: a use case with too few or too many steps; “step 3 carries no requirement”; “a requirement with no text”; “no measure: nothing says how to check it”; a story sentence with no source; two items sharing one id; a use case with “no alternate pathway: what happens when a step cannot complete?”; a requirement carried only by summary steps, so no real user goal ever reaches it.
On the slides: slide 9.
Breaking the code on purpose
The usual name is mutation testing, and each broken copy is a mutant. This manual calls them breaks, and the slides call them planted bugs. A break that makes some test fail is caught (or “killed”). A break that every test still passes got past (or “survived”), and that points at a missing test.
A small example
if (wallet.Money >= price) // the real code: exact money is enough
if (wallet.Money > price) // a break: paying the exact price now fails
If every test still passes with the second line, then no test buys something with exactly enough money. That is a missing test, and the report names the line.
The kinds of break the C# adapter makes, in plain words:
- Conditions: turned around, forced always true, forced always false.
- Operators:
<made<=,==made!=,&&made||, plus made minus. - Values: a number made one more, true made false, a text emptied, a function argument replaced by its default.
- Structure: the two outcomes of a
? :swapped, an else branch removed, a statement removed, the whole method made to return at once.
Each statement gets at most 24 breaks, one of each kind first, so a busy statement cannot crowd out the kinds of break that matter most.
On the slides: slide 10.
Why it is fast enough
Building each broken copy separately would mean hundreds of compiles. Instead, every method is copied once per break inside one build, and a switch picks which copy runs. One compile serves every break. A copy that does not compile is found by the line and column of its error and dropped, and the rest are rebuilt.
Each break then runs the tests alone, in its own copy of the build in a temporary folder. The live project is never touched.
Where these numbers come from
Measured On 30 October a whole pass took about 13 minutes on SemVer (246 breaks), about 9 on JsonObject (582) and about 74 on osu!’s scoring rules (563), each break run alone in a copy of the project, two at a time, at low priority on a busy machine. osu! is slow because every break of a constructor rebuilds the game.
The right test must notice
It is not enough that some test fails when a requirement’s code is broken. It should be that requirement’s own test. Otherwise the requirement is only protected by accident: a test written for something else happens to fail today and may not tomorrow.
So each test says what it checks, on the line above it:
/// CHECKS: W-1
[Test]
public void Pay_TakesExactlyThePrice() { ... }
A requirement whose named test misses its own breaks is a hole, even when another test caught them. And a test named as a requirement’s check that catches none of that requirement’s breaks is a hole too, so listing more tests can never make the score look better.
Breaks that change nothing
Some breaks get past every test because they do not change what the program does. Example from the kit’s own example: in Store.Buy a line with none left is already refused two statements earlier, so changing > 0 to >= 0 changes nothing the program can do. No test can catch that, and none should. These are called equivalent breaks.
The danger is calling a real missing test “equivalent” to make it go away. So the loop accepts an equivalent in only two ways:
- Proven by the compiled code. C# compiles to IL, the instructions the runtime actually runs. If the broken copy’s IL is identical to the original’s, the two are the same program (a technique called Trivial Compiler Equivalence, Papadakis and others, 2015). The loop checks this itself.
- Declared with a reason. Otherwise a person or agent writes down why it changes nothing, and names the methods the reason depends on. If those methods change later, the reason lapses and the break is counted as a hole again.
And if a test ever does catch a break that was declared equivalent, the declaration was wrong, and that is reported as a hole. This has happened: breaks that looked harmless could change the result, and both are now held by tests instead.
On the slides: slide 10.
Claims and locks
A claim says “this method belongs to requirement X”. The loop pins each claim to a fingerprint of the method’s statements. If someone adds a line to that method or deletes one, the fingerprint changes and the claim goes stale. A new line can never slip in under an old claim without being looked at.
The pins are kept in a small file (claims.lock). If that file goes missing, every claim is unpinned until a person pins them again with a reason. A missing lock never quietly re-pins itself; a reviewer found that it used to, and it was fixed.
The node graph, and flying through it
Everything the loop knows about a feature is one graph: who asked for what, which code does it, every way that code was broken, and which test noticed. The graph of osu!’s scoring rules has 1,062 nodes and 5,675 edges, SemVer’s has 448 and 1,554, JsonObject’s 1,091 and 3,840. Each is stored as SQLite and N-Triples. The fly-through of osu!’s graph draws such a store as it is, with nothing added and nothing left out. The pictures behind the slides are stills from the same page, of osu!’s scoring rules.
How it is laid out
The store has no shape of its own, so the demo gives it one you can read from top to bottom, one ring for each kind of node. On osu!’s scoring rules:
- Why it exists: 1 source (the code’s own doc comments), 30 requirements (all written by hand, none inferred), 1 owner’s question, 5 use cases (one epic and four under it) and their 17 steps.
- Claude’s reasoning: a chain and its steps, each citing what it stands on. This trial wrote no chain, so the ring is empty; argument chains has one, run on the kit’s example.
- What the code does: 34 methods in a ring, sorted by name, so a class’s methods sit together.
- What proves it: 200 branch cells (191 branches and 9 for what a judgement moves), 563 breaks and 168 tests, of which 151 ran. Each branch and each break sits at the angle of the method it belongs to, so a method and everything under it line up.
How code maps to requirements
Two kinds of edge do it. A branch maps to a requirement: 339 of those. A branch is at a method: 191. Together they take you from any decision in the code to who asked for it, and from a requirement to all the code that answers to it. A decision no requirement owns is a hole, and the loop lists it.
What the red points are
The 277 breaks no test caught. Red is what the graph records, not a verdict: 250 got past every test that ran, 13 are proven by compiled code to be the same program and 14 did not run. The 250 are the real work: none has a written reason. That is what the 53.4% means: 286 caught of the 536 live breaks, leaving out the 13 and the 14. It is what three test files hold, not what the game’s whole test project holds.
The demo
Drag to orbit, scroll to zoom, W A S D to fly, Q and E for down and up. The tour follows one real thread: a source, a requirement, the step that needs it, Claude’s reasoning, the code, a branch, a break of that branch, the test that caught it, and a break that got past. The Show menu narrows it to one question: how code maps to requirements, every break and what caught it, Claude’s reasoning, requirements and use cases, or what the graph neural network reads.
It is a tool in the kit, for any graph the loop has written. --hide keeps names you choose off the page:
python graph_flythrough.py <store.sqlite> --out page.html --hide "NameA|NameB"
The knowledge graph
A graph here is just a list of things (nodes) and the links between them (edges). The things are requirements, use case steps, places in the code, branches, breaks, tests and reasoning steps. The links say how they relate, for example:
requirement W-1 is checked by test Pay_TakesExactlyThePrice
break #312 was caught by test Pay_TakesExactlyThePrice
method Pay is claimed by requirement W-1
The graph is the record: the 18 checks read it, and a later agent reads it instead of re-deriving what an earlier one found. It is written on every run as a SQLite database and as an N-Triples text file, a standard format any graph tool can open.
How big it gets
Measured The graph of osu!’s scoring rules holds 1,062 nodes and 5,675 edges, SemVer’s 448 and 1,554 and JsonObject’s 1,091 and 3,840, each read from its run’s line in the morph log.
The documents are nodes too. When a number or a behaviour changes, something has to say which documents go stale, so a model can declare its documents and each section, slide and page becomes a node. A link from a requirement, stat, goal or method to a document exists only when the document names it literally, or an authored line says so and quotes the words it stands on. Likeness never makes a link. doc_nodes.py what-to-update <node> answers which documents describe a node. A document naming a requirement the model does not hold is a hole. A requirement no document describes is counted and named, not failed, because most models have never declared any documents.
On the slides: slide 13.
The 18 checks
Each check is named after the mistake it stops. Each one is proven by a planted mistake: the tool’s own selftest builds a small model with that exact mistake in it and requires the check to catch it. A check that has only ever passed cannot be told apart from one that is switched off.
The numbering skips L17 and L18. L17 is a planned check from the research (vacuity: a requirement’s check must fail when its condition is forced off), not built yet.
On the slides: slide 9.
Must and may: what “closed” means
The research behind the loop looked at how logicians describe “complete”. The closest fit is a Modal Transition System (Larsen and Thomsen, 1988). Every behaviour is marked either must (it has to happen) or may (it is allowed, not yet decided). A specification is finished when no “may” is left: every open choice has been settled one way or the other.
Here, the authored requirements are the musts. The inferred ones, the things the steps need that nobody wrote, are the mays. Each may is a question for the owner, who settles it as a must or as “either is fine”. On SemVer, “Parse throws on exactly the strings TryParse refuses” is one such may: the spec says what is valid, not how a class refuses, and no one has said.
The honest limit
Breaking code can show that requirements are not closed. It can never prove they are. “Closed” always means “closed against the breaks we tried and the tests that watched”. A new kind of break might find something new.
On the slides: slide 13.
The requirements map to the code, and the map finds mistakes
This is an early prototype. Every requirement links to the code it owns, in one graph. The graph's criteria and the breaks do most of the finding so far. A graph neural network (GNN) was tried on the same map and switched off on 3 November, because a plain rule beat it; the owner head does its job now (the GNN).
What the map has found
Measured 7 of the 16 requirements taken from the SemVer spec have no test about them, including every rule about which strings are valid. Which test is about what was our reading of each test, written down before any break ran.
Measured 110 of the 191 branches of osu!’s scoring rules have a break that got past the three test files that were run, and every one of the 191 is claimed by a requirement, so those 110 are tests missing, not requirements missing. Read it as what those three files hold.
Measured On public code: a public asteroid game's document modelled as 48 requirements. 10 rest on a function's behaviour with no constant behind them, and the parse check caught a document row missing a column. Language-neutral parts only, with no breaks, since there is no JavaScript adapter; a thin graph, where a green result means little.
Measured On the toolkit's own code: 885 of the 1,532 breaks caught. See the tool on itself.
Open That trial is done, on C#: see code you can read yourself. There is no C++ adapter, so on C++ it would run the language-neutral parts only.
Switched off on 3 November
On the tool’s own code the network lost to a plain rule, so it is off. It wakes when a project’s graph reaches 5,000 entities, and must then win by more than 0.02 on code it has not seen. Its jobs are done by two small checking heads and by reading the test runs; this section is the record of what it did while it ran.
How the GNN works, what it records, and how it tries to improve
How it works
- It reads. Each node starts as its own words, hashed to 512 numbers.
- It listens. GraphSAGE layers let each node hear its neighbours, relation by relation. After that a node's numbers say something about what surrounds it.
- It ranks. A scorer ranks every requirement for a decision that has no owner.
- It asks. Below 70% sure it abstains. Above that, its answer goes out as a question for a person or an exact check. It is never a verdict.
It never reads breaks, tests or chains. Those hold the answer, and new code would not have them. Two ways that answer leaked in were measured and closed before any score here was taken. Whole methods are held out when it is tested, so each score is on code it has not seen.
What it records
- Its score beside the plain rules, on each pass run with
--gnn(stats.jsonl): the GNN, the best plain rule, the words alone, random and the same method, in mean reciprocal rank, where 1.0 is always ranked first. - Each job against the best plain rule (
kg/gnn_tasks.jsonl): who owns new code, which tests catch a break, which steps carry a requirement, which breaks are the same program. The results. - The shape of what it reads: its nodes and links, how many are owner links, how many methods, and how those links pile up: how few requirements hold most of them.
- Breaks ranked (
kg/survivor_rank.jsonl): how many of the survivors a learned order finds in the first fifth of a run, on average over ten seeds, against the plain rule and random. One seed flatters, so only the mean over seeds is recorded. - Its curve (
gnn_curve): its score by how much of the links it can see. - How sure it was: at 70% sure or more, how often it was right and on how much of the new code, by seed.
- Every tune (
gnn_tune.jsonl): each variant's score, and what was kept.
How it tries to improve
Its lead over the plain rules is itself a stat. When that lead falls as more tests arrive, the telemetry reports it as worsening and names reshaping the graph as a next job, beside writing tests. A tune is due after 5 passes without one anyway. A standing tick or a person then runs it: graph_gnn.py tune <graph>. A tune redraws what the GNN reads, with each relation left out and each derived link added. A change is kept only if it wins by 0.02 on the tuning seeds and again on a seed it was not chosen on.
So far
Measured Tunes have been run and nothing was kept: no variant won by the bar of 0.02. It attempts to improve itself, and the attempt has not paid yet.
Open A project's run plan records whether each learned job wins there. 3 of its 7 lines reach what a pass runs, wired by the tool's own tick, and all 3 are off today because a plain rule wins. The other 4 are measured only.
The GNN: who owns new code?
When someone adds code, the loop needs to know which requirement it belongs to. The GNN (a graph neural network, a learning model that reads the graph) suggested an answer, until it was switched off on 3 November, because a plain rule beat it. It looked at what was already linked around that code and ranked the requirements from most to least likely owner.
How it was tested fairly
It was trained on part of a project’s graph, with whole methods hidden from it. Then it was asked to rank owners for the hidden methods, code it had never seen. It was compared against simple rankers that do not learn, such as “the requirements already linked to this method” or “the requirement whose words overlap the code’s names most”. It was only allowed to make suggestions if it beat them.
What the numbers mean
MRR (mean reciprocal rank) scores a ranking: the right owner ranked first scores 1, second scores ½, third ⅓, and so on, averaged over every test case. An MRR of 0.75 means the right owner is usually first, sometimes second.
Where these numbers come from
Measured On the tool’s own code, with whole methods held out and a mean over five seeds, the GNN scored MRR 0.65 against 0.72 for the best ranker that does not learn: it loses, and is switched off there. Runs differ by about a tenth, so a single number is never the whole story.
Measured It only wins on a large enough graph that is about half claimed: see Seed it first, then it flies.
Measured The design is to abstain: below 70% sure the GNN says “not sure” and asks. How often it is right at 70% and on how much of the new code was measured on one project of ours, and is not repeated here because that project is not public.
Its suggestions were always questions for a person, never verdicts. One lesson from building it: the first version did worse than the simple rankers, because the links it was asked to predict were also links it could read. Separating the two fixed it.
Two lists: what they train for, and what they could train with
Goals are how we understand what we are training for, and trying for a goal does not mean reaching it. So every goal is listed with what it was measured against and whether it got there, beside a second list: every stat and piece of metadata on file, what it does today, and which goals use it. The two work on each other through the node graph. A goal that loses to a plain rule asks for what stands behind that rule as an input. A stat no goal reads is a possible input, or something worth predicting. A relation in the graph that nothing predicts is a goal on offer.
Tried, and did not work: a goal as a vector
Measured We turned each goal into a list of numbers measured before training and fitted a direction to what paid off, then tested it on graphs it had never seen, with its bars and predictions committed first. There is a weak direction for what a network can learn, and none for what saves work. The two point about 95 degrees apart. Copying a goal's sources could game it, and it ranked a random relation high. It rests on 13 graphs in 5 domains, so it can show a direction is not general, not that none exists.
Measured What did hold: goals where each answer belongs to about one source beat a plain rule far more often, right 80% of the time against a 48% base, on 44 goals across 5 graphs. And ordering tests by a plain rule already saves most of what ordering them can: a learned order beat it on 2 of 10 goals.
Both lists are generated from the records, a job no goal claims fails the check, and each run logs what moved. See the Goals List and the Stats List.
The tool on its own networks
The requirements for each learned part are a subgraph of their own: one use-case model per part, with the stats it reads and records as a closed list inside it, all joined in one graph of 3,298 nodes and 8,863 edges, where 30 stats are shared across parts. The loop then ran on each part, breaking its code, and each part was run on a graph of itself.
What it measured, on 30 October
Measured No part beat the plain rules on its own code. Who owns new code: 0.65 against 0.83. Which tests catch a break: 0.83 against 0.92. Which steps carry a requirement: 0.66 against 0.81. Which breaks get past the tests: 0.15 against 0.19. Two were too thin to say: which breaks change nothing, and where to add a test. One seed each, so these are not means.
Measured 48 requirements, 9 of them inferred. 144 stats, used 233 times across the parts, 32 of them read by nobody. 1,174 of 1,816 breaks caught, 642 got past. In the first, partial run, 115 of the 369 catches came only from new guards written from the requirements, because the tool's own selftests had passed those breaks.
Measured Every break has a verdict: none is left unrun. The first run left most of them unrun, so its counts were lower bounds; these counts are not.
What it found, none of it fixed yet
- With the guard that keeps the answer out of what a network reads removed, the network's own selftest still passes. Only the new guard catches it.
- One win is probably not one: which breaks change nothing, on the tool's own code, learned from labels written before a fix to how breaks are compared. Inferred, not re-run. On both projects every example it learns from is a break a pass already removes, so its score says nothing about the survivors a person reads.
- The bar for showing a proposal has no margin, so 0.51 against 0.50 passes, and it leaves out the plain rule that actually wins on one project.
- The run plan reads only the last row, and the row names no seed, so two losing seeds and one winning one switch a part on.
- The code that writes a score is not held by a test: 32 of 32 breaks of it got past.
- The Python adapter only breaks statements inside functions, so the thresholds and the guard list themselves were never broken by it.
Running the rules on the improvement model itself, a level up, is still open.
A fresh reviewer every pass
After each pass, a new agent that has not seen the work is asked one question: how could each check pass while the requirement is not really closed? A fresh agent is used because one that built the work tends to confirm it.
What they found
Measured The first reviewer found 9 such ways, all fixed. The second found 10; 8 are fixed and 2 are open. Three examples:
- Deleting a line made the score better. Now the claim’s fingerprint moves and the claim goes stale (claims).
- Naming more tests as a requirement’s checks could only remove holes. Now a named test must catch a break of its own requirement (the right test).
- A deleted lock file re-pinned itself in silence. Now a missing lock leaves every claim unpinned.
Open A structured claim on each reasoning step (so a step that says “or” where the code says “and” fails).
Every fix lands with a planted mistake the selftest must catch, the same rule as for the 18 checks.
On the slides: slide 23.
Argument chains: holding the agent’s explanation to the requirements
Breaking the code checks the code. A chain checks the agent’s explanation of the code. When an agent finishes a change, it writes down, step by step, why the change does what the requirements ask: each step is one claim in plain words, plus what that claim stands on. The loop then checks every step against the requirements, the code and the graph.
A chain is a written argument the agent produces on purpose, like showing your work. It is not a recording of the model’s private thinking, and the tool does not try to extract that.
What a chain looks like
A real one, run on a copy of the kit’s example, a store’s wallet. It names the question it set out to answer, then its steps; each step says one thing and cites what it stands on: a requirement’s id, a question put to the owner, or a method in the code.
chain:
id: CH-EXAMPLE
question: "Does a purchase stop when the stock is gone?"
steps:
- says: "A limited line sells its quantity and is then sold out"
cites: [W-3, Store.Buy]
The four checks
- Grounded. Every cite must exist: a requirement in the model, a question the owner was asked, or a method the code really has. A step that cites nothing, or cites something that is not there, fails the run.
- Same words. The step’s words should share at least one word with what it cites. If they share none, the step is flagged for a person (“a new finding, or a mistake”). This is a hint, not a failure.
- Joined in the graph. For every pair of things one step cites, the graph must already connect them within four links. The shortcuts that would make everything look connected are not counted: the chain’s own links, the shared “space” node, and the records of breaks and tests. A pair the graph does not join is either a link the model is missing or a mistake in the argument, and a person decides which.
- Blind spots. Every inferred requirement (an owner’s open question) that no step cites is listed: the chain never looked there.
The chain also becomes part of the graph: a node for the chain, a node for each step, and a link from each step to everything it cites. A later agent can see which explanations rest on which requirements.
How a chain meets the code
The chain does not sit beside the graph: it goes in. Take that step, “A limited line sells its quantity and is then sold out”, citing W-3 and Store.Buy. The step becomes a node with an edge to each of those, and both were already nodes. Then the graph walks from one to the other. Here a branch of that method maps to W-3, so the two are 2 links apart, and the report says so: “grounded; the graph joins every pair it cites”. The picture on the slide is this step’s walk through the example’s graph, on down through the code, a branch, a break of that branch and the test that caught it.
Run end to end on 30 October, on a copy of the kit's example
Measured A chain with real cites: exit 0, and the chain and its step are in the graph with their cites. The same step citing a requirement that does not exist: exit 1. Citing a method the code does not have: exit 1. Citing nothing: exit 1.
Run on the kit’s example, 30 October
Measured CH-EXAMPLE: 1 step, grounded, and the pair it cites already joined in the graph, 2 links apart. The owner’s one open question, W-4 (what a refused purchase does to the stock), is a blind spot: the chain never looked at it. The whole loop on that copy held.
What it does not check yet
Open Whether a claim is true of the code. A real step, in one of our own runs, said “or” where the code needs both, and passed. A fresh reviewer found it. The fix being built: a step can carry a precise claim in the same form a requirement’s effect uses (for example, which two values a method branches on), and the code check tests it the way it tests an effect.
Open Whether the code a step stands on is tested. Next: a chain is accepted only if no break that got past the tests sits in code one of its steps stands on.
On the slides: slide 11.
Buzzsaw testing: attacking the assumptions
Breaking the code attacks the code. Buzzsaw testing attacks what the requirements assume. Every model takes some things as given:
- Inferred requirements: things a step needs that nobody wrote down. Only the owner can say if they are right.
- Entry conditions: facts that come from outside the model, such as “the string a caller hands in”.
- Declared equivalents: breaks someone said change nothing, with a reason.
For each one, buzzsaw flips the answer (“what if this is not true?”) and walks the graph to find everything that rests on it: the later steps of every use case that relies on it, the requirements those steps carry, their effects, the code, and the tests that would have to change. That set is the assumption’s blast radius. The widest ones go to the owner first, because the most depends on them.
Measured on the public trials
Measured SemVer: 11 assumptions, 3 inferred requirements and 8 entry conditions, and the owner’s widest question reaches a radius of 7. JsonObject: 19, 6 inferred and 13 entry conditions, the widest question a radius of 57. osu!: 15, all entry conditions, with no inferred requirement and so no question to rank.
A lesson from building it
The first version said every assumption reached nearly every step and every test, so it could not rank anything. The summary use case (the epic) repeats requirements the detailed use cases carry, and walking on from the summary reaches everything. Starting the walk only from the most specific step that carries an assumption fixed it.
What it cannot do: it does not know which answer is right, and it measures how much depends on an assumption, not how likely it is to be wrong. Next: the same attack on the values a model takes from outside, by changing each value across its range and running the tests (Open).
Where the model asks who, what, why
Good requirements answer seven questions: who acts, what they want, why, when it starts, where it happens, what must be true before, and what is true after. The model has three levels where these can be asked: a use case, a step inside it, and an extension (what happens when a step cannot complete). Each level has a field for some of the questions and none for others.
Measured on SemVer
Measured Use cases (4): all seven answered in every one. The model’s rules require them.
Measured Steps (15): what and why answered in all 15; who, when, where, before and after are never asked, because a step has no field for them.
Measured Extensions (4): what, why, when and after answered; who, where and before never asked. That is 74 answered and 8 never asked; the 8 are the same on every model.
So the gaps are not in the summaries but one level down, in the steps between them. These counts go into the morph log every pass. Open Whether asking more at the step level makes agents more accurate, or just slower, has not been measured.
Shaping the graph for the GNN
Not run since the network was switched off on 3 November, because a plain rule beat it. This is the record of what the tunes found.
The GNN learns from the graph’s links, and not every kind of link helps it. A tuning pass tries leaving out one kind of link at a time (for example the link from every branch to the one shared “space” node, which ties everything to everything) and measures the GNN again on code it never saw. A change is kept only if it helps on the tuning runs and again on a separate run it was not chosen on, so that a lucky result is not mistaken for a real one. The kept choice is written beside the graph, and the GNN reads it from then on.
The stats decided when to tune: when the GNN’s lead over the simple rankers stopped growing, or every five passes otherwise (how the stats choose).
The first tune, on the graph the network was built on
Measured The full graph set the score to beat, and run-to-run noise alone was about 0.02. Six variants were tried. None beat the full graph by the required 0.02, so nothing was changed. The best, leaving out the use-case structure links, gained a little; leaving out the shared “space” node gained less.
Measured The links the GNN relied on most were “binds requirement” and “is at”, which ties a branch to its method.
The second tune: adding links, three runs each
Measured Adding links the graph lacks did not help beyond the noise: linking the methods of one class, and linking requirements one step carries together, each gained under 0.01. Leaving out the use-case structure links was the only change never worse on any run, and it stayed under the 0.02 bar. Nothing was changed.
What that says: on this graph the GNN already gets what the structure can give it. The next gains are more likely from more graphs to learn from and better descriptions of each node than from reshaping this one.
Part 3 · What was measured
The test subjects
Every number in this manual comes from code you can read, or from the toolkit’s own source. Three subjects were chosen, each for a reason.
- Unreal Engine’s
SemVerparser is the academic one. A published spec, Semantic Versioning 2.0.0, says exactly what the class must do, so every requirement can cite a numbered rule. It has 68 statements and 2 tests. - Unreal Engine’s
JsonObjecthas a contract from three places: its own doc comments, Microsoft’s publishedEqualsandGetHashCodecontract, and RFC 8259. It has 289 statements and 43 tests. - osu!’s scoring rules are the game: a shipped open-source rhythm game (ppy/osu, MIT), held to its own doc comments, with the 151 test cases of three test files. It has 191 statements.
No code was changed and no test or product code was written: only a requirements model for each, and /// CHECKS: comments in a scratch copy of the tests. The numbers, the bugs and how to re-run them are in on code you can read for yourself. The kit’s own example, a store’s wallet, is the one every install runs.
Breaking code one way flatters it
A coverage tool said SemVer was 90% covered by lines run, and 85% by branches. Breaking every decision every way the tool knows made 246 breaks, 236 of them live, and 30 got past every test. 14 of those 30 sit on lines the coverage tool marks as covered.
Nothing about the code changed. Only how hard it was tried: a line that runs is not a line that is checked. Coverage says a test ran the line; only a planted bug on that line says whether any test would notice it being wrong. This is a known result, not only ours: Inozemtseva and Holmes, ICSE 2014, found coverage only low to moderately tied to how many planted bugs a suite catches, once its size is allowed for. The 30, by line, are in the public-code trial.
That is why every pass records how wide the net was beside what got through it, and why a score from a narrow net, or from lines run, should never be trusted on its own.
On the slides: slide 4.
The tool runs on itself
The loop’s own requirements are written as a model too, from the author’s own words with their dates: 8 use cases, 18 authored requirements and 2 inferred ones, and the model passes the 19 rules. A Python adapter breaks the loop’s own decision code the same way the C# adapter breaks a game’s, and the loop’s selftests are the tests.
The model is expected to stay open in places, and to say so: it must report that the installer is the newest piece, that the two targets are not measured, and that a reviewer every pass is still a practice rather than a check.
The first pass on its own code (29 October, 65 minutes)
Measured 1,626 breaks of the tool’s own decision code. 94 were proven to change nothing by comparing compiled bytecode. Of the 1,532 that changed the program, its selftests caught 885 (58%) and 647 got past every test. 174 of 392 statements are held by a test (44%).
So the tool is less tested than SemVer (71% of its statements held, against its own 44%) and than JsonObject (61%), and about as tested as osu!’s scoring rules under three test files (42%). Its tests are planted mistakes that each prove one rule; they were never written to catch every break of every line. That gap is the next work, and these numbers are on code you will hold: the kit ships this source.
Measured The model passes all 19 rules; after that first pass its graph held 2,335 nodes and 6,259 edges; 2 questions stay open for the owner.
Two lessons from getting it to run
The first attempt ran all 1,626 breaks, then crashed while writing the summary, and the hour of runs was lost. The tool now saves every verdict before it builds the summary, and reuses them when the code and tests are unchanged. The second attempt refused to start, correctly, because a tool’s own test was failing while it was being edited. Rule since: never edit a tool while a run is using it.
On the slides: slide 23.
On code you can read for yourself
Three trials on 30 October 2026, on code outside our own project: two parts of Unreal Engine's own C# and the scoring rules of a shipped open-source game. Every number below was measured on these runs, and every defect was reproduced by running it. No test and no product code was written for any target: only the requirements model, and /// CHECKS: comments in a scratch copy of the tests. Findings were proven with throwaway calls that are not part of any result.
What was tested
- Unreal Engine 5.7,
EpicGames.Core/SemVer.cs. Requirements: the public Semantic Versioning 2.0.0 spec, every numbered rule either mapped or excluded with a reason. Tests:EpicGames.Core.Tests, 2 of them on this class. - Unreal Engine 5.7,
EpicGames.Core/JsonObject.cs. Requirements: the class's own doc comments, Microsoft's publishedEqualsandGetHashCodecontract, and RFC 8259. Tests: 43 on this class. 37 C# files under the engine'sSource/Programsname this class. - osu!, a shipped rhythm game (ppy/osu, MIT, commit 845336e),
ScoreProcessor.csandHitResult.cs. Requirements: their own doc comments. Tests: the 151 test cases inosu.Game.Tests/Rulesets/Scoring. The project has many more, which this pass did not run per break: see below.
Unreal's source is available to anyone with an Unreal licence, not to the general public, so this file gives counts, method names and calls we wrote, and no engine code.
Check it yourself: three faults, each against a rule anyone can read
- Unreal's SemVer class (
Engine/Source/Programs/Shared/EpicGames.Core/SemVer.cs):var v = SemVer.Parse("1.0.0-x-y-z.--"); v.CompareTo(v);throwsArgumentOutOfRangeException. The SemVer spec, rule 9, lists that version among its valid examples. Re-run on the installed 5.7 source on 3 November. - Unreal's JsonObject:
x.Equals(x)on{"a":null}throwsNullReferenceException. Microsoft's Object.Equals contract saysx.Equals(x)returns true and that Equals must not throw. Re-run on the installed 5.7 source on 3 November. - osu!'s ScoreProcessor: before anything is judged,
TotalScoreWithoutModsreads 1,000,000 whileTotalScorereads 0. Its own doc comment says it is the points awarded. Re-run on a fresh clone at that commit on 3 November.
The numbers
A break here is a planted bug: one small change to the code on purpose, a mutant in mutation testing, made on its own copy and run against the tests. "Live" leaves out breaks that did not compile and breaks whose compiled code was proven identical to the original. "Caught" means a test failed; "got past" means every test still passed.
SemVer
- Coverage tool, for comparison: 90% of lines, 85% of branches.
- 246 breaks made; 236 live; 206 caught (87.3%); 30 got past every test.
- 14 of those 30 sit on lines the coverage tool marks as covered.
- 48 of 68 statements held by a test; 18 have a break that gets past.
- 7 of the 16 requirements taken from the spec have no test about them, including every rule about which strings are valid. Which test is about what was our reading of each test, written before any break ran.
The 30 planted bugs no test caught, by line
Every one is in Engine/Source/Programs/Shared/EpicGames.Core/SemVer.cs as it ships in Unreal Engine 5.7, and every line number below was matched against that file. Each was one change, on its own copy, and the class's 2 tests (PrecedenceTests and PrereleaseTests, in EpicGames.Core.Tests) still passed. To check one, make that change in your own copy and run those tests: none fails. No engine code is printed here, only where each bug was and what changed.
Parse: line 54: argument 1 of the call toArgumentExceptionreplaced by its default value; line 54: the statement deleted.TryParse: line 68: theifcondition forced to true; line 76:falsechanged totrue.GetHashCode: line 81: argument 2 of the call toAsSpanreplaced by its default value; line 81: argument 1 of the call toGetHashCodereplaced by its default value; line 81: argument 2 of the call toGetHashCodereplaced by its default value; line 81: the constant 0 changed to 1; line 81: the value it returns replaced by its type's default.Equals: line 84: argument 1 of the call toEqualsreplaced by its default value; line 84: the value it returns replaced by its type's default.CompareTo: line 90: argument 1 of the call toComparereplaced by its default value; line 90: argument 2 of the call toComparereplaced by its default value; line 90: the value it returns replaced by its type's default.ToString: line 93: the value it returns replaced by its type's default.Compare: line 106: argument 2 of the call toGetFieldreplaced by its default value; line 107: argument 2 of the call toGetFieldreplaced by its default value; line 142: the second<on the line changed to<=; line 155: the constant 1 changed to 2.CompareFields: line 175: argument 1 of the call toIsNumericreplaced by its default value; line 175: theifcondition forced to true; line 177: argument 1 of the call toIsNumericreplaced by its default value; line 177: the constant 1 changed to 2; line 181: the constant 1 changed to 2.IsNumeric: line 192: the constant 0 changed to 1; line 194: the first<on the line changed to<=; line 194: the first>on the line changed to>=.operator!=: line 210: argument 1 of the call toEqualsreplaced by its default value; line 210: the value it returns replaced by its type's default.operator<=: line 220: the constant 0 changed to 1.
JsonObject
- Coverage tool: 77% of lines, 69% of branches.
- 582 breaks made; 429 live; 299 caught (69.7%); 130 got past every test.
- 67 of those 130 sit on lines the coverage tool marks as covered.
- 176 of 289 statements held by a test; 82 have a break that gets past.
- 6 questions only the class's owner can answer.
osu! scoring
- Coverage tool, from those 151 tests: 72% of lines, 78% of branches.
- 563 breaks made; 536 live; 286 caught (53.4%); 250 got past those 151 tests.
- 145 of those 250 sit on lines the coverage tool marks as covered.
- 81 of 191 statements held by those tests; 110 have a break that gets past.
- Read 53.4% as what those three test files hold, and as low for the game as a whole. We applied 7 of the surviving breaks by hand and ran the rest of
osu.Game.Tests:TestFailScore,TestResetFromReplayFrameandAccuracyRankAndTotalScoreOfLazerScorePreservedcatch some of them. We could not finish that project on this machine: its test host crashed on the clean copy and the broken one, at different points, with and without the visual tests, so what the rest of it would catch is unknown. - This pass ran before property bodies were walked, so 5 statements in property bodies got no break.
Checking the tool's verdicts without the tool
We applied breaks the tool said no test notices, by hand, to clean copies, and ran the tests ourselves.
- SemVer: 6 breaks at once (including
CompareToalways answering "equal" andTryParseno longer testing the match): all 90 tests that run still pass, 1 is skipped. No other test project names the class. - JsonObject: 8 breaks, each alone, against both test projects that name the class. 7 pass everywhere:
RemoveFieldremoves nothing,Equalsanswers true for anything,GetHashCodeis always zero, the file reader stops skipping a byte order mark, the file reader stops skipping comments,Parserefuses trailing commas, a missing double field answers true. - The 8th is the tests' credit. JSON
falsereading astruepasses the class's own project, butUnrealBuildTool.Testscatches it. We had first read that as uncaught, and running the second project corrected us. - 8 of
UnrealBuildTool.Tests' 116 tests fail on the clean copy in our setup, so they could catch nothing here.
What it found
Each of these fails when run, on the code as shipped.
SemVer against the spec
ComparethrowsArgumentOutOfRangeExceptionon valid versions, including the spec's own example compared with itself:v = SemVer.Parse("1.0.0-x-y-z.--"); v.CompareTo(v);. Any pre-release ending in a hyphen does the same.- A hyphen is treated as a separator, but the spec separates identifiers only at dots.
1.0.0-alpha-10sorts above1.0.0-alpha-9(the spec says below).1.0.0-2sorts above1.0.0-1-2(below). 1.0.0-a.band1.0.0-a-bcompare as equal whileEqualssays they differ.- It accepts strings the spec rules out:
"1.0.0\n", and non-ASCII digits such as an Arabic-Indic zero in1٠.0.0. Both come from .NET's meaning of$and\din the spec's regular expression. - Reach: no other C# file under the install's
Source/ProgramsorPluginsnames this class, so nothing in that tree is hurt today.
JsonObject against Microsoft's contract
EqualsthrowsNullReferenceExceptionon any object with a null field, evenx.Equals(x). The contract says it must not throw, and thatx.Equals(x)is true.- Two objects that compare equal have different hash codes, which the contract forbids. A
HashSetof two equal objects holds 2. We found noHashSet<JsonObject>orDictionary<JsonObject, ...>in the engine's C#.
osu! against its doc comments
TotalScoreWithoutModsreads 1,000,000 right afterApplyBeatmap, whileTotalScorereads 0, andPopulateScorebefore the first hit copies that 1,000,000 into the score. The field's doc says "The total number of points awarded for the score without including mod multipliers", andReset's doc says "Resets this ScoreProcessor to a default state", butResetnever clears it. We did not trace what the game does with the value afterwards.- After a second
ApplyBeatmapon one processor,MinimumAccuracyreads 1 before any hit. We did not check whether the game ever applies a beatmap twice.
JsonObject: the owner's questions, not defects we can call
Equalsanswers false for two parses of the same text when it holds an array or an object.- A repeated name keeps the last value at the top level and throws one level down.
{"A":1,"a":2}silently keeps one field, and throws one level down. RFC 8259 only says duplicates are unpredictable between programs. Readaccepts comments andParserefuses them. A root that is not an object throwsInvalidOperationExceptionfromReadandJsonExceptionfromParse.{"a":1e999}parses, thenToJsonStringthrows.{"id":9007199254740993}writes back as...992, which RFC 8259 allows.TryGetIntegerFieldaccepts the string"12".TryGetEnumField<DayOfWeek>accepts"99". On a German machineTryGetDoubleFieldreads the string"1.5"as 15.
What it found nothing on
- SemVer: 13 valid strings taken from the spec, 15 of the 19 invalid strings we tried, the printed precedence chain, and build metadata were all correct. All 21 breaks of
GetFieldand all 6 ofCompareNumericFieldswere caught. - JsonObject:
ToSystemJsonObject(8 of 8 caught),ToJsonNode(10 of 10),Read(5 of 5),ContainsField(3 of 3). A 562 KB file with long escaped strings read correctly, and from that we judged the pooled buffer inReadlatent. That was wrong, as the faults below show: "all breaks ofReadcaught" did not meanReadwas right. - osu!: all 41 breaks of the nine
HitResultquestions the tests ask about (IsHit,IsBonus,AffectsComboand the like) were caught, and so were all 8 ofApplyBeatmap, all 11 ofupdateScoreMultiplierand all 5 ofGetComboScoreChange. By hand, a reverted hit put the score, combo and accuracy back, and each rank's cutoff gave that rank.
Who found what
Be accurate about this. The defects above were found by an agent reading the code against each rule while writing the model: the requirements step. The breaks did not find them. The breaks measured how much the tests hold, and showed where no test looks: 9 breaks of JsonObject.Equals got past every test, and that is where the crash was. No graph neural network (GNN) played a part in any finding here.
More faults, found by reading the code carefully, each proven by the harness on the code as shipped:
ReadandTryReadparse a file over 128 KiB from a pooled buffer they have already returned. A valid 173 KB file with a very long name came back with other fields silently replaced, whileParseof the same text read it correctly, and a second file madeReadthrowFormatException.- Name matching uses a culture comparer, so names that are different strings can count as one field.
RemoveFieldandAddOrSetFieldValueignore the empty name, which RFC 8259 allows.NaNand infinity can be set, and thenToJsonStringcannot write them. An object parsed 64 levels deep cannot be written either: it reports a cycle that is not there.TryGetEnumFieldthrows on a struct that is not an enum.CreateDeepCopyshares reference values it does not know how to copy, and throws on keys that differ by case and on a multi-dimensional array.
What went badly
The trial's biggest finding was in our own tool. Real code broke it five ways on the first attempts, and a sixth way on osu!, none of which our own examples had shown:
- It silently skipped code. Any method with overloads, any operator, any generic method with a
whereclause and any property body was left out, and the report still read as if the whole file was held. On SemVer it saw 35 of 68 statements, missing the privateComparethat does all the comparing. On JsonObject it saw 213 of 288. - It crashed on read-only files, which is how a Perforce workspace keeps them, on the first run.
- It mistook a nested type's constructor for a method, and stopped.
- One mutant with broken braces stopped the whole run.
- An endless loop in a mutant tested alone hung the run: 11 minutes on one test host before we killed it.
It also could not read MSTest or xUnit tests' /// CHECKS: lines, only NUnit's, and it took a parameterized test's arguments for part of its name, so it reported named tests as never run. Because that fix came after osu!'s pass, its per-requirement check results are not quoted here. All of these are fixed, each with a planted test. A field's value and an indexer are still not walked, and the report now names them.
Our earlier figures on a game feature of ours came through nearly unchanged, because that code has no overloads: no existing verdict changed, and its 7 new breaks found one real gap, which a test now closes.
The budget did not hold. We set 45 minutes and 150K tokens per trial and spent about 100 minutes and over 260K output tokens, most of it on the tool's own bugs. A full pass took 13 minutes on SemVer, 9 on JsonObject and 74 on osu! (each constructor break rebuilds the game), at low priority on a busy machine. The earlier-budgeted 60 minutes for osu! did not hold either.
Re-running it
The models are in t7/ beside this file. For the Unreal trials, copy EpicGames.Core, EpicGames.Core.Tests and their props files from your own engine to a scratch folder and run closed_loop.py on the model; numbers may differ on another engine version, and these are from 5.7. For osu!, clone ppy/osu at commit 845336e, keep the three test files under osu.Game.Tests/Rulesets/Scoring, and run it the same way.
A run on public code: the asteroids mining loop
Numbers from a private repository are hard to trust from outside. So the loop also ran on a game whose code anyone can read: the asteroids mining loop of starfighter, a browser game in JavaScript. Its repository already holds a use-case document for the mining loop (ASTEROID_USE_CASES.md) and the list of the 61 constants that govern rocks, gems, the belt and the mining drones.
The kit cannot break JavaScript yet (it has adapters for C# and Python), so this run uses only the parts that do not need the code broken: the model’s 19 rules, the graph, the checks that read only the graph, buzzsaw and the question grid. The model says so itself, and the other checks report “not held” rather than passing.
Measured
Measured The document became 7 use cases and 48 requirements: 44 traced to the code, 4 open questions for the owner. All 61 constants are claimed by some step.
Measured 10 requirements rest only on what a function does (for example “a destroyed rock spills a gem”), with no constant behind them. The constant list cannot see them; only breaking the code could hold them. That is exactly where a JavaScript adapter would pay.
Measured Of the checks that need no code: 10 requirements with no source standing behind them, 34 resting on a single source, 4 open questions, and none carried by nothing or ending nowhere. Buzzsaw ranks the owner’s first question as “which collision applies ramming damage?”: five later steps rest on it.
Measured The tool’s own cross-check found a row in the document with a missing column before anything else ran.
A thin graph, on purpose
The graph holds 186 nodes and 396 edges, with no code and no tests in it. That is the thin case this manual warns about (why): the checks that ran are real, but a clean result here would mean little, and a learned part would have nothing to learn from.
Self-improvement, on by default
The tool tests itself, and that is on unless you turn it off. It happens at two sizes.
Every pass. A pass writes one row of stats about itself to the morph log, checks those stats against the tool's own requirements for improving, and prints what to improve next. Nothing about your code changes.
The standing tick. A fresh agent with no memory takes one item, measures the same stat before and after the change, and records whether it was worth it. Both ship in this kit, with the self-improving qualities we built for our ARC3 harness. Nothing starts a tick for you: TICK.md is its runbook, and you schedule it.
Turning it off
Run with --no-self-improve, or put self_improve: false in the model's loop: block. The pass then records no row and prints no picks. There may be some token saving, but on a pass it is small; the larger spend is the tick, which only runs if you schedule it, and stops when you stop scheduling it or create .loop_tick.STOP beside the tools.
Where these numbers come from
Measured A pass: 44 stats in a row, about a third of a second to collect them, three printed lines for the agent to read. On the kit's Python example, a clean install, with it off a pass wrote no row and printed 295 fewer characters.
Measured A tick: the app counted 197,134 tokens for the first tick and 291,042 for the second, over 9 and 22 minutes.
One shared graph, not networked in this version
Ideally everyone working on a project shares one graph, because it also works as a retrieval store (retrieval-augmented generation, RAG): an agent looks up what is already proven instead of re-reading the files to work it out again. We share ours across the team that way, in the belief that it saves tokens. This version is not networked, and each checkout keeps its own graph; you could add sharing in whatever way suits your network.
Where this is not measured
Open How far a shared graph cuts what an agent reads has not been measured. To learn who owns a method, what it must do and which tests catch it, the graph holds rows for that method and the files hold the model, the source and the tests; no agent has been timed using either, so a saving is a belief and not a measurement.
What it has found so far
Where these numbers come from
Measured 19 ways a check could pass for the wrong reason, found by fresh reviewers; 17 fixed, each with a planted mistake a selftest must catch (see reviewers).
Measured 11 stats were recorded with nothing reading them. The audit first said 6, and that was itself a bug: it counted test fixtures and the recorder as readers. Now 0: each stat triggers something, or has a reader with a date.
Measured 33 requirements on its own stats, 2 records banked for a learner that does not exist yet, and 5 triggers firing now. Each trigger that fires is a work item. It was 12 until a tick found that 8 of them fired on every pass whatever happened.
Measured 9 faults its first measuring round found in itself: 5 in the harness, 4 in the tool, all fixed.
Measured Its optional parts: of the run plan's 7 lines, 3 reach what a pass runs, and on today's evidence all 3 are off on both projects, because a plain rule wins by more than the 0.02 a learned part would need. The other 4 are marked measured only: they record a verdict and change nothing, and three of the verdicts behind the plan flip with the seed alone. Its own tick found that none of the 7 reached a pass by code before it wired these.
Measured 58% of the breaks of its own code were caught, 885 of 1,532, on its first self pass; a second pass on 30 October caught 989 of 1,637 (60%). The tool is less tested than the feature it tests.
Open 0 of the 20 measured outcomes its improvement learner needs, so rules choose the work for now, not a model. These numbers show what it has found, not yet how fast it gets better.
The loop, in pseudo C++
The toolkit is written in Python. This is the same logic for a C++ reader; the lower-case names are the real ones in the code.
// One pass. Self-improvement is on by default.
Verdict run(Model& m, bool selfImprove = true) {
Breaks b = Mutate(m.code); // one compile
Results r = RunEachAlone(b, m.tests);
Graph g = KnowledgeGraph(m, r);
Holes h = Criteria(g); // 18 checks
if (self_improve_on(m, selfImprove)) {
Row row = PassStats(m).collect(); // 44 stats
MorphLog(m).add(row);
auto fired = improvement::fires(rules, log);
Print(NextImprovement(log, fired).picks());
}
return h.ours.empty() ? HOLDS : OPEN;
}
// One tick. A fresh agent, one item, no memory.
void Tick() {
Start s = loop_tick::start(); // the lock
if (s.stopped || s.busy || s.idle) return;
if (s.epochStarts) Brainstorm(); // every 12
RunSelftests(); // red = item
telemetry_audit(); // unread = item
Item i = Choose(improvement::check(), board);
Stat before = i.stat.Measure();
i.Do(); // one change
Stat after = i.stat.Measure();
if (s.epochEnds) {
run_plan::decide(); // parts on or off
improvement::graph(); // did it work?
}
loop_tick::finish(i, before, after, worth);
}
Three ticks in a row worth "no" and the tick sleeps until something it watches changes, so an idle loop costs almost nothing. The improvement graph is the same shape one level up: stats are its code, triggers its requirements, ticks its tests.
Telemetry: measuring growth both ways
A loop can look better by trying less, and look worse by trying more. So every number the loop records is one of five kinds, and they are read together:
- Net
- How hard the pass tried: how many breaks, how many kinds, how much code. Should grow, then hold.
- Closure
- How much of what it tried was caught, claimed or answered. Should grow towards the net.
- Open
- What is still open, and whose it is. Should shrink; the owner’s part only shrinks when the owner answers.
- Cost
- What a pass costs in time. Should not grow faster than the net.
- Tool
- The tool itself: its size, and how much of it its own planted mistakes prove.
A pass where the net grew and the open count grew too is progress, not a step back. The list of stats is revisited every pass, and a new one is added whenever a pass shows growth the existing ones miss.
On the slides: slide 19.
How the stats choose what to improve next
Every pass, each stat in the morph log is read as a trend over the passes and given one word: improving, worsening, stalled (the last three values barely moved), done (an open count stuck at zero), or too few (fewer than three passes have it). The word compares the stat to the way its kind should move: the net and what was caught should rise, what is open should fall.
The next improvement is then chosen by the stats, not by a person’s feeling. Each area of the tool (tests, the GNN, the kinds of break, speed, the owner’s questions, assumptions, coverage) names the stats that speak for it. A worsening stat scores 3, a stalled one 2, and an area scores its worst stat. The GNN tune was also due every five passes, until the network was switched off on 3 November, because a plain rule beat it. The pick, and the trend lines that chose it, are printed after every pass.
What the first picks were
Measured Over its first recorded passes the graph grew, and the stats’ first pick was the GNN tune, because it had never been run (the tune).
Measured A bug the selftest caught while building this: adding up an area’s stats ranked two related GNN stats, both stalled, above a survivor count that was getting worse. An area now scores its single worst stat.
Stats that might predict others
The loop also looks for stats that move before others: does a change in one stat this pass go with a change in another the next pass? Each pair says how many passes it rests on. With fewer than eight it is labelled an anecdote, and pairs where one stat is simply made from the other are labelled mechanical. Every pair so far rests on a handful of passes, so they are anecdotes to watch, not findings.
The report every 20 passes
Every 20th pass writes a report: each stat’s trend, how much the graph grew (in total and by kind of node) for the feature’s model and for the loop’s own requirements model, the predictive candidates, the GNN tunes so far, and what the stats pick next.
Part 4 · Limits and targets
What it can learn from its own stats
Every pass leaves a verdict for every break: which kind of break, on which statement, in which method, and whether a test caught it. That is training data. A small model trained on it ranked the breaks most likely to get past the tests first (retired on 3 November: the break-order head now does that job), so a pass can show its holes early, and an agent writing new code can be told where its tests are likely to be weak before anything runs.
The test that lets it improve
It is only scored on code it never saw: whole methods are held out, and the score is how many of the real survivors it finds in the first fifth of its ranking. Random order finds a fifth; the plain rule to beat ranks each kind of break by how often that kind survived before.
Where these numbers come from
Measured Within one game feature of ours, the learner found more of the survivors in the first fifth than the plain rule did, at every one of ten seeds; no count is given because the feature is not public.
Measured Within the tool's own code: 30% over ten seeds, against the plain rule's 25% (random 20%). It won at nine of the ten, and the first seed happened to be the one it lost.
Measured Across projects it is worse than random: the words it learns belong to one project. Only the plain rule carries over. So, again: seed first, then it flies.
The real test of learning is over time: trained on the earlier passes, scored on the next pass's new code. That needs every pass's verdicts kept; until 29 October each pass overwrote the last, so the loop now archives them.
Planned rounds of improvement
The standing loop works in epochs of twelve ticks. The first tick of an epoch is a brainstorm: a fresh agent reads the stats, the tick log and the results, and ranks ideas by the one stat each should move. The last tick is a benchmark: the same fixed measures every time, on inputs no tick may change. An epoch counts as an improvement only if a number moved the right way and none moved the wrong way.
The code-writing side
Planned Connect what it learns to the agent that writes the code: before it writes tests, hand it the statements of its change most likely to get past them, and the requirement the owner head thinks its new method belongs to.
The loop on its own improvement
The loop records a great deal about itself: every pass records 44 stats about itself, and the learned parts leave evidence of when they help. Recording a number is only worth it if something reads it and acts. On 29 October an audit found numbers that nothing read, and many more that were only printed.
So the improvement of the loop now has requirements of its own, written the way the product's are. Each number has a requirement saying what pattern in it should trigger something, what that is, and why the number is kept: a test that catches nothing for two passes is a speed item; an excuse for a harmless break that lapses is a hole; the graph not growing for three passes means the model is not being seeded. A number kept for a learner that does not exist yet must name that learner and a date. Past the date, unread, it fails.
Three levels, one graph
- Level 1, the product. The loop holds the code to the product's requirements.
- Level 2, the improvement. The loop holds itself to its improvement requirements, with the same kind of closure rules: every number covered, every trigger that fires acted on, every action naming the number it meant to move and its result measured, and an action that never moves its number retired.
- Level 3 is level 2 again. How well the improvement graph is doing (numbers covered, triggers acted on, actions that worked) is itself written as numbers with requirements, so they sit in the same graph. The tower closes on itself.
The same learning applies one level up: once enough actions have measured results, a model can learn which kind of action moves which number, and it is switched on only where it beats the plain rule, like everything else.
Where these numbers come from
Measured After the audit of 29 October gave each number a reader or a trigger, none is read by nothing. 14 now drive a decision in code and 28 are covered by a trigger.
Measured The first run of the triggers fired 12: among them, 5 tests in the tool's own suite that catch no break at all, 8 of the who/what/why questions never asked on each model, and one requirement whose breaks were all caught last pass and are not now.
Open No action has a measured result yet; the improvement graph becomes learnable at 20.
How new is it?
An independent agent with web search compared the loop with published work on 29 October. Its answer: about eight of the ten parts are known. Requirements kept as data with gaps turned into questions are close to GitHub's Spec Kit and to Kiro's requirements turned into tests. Compiling all breaks at once and checking only what changed is what Stryker.NET already does. Breaks as feedback to an AI agent is Meta's mutation-guided test generation. Suggesting which requirement owns code is standard trace-link recovery.
Medium confidence What looks new is the whole used as an agent's "done" gate: every break in requirement-owned code must be caught by that requirement's own test, proven to change nothing by compiled code, explained with a reason that lapses when its code changes, or settled by the owner. Pieces of that exist (TestSuiteBench ties each break to one requirement and one test); the gate does not seem to.
It also said two things are aimed slightly off, and each could change the outcome more than polish: the breaks are syntactic while agents' mistakes are about intent; and the GNN ignores its strongest signal, which requirement's tests catch the new code's breaks. Both are on the work board.
Where it stands among the other tools
We set out to find every agent requirements tool we could, rank them, rank this kit beside them by the same rubric, and take the abilities they have that it lacks. The danger in a comparison like this is a rubric built from what one's own tool happens to do, which proves nothing. So the rubric was committed first, before any tool was scored, with its weights, its predictions and the checks on the order.
It has 17 criteria in two groups. Ten are what other tools are good at and this kit may not be: asking a person good questions, gathering from conversation or documents, ease of first use, install friction, editor and agent integration, working on an existing codebase, a team's review workflow, adoption, price and licence. Seven are what the loop was built for: finding holes in the requirements, tying them to code, tying them to tests, proving a test catches a break, checking the agent's argument for a change, keeping requirements true over time, and measured evidence that it works. Each criterion says how a score from 0 to 3 is earned and what evidence counts. Two weightings were fixed before scoring: every criterion once, and the nine a typical team names first (a judgement, not a survey).
Every score carries one kind of evidence. READ is the tool's own page, with its link. TRIED is something we ran, which only our own kit can have, since nothing of anyone else's was installed. ABSENT is a search of named pages that found nothing: a search result and not a proof, worth 0. SECOND_HAND is a third party, or a search summary of a page we did not fetch, and is capped at 2. CLAIM is a tool's own marketing with no measurement behind it, capped at 1, because a claim is not a measurement. A check, landscape.py --check, refuses a score with no page behind it, a tool with no link and a criterion nobody scored, and its selftest plants each refusal and sees it fire.
Measured Read on 2026-10-30: 31 tools, 28 of them ranked beside the kit, all 31 read at their own pages and 15 with released code. 493 scores: 258 read at the tool's own page, 213 not found on the pages read, 13 second-hand, 7 marketing claims, 2 run by us.
Measured With all 17 criteria counting once the kit scores 70.6 of a possible 100 and is 1st; next is Rebuild Dossier at 60.8. On the 9 criteria a team names first it scores 48.1 and is 10th, tied with Devin; OpenSpec leads at 70.4. On the deep criteria alone it scores 95.2 and is 1st; Rebuild Dossier is next at 61.9.
Open Our place depends on the weights: yes. Over 10,000 random weightings the kit is in the top third in 99.9% and first in 90.0%, median place 1, range 1 to 13. Leaving out any one criterion leaves it 1st. Granting every absence on a deep criterion one point, or counting no marketing claim, leaves its place unchanged. Scoring ourselves one point lower on every deep criterion gives 2nd at equal weights; one point lower on every score gives 8th at equal weights and 22nd on the team list.
What other tools do better, by what taking it would move:
- Adoption: we score 0 of 3, the best is 3 (GitHub Spec Kit, BMAD-METHOD, OpenSpec, Task Master and 1 more). Closing it would take the team-list place to 7th.
- Editor and agent integration: we score 1 of 3, the best is 3 (GitHub Spec Kit, OpenSpec, Traycer, Agent OS and 2 more). Closing it would take the team-list place to 8th.
- Ease of first use: we score 1 of 3, the best is 3 (Superpowers, Claude Code plan mode, Cursor plan mode). Closing it would take the team-list place to 8th.
- Asks the human good questions: we score 1 of 3, the best is 2 (GitHub Spec Kit, BMAD-METHOD, OpenSpec, Tessl and 7 more). Closing it would take the team-list place to 10th.
- Gathers from conversation or documents: we score 1 of 3, the best is 2 (Task Master, ArDoCo, SpecTrace (spec-trace), Elicitron). Closing it would take the team-list place to 10th.
- Install friction: we score 2 of 3, the best is 3 (Claude Code plan mode, Cursor plan mode, Devin, ChatPRD). Closing it would take the team-list place to 10th.
- Team and review workflow: we score 1 of 3, the best is 2 (AWS Kiro, OpenSpec, Tessl, Traycer and 7 more). Closing it would take the team-list place to 10th.
- Measured evidence that it works: we score 2 of 3, the best is 3 (TDAD (Test-Driven Agentic Development)). Closing it would take the team-list place to 10th.
A system of record for requirements, with AI features added (a person works in it) (4)
- Copilot4DevOps (Modern Requirements): not stated on the pages read (a hosted add-on), no code released, rank 19 on equal weights.
- ChatPRD: not stated on the pages read (a hosted product), no code released, rank 22 on equal weights.
- Jama Connect Advisor: not stated on the pages read (a hosted product), no code released, rank 23 on equal weights.
- IBM Requirements Quality Assistant with DOORS Next: not stated on the pages read (a licensed product), no code released, rank 25 on equal weights.
A workflow that turns an idea into requirements, a design and tasks for a coding agent (8)
- GitHub Spec Kit: MIT, code released, rank 3 on equal weights.
- OpenSpec: MIT, code released, rank 4 on equal weights.
- BMAD-METHOD: MIT, code released, rank 5 on equal weights.
- Superpowers: MIT, code released, rank 5 on equal weights.
- Agent OS: MIT, code released, rank 9 on equal weights.
- Task Master: MIT with Commons Clause, code released, rank 10 on equal weights.
- Tessl: not stated on the pages read, no code released, rank 13 on equal weights.
- AWS Kiro: not stated on the pages read (a product with paid plans), no code released, rank 17 on equal weights.
A planning mode inside a coding agent or editor (3)
- Claude Code plan mode: no open-source licence stated on the pages read, no code released, rank 8 on equal weights.
- Cursor plan mode: not stated on the pages read (a product with paid plans), no code released, rank 13 on equal weights.
- GitHub Copilot Workspace: not stated on the page read, no code released, not ranked: Retired. Its user-manual repository was archived and made read-only on 2025-09-02 (page fetched); search summaries give a sunset of May 2025 and name the Copilot coding agent and VS Code agent modes as the replacement. Kept on the list so the gap in it is on the record, and not ranked because nobody can use it..
An agent that plans and writes the code itself, requirements being a step inside (2)
- Devin: not stated on the pages read (a product with paid plans), no code released, rank 19 on equal weights.
- TraceDev: no software licence; the paper is under arXiv's non-exclusive licence, no code released, rank 24 on equal weights.
Recovers links between documents, requirements, code and tests (3)
- LiSSA: MIT, code released, rank 17 on equal weights.
- ArDoCo: MIT, code released, rank 19 on equal weights.
- Trust-Aware Multi-Agent Traceability: no software licence; the paper is under arXiv's non-exclusive licence, no code released, rank 25 on equal weights.
Research prototype that has agents gather requirements, from simulated people or from documents (3)
- iReDev: MIT, code released, rank 16 on equal weights.
- Elicitron: no software licence; the paper is CC BY-NC-ND 4.0, no code released, rank 28 on equal weights.
- MARE (multi-agent requirements engineering): no software licence; the paper is under arXiv's non-exclusive licence, no code released, rank 29 on equal weights.
Makes tests the carrier of the requirement (6)
- Rebuild Dossier: MIT, code released, rank 2 on equal weights.
- SpecTrace (spec-trace): MIT, code released, rank 5 on equal weights.
- spec-kit-trace: MIT, code released, rank 10 on equal weights.
- TDAD (Test-Driven Agentic Development): MIT, code released, rank 12 on equal weights.
- Mutahunter: AGPL-3.0, code released, not ranked: A mutation tester, not a requirements tool. Listed because it is the nearest released tool to one criterion, proving a test catches a break, and the claim of difference there should be read against it..
- TestSuiteBench: no software licence on the page; the paper is CC BY 4.0, no code released, not ranked: A benchmark of 165 applications, not a tool to run on your own code. Listed because it ties each mutant to one specification requirement and to the test that kills it, which is the nearest published neighbour to what the kit's breaks do..
A workspace that runs several coding agents side by side, with a planning skill among others (1)
- Traycer: MIT, code released, rank 13 on equal weights.
An add-in that scores the wording and consistency of requirements held elsewhere (1)
- QVscribe: not stated on the pages read (a licensed add-in), no code released, rank 25 on equal weights.
Open The scorer is not independent for one row. The kit was scored by the lane that built it, by the same anchors, which is why the order is also shown with a point taken off each of our scores. Seven of the seventeen criteria are the ones this kit was designed to do well on, so equal weights favour it; the team list and the list of what it lacks are the corrective.
Open What is not verified. Documentation only: nothing was installed or run, so a feature a page does not mention scores as not found, and a tool may do more than its pages say. Star counts are as the pages showed them on the day. Where a first-party page could not be fetched, a search summary or a third-party page was used, and it is marked and capped. Tools change, so each score is a reading of a page on one day.
The abilities in the list above are on the work plan, each with a done-when. Where an idea comes from an MIT-licensed tool it is credited and no code is copied; an idea from a proprietary tool is an idea only, and nothing proprietary is used.
What it is not, yet
The first trial on code anyone can read
Measured The C# adapter was skipping code without saying so. On Unreal's own SemVer.cs it saw 35 of 68 statements, including none of the private method that does all the comparing, and on JsonObject.cs 213 of 288. It dropped overloaded methods, operators and generic methods with a where clause, and does not walk property bodies. The report still read as if the whole file was held. An earlier target of ours has no overloads, which is why its numbers never showed it.
Open All fixed, each with a planted case: naming every overload and operator, a report that names what it did not walk, a read-only tree such as a Perforce workspace, a constructor of a nested type, one bad break stopping a whole run, and a break that hangs a test forever. A field's value and an indexer are still not walked, and the report says so. Re-running that target changed no verdict.
- A prototype. Version 0.1. Its own model still names holes.
- A thin model misleads. A green report over a small model reads as proof and is not (why).
- Setup comes first. A problem story, requirements, use cases and tests a machine can run, before it helps at all.
- Not a chatbot. It is a harness for agents that build systems. It does not answer questions.
- It refutes, it never proves. “Closed” means closed against the breaks it tried (why).
- Feel is not checked. Visuals, timing and feel still need a person to look.
Used where it does not fit, it can be worse than not using it at all.
On the slides: slide 15.
The two targets
These are goals for the loop at its full potential, used where it fits. Neither has been measured yet.
95% accuracy or better
Target Agents using the loop on work it fits should get the task right at least 95 times in 100. The remaining 5% would be bugs in our own code.
How it will be measured: answers graded blind, counted per failure pile, with a hidden acceptance test deciding whether the task was done.
Up to 90% fewer tokens
Target On average, up to 90% fewer tokens per closed requirement, because a later step looks up what the graph already holds instead of re-reading and re-deriving it. This number is always being re-evaluated.
How it will be measured: from each run’s own usage records (the tokens the model read and wrote), per closed requirement.
The target is about larger systems held across many changes, where re-reading and re-deriving is what costs; that has not been measured yet.
What is next: from a rough epic to code that is right
The tool’s main intent is to take the requirements for a game from a rough epic use case and grow them into a larger set, so that the code written from them is as accurate as it can be. The question behind it is a question and not a result: if the requirements for something very difficult are complete and checked, how far does that move what an AI can build? The direction is better tools for making video games.
Measured How much of existing code its tests hold, on three public targets (on code you can read for yourself).
Open The epic-to-code test has run, and its results are held for a paper and are not on these pages, so nothing here rests on them. The test starts from one rough epic for a small game whose right answers come from an independent published source. Agents write the code from the grown requirements, and hidden tests written from that source, never from our model, grade it blind. Tokens are recorded, with how many requirements the epic grew into and how many the hidden tests needed that it missed.
Part 5 · Using it
Install
Get it from github.com/rj45Thompson/closed-loop-kit, under the MIT licence.
python install.py # install for this user
python install.py --check # check only, copy nothing
python install.py --uninstall # remove what it put down, nothing else
It puts the skill where Claude Code finds it and the tools beside it. It needs Python 3.10 or newer and PyYAML. PyTorch (for the graph network, which is switched off), the .NET SDK (for the C# adapter and the example) and PowerShell 7 (for the IL comparison) are optional; if one is missing it says what that piece would have done, and never pretends it ran.
Then it runs every selftest in the installed copy, and the whole loop on the example: a small wallet and store in C#, with NUnit tests that each name the requirement they check. The example must hold: everything closed except one question that only its owner can answer.
What a clean install showed
Measured Installing the built kit into an empty folder on 29 October: every selftest passes with PyTorch, scikit-learn and the .NET SDK (14 of 20 on Python and PyYAML alone, the rest named as skipped), and the loop over the example’s 58 breaks reported HOLDS.
On the slides: slide 16.
Other languages
The model, the graph, the checks and the telemetry do not care what language the code is in. Only the part that breaks code does:
- C#: any .NET project, through
dotnet test. - Python: breaks code through its syntax tree; the tests are commands.
- Anything else (C++, Blueprints): write an adapter that produces one result file in the same format the C# adapter writes: for each requirement, whether its code has the right shape, and for each break, whether a test caught it and which.
On the slides: slide 16.
The harness matters more than the prompt
Most of the gain is in how the agents are set up, not in what they are asked. The loop is a harness: a model to act on, checks that can fail, a record that outlives the context window, fresh reviewers, and measurements of its own progress. Keep improving your own: the agents’ instructions, their tools, and what they must prove before they say “done”. Measure each change you make rather than going by feel.
What would help most in return:
- Measurements: what it caught and what it missed, on your own large systems.
- Failure piles: the kinds of failure you see, and how often.
- A wrong pass: any check you can make pass for the wrong reason.
On the slides: slide 16.
Words used here
- Break, planted bug (mutant)
- A copy of the code changed on purpose so it is wrong: one small change, run against the tests.
- Caught (killed)
- A break that made at least one test fail. Good.
- Got past (survived)
- A break every test still passed. A missing test, unless it is equivalent.
- Equivalent
- A break that does not change what the program does.
- Requirement
- One rule the system must follow, with an id, a source and the code it points at.
- Authored / inferred
- Written by a person / found missing by the loop. Inferred ones are questions.
- Use case
- Who wants what, as numbered steps.
- Claim
- “This code belongs to that requirement.”
- Lock
- The recorded, verified state. A later change makes what it touches stale.
- Node, edge
- A thing in the graph, and a link between two things.
- Check (criterion)
- One of the 18 rules the graph is held to, each named after the mistake it stops.
- Planted mistake
- A deliberate error the selftest makes to prove a check can fail.
- Must / may
- Decided / not yet decided by the owner.
- MRR
- A score for a ranking: 1 for right-first, ½ for second, ⅓ for third, averaged.
- Acceptance test
- A test written before the work starts and hidden from the agent.
- Morph log
- The file that gets one line of numbers per pass.