How to use AI for games

← All projectsThe fly-throughThe manual
The real graph of osu!’s scoring rules: 1,062 nodes, drawn from the store the loop wrote.

TNT Tools · internal training manual

How to use AI for games

Adapted for Heavy Iron. Requirements first, tested by AI, so nothing is a guess.

BUG REPORT · FOUND BY AI · CHECK IT YOURSELF

UNREAL ENGINE 5.7

Unreal’s own SemVer class, in its C# tooling, throws on the SemVer spec’s own example.

SPEC

semver.org, rule 9, lists 1.0.0-x-y-z.-- as a valid version

CODE

Engine/Source/Programs/Shared/EpicGames.Core/SemVer.cs, in the 5.7 install

TRY

var v = SemVer.Parse("1.0.0-x-y-z.--"); v.CompareTo(v); // ArgumentOutOfRangeException

TESTS

The class has 2 tests, and both pass.

Three faults like this one, in code you can read, and how to check each

Slide 1: what the speaker says

This is how we use AI at TNT Tools: our internal training manual, adapted for you. It opens on one fault you can check in a couple of minutes with the engine you already have. The SemVer spec, rule 9, lists 1.0.0-x-y-z.-- among its valid versions. Unreal Engine 5.7’s own SemVer class, in EpicGames.Core, parses it, and then throws ArgumentOutOfRangeException when it compares that version with itself. Any pre-release ending in a hyphen does the same. The class has 2 tests, and both pass. An AI agent found it by reading the class against the spec, rule by rule, while writing down what the class must do; re-run on the installed 5.7 source on 3 November. Slide 3 has three faults like it, each with the public page that shows it. Behind this slide: The real graph of osu!’s scoring rules: 1,062 nodes, drawn from the store the loop wrote.

The new tool

A skill that installs itself, and puts requirements first on every build

You ask Claude to build something. Before any code, it writes down what the thing must do, and you sign that off.

1

REQUIREMENTS

Write it down

What it must do, each line saying where it came from. Anything missing becomes a question for you.

2

USE CASES

Plan the play

Who does what, step by step, and what should happen at each step. The main path at least.

GATE

REVIEW 1

I reviewed it

You read one page and press one button. No code is written before that, for that exact version.

3

CODE

Build, step by step

Each change says which step it serves, and why.

GATE

REVIEW 2

Read the intent, not the diff

What it meant to do at each step, and the problem each change solves. A change that serves no step is flagged.

4

EVERY RUN

Fly through it

A 3D map of what was built and why: requirements, steps and code, to fly through at the end.

TNT Tools · closed loop prototype 0.1

Install it, in plain words

2

Slide 2: what the speaker says

This is the tool. Claude installs it as a skill, and from then on every time you ask it to build something it starts with the requirements, not the code. First it writes down what the thing must do, each line citing where it came from, and turns every gap into a question for you. Then it writes the use cases: who does what, step by step, and what should happen, the main path at least. You read that one page and press I reviewed it; no code is written for any other version. While it codes, every change names the use case step it serves and says why. The second review is of those intentions, not the diff: a step with no intent, a change with no step and a change that solves no problem are each flagged. At the end of every run there is a fly-through, a 3D map of the requirements, the steps and the code.

The real graph of a shipped open-source game's scoring rules

Check each one yourself

Three faults an AI found by reading code against its own spec

UNREAL ENGINE 5.7 · SemVer.cs

Comparing the spec’s own example version with itself throws.

1.0.0-alpha-10 sorts above 1.0.0-alpha-9. The spec says below.

The rule: semver.org, rules 9 and 11

UNREAL ENGINE 5.7 · JsonObject.cs

Equals throws NullReferenceException on any object with a null field, even x.Equals(x).

Try {"a":null}.

The rule: Microsoft’s Object.Equals: x.Equals(x) is true, and Equals must not throw

OSU!, OPEN SOURCE · ScoreProcessor.cs

TotalScoreWithoutMods reads 1,000,000 before the first hit, while TotalScore reads 0.

Its doc says it is the points awarded.

The rule: its own doc comment, on GitHub

How: an AI agent read each class against its spec or its docs, rule by rule, while writing down what the class must do. Each fault was then run on the code as shipped. Nothing in the targets was changed or added.

TNT Tools · closed loop prototype 0.1

Each fault, and how to re-run it

3

Slide 3: what the speaker says

Three faults on code outside our own project, each against a rule anyone can read. Unreal Engine 5.7’s SemVer class, in EpicGames.Core under Engine/Source/Programs, throws ArgumentOutOfRangeException when it compares 1.0.0-x-y-z.--, a version the spec lists as valid in rule 9, with itself; and it treats a hyphen as a separator, so 1.0.0-alpha-10 sorts above 1.0.0-alpha-9 where rule 11 puts it below. Its JsonObject’s Equals throws NullReferenceException on any object holding a null field, even when compared with itself, where Microsoft’s published contract says x.Equals(x) is true and Equals must not throw. Both were re-run on the installed 5.7 source on 3 November. osu! is a shipped open-source rhythm game: at commit 845336e, TotalScoreWithoutMods reads 1,000,000 right after ApplyBeatmap while TotalScore reads 0, and the field’s doc comment says it is the points awarded without mod multipliers; that one was re-run on a fresh clone of that commit on 3 November. An agent found all three by reading each class against its rules while writing the requirements; the planted bugs on the next slide did not find them. Behind this slide: osu!’s real graph.

Every break of osu!’s scoring rules and the test that caught it. Red: no test caught it, and each one has to be explained.

Why the tests did not catch it

A line that runs is not a line that is checked

COVERAGE

90%

of SemVer.cs’s lines run when its 2 tests run. Coverage counts lines run. It cannot tell whether a test would notice that a line is wrong.

PLANT A BUG, RUN THE TESTS

30

of the 236 planted bugs that compiled and changed the program left every test green, each tried alone. Our count, by mutation testing: change one thing, like a < into <=, and see if any test fails.

Try one yourself: make JsonObject.Equals answer true for anything. No test that passed before fails, in either test project that uses the class.

Published, not ours: coverage is only weakly tied to how many planted bugs tests catch, once suite size is allowed for. Inozemtseva and Holmes, ICSE 2014.

TNT Tools · closed loop prototype 0.1

What was tested, and how

4

Slide 4: what the speaker says

The faults were found by reading. This slide is about why the tests let them through. On Unreal Engine 5.7’s SemVer class, which follows the public Semantic Versioning 2.0.0 spec and has 2 tests, a coverage tool reported 90% of lines and 85% of branches run. Coverage only says a line ran. So we planted bugs, the standard way called mutation testing: change one thing in the code, such as a less-than into a less-than-or-equal, a check removed, or a method made to answer a constant, then run the tests and see if any fails. We made 246 such changes, one at a time, each in its own copy. 236 counted (the rest did not compile, or compiled to the same program as the original): the tests caught 206 and left 30 green, and 14 of those 30 were on lines the coverage tool counts as run. That count is ours, and the recipe to re-run it is in the manual. The example on the slide you can check with the engine you have: we made JsonObject’s Equals answer true for anything, and no test that passed on the clean copy failed, in EpicGames.Core.Tests or in UnrealBuildTool.Tests. The general point is not ours: Inozemtseva and Holmes measured it on large Java projects and found coverage only low to moderately tied to how many planted bugs a suite catches once its size is allowed for. Behind this slide: every planted bug in osu!’s scoring rules. Red: no test caught it.

osu!’s scoring graph from below: the ring of tests nearest, the requirements furthest.

Measured, in plain words

What one run found about the tests themselves

7 of 16

rules we took from the SemVer spec for this class have no test about them, including every rule on which strings are valid. Read its 2 tests in SemVerTests.cs beside semver.org.

3 questions

the spec leaves unsaid, such as how the class should refuse a bad string. They go to the code’s owner as questions. The tool never answers them itself.

206 of 236

planted bugs in SemVer.cs made a test fail, counting those that compiled and changed the program. In JsonObject.cs, 299 of 429. In osu!’s scoring, 286 of 536, against the 151 tests we ran. Our counts.

13 min

for the whole SemVer run, each planted bug built and tested in its own copy, never in the live project. osu! took 74: a change to a constructor rebuilds the game.

TNT Tools · closed loop prototype 0.1

Where each number comes from

5

Slide 5: what the speaker says

These are the three public trials of 30 October, read in plain words. The first number you can check by reading: Unreal Engine 5.7’s SemVer class has 2 tests, and of the 16 requirements we took from the Semantic Versioning 2.0.0 spec for it, 7 have no test about them, including every rule about which strings are valid; which test is about what was our reading of each test, written before any bug was planted. The 3 questions are the model’s own: the spec says what is valid, not how a class refuses a bad string, and nobody has said which; they go to the owner, never settled by the tool. The planted-bug counts leave out changes that did not compile or compiled to the same program as the original. osu!’s count is what three test files hold, 151 tests, and reads low for the game as a whole: its full test project crashed on this machine on the clean copy too. The SemVer pass took 765 seconds, about 13 minutes, at low priority on a busy machine. The faults themselves are on slide 3. Behind this slide: osu!’s scoring graph from below: the ring of tests nearest, the requirements furthest.

Inside the graph, among the methods.

The key is accuracy

The gap between a guess and the truth is what people call AI slop

Accuracy pays everywhere

Every value an AI guesses is one somebody has to check. A value read from a source, with the source named, is one they can check.

Why the gap opens

Prose can’t be checked. Tests pass for the wrong reason. The context window forgets.

At TNT Tools: no slop

That is the standard. Every value is read from a source, or marked absent with its reason.

What that means in this deck: every fault on the opening slides links to the public page that shows it, and every count that is only ours says so.

A value is either read from a source, which it names, or marked absent, with the search that looked for it named. There is no third kind: a gap is never filled with a plausible number.

TNT Tools · closed loop prototype 0.1

What accuracy has been measured, and on what

6

Slide 6: what the speaker says

The key is accuracy. Every value an AI guesses is one somebody has to check later. The gap between what AI guesses and what is true is what people deem AI slop. So the rule here: every value is read from a source, which it names, or marked absent, with the search that looked for it named, and a gap is never filled with a plausible number. For this deck that means every fault on the opening slides links to the public page that shows it, and every count that is only ours says so. The numbers on code are on slide 5. Behind this slide: Inside the graph, among the methods.

Requirements at the top, code below, in osu!’s scoring rules: the orange lines are the links that say which requirement owns which decision. It does look like a waterfall.

Back to 1999!

Requirements first, reviewed before the code.

This is how we review our code: we pour the effort into requirements first, review them before any code, then iterate. Time to learn systems analysis and design again.

More requirements, gaps filled, loop closed, nothing is a guess: slopless code.

What writing them first found in Unreal’s SemVer parser

16

requirements taken from the public SemVer spec, every numbered rule mapped or excluded with a reason

4

faults against that spec, found while writing them, each shown by running the code

7

of the 16 have no test about them, including every rule on which strings are valid

Reading the code against each rule found the faults. Planting bugs on purpose then showed where no test looks.

Behind: the links from requirements down to the code they own.

TNT Tools · closed loop prototype 0.1

Requirements first, in plain words

7

Slide 7: what the speaker says

Back to 1999. This is how we review our code: first by pouring our effort into requirements. Time to learn systems analysis and design again: requirements written and reviewed before the code, then iterated on milestones like everything else. More requirements, gaps filled, loop closed, nothing is a guess, which means slopless code. The card is the public SemVer trial: the requirements were written from the Semantic Versioning 2.0.0 spec, every numbered rule either mapped to one or excluded with a reason, and writing them against the code found 4 faults that fail when run, such as Compare throwing on the spec’s own example. Be exact about who found what: an agent reading the code against each rule found them, not the planted bugs. What the requirements and the planted bugs add is the count of what no test holds: 7 of the 16 requirements have no test about them. Behind this slide: the links from requirements down to the code they own, in osu!’s scoring rules.

The requirements and the use cases of osu!’s scoring rules: one epic, four use cases, 30 requirements.

Requirements make requirements

It extracts new requirements from the ones you have

Automatic

Each use case step needs something. What no requirement provides becomes a new one, marked inferred, and goes to the owner as a question. On SemVer it inferred 3 beside the 16 the spec states; on osu!’s scoring, none.

Complete in, accurate out

The more complete the requirements, the more accurate the prompt built from them. Start with the ones that run through the most code.

Epic to code: run, results held

The aim is a rough epic grown into a larger set, then code written from it. These trials started from a spec or a game’s own docs. The epic-to-code test has run. Its results are held for a paper and are not on these pages, so nothing here rests on them.

Behind: osu!’s requirements and use cases: one epic, four use cases, 30 requirements. We wrote all 30; the walk inferred none.

TNT Tools · closed loop prototype 0.1

How requirements make requirements

8

Slide 8: what the speaker says

The tool’s main intent is to take the requirements for a game from a rough epic use case and grow them into a larger set, so that the code written from them is as accurate as it can be. The first card is the mechanism: it walks each use case, and when a step needs a condition that nothing provides, it marks an inferred requirement and sends it to the owner as a question, never as a decision. The second card is our judgment from building with it; no run has measured it directly. The third is the honest state. The trials on public code started from a published spec or from a game’s own doc comments, so their requirements were written from existing words: on osu! all 30 were written and none inferred, and on SemVer 3 of 19 are marked inferred. The end-to-end test, a rough epic grown into requirements and code written from them, graded by tests from an independent source, has run, and its results are held for a paper and are not on these pages: slide 18 says what it takes. One earlier try gives no number to lean on: when two features of ours were joined, the walk did not find the places they meet. Behind this slide: osu!’s requirements and use cases.

The graph one pass leaves behind.

How it runs

Write the model, then one command

Model

what it is for, what it must do, who does what; 19 rules check it

Plant bugs

in every decision, every way, built in one compile

Test

each planted bug alone; some test must fail

Graph

every piece a node; 18 checks read it

Verdict

HOLDS, or OPEN with a list of what is missing

python closed_loop.py run <model> --out REPORT.md

You fix your part: a missing test, a line no requirement asks for, a planted bug to explain. Questions about what the code should do go to its owner. Run again, and lock it once nothing of yours is open.

TNT Tools · closed loop prototype 0.1

One pass, end to end

9

Slide 9: what the speaker says

One pass runs from the model to a verdict. Every requirement, use case step, place in the code, decision, planted bug, test and reasoning step becomes a node in a graph, and each of the 18 checks stops one way of being wrong, proven by a mistake planted in its own selftest. Holes that are yours you close. Questions only the owner can settle go to the owner. Behind this slide: The graph one pass leaves behind.

One stretch of the breaks up close: each point is one way of breaking one decision.

Planting bugs on purpose

Mutation testing: every decision, changed every way

Conditions

negated, made true, made false

Operators

< to <=, equality swapped, && to ||, arithmetic swapped

Values

a number plus one, a bool flipped, a text emptied, an argument defaulted

Structure

ternary arms swapped, an else removed, a statement removed, an early return

When no test catches a planted bug, there are only two honest answers

It is the same program

The change did not change the program: proven when the compiled IL is identical. Otherwise it needs a written reason, which lapses when that code changes.

A test is missing

So write it. Each requirement’s claim on its code is pinned to that code’s fingerprint, so an edited line reopens the claim. Our own “same program” calls have been wrong before, and tests now hold those cases.

Behind: planted bugs up close. Each point is one change to one decision.

TNT Tools · closed loop prototype 0.1

What a planted bug is, and when it changes nothing

10

Slide 10: what the speaker says

This is mutation testing: each planted bug is one small change to the code, and a test should fail because of it. Up to 24 planted bugs a statement, one of each kind first. One limit on every: until this week the C# adapter skipped overloads, operators and property bodies without saying so, and it still does not walk a field's value or an indexer, which the report now names; slide 15 has the numbers. A planted bug that every test passes is either a missing test or the same program. Same program is proven by comparing compiled code, Trivial Compiler Equivalence from Papadakis and others, 2015. Anything else needs a reason in writing, and a declared equivalent that a test ever kills is a hole, not a pass. Behind this slide: one stretch of the planted bugs up close: each point is one change to one decision.

One real thread through the graph of the kit’s example: a requirement, Claude’s step citing it and the code, a branch of that code, a break of the branch and the test that caught it

Checking Claude’s reasoning

Claude writes its reasoning as a chain; the graph checks every step

says: "A limited line sells its quantity and is then sold out"
cites: [W-3, Store.Buy]

Where it meets the code: the step becomes a node citing two nodes already there. A branch of that method maps to that requirement, so the two join, 2 links apart.

Grounded: a cite that is not there fails the run. Joined: what a step cites must connect within 4 links. Read for truth: a reader holding the whole source reads every step for what is wrong. A real step once said “or” where the code needs both, and grounding alone passed it.

1 of 1

owner’s question this chain never reached

OWNER: where it is stated

W-3: the requirement

A use case step needs it

Claude’s step cites both

Store.Buy: the code

A branch of it maps to W-3

A planted bug in that branch

The test that caught it

TNT Tools · closed loop prototype 0.1

The chain checks, and what they do not check yet

11

Slide 11: what the speaker says

This is how we go after hallucination. When Claude finishes a change it writes down, step by step, why the change does what the requirements ask. A chain is a written argument it produces on purpose, like showing your work; it is not a recording of the model's private thinking. Here is how it meets the code. The chain and every step become nodes in the same graph as the code. Each step has cites edges to the requirement and the method it stands on. A cite to something that is not a node fails the run, so a made-up requirement or a method the code does not have cannot pass. Then the graph walks from one cite to the next: here W-3 and Store.Buy meet because a branch of that method maps to W-3, two links apart. If they do not meet within 4 links, the step is reported. An owner's question no step reached is listed as a blind spot; this chain never reached W-4, what a refused purchase does to the stock. The step and the picture are from the kit's example, which you can run yourself: the chain was added to a copy of it and the whole loop run, and it held. Be precise about the limit of the grounding: it checks what each claim stands on, not whether the claim is true; that is the job of the full-source review, which reads every step against the whole source and is on by default, and no check yet walks on from a step to whether the code it stands on is tested, though the links are there, as the picture shows. It removes the hallucination that invents a source; it cannot promise to remove all of it. And one real step is the proof: a fresh reviewer found it said or where the code needs both. It was grounded and joined, and it was still wrong.

The real node graph of osu!’s scoring rules, a ring per kind of node: requirements at the top, then use cases, code, branches, breaks and tests

The node graph

Everything is one graph, from why to proof

How it is laid out

One ring per kind of node, top to bottom: why it exists, what the code does, what proves it. Each decision and each planted bug hangs under its method.

How code maps to requirements

339 links say which requirement owns which decision in the code. 191 say where that decision sits. A decision nobody owns is a hole.

Fly through it: the demo

W A S D, or follow one requirement down to the test that holds it.

1 source

30 requirements, all written

5 use cases, 17 steps

34 methods

200 branch cells

563 planted bugs, 277 not caught

168 tests

TNT Tools · closed loop prototype 0.1

The graph and the demo, in plain words

12

Slide 12: what the speaker says

This is the actual graph of osu!’s scoring rules, the rules of a shipped rhythm game (ppy/osu, MIT licence), drawn from the store the loop wrote: 1,062 nodes and 5,675 edges, nothing invented and nothing left out. The store itself is SQLite and N-Triples; the rings are how the demo draws it. Each ring is one kind of node. Each decision and each planted bug sits at the angle of the method it belongs to, so a method and everything under it line up. The mapping is two kinds of edge: a branch maps to a requirement, and a branch is at a method. That is how you get from any line of decision in the code to who asked for it, and back. The model has one epic use case, four use cases under it and 30 requirements, all written from the code’s own doc comments; this trial wrote no argument chain, so that ring is empty. The red points are the 277 planted bugs no test caught: 250 passed every test that was run, 13 are proven by compiled code to be the same program and 14 did not run. The 250 are the work, and they are 250 of the 536 that compiled and changed the program only for the three test files that were run, not for the game’s whole test project. The fly-through is the same data in three.js: drag to orbit, W A S D to fly, and a tour that follows one real thread. It also has views for just the code-to-requirement links, just the planted bugs, and just what the graph neural network reads.

Inside the graph.

The record

The graph remembers; closed means no “may” is left

MUST

Authored requirements

The owner wrote them. Each must hold, and a test must notice when it does not.

MAY

Inferred requirements

What the chain needs and nobody wrote. Only the owner settles each one.

Planted bugs can show the requirements are not closed, never prove they are: closed only means closed against the bugs tried and the tests that watched.

TNT Tools · closed loop prototype 0.1

The graph, and must and may, in plain words

13

Slide 13: what the speaker says

The graph is the record, and the 18 checks read it. The closest formal idea of closed is the Modal Transition System of Larsen and Thomsen, 1988: every behaviour is marked must or may, and a specification is refined until no may is left. Here the authored requirements are the musts, and the inferred ones the chain needs are the mays. The owner's open questions are exactly those mays. Behind this slide: Inside the graph.

Only the links from code to requirements.

Early prototype

The requirements map to the code, and the map finds mistakes

Every requirement links to the code it owns, in one graph. Its checks and the planted bugs do most of the finding so far. A graph neural network (GNN) was tried on the same map and switched off: a plain rule did better (slide 24).

7 of 16

requirements taken from the SemVer spec that no test is about, every rule on which strings are valid among them

110 of 191

decisions in osu!’s scoring rules hold a planted bug no test caught, each belonging to a requirement that should have a test

10 of 48

requirements on a public asteroid game resting on behaviour with no constant behind it

On slide 3: the three faults, each beside the public rule that shows it.

Behind: only the links from code to requirements.

TNT Tools · closed loop prototype 0.1

What the map has found, and on what

14

Slide 14: what the speaker says

It is an early prototype, and it has already found a lot of problems by mapping code to the graph. Be accurate about who found what: so far it is the graph's checks and the planted bugs. The graph neural network was tried for which requirement owns new code and is switched off, because a plain rule beat it; slide 24 shows that, and the head that does the job now. The first number is a hole in the map itself: on SemVer, 7 of the 16 requirements taken from the spec have no test about them, found by reading each test against the requirements, before any bug was planted. The second is osu!’s scoring rules: all 191 branches the tests cover are claimed by a requirement, and 110 of them hold a planted bug that passed the three test files that were run, so those are tests missing, not requirements missing. Read it as what those three files hold, and as low for the game as a whole. The asteroid run is public code, language-neutral parts only, with no planted bugs, because there is no JavaScript adapter yet. Slide 3 is the three trials on source your team can read, on the same terms. Behind this slide: only the links from code to requirements.

A thin graph: the public asteroid game's, 186 nodes against osu!'s scoring rules, 1,062. Green on this much says little.

Honest limits

When to use it, and what it is not yet

Use it when

  • many changes will touch the same code
  • a rule must still hold after the tenth edit
  • agents run long, across context windows

Skip it when

  • it is a one-off prompt or a quick script
  • there is no model yet, and no runnable tests
  • the question is how it looks or feels

Built for large, complicated systems. To build something simple, do not use it. It has not been measured on large systems yet.

A prototype

Version 0.1. Bugs are planted in C# and Python only: no C++ or Blueprint adapter yet. On Unreal’s SemVer.cs its C# adapter first saw 35 of 68 statements, and the report did not say so. Fixed, with planted tests.

A thin graph misleads

A clean report over a few requirements reads as proof, and is not.

A full pass takes time

13 minutes for SemVer’s 246 planted bugs, 74 for osu!’s 563.

Behind: a thin graph, 186 nodes. osu!’s scoring rules, with their breaks, have 1,062.

TNT Tools · closed loop prototype 0.1

Each limit, and each target, in plain words

15

Slide 15: what the speaker says

It is for building large, complicated systems. To build something simple, do not use it: there it is only overhead. It has not been measured on large systems yet. On C++, such as an Unreal game's own code, only the checks that read the graph run, because nothing plants bugs in C++ yet. The prototype card is the first trial on code your team can read: the C# adapter skipped overloaded methods, operators and property bodies without saying so, so on Unreal's SemVer.cs it saw 35 of 68 statements and on JsonObject.cs 213 of 288. An earlier target of ours has no overloads, which is why its numbers never showed it. All of that is fixed with planted tests, and the counts on slide 4 and slide 5 are from the fixed adapter, which walks all 68 statements; re-running that target changed no verdict, and the report now names what it still does not walk: a field's value and an indexer. How thin is too thin is not known yet. The two targets are for agents writing code using the loop, and neither has been measured there. The token figure is always being re-evaluated. Behind this slide: A thin graph: the public asteroid game's, 186 nodes against osu!'s scoring rules, 1,062. Green on this much says little.

The example's own graph, the one you get when the install holds: 122 nodes.

Try it

Install it; the example must hold

python install.py

From github.com/rj45Thompson/closed-loop-kit, MIT. Puts the skill into Claude Code and the tools beside it, runs every selftest, then the whole loop on both examples. Needs Python 3.10+ and PyYAML; names PyTorch and the .NET SDK when absent.

python install.py --check

python install.py --uninstall

Clean install: every selftest passes and both examples hold with PyTorch, scikit-learn and the .NET SDK. On Python and PyYAML alone 14 of 20 pass, and the rest are named as skipped.

What we would most like back

  • measurements from your own large systems: what it caught, and what it missed
  • the kinds of failure you see, and how often
  • any check you can make pass for the wrong reason

Most of the gain is in how agents are set up. Keep upgrading your own harness, and measure each change you make to it.

Behind: the example’s own graph, 122 nodes.

TNT Tools · closed loop prototype 0.1

What the installer checks

16

Slide 16: what the speaker says

The installer checks what it needs, copies the skill and tools, runs every selftest in the installed copy, then runs the whole loop on the examples, a C# wallet and store and its Python twin, which must hold. Any other language writes one verdict file. The manual has every claim in this deck in plain words, with the test behind it. Behind this slide: The example's own graph, the one you get when the install holds: 122 nodes.

The tests: the green ring, and the breaks each one caught.

What AI is for

There is no such thing as “we don’t use AI”

FIRST

A first-class testing tool

Write every line by hand if you like. It still tells you which of your tests cannot fail, and which code no test notices.

SECOND

A code writer

When it writes code, it is held to the same requirements and the same tests as everyone else.

Keep your tests. There is no reason to throw away really well done tests: that is the baby with the bathwater. They become the judge of what AI does.

Behind: the tests, and every planted bug each one caught.

TNT Tools · closed loop prototype 0.1

AI as a tester first, in plain words

17

Slide 17: what the speaker says

The reality is that AI is a first-class testing tool primarily and a code writer second. A team that writes all its code by hand can still point it at its tests: every bug it plants must fail a test, and one that does not is a test you do not have. Nothing here asks you to throw away tests you already trust. They are what it is measured against. Behind this slide: The tests: the green ring, and the planted bugs each one caught.

osu!’s scoring graph, the real graph the loop wrote, seen from a distance.

What is next

From a rough epic to code that is right

What the tool is for

Take the requirements for a game from a rough epic use case, grow them into a larger set, and write the code from that set as accurately as we can measure.

The question

If the requirements for something very difficult are complete and checked, how far does that move what an AI can build?

Measured so far

How much existing tests catch: 87.3%, 69.7% and 53.4% of the bugs we planted in three public targets made one of their tests fail.

RUN, RESULTS HELD

The epic-to-code test has run. Its results are held for a paper and are not on these pages, so nothing here rests on them.

The direction is better tools for making video games.

TNT Tools · closed loop prototype 0.1

What is measured, and what is not

18

Slide 18: what the speaker says

The tool’s main intent is to take the requirements for a game from a rough epic use case and grow them into a larger set, so that the code written from them is as accurate as it can be. The question on the right is a question and not a result: if we have the requirements to do something very difficult, how does that change the landscape for the AI? What is measured is narrower. On three public targets, a version parser, a JSON object and a game’s scoring rules, it measures how much of the bugs planted in existing code its own tests catch. What is not shown here is the thing the intent names. The epic-to-code test has run, and its results are held for a paper and are not on these pages, so nothing here rests on them. The trials on these pages started from a published spec or a game’s own doc comments. The test starts from one rough epic for a small game whose right answers come from an independent published source. Agents write the code from the grown requirements, and hidden tests written from that source, never from our model, grade it blind. Tokens are recorded, and so are how many requirements the epic grew into and how many the hidden tests needed that it missed. The direction is more advanced tools for making video games.

Telemetry · read from the loop’s own records

What the running loop reports

67

proof items pass, of 107 shown

2,365

entities in the loop’s own graph, 6,272 edges

60.4%

of planted bugs caught on the tool’s own code, pass 2 of 2

14 of 29

optional parts on

Off

token saving, one setting for all the parts

Proof board

PassPass: 6767Fail0Unknown0Not runNot run: 4040Held, not shownHeld, not shown: 5858

A proof item is a bar written before its run, with the record that passed or failed it; not run means that run has not happened yet. 58 proof items and 64 benchmark entries are held, counted and not shown

The loop’s own graph

Breaks (mutants)Breaks (mutants): 1,6391,639Branches (decisions)Branches (decisions): 426426TestsTests: 131131Code (methods)Code (methods): 3838Use case stepsUse case steps: 3232RequirementsRequirements: 2020

Six largest of 14 kinds of entity

Optional parts, stat against threshold

Self-improvementmeasured outcomes 0 of 20measured outcomes 0 of 20

Token saving: off. Every part, on or off, has a threshold: the value its stat must reach before the switch file lets it wake. 20 of the 29 are not set yet, and a retired part is not shown.

Pass logs and the loop’s own verdicts on each stat

The kit’s C# example

6 passes on file. Verdicts on 7 stats: 4 stalled, 3 done.

The kit’s Python example

14 passes on file. Verdicts on 7 stats: 4 stalled, 2 done, 1 improving.

The tool’s own code

2 passes on file. Verdicts on 7 stats: 7 too few.

One game feature

13 passes on file. Verdicts on 7 stats: 6 improving, 1 stalled.

A verdict needs three passes; with fewer the loop says too few. A rise in the share caught does not show the tests got better.

TNT Tools · closed loop prototype 0.1

The live telemetry page

19

Slide 19: what the speaker says

Written by autobot/telemetry.py from the same record as the live page, so the numbers cannot differ. The proof board counts pass, fail, unknown and not run apart, and counts what is held without showing it. The share of breaks caught is the loop measuring its own code; a rise in it does not show the tests got better. Every run measured here was small, and nothing on this slide speaks to large systems. The live page draws the same data with WebGPU and refreshes as runs go: the live telemetry page.

Experimental

What runs unproven, and what was retired

The graph network

Retired on 3 November: on the tool’s own code it lost to a plain rule. It is tried again only when a project’s graph reaches 5,000 entities. Whether it suits large projects is not tested.

Self-improvement

It records its own stats and names what to improve next. Whether that raises results has not been measured.

Two targets

95% or better accuracy where the loop fits, and up to 90% fewer tokens per closed requirement. Aims, not results.

The measured results are on the slides before this one. These are kept for anyone who wants to try them.

TNT Tools · closed loop prototype 0.1

What is measured, and what is not

20

Slide 20: what the speaker says

Self-improvement runs and the graph network is retired; neither is shown to improve results. The graph network is switched off, because a plain rule beat it on the tool's own code; it is tried again only when a project's graph reaches 5,000 entities, and whether it suits large projects has not been tested. Self-improvement names the next thing to work on, and whether that raises results is not measured. The two targets are aims.

The tool's own graph: its own requirements, its own code, its own breaks.

Experimental · It tests itself

Self-improvement is on by default

ON

Every pass

Records 44 stats about itself, checks them against the tool’s own rules for improving, and names what to improve next.

Overhead: about a third of a second and 3 printed lines.

OFF

Some tokens saved

Run with --no-self-improve, or set self_improve: false in the model. A pass saves little.

The real spend is the standing tick, which ships in the kit and which you schedule: about 200,000 to 290,000 tokens a tick so far.

NOT NETWORKED HERE

One shared graph

We share ours across the team as a retrieval store (RAG): agents look up what is proven instead of re-reading it. That it saves tokens is our belief, not a measurement.

This version keeps one graph per checkout. Add sharing in whatever way suits your network.

Behind: the tool’s own graph.

TNT Tools · closed loop prototype 0.1

Self-improvement, in plain words

21

Slide 21: what the speaker says

Self-improvement is on by default because the tool tests itself. Every pass writes one row of stats, checks them against requirements on those stats, and prints what to improve next. That is cheap. A standing tick costs far more: a fresh agent takes one item, measures before and after, and records whether it was worth it. Tokens here are the app's count for each agent, and nothing starts a tick for you. How far a shared graph cuts tokens has not been measured in this version, which is not networked, and no agent has been timed using a graph in place of the files. Behind this slide: The tool's own graph: its own requirements, its own code, its own breaks.

Inside the tool's own graph.

Experimental · The loop, in pseudo C++

One pass, then one tick

// One pass. Self-improvement is on by default.
Verdict run(Model& m, bool selfImprove = true) {
  Breaks  b = Mutate(m.code);       // one compile
  Results r = RunEachAlone(b, m.tests);
  Graph   g = KnowledgeGraph(m, r);
  Holes   h = Criteria(g);          // 18 checks

  if (self_improve_on(m, selfImprove)) {
    Row row = PassStats(m).collect();  // 44 stats
    MorphLog(m).add(row);
    auto fired = improvement::fires(rules, log);
    Print(NextImprovement(log, fired).picks());
  }

  return h.ours.empty() ? HOLDS : OPEN;
}

// One tick. A fresh agent, one item, no memory.
void Tick() {
  Start s = loop_tick::start();     // the lock
  if (s.stopped || s.busy || s.idle) return;
  if (s.epochStarts) Brainstorm();  // every 12
  RunSelftests();                   // red = item
  telemetry_audit();                // unread = item
  Item i = Choose(improvement::check(), board);
  Stat before = i.stat.Measure();
  i.Do();                           // one change
  Stat after = i.stat.Measure();
  if (s.epochEnds) {
    run_plan::decide();            // parts on or off
    improvement::graph();           // did it work?
  }
  loop_tick::finish(i, before, after, worth);
}

The same shape, one level up: stats are its code, triggers its requirements, ticks its tests. Three ticks worth “no” in a row and it sleeps until something changes.

TNT Tools · closed loop prototype 0.1

The recursive graph, in plain words

22

Slide 22: what the speaker says

The toolkit is written in Python; this is pseudo C++ for a C++ audience, and the lower-case names are the real ones in the code: self_improve_on, improvement fires and check, loop_tick start and finish, telemetry_audit, run_plan decide, improvement graph. Left: one pass, which is what an agent runs. If self-improvement is on, it writes a row and prints the stats' picks. Right: one tick, the standing job. A fresh agent, no memory, chooses one item: a red selftest first, then an unread stat, then a fired trigger, then the board. It measures the same stat before and after. Every 12 ticks is an epoch: the first tick brainstorms, the last benchmarks, re-decides which parts run and asks whether the improvements worked. The improvement graph is checked the way the requirement graph is. Behind this slide: Inside the tool's own graph.

Every break of the tool's own code. Red: no test caught it. It is less tested than the SemVer parser it was run on (87.3%), and you can see it.

Experimental · Self-improvement · measured

What it has found in itself

19

ways a check could pass for the wrong reason, found by fresh reviewers. 17 fixed, each with a planted mistake a selftest must catch.

11 → 0

stats recorded with nothing reading them. Each now triggers something, or has a dated reader.

33

requirements on its own stats. 5 are firing now, and each one that fires is a work item.

9

faults its first measuring round found in itself: 5 in the harness, 4 in the tool. All fixed.

3 of 7

run plan lines that reach what a pass runs; the run plan says which learned parts a pass switches on. Its own scheduled run found none did by code, wired 3, and marked 4 measured only.

58%

of the bugs planted in its own code caught on its first pass over itself, 885 of 1,532; a second pass caught 989 of 1,637 (60%). It is less tested than the SemVer parser it was run on (87.3%).

Not yet: 0 of the 20 measured outcomes its improvement learner needs, so rules choose the work for now, not a model.

Behind: every bug planted in its own code. Red: no test caught it.

TNT Tools · closed loop prototype 0.1

Every self-improvement stat, where it comes from

23

Slide 23: what the speaker says

The audit of its own stats is itself a result: it reported 6 unread stats, and when the audit was fixed the true number was 11. All 11 now have a reader. The requirements on its stats are a second layer of the same idea: a pattern in a stat fires, and what fires becomes a work item. This is young. The standing tick has run only a few times, and no improvement has yet been measured before and after by the learner, so these numbers show what it has found, not yet how fast it gets better. Behind this slide: Every break of the tool's own code. Red: no test caught it. It is less tested than the SemVer parser it was run on (87.3%), and you can see it.

Experimental · What we switched off

A part that does not beat a plain rule is switched off

Every learned part had to beat a simple rule that does not learn. The ones that lost are off, and the record says when one may come back.

OFF · THE GRAPH NEURAL NETWORK (GNN)

Who owns new code 0.65 vs 0.72: on the tool’s own code it lost to a plain rule, so it is off.

Retired on 3 November. One rule stays in the switch file: when a project’s graph reaches 5,000 entities it is tried again, and must win by more than 0.02 on code it has not seen. A score of 1.0 means the right answer is always ranked first.

OFF · ITS OTHER JOBS

Which test catches which planted bug is now read off the test runs, and a bug that changes nothing is proven by the compiled code. Nothing is left to guess.

OFF · TWO MORE

A learned pick of a step’s requirements lost to plain similarity, and neither is used: the author binds each step’s requirements, and the model rules refuse a step that binds none. A learned order for running the planted bugs was retired, because the break-order head does that job.

TNT Tools · closed loop prototype 0.1

The graph network’s whole story, and its numbers

24

Slide 24: what the speaker says

Every learned part of the tool was held to one rule: it must beat a plain rule that does not learn, on code it has not seen, or it is switched off. The graph neural network lost. On the tool's own code it ranked the right owner of new code at 0.65 against the plain rule's 0.72, so it was retired on 3 November. One rule stays in the switch file: when a project's graph reaches 5,000 entities it is tried again, and it must win by more than 0.02. Its other jobs need no guessing any more: which test catches which planted bug is read off the test runs themselves, and a bug that changes nothing is proven by comparing compiled code. Two other learned parts are off as well: the pick of a step's requirements lost to plain similarity, and neither is used now, because the author binds each step's requirements and the model rules refuse a step that binds none, and the learned order for running the planted bugs, which did beat its plain rule, was retired because the break-order head shown next does that job. Switching off what does not beat a simple rule is the point: the record says why each part is off and what would wake it.

Experimental · What does the job now

Two small checking heads, each beating the best plain rule

A head is one weighted sum of four measured signals, such as how alike two texts are and the words they share, trained on the project’s own record. Cheap, and its weights can be read.

ON · THE OWNER HEAD

Which use case step owns a piece of code, before it has tests

0.549 right first time, against 0.479 for matching words, the best plain rule. Measured on 71 places in one feature of our own game.

ON · THE BREAK-ORDER HEAD

Which planted bugs to run first

54% of the bugs no test catches land in the first quarter of the run, against 21% for the best plain rule; random order would find one in four. On 1,335 planted bugs in the same feature.

And what needs no learning: which test catches which bug is read off the runs, and code links to a use case step by running that step’s own tests.

TNT Tools · closed loop prototype 0.1

How each was tested, and what replaced what

25

Slide 25: what the speaker says

What does the graph network's jobs now. Two small checking heads, each a linear model: one weighted sum of four measured signals, trained on the project's own record, so it is cheap and anyone can read its weights. Both were held to the same rule as the network, and both passed it with their bars committed before the run. The owner head says which use case step owns a piece of code before that code has tests: it was right first time 0.549 of the time, against 0.479 for matching words, the best plain rule, on 71 places in one feature of our own game. The break-order head decides which planted bugs to run first, so the ones no test catches show up early: 54% of them landed in the first quarter of the run, against 21% for the best plain rule, where a random order would find one in four; that was on 1,335 planted bugs in the same feature. Both are on one feature of our own, not on public code, so read them as a first measurement. The rest needs no learning at all: which test catches which bug is read off the test runs, and code is linked to a use case step by running that step's own tests.

The tool's own graph from above.

Experimental · What you are getting

Where it comes from, and how new it is

Where it comes from

A mutation of the self-improvement loop of ARC3, an agent harness we built for a benchmark of puzzle games, and that loop's failure is why it distrusts its own numbers. Ours is internal, so this version was made for you.

How new it is

A separate agent with web search, which had not built the tool, found about 8 of its 10 parts are known work. The whole, used as an agent’s done gate, looks new, at medium confidence.

A small edge, for now

As harnesses improve, this will become common knowledge.

TNT Tools · closed loop prototype 0.1

Where it comes from, and what it measured there

26

Slide 26: what the speaker says

This is the method we use ourselves at TNT Tools. The harness comes from ARC3, an agent harness we built for a benchmark of puzzle games. That project's autonomous self-improvement run went 16 days and found nothing that generalized: a counter that could be gamed rose while the honest number stayed flat. That failure is why every stat here must have a reader, a control and a fresh reviewer. As harnesses improve this will become more common knowledge, so the edge is small and it is for now. Our graph is shared across the team as a retrieval store; this version is not networked, and you could add that in whatever way suits your network. Behind this slide: The tool's own graph from above.