Benchmark

This is the record of every test run on the closed loop, kept so that a reader who doubts it can check. The closed loop is a set of tools that hold a coding agent’s work to written requirements: it breaks the code on purpose, in small ways, and asks whether a test notices. It claims to make an agent’s changes match what was asked. Whether that is true is what this page is for. Each entry gives the question a test asked, the bar it had to clear and whether the bar was fixed before the result, how it could have failed, the command that runs it again, the result read from the file that holds it, and what it cost.

48tests on record
34passed
9failed
5unclear
0not finished
21 of 48had the bar written down first

Where it stands today

48 tests are on record. 34 passed, 9 failed, 5 are unclear and 0 are not finished. A test passing here means a stated bar was met on stated data. It does not mean the loop is proven right.

What failed

What is unclear

What is not known

The whole list is at the end of this page.

Words used on this page
break
A small deliberate change to the code, such as flipping a comparison, made to see whether any test fails.
caught, survived
A break is caught when a test fails because of it, and survives when every test still passes.
kill rate
The share of the breaks that were caught.
hole of ours
A place where the loop's own checking lets a mistake through, and the fix is the loop's to make.
bar
The result a test must reach to count as passed, written down before the result is known. Whether it was is shown on every entry.
plant
A failure put in on purpose to show that a check can notice it.
plain rule
A simple rule that does not learn, such as counting past catches. A learned part has to beat the best one to earn a place.
network
The small learned model that guesses which requirement owns a piece of code.
seed
One random start of a run. Results are given over several seeds, because one seed can flatter.

Is it working?

Every test, failures first. Open an entry for how it can fail, how to run it again and what it cost.

Jump to: Failed 9 · Unclear 5 · Passed 34

Failed: 9

These are the most useful entries. Each one is a claim that did not hold, or a bar that was missed.

✗ Failed

How full must the graph be before the network beats the plain rules?

Oct 29

The question. Giving the network 10, 25, 50, 75 and 100 percent of the links, at what fullness does it first beat every plain rule that does not learn, at every seed?

The result. On the one graph where it won, the network scored 0.771 at full against 0.666 for the best plain rule. The tool's own graph beat the best rule at half full and then fell back, and the two small examples never beat it. Two seeds.

Read from the fullness curve rows.

Caution. Two seeds is thin. A later kill-count rule scored 0.845 on that same graph, so the plain rule it beat was not the strongest one.

The bar. The crossover, defined in the tool before the run, is the least fullness from which the network beats every plain rule at every seed. Written down first, in commit d3e1eddb.

How it can fail, how to run it again, what it cost
Ways to fail
  • No fullness from which the network beats the best plain rule at every seed. (fails by missing its bar: autobot/gnn_curve_1029.jsonl)
Run it again
py -X utf8 autobot/gnn_curve.py <graph.sqlite> --fractions 0.1,0.25,0.5,0.75,1 --seeds 0,1
Cost
Not recorded; run on CPU on 10-29.
Why this state
On three of four graphs the network did not stay ahead of the best plain rule. On one graph it did, from half full.
✗ Failed

Do fresh reviewers find holes the loop passes?

Oct 29

The question. A fresh agent that has not seen the work reads the loop's code and its criteria and ranks the ways a criterion passes while the requirement is not closed. How many did two of them find?

The result. The first fresh review ranked 9 holes in the loop's checks and the second ranked 10. The first one ran nothing; the second marks each hole as verified, reasoned or unverified.

Read from autobot/research_REVIEW_2_1029.md, a number in its written text; autobot/research_REVIEW_1_1029.md, a number in its written text.

Caution. Both reviewers are language models. The loop is a practice that expects a person's review too, which has not been done.

The bar. None fixed first. The loop's own criteria claimed to be complete. Not written down first. The reviews were asked for after the criteria were written and no count was predicted.

How it can fail, how to run it again, what it cost
Ways to fail
  • A reviewer finds a hole the loop's checks pass. Both did. (fails by missing its bar: autobot/research_REVIEW_2_1029.md)
  • Each hole names a plant that reproduces it, and the board marks one fixed only with its plant. (failure planted and seen to fire: autobot/research_REVIEW_1_1029.md)
Run it again
Cannot be run again by one command. Two agent runs that read the code; the reviews are autobot/research_REVIEW_1_1029.md and autobot/research_REVIEW_2_1029.md.
Cost
Not recorded.
Why this state
The first review ranked 9 holes and the second 10. The criteria were not complete, and the reviews are why they are more now.
Read more
Manual: fresh reviewers
✗ Failed

Every break of the learned parts, with a verdict

Oct 30

The question. Of the breaks the loop made in the learned parts' own code, what share does a test catch, now that every break has a verdict and the earlier counts are no longer lower bounds?

The result. 1,174 of 1,816 breaks caught across the six parts, 64.6 percent, and 642 got past every test. By part: 695 of 1,077, 100 of 176, 37 of 71, 84 of 104, 161 of 267 and 97 of 121.

Read from autobot/proofs/B1.md, a number in its written text.

Caution. It ran on the live tree rather than a snapshot. A change to code under test makes its verdicts stale and they rerun. The public counts still carry the older, lower-bound numbers until the next release.

The bar. The plan item's done-when, committed first: every break has a verdict. The loop's own standing line for a project is 80 percent caught. Written down first, in commit 161d20fc.

How it can fail, how to run it again, what it cost
Ways to fail
  • A job below the 80 percent line. Four of the six are. (fails by missing its bar: autobot/proofs/B1.md)
  • Any break with no verdict. None remain. (fails by missing its bar: autobot/proofs/B1.md)
Run it again
py -X utf8 autobot/gnn_subgraphs.py run --budget-min 180
Cost
245 minutes. 16:47 to 20:52 at below-normal priority on two workers, from the proof. Compute, not tokens.
Why this state
Only 2 of the 6 learned parts reach 80 percent caught, and 642 breaks got past every test.
✗ Failed

Do the plan's switches hold at more seeds?

Oct 30

The question. The run plan switches parts on or off by one seed's score. Would other seeds flip any switch, and was the number quoted in the deck the typical one?

The result. Over ten seeds the learned ordering scored 0.502 on the first project (range 0.345 to 0.636) against 0.313 for the plain prior, ahead at 10 of 10. The deck and manual were corrected to means over seeds.

Read from the board's record of the seeds.

The bar. A guess written on the board before the run - 1 to 3 seeds, 0 or 1 switch flipping - with the flip defined as a gap against the 0.02 margin. Written down first, in commit d23596ec.

How it can fail, how to run it again, what it cost
Ways to fail
  • A switch flips at another seed, or the quoted number was the best of the range. (fails by missing its bar: autobot/CLOSED_LOOP_TASKS.md)
Run it again
Cannot be run again by one command. The seed script was kept in the scratch folder of the session that ran it and is not in the repository. The rows are in autobot/tick_runs/e1_3_*.jsonl.
Cost
72 minutes at below-normal priority on two threads, no tokens beyond the tick. From the board.
Why this state
Seed 0 was the best of ten on the first project, so the public 64 percent was the top of a range. The honest figure was 50 percent. On the second project, seed 0 alone put the switch off.
✗ Failed

Does each learned part beat plain rules on the loop's own code?

Oct 30

The question. Each learned part of the loop was made a project of its own and the loop was run on it. Did any learned part beat the best plain rule on its own code?

The result. 1,812 breaks made across the six parts: 369 caught, 205 survived, 1,238 not run. 115 of the 369 catches came only from the new guards, not from the tool's own selftest.

Read from autobot/gnnmodels/RESULTS.md, a number in its written text.

Later work changed what this means: see Every break of the learned parts, with a verdict.

Caution. 1,238 breaks had no verdict when this was written, so the counts are lower bounds. The next entry gives every break a verdict; the published figures here still carry the older counts until the next release. Own-graph runs used one seed.

The bar. Whether the network beats the best plain rule that does not learn, the criterion the network is judged by. Not written down first. The criterion was the standing one; the run's own tool and its results landed in one commit, so no bar was committed first.

How it can fail, how to run it again, what it cost
Ways to fail
  • The network does not beat the best plain rule, or the graph is too thin to say. (fails by missing its bar: autobot/gnnmodels/RESULTS.md)
  • Planted known-bad inputs for the new guards must be caught. (failure planted and seen to fire: autobot/gnnmodels/RESULTS.md)
Run it again
py -X utf8 autobot/gnn_subgraphs.py summary
Cost
Not recorded as a single figure.
Why this state
None of the six beat a plain rule on its own code. Four lost and two were too thin to say.
✗ Failed

Can a goal be turned into a vector that points at goals worth training?

Oct 30

The question. Leave one domain out. Does a direction learned from the others point at goals that the network can learn, and at goals that are useful, on a graph it has never seen?

The result. Learnable: pooled rank correlation 0.295, but it met the bar in 2 of 5 domains and 0 of 8 single seeds. Useful: 0.409 with the same shortfall. The two directions are 95 degrees apart. A permutation test alone would have called both a yes.

Read from autobot/GOAL_VECTOR.md, a number in its written text.

The bar. Written before any result and committed with the tool, then the predictions sealed in a second commit before any was trained. Written down first, in commit c59a8e2a.

How it can fail, how to run it again, what it cost
Ways to fail
  • The pooled rank correlation fails the bar on a held-out domain. (fails by missing its bar: autobot/GOAL_VECTOR.md)
  • The plant selftest on invented graphs: a copy of a relation must be refused by the useful direction. (failure planted and seen to fire: autobot/GOAL_VECTOR.md)
Run it again
py -X utf8 autobot/goal_vector.py direct
Cost
Compute from 12:55 to 14:05 on 10-30; tokens were not logged separately.
Why this state
A weak direction for learnable, none for useful. It failed its bars and no page was made from it.
✗ Failed

A full pass on the game feature the loop was first built on

Oct 30

The question. Of the breaks made in this feature's code that a test could judge, what share did a test catch, and are there holes of the loop's own?

The result. Kill rate 0.9653: 974 caught, 55 survived, 77 holes of ours still standing. Of the 55 survivors, 20 are declared to change nothing.

Read from the game feature's pass log.

Caution. A kill rate counts only breaks whose compiled code differs and which a test could judge. It rises when an equivalent is declared, which is a claim, not a proof.

The bar. Exit 0 means no hole of the loop's own stands. No kill-rate bar is written before a pass. Not written down first. A pass reports a rate and a count of holes. The rule set is in code; no per-pass number was committed first.

How it can fail, how to run it again, what it cost
Ways to fail
  • A break no test catches, or a hole of the loop's own, makes the pass exit 1. (fails by missing its bar: the pass log)
Run it again
py -X utf8 autobot/closed_loop.py run <the feature's model>
Cost
Compute at below-normal priority on two workers, about 23 minutes; tokens not logged.
Why this state
974 of 1,104 breaks caught, 55 survived and 77 holes of the loop's own stand, so the pass exits 1.
Read more
Manual: the criteria the growth page
✗ Failed

A pass on the loop's own code

Oct 30

The question. Run on its own tools, how much of its own code do its tests hold?

The result. Kill rate 0.6042: 989 caught, 648 survived, 387 holes of ours.

Read from autobot/selfmodel/stats.jsonl, value of kill_rate where time=2026-10-30T16:28:41; autobot/selfmodel/stats.jsonl, value of holes_ours where time=2026-10-30T16:28:41.

Caution. The tool's own code is the least held of anything it has checked.

The bar. Exit 0 means no hole of the loop's own stands. No kill-rate bar is written before a pass. Not written down first. As for any pass, the rule set is in code and no per-pass number was committed first.

How it can fail, how to run it again, what it cost
Ways to fail
  • A break no test catches, or a hole of the loop's own, makes the pass exit 1. (fails by missing its bar: the pass log)
Run it again
py -X utf8 autobot/closed_loop.py run autobot/selfmodel/model --gnn --fresh
Cost
Compute at below-normal priority; tokens not logged.
Why this state
989 of 1,637 breaks caught, 648 survived, 387 holes of the loop's own stand.
✗ Failed

Joining two finished features, does the graph name the seams?

Oct 30

The question. Two finished features of one game were written as models and joined. Does the graph name the places where one needs what only the other provides?

The result. Against 23 real seams: the graph walk 0, the graph at best 2, exact names 2, a word search 10, an agent reading both models 20 with 4 false and 3 missed.

Read from autobot/t8_results.md, a number in its written text.

Caution. One pair, not blind, so the reading figure is an upper bound. The claim that joined products remix with no errors is not supported, and the deck and manual stopped saying it.

The bar. The list of real seams was written and hashed before any code was opened, and the scoring rule was fixed with it. Not written down first. The list and its hash were recorded in a file before the code was opened, but the file was committed together with the result, so the order cannot be shown from history.

How it can fail, how to run it again, what it cost
Ways to fail
  • The graph names fewer of the real seams than a plain word search. (fails by missing its bar: autobot/t8_results.md)
  • A planted provider must change the causal count: it moved it from 7 to 6, so the zeros are not a dead instrument. (failure planted and seen to fire: autobot/t8_results.md)
Run it again
py -X utf8 autobot/t8/count_scores.py
Cost
353,000 tokens, 21 minutes. From the board: 21 of 60 minutes budgeted, 353K tokens against 200K, over by 153K.
Why this state
The graph named at best 2 of 23 real seams, and none of the 8 open or in conflict. A word search found 10, an agent reading both models 20.

Unclear: 5

A result exists and it does not settle the question: too few runs, one judge, or a test that both sides passed.

? Unclear

Does it matter which requirements are written first?

Oct 29

The question. Giving the network a fraction of the requirements, does the order they are chosen in change how soon it beats the best plain rule, in random order, most-methods-spanned first, or most-branches-owned first?

The result. On the larger graph with the three most cross-cutting requirements first (10 percent), the network scored 0.736 against 0.666 for the best plain rule. In random order at 10 percent it scored 0.414. One seed.

Read from the seed-order rows.

Caution. One seed and two graphs. The deck and manual state the finding as a lead.

The bar. None. The question was asked on the board; no bar was fixed before the run. Not written down first. The aim was to find the smallest seed of requirements that lets the network pay. No threshold was set, and the run used one seed.

How it can fail, how to run it again, what it cost
Ways to fail
  • No order lets the network beat the best plain rule at any fraction. That is what happened on the loop's own model. (fails by missing its bar: autobot/gnn_seed_curve_1029.jsonl)
  • A plant shows the pillar order is not just the order of the ids. (failure planted and seen to fire: autobot/CLOSED_LOOP_TASKS.md)
Run it again
py -X utf8 autobot/gnn_curve.py <graph.sqlite> --by pillar,spread,random --seeds 0
Cost
Not recorded; run on CPU on 10-29.
Why this state
On the one graph with many requirements, the three requirements that span the most methods gave the network what a random seed needs all 27 for. On the loop's own model no order beat the plain rule. One seed.
Read more
Manual: seed it first
? Unclear

Are the documents tool's own checks held by breaks?

Oct 30

The question. The loop breaks code and asks whether a test notices. Break the documents tool's checks the same way: do its own tests notice?

The result. One class of checks: 83 of 83 breaks caught on the second pass. Another: 191 caught, 1 proven the same program, 1 got past on the second pass. The first passes let 17 and 29 through.

Read from autobot/proofs/C1.md, a number in its written text.

Caution. Only five classes in the tool were broken. The readers and the overlay are held by plants alone.

The bar. None set before the passes. The plan asked only whether the plants are themselves held. Not written down first. No bar was written before the three passes. The result is reported as found.

How it can fail, how to run it again, what it cost
Ways to fail
  • A break of the tool's own checks that no test catches. (fails by missing its bar: proofs/C1.md)
Run it again
py -X utf8 autobot/closed_loop.py run autobot/selfmodel/model --gnn --fresh
Cost
Compute at below-normal priority on a copy; tokens were not logged separately.
Why this state
The first pass let 46 of 281 breaks through; after new tests the second pass caught all but one, and one more removal has not been through a further pass.
? Unclear

Does every tool have a selftest, and does it pass?

Oct 30

The question. Each tool should be able to test itself with --selftest. Which tools have one, which pass, and which have none?

The result. Of the tools surveyed, those with a selftest that ran and exited 0, those that failed, those that take no such flag and those with none are counted on this page, with the lines they printed (a floor on the checks) and the tools without one by name.

Read from autobot/benchmark_selftests.json, key counts.none; autobot/benchmark_selftests.json, key counts.runs; autobot/benchmark_selftests.json, key counts.fails.

Caution. Lines printed are not checks; one line can stand for several, and some tools print only a summary. A tool still being built by another item failed its selftest in an earlier measurement the same hour and passed in the next, so the class count is a reading at one moment.

The bar. None fixed first. A class count taken now. Not written down first. The survey is new; there was no standing rule about selftests before this record.

How it can fail, how to run it again, what it cost
Ways to fail
  • A tool whose selftest exits non-zero, rejects the flag, or has none is listed by name. (fails by missing its bar: autobot/benchmark_selftests.json)
  • A tool on disk that mentions a selftest and is not in the survey makes the check refuse. (failure planted and seen to fire: autobot/benchmark.py --selftest)
Run it again
py -X utf8 autobot/benchmark.py --survey-selftests --tools-root <a copy of the tools folder>
Cost
About 31 minutes of wall time on a busy machine, 8 selftests at once, taken by prove_all.py; the longest single selftest took 30 minutes beside other work.
Why this state
Every selftest that ran passed in the last measurement, 68 of 68, taken by benchmark.py --survey-selftests on 2026-11-01 on a copy of the tools (81 tools in all); the measurement before it was 67 of 67 on the tools in place, then the measurement before it was 60 of 60 on a copy, then 60 of 60, then 57 of 57, taken by prove_all.py with the selftests running several at once. In the measurement before, autobot/dictation_proof.py had exited 1 while X5 was still being built. A passing selftest says little about how much it checks, and 12 tools have none.
? Unclear

Did each standing tick find something worth its minutes?

Oct 30

The question. A recurring tick picks one improvement and does it. Was each tick worth the minutes it used?

The result. Five ticks on 10-30, five answers of worth yes. 156 minutes in all.

Read from autobot/loop_ticks.jsonl, count of rows where worth=yes; autobot/loop_ticks.jsonl, sum of minutes.

The bar. None. The tick judges its own worth. Not written down first. Each tick records its own answer, so there is no outside judge and no bar.

How it can fail, how to run it again, what it cost
Ways to fail
  • A tick answering no. None has. (never exercised)
Run it again
Cannot be run again by one command. Each tick is an agent run. The ledger it writes is autobot/loop_ticks.jsonl.
Cost
156 minutes. From the tick ledger.
Why this state
All five ticks said yes about themselves. That is five opinions of the party that did the work.
? Unclear

A real argument chain on the kit's example

Oct 30

The question. Does one step of Claude's reasoning, citing a requirement and a method of the kit's example, ground and join in the graph the loop writes, with the whole loop still holding?

The result. The step was grounded and its two cites joined 2 links apart. One owner question was left as a blind spot the chain never reached, and the loop held.

Read from autobot/proofs/X1.md, a number in its written text.

Caution. The example is small and built to hold, and the chain has one step. Run it on a copy of the kit with CLOSED_LOOP_ROOT set to the copy, never on the shipped example, whose graph it would change. The picture on the chain slide is drawn from this run's graph.

The bar. The checks as the manual states them before this run: every cite must exist, and every pair a step cites must join within four links. Not written down first. No bar was committed for this run. It was made to take a real step and its graph for the deck, so it is an observation of the stated checks, not a test with a bar fixed first.

How it can fail, how to run it again, what it cost
Ways to fail
  • A cite that is not in the graph fails the run with exit 1. The manual records three planted cases run on a copy of the example: a requirement that does not exist, a method the code does not have, and a step citing nothing. (failure planted and seen to fire: closed_loop_kit/docs/manual/closed-loop-manual.html)
  • A pair of cites the graph cannot join within four links is reported. None was exercised here. (never exercised: proofs/X1.md)
Run it again
py -X utf8 autobot/closed_loop.py run closed_loop_kit/example/model --chain autobot/proofs/X1_chain.yaml --no-self-improve
Cost
A few minutes of CPU at below-normal priority on a snapshot of the tools; no tokens beyond the agent's own.
Why this state
One step, one run, and no bar fixed first. It shows the checks run end to end on a real step and that the loop still holds; it does not show they catch a wrong one.
Read more
Manual: argument chains

Passed: 34

Passing means the stated bar was met on the stated data. It does not mean the loop is right in general.

✓ Passed

The kit's C# example

Oct 29

The question. On the small example shipped with the kit, does the loop reach a clean pass a stranger can reproduce?

The result. 54 caught, 3 survived and declared to change nothing, 0 holes of ours.

Read from closed_loop_kit/example/stats.jsonl, value of kill_rate where time=2026-10-29T21:42:35.

Caution. The example was worked up to 1.0 on purpose by declaring equivalents and taking the lock. A rate of 1.0 here is built, not found.

The bar. Exit 0 means no hole of the loop's own stands. Not written down first. The example is built to hold. That a clean example holds shows the loop can pass, not that it finds anything.

How it can fail, how to run it again, what it cost
Ways to fail
  • A break no test catches on the example. Three survived and are declared to change nothing. (fails by missing its bar: closed_loop_kit/example/stats.jsonl)
Run it again
py -X utf8 autobot/closed_loop.py run closed_loop_kit/example/model
Cost
Not recorded.
Why this state
Kill rate 1.0 on the example, after the example was worked to hold.
✓ Passed

The kit's Python example

Oct 29

The question. The same, for a stranger without a .NET toolchain.

The result. 66 caught, 3 survived and declared to change nothing, 0 holes of ours.

Read from closed_loop_kit/example_py/stats.jsonl, value of kill_rate where time=2026-10-29T21:40:28.

Caution. Built to hold, as the C# example. It also exposed that the loop was blind to the Python tests' check lines; that was fixed.

The bar. Exit 0 means no hole of the loop's own stands. Not written down first. As for the C# example.

How it can fail, how to run it again, what it cost
Ways to fail
  • A break no test catches on the example. Three survived and are declared to change nothing. (fails by missing its bar: closed_loop_kit/example_py/stats.jsonl)
Run it again
py -X utf8 autobot/closed_loop.py run closed_loop_kit/example_py/model
Cost
Not recorded.
Why this state
Kill rate 1.0 on the example after 14 passes.
✓ Passed

Does the example run on a machine with only a newer .NET?

Oct 29

The question. Does the kit's C# example run on a machine that has .NET 9 or 10 but not the version it targets?

The result. It would not run at first. With a roll-forward setting 5 of 5 tests passed on .NET 9; without it, the run stopped.

Read from autobot/CLOSED_LOOP_TASKS.md, a number in its written text.

The bar. A budget of 10 minutes and 20K tokens, written first. Not written down first. The board line carrying the budget landed with the result.

How it can fail, how to run it again, what it cost
Ways to fail
  • Asked for a .NET version the machine lacks, the run stopped with 'You must install'. That shows the check can fail. (failure planted and seen to fire: autobot/CLOSED_LOOP_TASKS.md)
Run it again
Cannot be run again by one command. A one-off check on one machine.
Cost
15,000 tokens, 5 minutes. About 5 minutes and 15K tokens, from the board.
Why this state
Found that it would not, fixed it, and proved the fix both ways.
✓ Passed

Can the language-neutral parts model public code?

Oct 29

The question. Without any adapter, can the loop read a public game's use-case document into a model and say what is open?

The result. 7 use cases and 48 requirements, 44 derived and 4 inferred, all 61 constants claimed once the rows are read whole. The graph is thin: 186 nodes, no code, no tests.

Read from autobot/CLOSED_LOOP_TASKS.md, a number in its written text.

The bar. A budget of 45 minutes and 150K tokens, written first. No pass bar. Not written down first. The board line carrying the budget landed with the result.

How it can fail, how to run it again, what it cost
Ways to fail
  • A document row the parser cannot read stops the model. The check caught one missing a column. (failure planted and seen to fire: autobot/CLOSED_LOOP_TASKS.md)
Run it again
Cannot be run again by one command. One model built by hand from a document; the builder is in the asteroids folder.
Cost
60,000 tokens, 25 minutes. About 25 minutes and 60K tokens, from the board.
Why this state
A model was built from the document. It is thin by design: with no code, green means little.
✓ Passed

Can a fresh agent install the kit from the zip alone?

Oct 29

The question. A fresh agent with only the zip: does it get the kit installed and running, and what trips it?

The result. Installed from the zip alone. 11 papercuts and one real gap were found.

Read from autobot/CLOSED_LOOP_TASKS.md, a number in its written text.

The bar. A budget of 30 minutes and 250K tokens, written first. Not written down first. The board line carrying the budget landed with the result.

How it can fail, how to run it again, what it cost
Ways to fail
  • The agent cannot install, or stops on something the instructions do not say. (fails by missing its bar: autobot/CLOSED_LOOP_TASKS.md)
Run it again
Cannot be run again by one command. One agent run; not rerun by one command.
Cost
131,000 tokens, 10 minutes. About 10 minutes and 131K tokens, from the board.
Why this state
It installed, with 11 papercuts and one real gap, which were listed.
Read more
Manual: install
✓ Passed

Are the links from documents to requirements correct?

Oct 30

The question. When the loop says a manual section, slide or README paragraph describes a requirement, is that so? A wrong link is one whose document does not describe the node it is tied to.

The result. Seeds 101, 102 and 103 gave 20, 20 and 20 correct of 20. The riskiest links, every bare-member link and every goal-title link, were read whole: 51 of 51 correct.

Read from autobot/proofs/C1.md, a number in its written text.

Caution. The judge is the worker who wrote the rules, and 44 of the links are lines that worker authored. Nobody independent has read them. Only precision was measured, not recall.

The bar. At least 19 of 20 links correct on each of 3 fresh seeds, read against the source line. Written down first, in commit e1d2aec9.

How it can fail, how to run it again, what it cost
Ways to fail
  • Fewer than 19 of 20 correct on any seed. (fails by missing its bar: proofs/C1.md)
  • The first reading, before the seeds, found 2 wrong and 1 worthless of 36; each was fixed as a class with a plant before the seeds were drawn. (failure planted and seen to fire: proofs/C1.md)
Run it again
py -X utf8 autobot/doc_nodes.py --selftest
Cost
590,651 tokens, 62 minutes
Why this state
20, 20 and 20 correct against a bar of 19. The sample flatters, so the riskiest classes were also read whole, 51 of 51.
Read more
Manual: the documents are nodes too
✓ Passed

Does attaching documents change anything else?

Oct 30

The question. Adding documents to the graph must not move any other criterion, count or exit code. Did it?

The result. Old tools against new on five models: 0 lines differ other than the new criterion's, and exit codes are equal.

Read from autobot/proofs/C1.md, a number in its written text.

The bar. On the game feature and the kit's own model, no criterion's line other than the new one changes between the old code and the new, and the exit code does not change. Written down first, in commit e1d2aec9.

How it can fail, how to run it again, what it cost
Ways to fail
  • Any line other than the new criterion's differs between old tools and new, or an exit code differs. (fails by missing its bar: proofs/C1.md)
  • Each guard that keeps documents out of fingerprints, distance and the network's view was removed in memory and the figure moved. (failure planted and seen to fire: proofs/C1.md)
Run it again
py -X utf8 autobot/doc_nodes.py --selftest
Cost
Part of the C1 item; its cost is on the first C1 entry.
Why this state
0 lines differ across all five final models, exit codes equal.
✓ Passed

Does every link by name stand on the line it cites?

Oct 30

The question. For every link made by a name found in a document, does the file and line it cites really contain that name?

The result. 62 links on the kit's own graph, 127 on the game feature, 71, 75 and 65 on three outside trials: 0 mismatches.

Read from autobot/proofs/C1.md, a number in its written text.

The bar. 0 mismatches on every graph. Written down first, in commit e1d2aec9.

How it can fail, how to run it again, what it cost
Ways to fail
  • Any link whose cited line does not hold the name. (fails by missing its bar: proofs/C1.md)
Run it again
py -X utf8 autobot/doc_nodes.py --selftest
Cost
Part of the C1 item; its cost is on the first C1 entry.
Why this state
0 mismatches across five graphs.
✓ Passed

Does the documents tool refuse what it should?

Oct 30

The question. Does the tool refuse a fabricated requirement id, a renamed section, a missing file, an empty document set and an unparseable document, each in its own test?

The result. 94 checks in the selftest, each with a plant on a fixture.

Read from autobot/proofs/C1.md, a number in its written text.

The bar. The plan item asked for a plant for each guard. No number was set first. Not written down first. The plan item named plants but fixed no count before the selftest was written.

How it can fail, how to run it again, what it cost
Ways to fail
  • A planted fabricated id, renamed section, lost document, empty set or unparseable file must each be refused with a named reason. (failure planted and seen to fire: autobot/doc_nodes.py --selftest)
Run it again
py -X utf8 autobot/doc_nodes.py --selftest
Cost
Part of the C1 item; its cost is on the first C1 entry.
Why this state
The selftest prints PASS (94 of 94) and each plant fires.
✓ Passed

Do the growth charts refuse a bad log and draw only what the logs hold?

Oct 30

The question. Does the growth page draw every point from a dated log row, break a line where what was counted changed, and stop on a missing or unreadable log rather than guess?

The result. 26 selftest checks, and 11 of 11 deliberate breaks of its rules were caught.

Read from autobot/proofs/D2.md, a number in its written text.

Caution. The page reads live logs, so its numbers move. Fonts, print, a screen reader and colour-blind use were not checked by eye.

The bar. The plan item named a plant for each rule. No number was fixed first. Not written down first. The plan item's done-when asked for charts from every log and gave no numeric bar.

How it can fail, how to run it again, what it cost
Ways to fail
  • Each rule was broken inside a copy of the tool and the selftest had to go red on the check that guards it: lines for any readings, unreadable rows skipped, a join across a change, a missing log, absent read as zero, the gate not called. (failure planted and seen to fire: proofs/D2.md)
Run it again
py -X utf8 autobot/growth.py --selftest
Cost
445,494 tokens, 38 minutes
Why this state
All 11 rule breaks went red or were refused, and the control passed.
Read more
the growth page
✓ Passed

Does every recorded stat have a reader?

Oct 30

The question. The loop records many stats about itself. The audit says some are read by nothing. Is the audit itself counting honestly?

The result. The audit reported 6 stats with no reader; counting only real readers it was 11; now 0. 15 of 15 removals of a piece of the fix make a selftest fail.

Read from autobot/CLOSED_LOOP_TASKS.md, a number in its written text.

The bar. None fixed first. The idea was on the board; the measure was the audit's own count. Not written down first. The idea was on the board, but its measure and the result landed together.

How it can fail, how to run it again, what it cost
Ways to fail
  • Fifteen removals of a piece of the fix each must make a selftest fail. All did. (failure planted and seen to fire: autobot/CLOSED_LOOP_TASKS.md)
Run it again
py -X utf8 autobot/telemetry_audit.py --root .
Cost
22 minutes. 22 minutes from the tick ledger.
Why this state
The audit was under-counting. It said 6 unread; the honest count was 11; after readers were added it is 0.
✓ Passed

Do the plan's switches reach a pass at all?

Oct 30

The question. The run plan switches seven parts on or off. Does any switch actually change what a pass runs?

The result. 0 of 7 plan lines reached a pass by code before. 3 of 7 after. 14 of 14 removals of a piece of the fix fail a selftest.

Read from autobot/CLOSED_LOOP_TASKS.md, a number in its written text.

The bar. The check was written first so the reading before and after is one command. Not written down first. The check and the fix landed in one commit, so the order is not in the history.

How it can fail, how to run it again, what it cost
Ways to fail
  • Fourteen removals of a piece of the fix must each fail a selftest. The first run caught 13: an exit-code check was too loose and was tightened. (failure planted and seen to fire: autobot/CLOSED_LOOP_TASKS.md)
Run it again
py -X utf8 autobot/telemetry_audit.py --root .
Cost
41 minutes. 41 minutes from the tick ledger.
Why this state
Before the fix, 0 of 7 switches reached a pass by code, only through a sentence in a runbook. After, 3 of 7.
✓ Passed

Does a trigger that fires on every pass say anything?

Oct 30

The question. Improvement triggers fire when a stat changes. Some fired on every pass whatever happened. Do they carry information?

The result. Fired triggers 12 to 4. 13 of 13 removals of a piece of the fix fail a selftest.

Read from autobot/CLOSED_LOOP_TASKS.md, a number in its written text.

The bar. None fixed first. Found while doing a different item. Not written down first. Found while doing another item, so no bar existed before the finding.

How it can fail, how to run it again, what it cost
Ways to fail
  • Thirteen removals of a piece of the fix must each fail a selftest, on a copy with a passing control. (failure planted and seen to fire: autobot/CLOSED_LOOP_TASKS.md)
Run it again
py -X utf8 autobot/improvement.py check
Cost
11 minutes. 11 minutes from the tick ledger.
Why this state
12 triggers fired, 8 of which fired on every pass whatever happened. Now 4 fire.
✓ Passed

Is a change in one stat followed through the triggers, the pick, the lists, the growth page, the documents and the graph?

Oct 30

The question. The stats log, the improvement triggers, the pick of next work, the Goals List and Stats List, the growth charts, the documents and the graphs of the tool were separate tools that each passed their own selftest. When one stat worsens, does the change reach each of the others? One worsening of the owner network's score is planted on a scratch copy of the tool tree and followed through ten links.

The result. Before anything was wired 3 of 10 links held. After wiring, 9 of 10 hold on the plain run, in about 50 seconds and 53 MB, with the real tree untouched.

Read from autobot/proofs/E1.md, a number in its written text.

Caution. One stat. The planted rows and the tick are played by the tool, and the recovery row is planted, so link 6 proves the measurement and not that any work moved the stat. Other stats reach fewer links: no goal reads kill_rate, and 45 stats of the pass log have none.

The bar. Written in the tool's docstring before the first run. Every link gives a verdict and the evidence it read; a link nothing calls says NOT WIRED and what is missing, never a pass from a stand-in; the real tree is untouched; one chain run takes under 3 minutes and a copy under 150 MB. The docstring predicted 3 of 10 links would hold before wiring. Written down first, in commit 98e5a6e4.

How it can fail, how to run it again, what it cost
Ways to fail
  • A link reads FAIL or NOT WIRED. The first run found 6 NOT WIRED and 1 FAIL; wiring moved all but link 5, which stays NOT WIRED with its cost on file. (fails by missing its bar: proofs/E1.md)
  • A link passes without a cause: the control run, with nothing worsened, must not report PASS on links 1, 2, 3 or 6. (failure planted and seen to fire: proofs/E1.md)
Run it again
py -X utf8 autobot/harmony.py
Cost
Tokens not recorded here. A chain run takes about 50 seconds of one core at below normal priority; the plants take 7 minutes.
Why this state
The bars were met and the prediction held on all ten links. 9 of 10 links hold after wiring; link 5 (work ran) is not wired: nothing turns the pick into a launched run. The result is a gap found, not a claim that the loop is in harmony.
Read more
Manual: the loop on itself
✓ Passed

Does each link's check turn red when only that link is broken?

Oct 30

The question. A chain that has only ever passed proves nothing. For each wired link, one change to the scratch copy breaks only that link (a registry line, a store's edges, a one-line edit of one tool's copy, the old defect put back). Does that link stop reporting PASS, and does any other link go red that does not read what the broken one writes?

The result. 9 plants (the control and eight breaks) each turned their link off PASS. In the control, links 2, 3 and 6 also went red, which is their declared dependence on link 1.

Read from autobot/proofs/E1.md, a number in its written text.

Caution. The plants are made by the tool that is judged, on a copy. Each is one edit, so a break the plants do not imagine is not tried.

The bar. Written in the tool's docstring before the first run. Every wired link has a plant that turns it off PASS, and no link outside its declared dependence turns red. The control (nothing worsened) reports PASS on none of links 1, 2, 3 and 6. Written down first, in commit 98e5a6e4.

How it can fail, how to run it again, what it cost
Ways to fail
  • A plant leaves its link at PASS; the run says so ('the plant moved nothing') and exits 1. (failure planted and seen to fire: proofs/E1.md)
  • A plant turns red a link outside its declared dependence. The first matrix run found one: with nothing worsened, link 8 went red because it looked for the worsened value; it now looks for the values the run planted. (fails by missing its bar: proofs/E1.md)
Run it again
py -X utf8 autobot/harmony.py --prove
Cost
About 7 minutes for the nine plants, one core at below normal priority. Tokens not recorded here.
Why this state
9 of 9 plants turned their link off PASS and none turned a link red outside its declared dependence. Link 5 has no plant: nothing is wired to break.
✓ Passed

Does a finished tick rewrite the two lists, and does the next tick read what they ask?

Oct 30

The question. The Goals List and Stats List logged what moved only when someone ran the writer by hand. Does `loop_tick finish` now rewrite them, rebuild the growth page and name the documents of its stat, say so when a step is skipped, fail loudly when a step fails, and does `start` print what the lists ask? Does a pillar read the lists' check every cycle?

The result. The commit gate ran green on the commit that added the check to the pillars (11 checks). The tool's selftest has two new checks, FINISH and ASKS.

Read from autobot/proofs/E1.md, a number in its written text.

Caution. The pillar check is not commit-marked, so it runs in the full cycle, not on every commit. The first real finish writes the real lists and the public site's growth page.

The bar. The item's done_when in the plan: gnn_goals --write in the tick's finish, --check in the pillars, a tick reads what the lists ask. No number was committed before the work. Not written down first. The plan item states what must exist, not a number, and nothing was committed before the work. Harmony's link 7 reading PASS on a finished tick is the check that the writer ran.

How it can fail, how to run it again, what it cost
Ways to fail
  • A finish step is called and does nothing: the tool's selftest runs stub tools and counts which ran and which failed. (failure planted and seen to fire: proofs/E1.md)
  • The finish steps are removed one at a time from the copy and the matching harmony link goes off PASS (links 7, 8 and 9). (failure planted and seen to fire: proofs/E1.md)
Run it again
py -X utf8 autobot/loop_tick.py --selftest
Cost
The selftest takes seconds. Tokens not recorded here.
Why this state
The finish steps run in order, a failing one is counted and does not stop the rest, a missing tool is skipped with its reason, and the pillar's commit gate ran green with the new check.
✓ Passed

Ranking the other requirements tools by a rubric fixed first

Oct 30

The question. Can a ranking of agent requirements tools, our own kit included, be made so that every score has a page behind it and the rubric cannot be bent to flatter the kit?

The result. 32 tools and 17 criteria pass the check. The kit ranked first with equal weights and 10th of 29 under the other weights; two predictions about where it would fall were wrong.

Read from autobot/proofs/L1.md, a number in its written text.

Caution. The kit was ranked by its own makers, not blind. Its being first is a reading of a rubric they wrote and committed first, not an outside verdict.

The bar. The rubric, with 17 criteria, 3 weight profiles and predictions, committed before any tool was scored. Two amendments were committed before a single score. Written down first, in commit d4864a55.

How it can fail, how to run it again, what it cost
Ways to fail
  • The tool refuses a score with no evidence, and each refusal is planted in its selftest and seen to fire. (failure planted and seen to fire: proofs/L1.md)
  • A prediction the rubric made can be wrong. Two of six were. (fails by missing its bar: proofs/L1.md)
Run it again
py -X utf8 autobot/landscape/landscape.py --check
Cost
427,469 tokens, 27 minutes
Why this state
The check passes and the selftest has 43 checks. Two of the six predictions were wrong, which is what committing them first allows.
Read more
Manual: where it stands among the other tools
✓ Passed

A small tested class, against a public specification

Oct 30

The question. On a version-number class from a game engine's tools, with only two tests, written against a public specification anyone can read, what do the tests leave open?

The result. 246 breaks made, 236 live, 206 caught: 87.3 percent. Coverage said 90 percent of lines. 14 of the 30 that got past sit on lines coverage marks as covered.

Read from autobot/t7_results.md, a number in its written text.

Caution. Engine source is licensed, so only counts and method names are published.

The bar. A budget of 45 minutes and 150K tokens, written first. No pass bar for the rate. Not written down first. The budget was on the board; the result landed in a later commit and no bar for the rate was fixed first.

How it can fail, how to run it again, what it cost
Ways to fail
  • The run stops on code it cannot read or compile. It did, six times across the three trials (BP-4 to BP-8 and BP-10), and each cause was fixed with a plant. (failure planted and seen to fire: autobot/CLOSED_LOOP_TASKS.md)
Run it again
Cannot be run again by one command. Run on a licensed copy of the engine that is not in this repository.
Cost
Budget 45 minutes and 150K tokens; actual not recorded separately.
Why this state
The run completed and reproduced real defects by running them.
Read more
the loop on code you can read
✓ Passed

A large, well-tested class

Oct 30

The question. On a JSON class with 43 tests, does the loop add anything over a class that is already well tested?

The result. 582 breaks made, 429 live, 299 caught: 69.7 percent. Coverage said 77 percent of lines. 130 got past every test.

Read from autobot/t7_results.md, a number in its written text.

The bar. A budget of 45 minutes and 150K tokens, written first. No pass bar for the rate. Not written down first. As the first trial, the budget was fixed first and the rate had no bar.

How it can fail, how to run it again, what it cost
Ways to fail
  • Breaks that no test catches that coverage already marks covered. Many did. (fails by missing its bar: autobot/t7_results.md)
Run it again
Cannot be run again by one command. Run on a licensed copy of the engine that is not in this repository.
Cost
Budget 45 minutes and 150K tokens; actual not recorded separately.
Why this state
The run completed. It found 130 breaks that got past all 43 tests, 67 of them on lines coverage calls covered.
Read more
the loop on code you can read
✓ Passed

The scoring rules of a shipped open-source game

Oct 30

The question. On a public game's scoring rules (MIT licence), which anyone can open, what do its scoring tests leave open?

The result. 563 breaks made, 536 live, 286 caught: 53.4 percent of what three test files hold. Coverage said 72 percent of lines. 145 of the 250 that got past sit on lines coverage calls covered.

Read from autobot/t7_results.md, a number in its written text.

Caution. Read 53.4 percent as low for the game as a whole. Seven surviving breaks were checked by hand against the wider suite.

The bar. A budget of 60 minutes and 200K tokens, written first. No pass bar for the rate. Not written down first. As the first trial, the budget was fixed first and the rate had no bar.

How it can fail, how to run it again, what it cost
Ways to fail
  • A rate read on three test files only, which understates a suite of many more. (fails by missing its bar: autobot/t7_results.md)
Run it again
Cannot be run again by one command. Needs a clone of the game's public repository and a restore of its packages; the requirements model is in autobot/t7/t7c/model.
Cost
Budget 60 minutes and 200K tokens; actual not recorded separately.
Why this state
The run completed on the three test files used. It is not a rate for the whole suite.
Read more
the same run on a public game the loop on code you can read
✓ Passed

This benchmark's own check

Oct 30

The question. Does the benchmark refuse a proof with no entry, a command naming a missing file, a result that disagrees with its record, and a test with no way to fail?

The result. The record re-reads every number it names from its file. The selftest plants each refusal and each guard on held work, beside a clean control.

Read from autobot/proofs/V1.md, a number in its written text.

Caution. The record was backfilled by the worker who built the check. Nobody else has read it.

The bar. The plan item's done-when, committed before the work. Written down first, in commit 417044f4.

How it can fail, how to run it again, what it cost
Ways to fail
  • Each refusal is planted in a fixture and must fire, beside a clean control that must pass. (failure planted and seen to fire: autobot/benchmark.py --selftest)
Run it again
py -X utf8 autobot/benchmark.py --selftest
Cost
This item's cost is on the plan's cost log once it is closed.
Why this state
The selftest passes with every refusal planted and fired, and the check passes on the real record.
✓ Passed

The last slide says what the tool is for, and what is not measured

Oct 30

The question. Does the deck's last slide state the tool's main intent as a sentence, the hard-requirements question as a question, and plainly what is measured and what is not, with no claim about intelligence?

The result. The slide is the last public slide, 28 of 28, and the build finds nothing private.

Read from autobot/proofs/X4.md, a number in its written text.

Caution. A document check, not a measurement. What it says is measured is from t7_results.md; what it says is not measured is an end-to-end test whose results will be published with the benchmark.

The bar. The plan item's done_when, written before the slide: the intent, the question as a question, more advanced tools for making video games as the direction, and what is measured and what is not made plain. Written down first, in commit 0f5aa206.

How it can fail, how to run it again, what it cost
Ways to fail
  • A claim on the slide with no record behind it, or a number the proof cannot trace to a file, fails it. The proof traces each sentence. (never exercised: proofs/X4.md)
  • The public build refuses a name it must not carry. It was run on this slide and found none. (never exercised: closed_loop_kit/docs/public_build.py)
Run it again
py -X utf8 closed_loop_kit/docs/public_build.py --site <a scratch folder> --images closed_loop_kit/docs/deck/pictures
Cost
Not recorded.
Why this state
The slide holds each of the four parts, was looked at, and builds last in the public deck with the gate clean. No person has read it.
Read more
Manual: the two targets
✓ Passed

The kit's own requirements name the tests that catch their breaks

Oct 31

The question. No requirement of the kit's own model named a test as its check, so the kill rule had no votes on that graph and L16 listed every requirement. If the names are read from the verdicts of the last pass, never written, how many requirements have a real check, how many have none, and what does L16 read?

The result. 12 of 21 requirements have a real check, 1 is caught only by a crashing test, 3 have code no test catches, 4 have no code to break and 1 is not in the record. Kill votes went from 0 to 159 of 397 cells.

Read from autobot/proofs/A8.md, a number in its written text.

Caution. Not re-run on the current code, and the counting variant of the network was not re-measured against the new edges.

The bar. Each requirement names the tests that actually caught breaks of its code, copied from the record. A requirement none catches is left named as such. The kill rule has votes on the rebuilt graph. L16 is reported as it reads, whether or not it falls. Not written down first. The bar is the item's own words. It was not committed before the first run: the item was done in one pass, so no bar commit exists.

How it can fail, how to run it again, what it cost
Ways to fail
  • With the catching tests removed from the verdicts no requirement may name a check; a name is never written by hand. (failure planted and seen to fire: autobot/record_checks.py)
  • A check named by a label with a dot in it is one test and meets the break it caught; with the old cut at the last dot it does not. (failure planted and seen to fire: autobot/usecase_kg.py)
  • L16 does not fall. The names are fitted to the record, so what it can still find is the break caught only by a crash and the requirement nothing catches. It rose from 13 listed to 12 failing and 1 listed. (fails by missing its bar: proofs/A8.md)
Run it again
py -X utf8 autobot/record_checks.py autobot/selfmodel/model --check
Cost
Not recorded; a few minutes of CPU on the record, no pass.
Why this state
The names are read and the kill rule has votes (0 to 159 of 397 cells). L16 did not fall: it now reads 12 failing, which is a finding about the tests and not a pass. Three requirements have code that no test catches.
✓ Passed

A break with no verdict is drawn apart from one that got past every test

Oct 31

The question. The fly-through painted every break without a killed-by edge red, so breaks nothing had run looked like survivors. Drawn from the verdict record, are caught, got past and no verdict three states, and does the legend name only what the graph holds?

The result. On the shops graph the page read 55 breaks that got past every test and 75 with no verdict; all 130 were red before.

Read from autobot/proofs/A8.md, a number in its written text.

Caution. Contrast on a phone was not checked, and the longer legend crowds the tour card at a very small preview size.

The bar. Three states from the record on both pages, with a plant: the old rule must read more red than the new one on the same store. A legend line only for what the graph holds. Not written down first. Written in the item's own words and not committed before the first run.

How it can fail, how to run it again, what it cost
Ways to fail
  • On the test store two breaks have no killed-by edge and one got past: the old rule paints 2 red, the page must paint 1 red and 1 violet. (failure planted and seen to fire: autobot/graph_flythrough.py)
  • With the chain deleted the Reasoning line goes, and a state of a break shows only while the graph holds one. A store with no verdict table paints nothing red. (failure planted and seen to fire: autobot/graph_flythrough.py)
  • On the big-store test 10 breaks have no killed-by edge, 6 got past and 4 have no verdict: red must read 6. (failure planted and seen to fire: autobot/fly_manual.py)
Run it again
py -X utf8 autobot/graph_flythrough.py --selftest
Cost
Not recorded.
Why this state
The three states and the legend are held by selftests with plants, and a page built from the shops graph was read in a browser. The violet points were confirmed from the page data, not told apart by eye at the whole-graph stop.
✓ Passed

Is every number on a public page tied to a record that can say it moved?

Oct 31

The question. Each number on the deck, the manual and the README is a claim. Does a registry place every one of them (read from a record, written by a tool, a name and not a figure, or absent with its reason), re-read the records, and refuse a build when a number no longer matches its record, a record is gone, a new number is unregistered, or a claim reads unpublished work?

The result. 1,318 numbers on the public pages. 483 read from a record by 215 claims, 171 written by a tool and found fresh, 176 names and not figures, 488 marked absent (451 with no record read: 227 untied, 192 history, 32 unrecorded), 0 unaccounted, 0 stale.

Read from autobot/proofs/C2.md, a number in its written text.

Caution. Nobody independent read the claims. A figure claim is the weaker kind (70 of the 215). Number words are counted apart and not compared.

The bar. None fixed first. The refusals are the test: each planted and seen to fire, and the gate green on the whole corpus after the registry holds it. Not written down first. The item asked for a registry and a gate, not a rate, so no pass mark was written before the results; the share of numbers with no record is the finding and is counted, not a bar.

How it can fail, how to run it again, what it cost
Ways to fail
  • A record that moved, a record that is gone, a reworded sentence, a new unregistered figure, a claim on unpublished work, a malformed claim, a tool's page that would be written differently: each must refuse. A control with none of them must not. (failure planted and seen to fire: proofs/C2.md)
  • A copy of a trial's stats file changed (206 caught to 200): the public build refuses, names the slide and writes nothing. The same copy deleted: it refuses on 47 numbers. (failure planted and seen to fire: proofs/C2.md)
  • A figure claimed by its value and the words of its block can attach to another fact with the same figure and words. The check proves the digits equal the record, not that the sentence means what the record means. (never exercised)
Run it again
py -X utf8 autobot/claims.py selftest
Cost
A long session; tokens not recorded here. The gate adds about 18 seconds to a public build.
Why this state
36 plants in the tool's selftest and 5 in the public build's all fire beside clean controls, and the public build runs green on the corpus. It found 12 numbers stale or unsupported, all fixed. 451 of 1,318 numbers have no record read yet; that is a count, not a pass.
Read more
Manual: the tool on itself
✓ Passed

Does every proof on record still pass when run again, together, and does the runner refuse the ways to pass without proving?

Oct 31

The question. Run again on a copy of the tools, several at once, every command the record lists and every tool's selftest: how many pass, fail, give no answer, or are not run and why? Does the runner refuse a proof that hangs, one that exits 0 having checked nothing, and one that exits 0 while its own lines or the numbers its entry states say otherwise?

The result. 114 items on record: 71 passed, 1 failed, 1 gave no answer, 41 were not run (18 the record says no one command reruns, 12 skipped with a measured or read reason, 10 tools with no selftest, 1 in a folder kept untouched). 1,840 seconds of waiting for 7,636 seconds of work, 4.1 times faster than one after another, 8 at once at below-normal priority. The fast tier: 61 passed, 0 failed, 54 not run, 125 seconds.

Read from autobot/prove_all_runs.jsonl, value of counts.passes where label=e2-full-2; autobot/prove_all_runs.jsonl, value of items where label=e2-full-2; autobot/prove_all_runs.jsonl, value of counts.fails where label=e2-full-2.

Caution. Run on a machine other work was using at full load, once or twice per tier. A copy of the tools is not the tree, so what a command reads outside the copy fails there and is added by name or skipped with a reason. A pass says the command exited as intended, printed a check, and that the numbers its entry states still match the files they are read from; in most entries the command does not rewrite those files, so it says the record still matches files the command did not make.

The bar. Committed in the tool's docstring before the first full run. Nothing is dropped, and every item not run carries its reason. The three false passes each have a verdict of their own, planted beside a control. The fast tier finishes in under 5 minutes. A full run takes less than its longest single item plus 3 minutes. Afterwards the record's own check refuses nothing. The real tree is left alone. Written down first, in commit de60db5b.

How it can fail, how to run it again, what it cost
Ways to fail
  • A proof that hangs is waited on, or read as a pass. The plant is a program that sleeps for two minutes: it is killed at its limit and named, and the run takes seconds. (failure planted and seen to fire: autobot/prove.py)
  • A tool that exits 0 having checked nothing passes. Planted, and found for real in two tools' selftests, one of which printed no check in the form the others use. (failure planted and seen to fire: autobot/prove.py)
  • An entry whose command exits 0 but leaves a number different from the one the entry states passes. Planted in a fixture record with a command that rewrites the file. (failure planted and seen to fire: autobot/prove_all.py)
  • The run changes the real tree. The first full run did: a tool's selftest ran git commands that reached the real repository through variables the copy had handed it, and rewrote its index and one setting. Mended; the plant runs git add and commit in the copy and finds the original unchanged. (failure planted and seen to fire: autobot/prove_all.py)
  • Each guard of the runner broken one at a time in a copy; the selftest must go red on each. 22 of 22 did, after two plants were added for the two that got through. (failure planted and seen to fire: proofs/E2.md)
  • A copy of the tools is not the tree: a command that reads outside the copy fails there, naming the file. 8 of the 14 first findings were that, and each is now added to the copy by name or skipped with a reason. (fails by missing its bar: proofs/E2.md)
Run it again
py -X utf8 autobot/prove_all.py
Cost
About 31 minutes of wall time for a full pass on a busy machine, eight proofs at once; the longest single proof took 30 of them.
Why this state
Every command and selftest on record is run again in one command, each verdict says what the item proves and how it could fail, and the three false passes are refused. The first full run missed one bar: it rewrote the real repository's index and one setting through a copy that was not isolated; that is mended and planted. The second run: 71 of 114 passed, 1 failed (the record's own check, run before the record was brought current), 1 gave no answer, 41 were not run, each with its reason.
✓ Passed

Does the fly-through show the stats, the learned parts and the graphs working together from real records, and refuse to say a link holds from a stale one?

Oct 31

The question. The fly-through is the manual: a scene with a tree on the left and a page at the bottom. Does it now hold four worlds read from the stores the tool wrote (a game, the tool's own code, the learned parts, the improvement model), each saying how many nodes are drawn of how many; the chain's ten links as a path of stops, each saying PASS, FAIL or NOT WIRED from a record of a real run; and growth, the plan, the benchmark and the proof as pages? And does it refuse a link's state from a record made on tools that have since changed, a held entry reaching a page, and an empty world drawn as nothing?

The result. The selftest of fly_manual.py has 67 checks, up from 33, and passes. 29 of 29 breaks of its new guards turn it red. The real run of the chain recorded 9 of 10 links holding and 9 of 9 plants turning their link off PASS.

Read from autobot/proofs/F1.md, a number in its written text.

Caution. The pictures were drawn by a software renderer (SwiftShader), in headless Chrome, at the sizes of a desktop and a phone; no real graphics card or real phone has run the page. The record of the chain was made once, on 2026-10-31, and the build refuses once any of its nine tools changes. The benchmark and proof groups ship only with --with-benchmark, because the benchmark page is held.

The bar. fly_manual tree gains plan, growth, proof and harmony; the recursion levels are visible as worlds you can fly between; each with its page. Written down first, in commit 161d20fc.

How it can fail, how to run it again, what it cost
Ways to fail
  • A link's state is read from a record made on tools that have since changed. Planted: a tool edited after the record, and one deleted; each is refused, and the control passes. A record that does not hold links 1 to 10 once each, names a state the page does not know, has a link with no evidence, or quotes a path on this machine is refused the same way. (failure planted and seen to fire: autobot/fly_manual.py)
  • A held entry reaches a page: its id, its title, its tool or a file only it reads. Planted with a fixture record that holds one entry; the public form passes and the leaked form is refused, and the plan, the proof and the benchmark group each leave a held item out. (failure planted and seen to fire: autobot/fly_manual.py)
  • A world with nothing in it is drawn as nothing. Planted: a store with no node, a store not on disk, an improvement model with nothing in it. Each keeps its tree row, its page, its numbers and a mark, and says why it is empty. (failure planted and seen to fire: autobot/fly_manual.py)
  • A big world is thinned and a red break is dropped, or the numbers on screen do not add up. Planted: every red break and all structure stay, the limit holds, the same store thins the same way, and a break with a verdict the page does not know stops the build. (failure planted and seen to fire: autobot/fly_manual.py)
  • Each guard the page gained made to do nothing, one at a time, in a copy: the selftest must go red. 29 of 29 did. The first run let 3 through (the proof's held-item filter, a relation the page was not told of, a world with no node dropped from the order); each got a plant, one needing its fixture made stricter, and the second run caught all 29. (failure planted and seen to fire: proofs/F1.md)
  • The chain is changed and its record is not run again, so the page cannot be built: that is the refusal working, and it costs about 14 minutes to clear (harmony.py --prove --record). (fails by missing its bar: proofs/F1.md)
Run it again
py -X utf8 autobot/fly_manual.py --selftest
Cost
About 14 minutes for the real chain run with its plants (864 seconds, below-normal priority); about 25 minutes for each browser run, two at once.
Why this state
The committed done-when is met: the tree has the levels as worlds and the harmony, growth, plan, benchmark and proof groups, each with its page. Every planted refusal fires, and 29 of 29 breaks of the new guards turn the selftest red. Not met: nothing has drawn it on a real graphics card or a real phone.
Read more
Manual: the loop on its own improvement Manual: flying the graph
✓ Passed

Does every new group, world and link work for a visitor in a real browser, with WebGPU and with WebGL 2, on a desktop, on a phone and with no graphics?

Oct 31

The question. A browser is driven the way a visitor would: each new tree group is clicked and opens its page, each world flies the camera and says how many nodes are drawn of how many, the goals and the stats are one click from each world and back, each of the ten stops is clicked and must say PASS, FAIL or NOT WIRED as the record does, the page loads no script from anywhere, and the same is done at phone width and with no graphics. Does everything the page promises happen, in both ways of drawing?

The result. The visitor made 105 checks with WebGPU and 105 with WebGL 2, took 34 and 34 pictures, and failed 0 and 0.

Read from autobot/tick_runs/f1_visit.json, key webgpu.passed; autobot/tick_runs/f1_visit.json, key webgpu.checks; autobot/tick_runs/f1_visit.json, key webgl.passed.

Caution. A software renderer in headless Chrome, not a graphics card or a phone. The benchmark page the Benchmark group links to cannot be built today (the record's own guard refuses it because entries added by another item name held items), so that link answers 404 on the scratch site; the group is held for that reason as well.

The bar. fly_manual tree gains plan, growth, proof and harmony; the recursion levels are visible as worlds you can fly between; each with its page. Written down first, in commit 161d20fc.

How it can fail, how to run it again, what it cost
Ways to fail
  • A new group or world does not open its page, or does not move the camera, or does not say how many nodes are drawn of how many. Each is a check of the visitor. (fails by missing its bar: proofs/F1.md)
  • A link page says a state other than the record's. Each of the ten stops is compared with the record file. (fails by missing its bar: proofs/F1.md)
  • Running and looking found nine faults in the first runs, before they shipped: the first world's layer labels drawn over another world, the same sentence twice under a title, an overview of the four worlds that was a white blob with a label out of frame, two links standing on one node, a world's label cut at the edge of a phone, a count that included the chain's ten stops, a pinch on a phone that zoomed the page and not the scene (the visitor's own aim), a name compared in the wrong case (the visitor's own), and a local server that refused connections under two browsers. Each was mended and the page was visited again. (failure planted and seen to fire: proofs/F1.md)
  • The page cannot be drawn: with WebGPU removed it falls back to WebGL 2, and with no canvas at all the tree and the pages still work. Both are run. (fails by missing its bar: proofs/F1.md)
Run it again
py -X utf8 autobot/fly_manual.py --visit <the built page> --renderer webgpu --out <folder>
Cost
About 25 minutes per renderer, two at once, below-normal priority.
Why this state
Both runs finished and every check passed: 105 of 105 with WebGPU and 105 of 105 with WebGL 2. The pictures were read.
Read more
Manual: the loop on its own improvement
✓ Passed

Does the skill reach five other agents in the file each reads, from one source, with a stale or invalid copy refused?

Oct 31

The question. The skill is a Claude Code skill. Can its front door, run and prove be written in the layout of Codex and OpenCode, Cursor, Gemini CLI, GitHub Copilot and Windsurf from skill/SKILL.md alone, each valid by that tool's own documentation and in the place it names, and can an install write one of them into a project without touching anything else?

The result. 7 agent names wrote their one file into a fresh project through the installed command, and nothing else; a user's AGENTS.md kept its own text byte for byte; the generator's selftest has 24 checks, 18 of them plants, all refused; a changed SKILL.md changed all five files on the next build.

Read from the pack_check log in proofs/L3.md.

Caution. The documentation was read through a fetch that summarises the page, not as raw text, and one page (Windsurf) has moved to another site since the item was written. No agent was run. Sizes under 6,000 characters are this item's own bar, not a limit any page states.

The bar. Committed in proofs/L3_prep.md before any layout existed. Five distinct files; each valid by its tool's documented format, under 6,000 characters and under the limits the pages state; each naming the front door, grow, ask, run and prove and how to get the command; all cut from SKILL.md and carrying its hash. A planted missing command, missing file, stale hash, bad front matter and hand edit are each refused, a changed SKILL.md changes all five files when the build runs again, and install --agent writes exactly one file, keeps a user's own text and refuses a file that is not ours. Written down first, in commit e03cd809.

How it can fail, how to run it again, what it cost
Ways to fail
  • A layout drops one of grow, ask, run or prove, or a layout file is missing. Each planted and refused. (failure planted and seen to fire: closed_loop_kit/agent_layouts.py)
  • A layout is cut from an older SKILL.md, or edited by hand with its stamp kept. Both planted: the kit check and install --agent refuse. (failure planted and seen to fire: closed_loop_kit/agent_layouts.py)
  • Front matter the tool does not document: a Cursor field it has no name for, globs set on an Apply Intelligently rule, front matter that is not YAML, a Windsurf trigger outside the four, front matter on a plain-Markdown file. Each planted and refused. (failure planted and seen to fire: closed_loop_kit/agent_layouts.py)
  • install --agent overwrites a user's AGENTS.md, a Cursor file that is not ours, or writes outside the project. Planted by giving it a user's own AGENTS.md with CRLF endings and a foreign Cursor file, and by comparing the home folder before and after. (failure planted and seen to fire: closed_loop_kit/pack_check.py)
  • An agent does not follow the file, or reads a path the documentation no longer names (Windsurf's page already calls the folder shipped here legacy). Nothing here runs an agent, so this cannot fail in the test. (never exercised)
Run it again
py -X utf8 closed_loop_kit/agent_layouts.py --selftest
Cost
Six documentation pages fetched. The builder's own tokens are not metered per item. The package run takes about three minutes of one machine.
Why this state
Seven agent names write five files, each valid and placed as the documentation read on 2026-10-31 names. That is all it shows: no agent other than Claude Code was run on these files.
Read more
Manual: where it stands among the other tools
✓ Passed

Drafting a model from a PRD, a ticket export, a transcript and a public proposal, every requirement citing its sentence

Oct 31

The question. Can a program number the sentences of a PRD, a ticket export, a transcript and a public requirements document, and refuse a draft that does not ground each requirement in a sentence or in a reason, while a second reader reports every rule sentence nothing cites?

The result. The public proposal (KEP-2185 of kubernetes/enhancements, Apache-2.0) numbered 153 sentences: 21 cited by 17 requirements, 132 set aside with a reason, none uncited. The PRD, ticket export and transcript each cited every sentence they did not set aside.

Read from the counts in grow_trial/results.json.

Caution. The drafts of the four shipped inputs are the builder's. 132 of the proposal's 153 sentences were set aside by the drafter with reasons (checklists, questionnaires, alternatives not chosen); 7 rule-looking ones among them go back to the producer as questions. The proposal's text is fetched and not shipped.

The bar. The plan item's done-when, committed in 88eb6aef: one fixture of each kind plus one public document, each requirement citing the sentence and line it stands on, the source_rules check reporting every rule sentence nothing cites, a requirement citing a sentence that is not there refused, counts on file. Written down first, in commit 88eb6aef.

How it can fail, how to run it again, what it cost
Ways to fail
  • A requirement citing a sentence that is not in the text is refused, one citing none is refused, one quoting words the sentence lacks is refused; each is planted and fires. (failure planted and seen to fire: proofs/L5.md)
  • A rule sentence nothing cites is reported by the sentence space (U7) and by source_rules by line; planted. (failure planted and seen to fire: proofs/L5.md)
  • Real markdown broke the first reading (283 half-line sentences on a 400-line proposal, then 3 false uncited hits); both fixed with a case each. Another real document may break it a third way. (never exercised)
Run it again
py -X utf8 autobot/grow.py --selftest
Cost
No agent tokens for the drafts of the shipped inputs; the proposal's draft is a script (grow_trial/runs/kep_make_draft.py). Counted in the cost of the item that held the proof.
Why this state
All four inputs and the public proposal grow a model the 19 rules accept apart from holes that are questions; every requirement carries its sentence or its reason. The reader is an agent; the tool cannot check that a requirement means what its sentence means.
Read more
Manual: where it stands among the other tools
✓ Passed

Does the kit run from one command with no clone, from a package, on a clean home folder?

Oct 31

The question. Can a stranger with Python and no clone run the shipped example through the whole loop with one uvx or pipx command, install the skill, and write another agent's file, with HOME, TEMP and the package cache all empty?

The result. uvx --from the wheel closed-loop-kit try ended HOLDS in 8.5 seconds from an empty uv cache (13.0 from the built source folder), on a machine other work was using. The package run made 25 checks and none failed; the wheel is 435 KB in 79 files.

Read from the pack_check log in proofs/L3.md.

Caution. Times are one run on a Windows machine that other work was using. The package was built by uv build from the built kit; nothing was published and the name closed-loop-kit was free on the index when asked. The full install here had the .NET SDK taken off the PATH, so the C# example was not run.

The bar. Committed in this proof before any package existed. uvx --from the built wheel closed-loop-kit try exits 0 and prints HOLDS from a cold cache with every home and temp folder inside a scratch folder; the same from the built source folder and from pipx run if pipx can be had; install --quick ends proved; the wheel holds the tools, the skill, the examples and the layouts; the privacy gate ran on exactly what the wheel was built from; nothing is published. Written down first, in commit e03cd809.

How it can fail, how to run it again, what it cost
Ways to fail
  • The wheel lacks a file the tools find by location. Planted by building a wheel without the Python example: try exits 1 and does not print HOLDS. (failure planted and seen to fire: closed_loop_kit/pack_check.py)
  • The route a user will take, uvx from the git address, builds from what the repository holds. It has not been run, because the repository does not carry the files until the main session pushes. The built source folder and the sdist stand in for it. (never exercised)
  • A machine with a warm package cache, a clone nearby or another Python would pass what a stranger fails. Each stranger gets a fresh uv cache and an empty working folder; a Python 3.10 interpreter was run once. A slow disk, no network and other systems were not. (never exercised)
Run it again
py -X utf8 closed_loop_kit/pack_check.py --pipx --json autobot/proofs/L3_pack_check.json
Cost
About three minutes of one machine per run, plus the package index for PyYAML and, with --pipx, for pipx. The builder's own tokens are not metered per item.
Why this state
From a local path install of the built wheel, the sdist and the built source folder, one command ran the example to HOLDS on a cold cache with the home inside a scratch folder, through uvx, through uvx on Python 3.10 and through pipx; the full install on Python alone ended READY. The git address itself is not tested.
Read more
Manual: where it stands among the other tools
✓ Passed

Does a stranger get a working, proved install from the README alone, and what does the first run cost?

Oct 31

The question. On a clean home folder, following only the README, does the public kit install, prove itself on its own examples and run the front door with every step giving what the README says and no file of this repository opened; and what does a first run cost in minutes, disk, steps and tokens?

The result. At the same full load, the full install on a machine with PyTorch and .NET and a cold package cache took 877.5 seconds on the shipped kit and 446.0 on the kit as it now stands (505.0 and 291.0 in earlier fixed runs, at full and at lighter load). On Python alone 149.0 seconds and 45.7. The quick install 110.7 seconds and 40.5. Steps that did not give what the README says: 1 on the shipped kit's full install, 0 on the fixed one. Files opened under the repository: 0.

Read from autobot/first_run.jsonl, value of install_seconds where label=final-full-machine; autobot/first_run.jsonl, value of install_seconds where label=base2-full-machine; autobot/first_run.jsonl, value of install_seconds where label=fix-full-machine.

Caution. Times are loaded times and the load differed between runs. The baseline rows were made before the load was recorded. One Windows machine, one Python for the launcher, no other operating system, no network failure.

The bar. Committed in the tool's docstring before any fixed run, the baseline having been measured first. Every README step gives its promised exit and text for the four strangers (quick and full, a machine with PyTorch and .NET and one with Python only). No Python process of the run opens a file under the repository, another run folder or the real home. The full install on the machine with PyTorch and .NET takes at most 60 percent of the baseline's time and the quick install at most 120 percent. Every thing that tripped a new user is written down with whether it was fixed. Written down first, in commit de60db5b.

How it can fail, how to run it again, what it cost
Ways to fail
  • A README step gives another exit or text. It did: the shipped kit's full install ended INSTALLED WITH FAILURES for all three clean full installs, on a selftest that fails on every Windows home folder. (fails by missing its bar: proofs/V2.md)
  • The clean folder lets the real home help the stranger (a warm package cache, a PATH holding dotnet, a temp folder elsewhere): the tool's selftest asks a child where the home, the temp folder and the package cache are, and checks the minimal stranger's python and PATH. (failure planted and seen to fire: autobot/first_run.py)
  • A kit that reads the repository would pass on a machine that has it: the audit logs such an open, and refuses itself if a file opened on purpose is not logged. (failure planted and seen to fire: autobot/first_run.py)
  • One run per scenario and label on a machine other work was using, at a load that changed between runs and was not recorded for the baseline. A gain the size of the load's swings cannot be told apart from the rows. (never exercised)
Run it again
py -X utf8 autobot/first_run.py --selftest
Cost
114,373 tokens. Tokens are the one fresh agent that followed the README to the first question. The runs themselves are compute: a full install of the shipped kit takes 13 to 15 minutes of one machine at below-normal priority.
Why this state
After the fixes all four strangers get every README step as promised, the full installs end READY and no file of the repository was opened; the shipped kit's full installs did not. How much faster is real in direction and not exact in size, because the machine's load changed between runs.
✓ Passed

Does the dictation panel turn speech into text and a reply beside the slide on both pages, and keep the key in the tab?

Oct 31

The question. With a key typed into the panel, does pressing Mic turn speech into text, send it to Claude and show the reply beside the slide, on the public deck and on the fly-through, through pauses and from any slide; does every failure say so in words with no substitute reply; and does the key stay in the tab?

The result. 195 checks passed with none failed, on both pages, with a synthesized voice for the microphone and a local stand-in for the API; 13 of 13 planted leaks and defects were found; the key was found in none of 13 browser places, the network, the console, the stub's log, the browser's profile or any built file.

Read from autobot/proofs/X5.md, a number in its written text.

Caution. A stub is our reading of the API, though the entry below measures what can be measured of the real one. A person's voice on a real microphone, other browsers, a real phone and a hostile browser extension were not tried.

The bar. The plan item's done-when and six predictions, written into the proof tool's docstring and committed before its first run; the take's bar and three more predictions written into the same docstring on 10-31 before those checks first ran: every check passes on both pages, no check waived; the dummy key is found nowhere it must not be, and found in each place it is planted; each named failure is said in words. Written down first, in commit e8a247df.

How it can fail, how to run it again, what it cost
Ways to fail
  • Planted leaks (the dummy key put in storage, the address, the page title, the console, a file, the address of a call) and the old code of each defect restored in memory (the Mic off at the first pause, no quiet timer, the button in the header, the slide not held, the box cleared on a reply, Close leaving the Mic on, the old resize) must each be caught, and a clean copy must stay quiet. (failure planted and seen to fire: autobot/dictation_proof.py)
  • The gate's selftest puts 16 unsafe things into a script or a page (a write to storage, a console line, a second address, a script with a source, a wider policy, a second copy, a key-shaped string) and each must be refused. (failure planted and seen to fire: closed_loop_kit/docs/dictation_panel.py)
  • A script with a source, a replaced fetch, a script added after a key is held, a page over plain http, silence, a browser with no speech, every word the recogniser can end on, a real network failure and ten API failures each make the panel say so in words and ask Claude nothing. (failure planted and seen to fire: proofs/X5.md)
  • The first 135 checks passed while five real bugs stood, because every test spoke six words, was answered at once and started at the top of the page. A use no test walks can still fail. (never exercised: proofs/X5.md)
Run it again
py -X utf8 autobot/dictation_proof.py --site <a built site folder>
Cost
Tokens not recorded here. The browser run takes about 15 minutes of one headless Chrome at below normal priority.
Why this state
All 195 checks passed on the last run, on the committed script. Five bugs the first run could not see were found on 10-31, fixed and covered, each with a plant. No call with a real key has been made: that is the owner's.
✓ Passed

Do the real API and the real speech service let the dictation panel through, with no key at all?

Oct 31

The question. A browser's route to the API is decided before any key is looked at. With no key anywhere, does the real API's preflight accept the panel's exact headers from the public site's origin, is its refusal readable from a browser, is the panel's version the newest published and each of its models named, and does the panel's own request get out of the page past its policy tag?

The result. 27 checks passed with none failed, on the real API, the real speech service and both published pages, with no key anywhere: the key header went out empty.

Read from autobot/proofs/X5.md, a number in its written text.

Caution. Nothing here has seen a reply from Claude through the API. The API checks the key before the version, the model and the body, so those are read from the public documentation, not tested.

The bar. Written into the tool's docstring before its first run: every check passes on both pages, a check that cannot run is a failure, and three controls that must be refused are refused: no opt-in header at the API, no opt-in header in the browser, a host the policy does not name. Written down first, in commit 70945741.

How it can fail, how to run it again, what it cost
Ways to fail
  • The same preflight and request without the opt-in header, and the same request to a host the policy does not name, must each be refused. All were. (failure planted and seen to fire: autobot/dictation_live.py)
  • The same tool on the published page before the fix failed on the script not being the current one, on the Mic stopping at the first pause and on the no-key wording. (fails by missing its bar: proofs/X5.md)
Run it again
py -X utf8 autobot/dictation_live.py --base https://rj45thompson.github.io/anchor-demo/harness
Cost
No tokens. About 3 minutes of one headless Chrome at below normal priority.
Why this state
All 27 checks passed on the published pages. It cannot say whether a real key is accepted or whether the model ids and output_config pass validation, because the API checks the key first.

How we know a test can fail

A check that has only ever passed proves nothing, because it might be unable to fail. So each entry lists the ways it could fail, and says whether a failure was ever planted on purpose and seen to be caught.

34 of 48have a failure that was planted and caught
12can only fail by missing a stated bar
2have a way to fail nobody exercised
68 of 81tools carry a selftest that ran and passed

Planted failures, entry by entry

  • The kit's own requirements name the tests that catch their breaks: With the catching tests removed from the verdicts no requirement may name a check; a name is never written by hand.; A check named by a label with a dot in it is one test and meets the break it caught; with the old cut at the last dot it does not.
  • A break with no verdict is drawn apart from one that got past every test: On the test store two breaks have no killed-by edge and one got past: the old rule paints 2 red, the page must paint 1 red and 1 violet.; With the chain deleted the Reasoning line goes, and a state of a break shows only while the graph holds one. A store with
  • Are the links from documents to requirements correct?: The first reading, before the seeds, found 2 wrong and 1 worthless of 36; each was fixed as a class with a plant before the seeds were drawn.
  • Does attaching documents change anything else?: Each guard that keeps documents out of fingerprints, distance and the network's view was removed in memory and the figure moved.
  • Does the documents tool refuse what it should?: A planted fabricated id, renamed section, lost document, empty set or unparseable file must each be refused with a named reason.
  • Is every number on a public page tied to a record that can say it moved?: A record that moved, a record that is gone, a reworded sentence, a new unregistered figure, a claim on unpublished work, a malformed claim, a tool's page that would be written differently: each must refuse. A control with none of them must not.; A copy of a tr
  • Do the growth charts refuse a bad log and draw only what the logs hold?: Each rule was broken inside a copy of the tool and the selftest had to go red on the check that guards it: lines for any readings, unreadable rows skipped, a join across a change, a missing log, absent read as zero, the gate not called.
  • Does every recorded stat have a reader?: Fifteen removals of a piece of the fix each must make a selftest fail. All did.
  • Do the plan's switches reach a pass at all?: Fourteen removals of a piece of the fix must each fail a selftest. The first run caught 13: an exit-code check was too loose and was tightened.
  • Does a trigger that fires on every pass say anything?: Thirteen removals of a piece of the fix must each fail a selftest, on a copy with a passing control.
  • Is a change in one stat followed through the triggers, the pick, the lists, the growth page, the documents and the graph?: A link passes without a cause: the control run, with nothing worsened, must not report PASS on links 1, 2, 3 or 6.
  • Does each link's check turn red when only that link is broken?: A plant leaves its link at PASS; the run says so ('the plant moved nothing') and exits 1.
  • Does every proof on record still pass when run again, together, and does the runner refuse the ways to pass without proving?: A proof that hangs is waited on, or read as a pass. The plant is a program that sleeps for two minutes: it is killed at its limit and named, and the run takes seconds.; A tool that exits 0 having checked nothing passes. Planted, and found for real in two tools
  • Does a finished tick rewrite the two lists, and does the next tick read what they ask?: A finish step is called and does nothing: the tool's selftest runs stub tools and counts which ran and which failed.; The finish steps are removed one at a time from the copy and the matching harmony link goes off PASS (links 7, 8 and 9).
  • Does the fly-through show the stats, the learned parts and the graphs working together from real records, and refuse to say a link holds from a stale one?: A link's state is read from a record made on tools that have since changed. Planted: a tool edited after the record, and one deleted; each is refused, and the control passes. A record that does not hold links 1 to 10 once each, names a state the page does not
  • Does every new group, world and link work for a visitor in a real browser, with WebGPU and with WebGL 2, on a desktop, on a phone and with no graphics?: Running and looking found nine faults in the first runs, before they shipped: the first world's layer labels drawn over another world, the same sentence twice under a title, an overview of the four worlds that was a white blob with a label out of frame, two li
  • Does each learned part beat plain rules on the loop's own code?: Planted known-bad inputs for the new guards must be caught.
  • Can a goal be turned into a vector that points at goals worth training?: The plant selftest on invented graphs: a copy of a relation must be refused by the useful direction.
  • Ranking the other requirements tools by a rubric fixed first: The tool refuses a score with no evidence, and each refusal is planted in its selftest and seen to fire.
  • Does the skill reach five other agents in the file each reads, from one source, with a stale or invalid copy refused?: A layout drops one of grow, ask, run or prove, or a layout file is missing. Each planted and refused.; A layout is cut from an older SKILL.md, or edited by hand with its stamp kept. Both planted: the kit check and install --agent refuse.; Front matter the tool
  • Drafting a model from a PRD, a ticket export, a transcript and a public proposal, every requirement citing its sentence: A requirement citing a sentence that is not in the text is refused, one citing none is refused, one quoting words the sentence lacks is refused; each is planted and fires.; A rule sentence nothing cites is reported by the sentence space (U7) and by source_rule
  • Does the kit run from one command with no clone, from a package, on a clean home folder?: The wheel lacks a file the tools find by location. Planted by building a wheel without the Python example: try exits 1 and does not print HOLDS.
  • Do fresh reviewers find holes the loop passes?: Each hole names a plant that reproduces it, and the board marks one fixed only with its plant.
  • Does it matter which requirements are written first?: A plant shows the pillar order is not just the order of the ids.
  • Does every tool have a selftest, and does it pass?: A tool on disk that mentions a selftest and is not in the survey makes the check refuse.
  • Does the example run on a machine with only a newer .NET?: Asked for a .NET version the machine lacks, the run stopped with 'You must install'. That shows the check can fail.
  • Can the language-neutral parts model public code?: A document row the parser cannot read stops the model. The check caught one missing a column.
  • A small tested class, against a public specification: The run stops on code it cannot read or compile. It did, six times across the three trials (BP-4 to BP-8 and BP-10), and each cause was fixed with a plant.
  • Joining two finished features, does the graph name the seams?: A planted provider must change the causal count: it moved it from 7 to 6, so the zeros are not a dead instrument.
  • This benchmark's own check: Each refusal is planted in a fixture and must fire, beside a clean control that must pass.
  • Does a stranger get a working, proved install from the README alone, and what does the first run cost?: The clean folder lets the real home help the stranger (a warm package cache, a PATH holding dotnet, a temp folder elsewhere): the tool's selftest asks a child where the home, the temp folder and the package cache are, and checks the minimal stranger's python a
  • A real argument chain on the kit's example: A cite that is not in the graph fails the run with exit 1. The manual records three planted cases run on a copy of the example: a requirement that does not exist, a method the code does not have, and a step citing nothing.
  • Does the dictation panel turn speech into text and a reply beside the slide on both pages, and keep the key in the tab?: Planted leaks (the dummy key put in storage, the address, the page title, the console, a file, the address of a call) and the old code of each defect restored in memory (the Mic off at the first pause, no quiet timer, the button in the header, the slide not he
  • Do the real API and the real speech service let the dictation panel through, with no key at all?: The same preflight and request without the opt-in header, and the same request to a host the policy does not name, must each be refused. All were.

The tools’ own selftests, as a class

81 tools were surveyed. 68 have a selftest that ran and exited 0, 0 ran and failed, 0 take no such flag, and 13 never mention one. The runs printed 1,088 lines that end in a verdict. That is a floor on the checks: one line can stand for several, and 3 passing tools print no per-check lines at all.

Without a selftest: autobot/answer.py, autobot/autobot4.py, autobot/bench.py, autobot/cut.py, autobot/doc_to_graph.py, autobot/doc_verify.py, autobot/x2b_measures.py, autobot/landscape/make_slide.py, closed_loop_kit/install.py, closed_loop_kit/pack_check.py, closed_loop_kit/docs/goal_slides.py, closed_loop_kit/docs/goals_pages.py.

What it cost

Cost is tokens and minutes of an agent working. 13 of 48 tests have a recorded cost, adding up to about 2,136,987 tokens and 663 minutes. 4 of those are read from the plan’s cost log and re-read by the check; the rest are figures written on the board at the time, which nothing re-reads. The other 35 tests have no cost recorded, which is itself a gap.

The most expensive, in tokens

Are the links from documents to requirements…
590,651
Do the growth charts refuse a bad log and draw…
445,494
Ranking the other requirements tools by a…
427,469
Joining two finished features, does the graph…
353,000
Can a fresh agent install the kit from the zip…
131,000
Does a stranger get a working, proved install…
114,373
Can the language-neutral parts model public…
60,000
Does the example run on a machine with only a…
15,000

Tests with no cost on record: Does every link by name stand on the line it cites?, Does attaching documents change anything else?, Does the documents tool refuse what it should?, Are the documents tool's own checks held by breaks?, Does the skill reach five other agents in the file each reads, from one source, with a stale or invalid copy refused?, Does the kit run from one command with no clone, from a package, on a clean home folder?, Drafting a model from a PRD, a ticket export, a transcript and a public proposal, every requirement citing its sentence, How full must the graph be before the network beats the plain rules?, Do the plan's switches hold at more seeds?, Can a goal be turned into a vector that points at goals worth training?; and 25 more.

What has not been tested

A list of what this record cannot say yet, so a reader does not mistake silence for a pass.

Some results are not published yet. More is on the way, and this page will grow when they are.

Where the bar was not fixed first

27 of 48 tests did not have their bar committed before the result. For these, the page cannot show that the bar was not chosen after seeing the answer. They are: Does the documents tool refuse what it should?, Are the documents tool's own checks held by breaks?, Do the growth charts refuse a bad log and draw only what the logs hold?, Does each learned part beat plain rules on the loop's own code?, A small tested class, against a public specification, A large, well-tested class, The scoring rules of a shipped open-source game, Joining two finished features, does the graph name the seams?, Can the language-neutral parts model public code?, Does the example run on a machine with only a newer .NET?, Can a fresh agent install the kit from the zip alone?, Does every recorded stat have a reader?.

Check it yourself

The record is a data file. This command re-reads every number named on this page from the file that holds it, and refuses when one has moved, when a command names a file that is gone, when a test has no way to fail, or when a written-up test has no entry.

py -X utf8 autobot/benchmark.py --check

Read more: Growth · Goals List · Stats List · Limits · The whole manual · The deck

Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.