How the closed loop has grown, drawn only from the dated records it keeps about itself. Every point is a reading in a log row that carries a date, and nothing is estimated, interpolated or filled. A line is drawn only through 3 or more readings, and is broken wherever what was counted changed (the kinds of break tried, the criteria checked). With fewer readings the page shows the readings and says so. Every reading here was taken between 10-29 12:02 and 10-31 22:07, so most of what the loop records has few readings; that is itself the finding.
What it is. Of the breaks the loop made in the code that a test could judge, the share a test caught. Breaks shown or declared to change nothing are left out, and so are breaks with no verdict.
What a rise does and does not prove. A rise proves a larger share of the judged breaks was caught, or that more survivors were explained. It does not prove the tests got better.
In the kit’s Python example the kill rate rose from 0.9565 to 1.0000 while the count of breaks caught stayed at 66 on every reading, so the rise came from breaks being explained, not from more being caught.
The rate leaves out breaks with no verdict. The largest share is in one game feature: 52 of 1,104 breaks had none.
The axis starts at 0.50 where no reading is lower.
The tool’s own code: its two readings are not comparable (the definition changed: the Python adapter now decides by bytecode which breaks change nothing, so breaks once set aside as equivalent are judged).
What it is. Breaks no test caught, and of those the ones nobody has yet explained as changing nothing. The unexplained ones are the open work.
What a rise does and does not prove. A fall in the unexplained line is closed work only if the survivors were caught, or shown to be the same program. A fall that comes from a person saying a survivor changes nothing is a claim, not a proof.
In the kit’s Python example the number that got past stayed at 3 and the number caught at 66 on every reading, while the unexplained part fell from 3 to 0: nothing more was caught, 3 were explained, as the same program or by a person.
The tool’s own code: its two readings are not comparable (the definition changed: the Python adapter now decides by bytecode which breaks change nothing, so breaks once set aside as equivalent are judged).
What it is. Places where the loop’s own checking still lets a mistake through and the fix is ours to make, counted over the criteria it checks. The number of criteria is recorded with each reading. Fewer is better.
What a rise does and does not prove. A fall means holes were closed. A rise does not mean the code got worse: the tool is changed between readings, and when it learns to look for another kind of gap the count rises on code that did not change.
The largest rise in the kit’s Python example went from 0 to 12 between 10-29 18:25:51 and 10-29 21:37:22, while the tool changed from 5,835 to 5,994 lines. The code under test was not what changed first; the tool was.
One game feature, holes of ours: the line is not joined between readings 1 and 2 (criteria checked 14 to 17).
Learned part: Which breaks will get past the tests, holes of ours: the line is not joined between readings 2 and 3 (criteria checked 17 to 18).
Learned part: Which breaks change nothing, holes of ours: the line is not joined between readings 2 and 3 (criteria checked 17 to 18).
Learned part: Where to add a test, holes of ours: the line is not joined between readings 2 and 3 (criteria checked 17 to 18).
Learned part: Who owns new code, holes of ours: the line is not joined between readings 2 and 3 (criteria checked 17 to 18).
Learned part: Which steps carry a requirement, holes of ours: the line is not joined between readings 2 and 3 (criteria checked 17 to 18).
Learned part: Which tests catch a break, holes of ours: the line is not joined between readings 2 and 3 (criteria checked 17 to 18).
What it is. How well the network that guesses which requirement owns a piece of code ranks the right one, as a score from 0 to 1 (mean reciprocal rank: 1 means the right requirement is always first). Only a pass that ran the learned part writes it.
What a rise does and does not prove. A rise means the network ranked the right requirement higher on that pass’s graph. The graph and the model change between passes, so two readings answer slightly different questions: it does not prove the network got better.
No reading of this stat: The kit’s C# example, The kit’s Python example, The tool’s own code, Learned part: Which breaks will get past the tests, Learned part: Which breaks change nothing, Learned part: Where to add a test, Learned part: Who owns new code, Learned part: Which steps carry a requirement, Learned part: Which tests catch a break.
What it is. How many requirements and how many reasoning steps each project’s model holds, at the latest reading it has.
What a rise does and does not prove. More is not better by itself. A model with more requirements has said more, not said it more correctly; it has grown only if it now covers more of the product.
Within every project the counts were the same on every reading: a model's size is set when it is written and did not change in the readings kept.
What it is. How many fields a pass writes into its log row: the stats tracked. Rows rebuilt by hand from older reports are not counted.
What a rise does and does not prove. A rise means more is recorded. It does not mean more is used, and the count also moves with which optional parts a pass ran: a pass without the learned part writes none of its fields.
What it is. Lines in the tool’s own source files (8 of them now), counted by each pass and written in its log row. The list of files is checked against the stats recorder’s on every build, and the build stops if it changes without being recorded here, so the line breaks wherever the files counted changed.
What a rise does and does not prove. A rise proves the tool got bigger. It does not prove it got better: that needs a stat that moved with it. A tool that grows while nothing it measures moves is added complexity.
From 7,975 to 7,995 lines between 10-30 16:58 and 10-30 20:52: 20 more (0%).
The line is not joined between 10-30 16:28 and 10-30 16:58: files counted 7 to 8.
What it is. Each improvement requirement watches one stat, names the trigger that fires and the action to take. Triggers that fired is how many tripped at that reading. Measured outcomes is how many ticks aimed at a stat have a before and an after reading of it.
What a rise does and does not prove. More requirements is more watching, not more improving. Fewer triggers firing is good only when the cause was fixed in the world and not in the trigger; the fall here came from fixing triggers that fired on every pass, which says the watching got better.
Measured outcomes were 0 on 4 of 4 readings: no tick aimed at a stat has yet had its before and after measured.
What it is. How long each improvement tick took, by mode, and the tick’s own answer to whether the item was worth doing, written when it ended.
What a rise does and does not prove. Minutes are cost, not growth, and bars are one per tick so no trend is drawn. Worth is the tick’s own judgement and not an outside check: 5 of 5 say yes, and a verdict that has not yet said no tells little.
Long passes launched by a tick: 3, collected: 3.
What it is. How many of the goals on the Goals List have a network trained for them, counted at the time each one’s code first appeared in git.
What a rise does and does not prove. More goals trained is more tried, not more reached. Trying is not reaching, and the reached count is a single reading, so it has no line.
Reached, one reading (10-30 12:56): 5 of 10 results reached their goal on paper, so 3 of 5 trained goals reached it on at least one project. A goal counts as reached only when it beats a plain rule at most seeds on code it has not seen. The log has 1 reading of this.
On the list: 5 trained, 4 planned, 1 uses another.
What it is. How many items of the plan are open, running and done, written each time one changes state.
What a rise does and does not prove. Open falling while done rises is the burn-down. A reading is written only when an item changes state, so the spacing says when work moved and not how fast it goes.
170 readings, 1800 minutes apart: read the line against the dates, not as a steady rate.
What it is. For each stat that may be shown in public, the loop’s value today (filled dot) and the most it could ever be (the upright tick). Each row has its own scale; its two end labels are its own. A ceiling is drawn only where arithmetic or an oracle argues one.
What it does and does not prove. The gap to the tick is room left. Each value is read from a record.
Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.