Breaking the code on purpose
The usual name is mutation testing, and each broken copy is a mutant. This manual calls them breaks, and the slides call them planted bugs. A break that makes some test fail is caught (or “killed”). A break that every test still passes got past (or “survived”), and that points at a missing test.
A small example
if (wallet.Money >= price) // the real code: exact money is enough
if (wallet.Money > price) // a break: paying the exact price now fails
If every test still passes with the second line, then no test buys something with exactly enough money. That is a missing test, and the report names the line.
The kinds of break the C# adapter makes, in plain words:
- Conditions: turned around, forced always true, forced always false.
- Operators:
<made<=,==made!=,&&made||, plus made minus. - Values: a number made one more, true made false, a text emptied, a function argument replaced by its default.
- Structure: the two outcomes of a
? :swapped, an else branch removed, a statement removed, the whole method made to return at once.
Each statement gets at most 24 breaks, one of each kind first, so a busy statement cannot crowd out the kinds of break that matter most.
On the slides: slide 10.
Breaks that change nothing
Some breaks get past every test because they do not change what the program does. Example from the kit’s own example: in Store.Buy a line with none left is already refused two statements earlier, so changing > 0 to >= 0 changes nothing the program can do. No test can catch that, and none should. These are called equivalent breaks.
The danger is calling a real missing test “equivalent” to make it go away. So the loop accepts an equivalent in only two ways:
- Proven by the compiled code. C# compiles to IL, the instructions the runtime actually runs. If the broken copy’s IL is identical to the original’s, the two are the same program (a technique called Trivial Compiler Equivalence, Papadakis and others, 2015). The loop checks this itself.
- Declared with a reason. Otherwise a person or agent writes down why it changes nothing, and names the methods the reason depends on. If those methods change later, the reason lapses and the break is counted as a hole again.
And if a test ever does catch a break that was declared equivalent, the declaration was wrong, and that is reported as a hole. This has happened: breaks that looked harmless could change the result, and both are now held by tests instead.
On the slides: slide 10.
Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.