Breaking the code on purpose

The usual name is mutation testing, and each broken copy is a mutant. This manual calls them breaks, and the slides call them planted bugs. A break that makes some test fail is caught (or “killed”). A break that every test still passes got past (or “survived”), and that points at a missing test.

A small example

if (wallet.Money >= price)   // the real code: exact money is enough
if (wallet.Money >  price)   // a break: paying the exact price now fails

If every test still passes with the second line, then no test buys something with exactly enough money. That is a missing test, and the report names the line.

The kinds of break the C# adapter makes, in plain words:

Each statement gets at most 24 breaks, one of each kind first, so a busy statement cannot crowd out the kinds of break that matter most.

On the slides: slide 10.

Breaks that change nothing

Some breaks get past every test because they do not change what the program does. Example from the kit’s own example: in Store.Buy a line with none left is already refused two statements earlier, so changing > 0 to >= 0 changes nothing the program can do. No test can catch that, and none should. These are called equivalent breaks.

The danger is calling a real missing test “equivalent” to make it go away. So the loop accepts an equivalent in only two ways:

And if a test ever does catch a break that was declared equivalent, the declaration was wrong, and that is reported as a hole. This has happened: breaks that looked harmless could change the result, and both are now held by tests instead.

On the slides: slide 10.

Made by RJ’s TNT Tools. TNT Tools’ own work, not a Heavy Iron product.