How can a detector fail if all its tests are green?

Because the tests check files that contain no violations. A correct detector returns empty on a clean file, and a broken one does too. The two results are identical, so the test passes in both cases and never tells them apart. In one measured case, a guard was blind across 1,029 of 6,309 lines —824 of them executable— with its three tests green the whole time.

Why break the rule in the real code and not in the test harness?

Because if the failure is simulated in the harness, the test goes red because of the harness and not the detector, and you are back where you started. The valid check is to mutilate the code being measured and require the test to go red — while also checking that the mutant compiled, or the red may come from a build error rather than the rule.

Where do you get a mutant without writing one?

From history. When you fix a detector, the previous version of the rule is already the mutant you need: git show HEAD:path/to/detector > /tmp/old and run the new pair against it. If the red case goes red, the pair discriminates; if it comes out green against both versions, the new tests are worth no more than the old ones. It costs a minute.

Does this only apply to linters and code guards?

No. It applies to anything that returns "I found nothing": tenant isolation tests, secret-leak checks, dependency audits, monitors. They all share the property that their failure does not complain: a zero from breakage and a zero from real absence look exactly the same.

My detector was blind across 1,029 of 6,309 lines and its three tests stayed green

Q: What is a "pair" of tests and why do you need both?

Two cases for the same probe: one that MUST fire and one that MUST NOT. With only the first, an instrument that says yes to everything passes just the same; with only the second, a blind one does. Each case covers the other's blind spot, so they only work together.

The guard worked. I had written it so that no session open in one project could touch another project's files, it had three tests, and all three were green. They had been green for months.

It was blind across 1,029 of the 6,309 lines it was watching. Eight hundred and twenty-four of them executable.

And the three tests kept passing, because all three looked at paths that were already clean.

The defect isn't in the detector: it's in how you test it

You test a detector with cases. The natural thing is to take them from the system itself: feed it the real files and check that it finds nothing, because the system is healthy. A hundred cases, all green, and the feeling of having measured something.

Nothing has been measured.

A bench made only of negatives over artefacts that are clean today comes out green with the correct rule and comes out green with a rule that doesn't look. A file with no violations returns empty in both worlds: the one where the detector works and the one where it broke three months ago. The two results are identical, so the bench cannot tell them apart.

What that bench measures is the watched artefact, not the watcher. And the watcher is precisely the piece whose failure never complains.

It's the disguised form of a very familiar error, which is why it slips through: the tests exist, they are well written, they are readable, and they pass. There is nothing to flag in code review. The defect isn't in any line; it's in the set.

The pair

The fix is cheap and fits in one sentence: every probe ships with two cases, not one. The one that MUST see something and the one that MUST NOT.

With only the first, an instrument that says yes to everything passes just the same. With only the second, a blind one does. You need both, and you need them together: each covers exactly the other's blind spot.

And there are two conditions that get forgotten the moment you write the second case, and they are what separates a real pair from a decorative one.

First: the red case has to fail through the real path. Breaking the rule in the test harness doesn't count. You have to break it in the code being measured, and require the test to actually go red. If mutilating the detector leaves the test green, what you were measuring was the harness. And it's worth checking that the mutant compiled: a test that goes red because the code doesn't build has demonstrated nothing about the rule.

Second: the detector is also tested against synthetic input. The real files aren't enough. You have to fabricate a text that DOES contain the violation and another that doesn't, and push both through the same seam the real path uses. If the test enters through another door — a helper, an already-constructed object — the red can come from the harness rather than the detector, and you're back where you started.

The mutant already exists, and it's free

This is the part with the highest return per minute invested, and almost nobody uses it.

When you fix a detector, the previous version of the rule is the mutant you need. It's in history, one command away:

git show HEAD:path/to/detector.py > /tmp/previous_version.py

Run the new pair of tests against that old version. If the red case goes red with it, the pair discriminates: it distinguishes the good rule from the blind one. If it comes out green against both, the new tests are worth no more than the old ones and nothing has been fixed yet.

It costs a minute. It's the difference between believing you repaired a detector and knowing it.

How the blind spot surfaced, which also teaches something

No test found it. Pulling the opposite thread did.

The guard was producing a false alarm: it warned every time a command redirected to any file at all, even /dev/null. Annoying and visible. While writing the missing case to fix that false positive, its silent twin appeared: writing directly into another project's file never warned at all. It had been like that from day one.

They weren't two defects. It was one, seen from both sides. The noisy side complained every day; the blind side was never going to complain.

From which comes a practical rule that holds for any detector: when you fix a false positive, go looking for the false negative that shares its cause. The noisy one leads you to the mute one, and the mute one is the expensive one.

What's left

Three things, and none of them costs money:

A bench of only negatives does not discriminate, however many cases it has. If all your tests assert "there's nothing here", they come out green with the good rule and with a dead one.
Every probe goes out with its pair, the red fails through the real path, and the mutant comes from history.
Every noisy alarm hides a silent twin. Go find it when you fix the loud one.

And the uncomfortable part: this happened in a guard of mine, written by me, with tests written by me. Discipline doesn't collapse while auditing someone else's work; it collapses exactly where the thing being checked is something you just wrote, because thinking it through feels like verification and isn't.