XiliuxI had a table in the database that was supposed to fill itself. Its job was to learn from failures: every time the system tried a variant of something and it didn't work, it saved it, to recycle later in another context where it might. A laboratory of failed attempts, piling up.
In production it had zero rows. It had been deployed for days and hadn't saved a single record. Meanwhile a neighbouring table —another memory, the one that notes which work is already exhausted so as not to repeat it— was growing normally.
The temptation is obvious: there's a bug in the write. I went looking for it, and it wasn't there.
Zero rows isn't the same as a write error
The path that saves into that table emits a warning if the write fails. I searched the logs for those warnings: zero. No write had failed.
That's a fact, not an absence of one. If the path had been taken and had failed, it would have left a trace. Zero traces and zero rows fit only one explanation: the write path never ran. Not ran-and-failed. Didn't run.
That's the difference between a real negative and a negative that was never put to the test, and they look the same unless you look for the positive control —something the log WOULD show if the path had been taken—. Without it, "healthy and quiet" and "dead" look identical.
Two mechanisms starving each other
Why didn't it run? Because of another mechanism, upstream, doing its job well.
That system has a negative memory: when it exhausts everything it knows how to try against a target, it notes it down, so as not to spend effort again on something it already knows won't pay. It's a sensible optimisation. But it sat before the phase that generated new variants —the phase that, on failing, would have fed the library—. As soon as a target went "exhausted", that phase was skipped entirely. And if the phase never runs, it never produces a failure to save.
Each mechanism, on its own, is correct. The negative memory avoids useless work. The library learns from failures. Together they formed a deadlock: the first starved the second. The library could only fill from a phase the other one switched off.
It's a recurring pattern: when you add a mechanism to contain what another one does, and the system starts behaving strangely, suspect the design before the code. The symptom —a table at zero— looked like a bug; it was a tension between two pieces that are each fine on their own.
A value with two meanings is two fields
There was a second problem, and it's the more instructive one. The library only saved one kind of failure: the one where the system processed the attempt and returned a normal result, without success. The other kind —when the attempt was blocked up front, without ever being processed— it discarded, with a reasonable argument: a block says more about the target's posture than about the variant.
But that argument falls apart when recycling. The variant blocked at one target is an untried candidate at another, with a different configuration. Discarding it was, precisely, failing to learn from the most common failure.
The fix was not to collapse the two meanings into one. "Processed failure" and "blocked failure" are two distinct signals, so they go in two distinct fields: two counters. That way the library saves both, and whatever recycles them orders them —the processed ones, more informative, first; the blocked ones, at the back, but present—. A value with two meanings isn't one field: it's two.
What was left
Two fixes on the same symptom. Neither touched the write, which was always correct. One broke the deadlock —letting the phase feed the library even when the target is "exhausted"—; the other split the two faces of failure. And the table at zero, which looked like the bug, was the messenger: a path that never ran and a condition too narrow, seen through the same hole.