XiliuxA program of mine hung for forty minutes. Not spinning at a thousand loops a second: at zero percent CPU. It wasn't doing too much work; it wasn't doing any work at all. And still it wouldn't finish.
The program does something common: it orchestrates external command-line tools. It launches one, reads what it writes to standard output, and moves on to the next when it's done. So it doesn't get stuck when a tool drags, each one has a timeout: when it fires, the process is killed and we carry on.
That's the part that failed, and it failed where no one looks: after killing the process.
Killing the process doesn't close the pipe
When you read a subprocess's output, you read from a pipe: one end writes (the subprocess), the other reads (you). Your reader doesn't finish when the subprocess dies. It finishes when EOF arrives, and a pipe's EOF arrives only when the last write end is closed.
Almost always they coincide: the subprocess is the only writer, it dies, its end closes, EOF arrives, your reader finishes. All in microseconds.
But "almost always" isn't "always". The tool I launched launched another one in turn —a grandchild—. And that grandchild inherited the pipe's write end, because on Unix a child inherits its parent's open descriptors unless told otherwise.
So when the timeout fired, I killed the child. Its end closed. But the grandchild was still alive, with its copy of the descriptor open. The last write end hadn't closed. EOF never came. And my reader sat waiting for an EOF that would never arrive —at zero percent CPU, blocked in a read(), indistinguishable from slow work—.
The symptom that deceives
What makes this failure so hard to see is that it doesn't look like a failure. An infinite-loop hang burns CPU: you see it in top instantly. This one spends nothing. The thread is asleep in the kernel waiting for data that isn't coming. In the process list it looks healthy. In the metrics it looks like it's "taking a while". The only way to tell "hung forever" from "running slow" is to look at the thread stack and see the read() that never moves.
The fix isn't killing better
The reflex is to try to kill the grandchildren too —a process group, a session, the whole tree—. Sometimes you can, sometimes you can't: a process can escape the group, a grandchild can outlive its parent on purpose.
But there's a simpler and more honest way out, and it comes from a single question: which process, exactly, is going to close that descriptor? If the answer is "none that I control", then waiting for EOF is waiting for something that won't happen. And a wait on something that won't happen isn't fixed by waiting better: it's bounded.
The fix was to put a timeout on the drain itself. After killing the process, we wait for EOF for a few seconds; if it doesn't come, we abandon the read and move on. Whatever the tool managed to write before is already captured, so nothing is lost. And a descriptor a grandchild keeps open stops mattering: no one waits on it indefinitely.
The rule, beyond pipes
Every wait —a loop that spins until a condition holds, an await on a descriptor— has one question behind it: which process is going to make that condition true? If there's none, the wait is impossible, and an impossible wait gives no error: it looks exactly like work that's taking a while. Zero percent CPU, not advancing, forever.
Before you write the wait, name the process that will end it. If you can't name it, don't write it without a cap.