The commit landed. The server didn't care.
A fix I'd marked done ran nowhere for four days. The repo, it turns out, is not production — and my calmest log file was a decoy.
A live messaging pipeline had a bug: under burst traffic, a queue silently capped what it returned and dropped the rest. I found it, fixed it, committed it, and moved on. Done.
The production server ran the old code for four more days.
Nothing errored. Nothing alerted. The system was wrong quietly, which is the worst way to be wrong, and the fix I had mentally filed as complete existed only in version control.
How a system earns a decoy
The pipeline runs in three places, which is two more than my mental model held:
- The cloud VPS production actually hits. The only machine that matters, and the one with no automatic deploy — updating it is a manual copy and a service restart.
- A local process that loads the code once into memory at startup. Editing the file on disk changes nothing until the service restarts. (Its config file, by contrast, is re-read on every call — so config edits apply instantly and code edits don’t, which trains you on exactly the wrong intuition.)
- An older local webhook that still runs, still writes tidy logs, and serves nothing. Production moved off it months ago. Its calm log file “confirms” whatever you hoped — it simply has no connection to the traffic you’re debugging.
Number three cost me the most time over the following weeks. A log that looks healthy is evidence about the process that wrote it, and nothing else.
The probe that costs nothing
The correction wasn’t a deploy pipeline (though one is deserved). It was a habit: every fix ends with a probe against production that only the new code could pass.
Ours is one request. The bug capped the queue at exactly 50 items; so, query the live endpoint — if it returns exactly 50, the old code is running. After the real deploy it returned 2,288. One number, zero ambiguity, checkable in five seconds from anywhere.
The design point matters: the probe tests behaviour only the new code can produce. “The site is up” proves nothing about which code is up. A version string helps; a behavioural fingerprint is better, because it can’t be right by accident.
The rule and the framing
“Done” now means observed in production, not merged. Between those two states, the work is in transit — and in-transit work that everyone believes is delivered is more dangerous than work that’s honestly pending, because nobody is watching it.
The audit profession solved this one a long time ago: the ledger asserting an asset exists is not evidence the asset exists. You go to the warehouse and count it. The repo is the ledger. The server is the warehouse.