Five faults
and how I found them

1 — A thousand restarts, and the lie underneath them

Symptom: a photo tagger restarting every eight minutes. 1,004 restarts. Alerts on both sides of every cycle.

• The alert said the container was restarting, so the container looked like the problem. It was not.
• Its database had run out of disk. Every request to the photo server returned 404, and one endpoint intermittently returned 500. The 500 was unhandled, so the process died, the restart policy brought it back, and it died again.
• The last crash is stamped 08:13:37. The database logged "ready to accept connections" in the same second. That timestamp is what confirmed the direction of causation rather than assuming it.
• The real damage was invisible. On each 404 the tagger recorded the photo as examined and found to contain nothing — it wrote that about 76,992 photographs it never looked at. 96% of its state was that lie. Those photos would never have been tagged again, and nothing would ever have said so.

Fixed by separating bad luck from a sick server: one 404 is an asset deleted mid-run and worth recording; twenty-five in a row means the server is unwell, and the sweep now abandons itself without marking anything.

Found in logs · timestamps to the second

2 — A service standing down at its own footsteps

Symptom: the same tagger managing five photographs in four hours, with sixteen back-offs.

• Two log lines, forty seconds apart, looked like separate events:

02:04:13 asset ...: timed out - will retry next sweep
02:04:53 GPU is in use by something else - backing off 600s

• They were the same event. The image request timed out because the model server was busy. The "is anyone else using the GPU?" check then also failed to get an answer — and that check treated no-answer as yes-somebody-else-is.
• So it slept ten minutes because its own work had made the server slow to reply.
• Worse by construction: model residency was two minutes and the sleep was ten, so every back-off guaranteed a cold reload, which made the server slow again, which tripped the check again. A cooldown longer than residency is self-defeating.

The check now answers free / busy / unknown instead of a boolean, because "a neighbour has the card" and "the server is too busy to talk" want different responses.

62s per photo → 20s

3 — The fix that was written and never ran

Symptom: a nightly backup filled a 669 GB disk in one night and took a photo library down with it.

• The backup had been archiving a network share onto local disk — copying the NAS onto the SSD, nightly, to protect a NAS that already is the second copy.
• A guard against exactly this had been written eight days earlier. It was correct. It never ran.
• Cron executes /usr/local/sbin/vault-backup. The fix went into the repository copy. The installed binary was three weeks older.
• A fix in a file nothing executes is not a fix. The repo version is now installed, and the previous one kept beside it so the difference is visible rather than assumed.

Both runaway archives turned out to be truncated and failed integrity checks — the disk filled mid-write — so the backup that consumed the disk could not have restored anything either.

423 GB reclaimed · 0 data lost

4 — Two machines on one LAN, talking via the public internet

Symptom: intermittent DNS and TLS errors, and a job that would have taken months.

• Errors read as network flakiness: "temporary failure in name resolution", "SSL: UNEXPECTED_EOF".
• The service was configured with the site's public hostname. Every request left the container, resolved through dynamic DNS, went out to the public IP and came back in through the router, a relay and a reverse proxy — to reach a machine two hops away.
• Measured, same request, four calls each: 171 ms public against 4 ms on the LAN. Another service in the same config file already used the LAN address.
• At two calls per photograph across 79,000 photographs, that round trip was about seven hours of pure latency — and every DNS failure cost a whole photo.

43× · network errors 16 → 0

5 — The flag that nearly got a hole punched in it

Context: a tool that finds an AI watermark on a photograph and paints it out.

• The detector scores candidates on shape: square in its box, fills about a third of it, symmetric both ways, solid at the centre. A four-pointed star passes all four.
• So does a star on the United States flag. And a yellow flower on a dress. And a diamond in knitted fabric. Run across two real albums with the safety check forced off, the tool claimed all three.
• No shape test can separate them, and I do not believe one exists — they are genuinely the same shape.
• What separates them is not shape but surroundings. The watermark sits on a quiet corner; a flag, a flower and a knit do not. A threshold written to protect fill quality turned out to be the only thing standing between the tool and a hole in somebody's photograph.

Caught by rendering a before/after contact sheet and looking at it, which is why that now runs in dry-run mode too: judging whether to trust a detector by looking at what it would do is impossible if the pictures only appear after it has done it.

60 cleaned · 26 correctly refused