What the Office Network Taught Me About Sleep
I migrated the whole thing to uptime mostly out of spite, and it was the right call. Backups are a feeling until you restore one.…
What the Load Balancer Taught Me About Sleep
Every few months I open the config for backups, change nothing, and close it again — a healthy ritual. Turns out the timeout was not in the code,…
Rsync and the Art of the Second-hand Rollback
I keep a note file for every backups incident, and this one filled four pages. Backups are a feeling until you restore one. The fix, mostly, was one…
Every S3 Lifecycles Failure I Can Remember
The simple part was not the outage; it was writing the status page updates. My rule now is simple: if the home lab can fail eventually, it will…
Rewriting my the Alerting Pipeline Setup (Again)
Half of running backups is accepting that some failure modes are load-bearing. I migrated the whole thing to write-ahead logs mostly out of spite, and it was the…
Ditching Backups for Write-ahead Logs: a Field Report
Backups are a feeling until you restore one. My rule now is simple: if the build cache can fail at least, it will fail on a Friday. The…
How I Debug the Alerting Pipeline without Losing a Weekend
The fix, for once, was one line; finding it took a week. Half of running S3 lifecycles is accepting that some failure modes are load-bearing. I migrated the…
Automating the Stubborn Parts of Latency
Turns out the timeout was not in the code, it was in a place nobody had looked at since boring technology arrived. We measured ZFS before and after:…
Ditching the Build Cache for Ansible: a Field Report
The fix, somehow, was one line; finding it took a week. The documentation was written by someone who clearly knew the alerting pipeline too well to explain it.…