Theo's Corner
dev / irl / thoughts
← Back
techlife

Things I've broken and what I learned from them

Theo|Jun 2026|~5 min read

Breaking things is how you actually learn. Not the comfortable "I tried a tutorial and it worked" kind of learning — the "I just took down something that was working and now I have to figure out why" kind. That's the learning that sticks. Here are some of mine.

Filling a root disk with backups

On one of my Wings nodes, I ended up with 199GB of Pterodactyl backups sitting on the wrong drive — the root filesystem instead of the NVMe where they were supposed to go. Root disk hit 100% and the node died. Nothing would start, logs were failing to write, the whole thing was just stuck.

What I learned: always check where backup targets are actually writing before assuming they're correct. Verify with df -h before and after configuring anything that generates large files. And have a plan for what to do when a disk fills up, because it will happen eventually.

Nginx config that worked locally but broke in production

I once spent an embarrassing amount of time debugging an nginx config that was syntactically valid, passed nginx -t, but was doing something completely wrong in production. The issue was config file load order — on that particular server, all nginx site configs needed to be prefixed with zzz- to load in the right order, and without that prefix a default config was taking precedence.

Always know your server's config loading order. And always run nginx -t before reloading, but know that passing the test doesn't mean it's correct — it just means it's valid.

Wiping work that wasn't committed

Early on, before I was properly disciplined about Git, I lost a meaningful amount of work by doing something that overwrote files I hadn't committed. I don't even remember exactly what — some combination of pulling changes and having local modifications that just got overwritten. It was gone and I had to redo it.

I've never not committed regularly since. The lesson took exactly one experience to stick permanently. Use Git, commit often, write messages that describe what changed. The ten seconds it takes is insurance against hours of redoing work.

The pattern

Looking back at all of these, the pattern is pretty consistent. Every one of them happened because I was moving fast and skipped a verification step. Check before assuming. Back up before changing. Verify after the fact. These aren't interesting rules — they're boring, obvious, and the most important things I've learned from years of breaking things.

You will break things. That's fine. The goal is to break them in ways that are recoverable and to learn something each time. A broken thing you understand is more valuable than a working thing you don't.