Knowing when something breaks — before I go looking

By this point there were several things running that I’d be annoyed to have quietly fail: the tunnel, the DNS blocking, the file share, the public box itself. The file-sharing part ended on the gap that made obvious — a service can look healthy while serving nothing, and I only found out by stumbling on it. What I needed was for the setup to tell me when something was wrong, instead of me discovering it by accident. That’s what this part is: giving the whole thing a sense of its own health, and a way to reach my phone the moment that health slips.

Two different questions

Monitoring answers two separate questions, and it’s worth keeping them apart.

The first is “how are things, right now and over time?” — the slow, browsable view. How much memory is the home box using this week, is the disk filling up, how warm is it running, how much work is the public box doing. This is the dashboard — Grafana, drawing on readings collected by Prometheus: charts I can look at when I’m curious or chasing a hunch, showing what normal looks like so that abnormal stands out.

The second is “is anything actually wrong this second?” — and that one I don’t want to have to go and look for. If the public box falls off the internet, or the home box stops answering, or a disk is about to fill, I want to be told, on my phone, without having opened anything. The dashboard is for when I choose to look; the alert is for when I’m not looking, which is most of the time.

The setup does both, and the second is the one that earns its keep.

How it’s put together

The pieces are ordinary and the design is simple. A small collector on each machine — the home box and the public box — reports that machine’s vital signs. A central service gathers those readings on a schedule and keeps a rolling history, so there’s both a live picture and a record of the last while. A dashboard layer turns that history into the charts. And an alerting layer watches the same readings against a handful of thresholds and sends a message — to Telegram, in my case — when one is crossed.

All of it runs on the home anchor, in isolated little packaged environments so each piece stays tidy and restartable without disturbing the others. The one deliberately careful detail is how the public box reports in: its collector answers only over the tunnel, not on the public internet. Health readings are exactly the kind of thing you don’t want to hand to any passing scanner — how busy a box is, when it’s under load — so the reporting rides the encrypted tunnel like everything else private in this build. The public box exposes nothing extra to the world to be monitored; it reports inward, through the same private channel.

The alerts that matter

I kept the alerts few and meaningful, because an alert that fires too often is an alert you learn to ignore — and an ignored alert is worse than none, since it trains you to swipe away the one that finally matters.

So the list is short and each one means something I’d genuinely want to act on: the public box is unreachable; the home box is unreachable; a machine is working unusually hard for an unusually long time; a disk is getting close to full; the DNS blocking has stopped answering. Each waits a little before firing, so a momentary blip doesn’t wake my phone — a thing has to be actually wrong for a couple of minutes, not just briefly noisy, before I hear about it. When one does fire, and again when it clears, a message lands on my phone. That “and again when it clears” matters as much as the alert itself: knowing something recovered on its own is as useful as knowing it broke.

And I tested them the way I’ve come to test everything in this build — by breaking the thing on purpose. I deliberately stopped the public box’s reporting to confirm the “unreachable” alert actually fired, and that the all-clear followed when I started it again. An alert you haven’t seen fire is a guess, not a safety net. This is the same lesson as checking the fail2ban filter really matched, or recreating the empty-share failure on purpose: an untested safeguard is only a hope.

The night it proved itself

The payoff came without warning, early one morning while I was asleep.

The public box locked up completely for about twelve minutes and then came back on its own. I didn’t see it happen. What I saw later was the record: an alert had fired when the box went unreachable, and a second message had followed twelve minutes later when it recovered — the whole event captured, timed, and resolved before I’d even woken up. Without the monitoring I’d have had no idea it had ever happened.

The interesting part was what the box’s own logs showed about why. The evidence pointed to something wrong beneath my server — not in anything I’d installed or configured, but in the physical host it runs on, the machine underneath the virtual one. I raised it with the provider, and their answer confirmed it: another customer on the same physical hardware had spun up a flood of servers at once and overloaded the shared host; my box was simply a bystander that got starved of resources for a few minutes while their automatic systems caught up. Nothing on my side was at fault, and nothing on my side needed fixing.

That’s the whole case for monitoring in one incident. Without it, the lockup would have been invisible — a twelve-minute gap in the middle of the night that I’d never have known about, and would have had no way to explain if I’d somehow noticed. With it, I had the timing, the recovery, and enough evidence to hand the provider so they could find the real cause. Monitoring didn’t prevent the problem; nothing on my side could have. What it did was turn an invisible, unexplainable blip into a documented event with a known cause — and told me, definitively, that the thing that broke wasn’t mine. Knowing a problem isn’t yours to fix is worth almost as much as fixing it.

What this bought

With this in place, the setup stopped being something I had to remember to check. It watches itself now, and stays quiet until there’s a reason not to be. Most days I never hear from it, and the silence is the information — it means everything that should be up, is. On the rare day a message arrives, it’s about something real, and it arrives before I’ve gone looking rather than after I’ve stumbled on it.

That changed how it felt to run all this. Up to here, trusting the setup meant periodically poking it to make sure it was still fine. Now trust is the default and doubt is the exception, flagged to me by the system itself. For something meant to run unattended for months, that’s the difference between a hobby you have to babysit and infrastructure you can leave alone.

The next part is the one this whole architecture was built for in the first place — reaching the thing at home I actually needed from outside, safely, with the front door to the house still firmly shut.