Taking control — and discovering you're not alone

You rent the box, and a minute later there’s an email with an IP address and a root password. The first job is to get inside it. The second, which I didn’t realise was urgent until I looked, is to lock it down — because the box was already being probed before I’d finished setting it up.

Getting in

The first connection is from the Windows PC:

ssh root@<the IP from the email>

It asks for the root password once, then gives a prompt on a fresh Ubuntu 24.04 box. That’s the appeal of a VPS — you’re on the machine directly, not clicking through a control panel.

The first thing worth doing is replacing password login with an SSH key. I copied the public half of a key pair to the server’s authorized_keys and confirmed I could log in with the key before disabling passwords. Confirm the new method works before turning the old one off — if you lock yourself out of a box you can only reach over the network, the only way back is the provider’s rescue console.

This is also where I set a rule I kept for every risky change afterward: keep a second SSH session open. Make the change in one session and test it in a second session that’s already connected, so if the change breaks access the open session is still there to undo it.

The setup that felt like enough

With key login working, I did the usual new-box tasks. Ran apt update && apt upgrade. Installed Nginx. Got the speed test running behind it. Issued a Let’s Encrypt certificate with certbot for SSL. It worked and it looked finished.

It wasn’t. I just hadn’t looked at the right log yet.

The log that changed the approach

Later, while looking at something unrelated, I opened the Nginx error log and found it was 16 MB. For a personal site almost nobody knew about, that’s far too large. Counting the failures in it gave one number:

# grep -c 'password mismatch' /var/log/nginx/error.log
67740

67,740 failed authentication attempts, from one IP address, against the HTTP Basic Authentication prompt protecting the site at the time — the login Nginx enforces with auth_basic, not something Nginx does on its own.

Nothing had been breached — the attempts failed. But the point is that a server on a public IP is under continuous automated attack from the moment it exists. Not because anyone targeted me — the entire IPv4 range is scanned constantly by bots trying default passwords and known exploits against anything that answers. Those attempts had been arriving the whole time the box “felt finished.” I just hadn’t been watching that log.

That’s when the setup turned into a hardening pass. “It works” and “it’s safe to leave on the internet unattended” are two different states, and I was only in the first one.

Closing the doors

The goal was to remove whole categories of attack before they reached anything that could be brute-forced.

Turn off password login at the protocol level. If SSH won’t accept passwords at all, password-guessing against SSH becomes impossible rather than just slow. Key-based auth only. I thought I’d already done this — I was wrong, which I’ll come back to.

Make the raw IP go dark. The attacks were hitting the bare IP address, not the domain. By default, Nginx answers a request to the raw IP by falling back to the first site it knows about, which routes scanner traffic into the real site’s auth. The fix is a catch-all default-server block that matches anything not addressed to the proper domain and returns Nginx’s non-standard 444, which closes the connection without sending an HTTP response. After that, a bot scanning the IP range gets nothing back. This reduced most of the log noise.

Add fail2ban, and verify it matches. fail2ban watches the logs and bans IPs that pass a failure threshold. The lesson here is worth more than the tool: a jail that loads cleanly but matches zero log lines gives you no protection while looking like it does. The filter has to match your actual log format, and the way to check is to test it against a real log:

# fail2ban-regex /var/log/nginx/error.log \
    /etc/fail2ban/filter.d/nginx-http-auth.conf
...
Lines: 67740 lines, 0 ignored, 67740 matched, 0 missed

I ran it against the log that started this — it matched all 67,740 lines, confirming the pattern was right. Loading without error is not the same as working, so verify it.

Add swap on the small box. The 1 GB server had no swap. With no swap, a memory spike leaves the kernel’s out-of-memory killer to terminate one or more processes to recover memory. A small swap file turns that into a brief slowdown instead of losing a service without warning.

The hardening that looked done but wasn’t

Back to the SSH password lockdown, because it’s the most useful lesson in the pass.

I had written a config file to disable password authentication. The setting was correct, and I moved on assuming SSH was key-only. It wasn’t — and I only found out by checking what the running SSH service actually reported, rather than what my config file said.

The cause is specific to this provider, but the lesson is general. SSH reads several drop-in config files that load in order, and when two set the same option, load order decides which wins. The provider ships its own SSH configuration, and one of its settings overrode mine. My setting was correct and completely inactive, sitting underneath theirs.

The fix was to place my hardening in a drop-in whose load order gave it precedence over the provider’s defaults, and one that survives a rebuild. But the fix matters less than the lesson: don’t assume a configuration change took effect because you wrote it — confirm it against what the running service reports. SSH prints its effective settings directly:

# sshd -T | grep -E 'passwordauthentication|permitrootlogin'
passwordauthentication no
permitrootlogin prohibit-password

That output is the running truth, assembled from every config file in load order — not the contents of one file. When I’d first “disabled” passwords, this command would have said passwordauthentication yes. “I changed the setting” and “the setting changed” are two different things.

What “done” meant after the pass

After the pass: password guessing against SSH impossible with password auth off; the raw IP silent to scanners; brute-force attempts banned after a few tries; memory spikes cushioned by swap. The log noise dropped to almost nothing, which was the proof the doors had closed.

None of these measures is sufficient on its own. Together they reduce the opportunities an automated attacker has to interact with the server at all — fewer surfaces answering, and the ones that remain answering only to keys and clean requests. That doesn’t make the box unbreakable, but it moves it from “working, and under constant probing” to “working, and not worth a scanner’s time,” which is the realistic goal for a box on the public internet.

Next was putting the first real service on it properly — the speed test that started the project.