Outages are rarely sudden
A disk that fills up, a memory leak that eventually triggers the OOM killer, a certificate that expires: almost all of these are visible in metrics well before they cause an outage. The gap usually isn't a lack of data, it's that nobody's monitoring it, or the monitoring exists but the alerts are too noisy to act on.
What to actually track
A reasonable baseline for most servers:
- Uptime and endpoint availability (is the service actually responding, not just is the process running)
- Disk usage, with an alert threshold well before 100 percent, disks rarely fail gracefully at capacity
- Memory and swap usage trends, not just a snapshot
- CPU load relative to what's normal for that server, sudden spikes matter more than absolute numbers
- Certificate expiry dates, tracked separately since they fail silently until the exact moment they don't
- Application and system logs, centralized somewhere searchable, not scattered across individual servers
The alert fatigue problem
Monitoring that pages someone for every minor blip trains them to ignore alerts, which defeats the purpose. The fix is tuning thresholds to what's actually actionable (a CPU spike that resolves in 30 seconds usually isn't worth an alert; one that persists for 10 minutes is), and making sure alerts route to someone who can actually act on them, not a channel that gets checked once a day.
Monitoring without maintenance is half the job
Catching a problem early only helps if there's a scheduled window to act on it. Pairing monitoring with regular, low-impact maintenance windows, where routine fixes and updates happen on a predictable schedule instead of only in response to an alert, is what actually turns 'we saw it coming' into 'it never became an outage.'