Ask most engineering teams if they have observability, and the answer is yes — there's a Grafana instance, a handful of dashboards, some alerts in Slack. Ask when someone last opened those dashboards outside of an incident, and the answer is usually: rarely, if ever.
That gap is the actual problem. Observability isn't a set of dashboards that exist — it's the ability to answer "why" quickly when something breaks. Those are different things, and building the first doesn't guarantee the second.
Infrastructure metrics answer the wrong question
CPU usage, memory usage, disk I/O — these are cheap to collect and easy to put on a dashboard, which is why they're usually where teams start. But they rarely answer the question that matters during an incident: is a user's request failing, and why?
Infrastructure metrics tell you about the machine. They don't tell you about the request. A CPU graph that looks fine can coexist with a service returning errors to 20% of users, and a CPU spike can be completely harmless if it's not affecting response times or error rates.
What to measure instead: the four signals tied to user experience
For most services, four signals — sometimes called the "golden signals" — get you most of the way to a useful incident response:
- Latency — how long requests take, split by success and failure (a slow error is a different problem than a slow success)
- Traffic — request volume, to give the other signals context
- Errors — rate of failed requests, ideally broken down by cause
- Saturation — how close a resource is to its limit, measured against what actually constrains that resource (queue depth, connection pool usage — not just raw CPU)
These map far more directly to "is a user having a bad experience right now" than infrastructure metrics do, and they're what should drive alerting.
SLOs turn signals into a threshold that means something
Metrics without a target are just numbers. An SLO — "99.5% of requests complete in under 400ms over a rolling 28 days" — turns latency into a number you can alert on meaningfully: not "latency went up" (which happens constantly and is mostly noise) but "we are burning our error budget fast enough that this trajectory is a problem."
Burn-rate alerting (alerting based on how fast you're consuming your error budget, not on a static threshold) is what actually reduces alert fatigue, because it only fires when the trend is bad enough to matter.
Traces are what makes "why" answerable across services
Once a request touches more than one service, metrics alone stop being enough to explain a slowdown — you can see that latency went up, but not where in the chain. Distributed tracing, with a consistent request ID propagated across services, is what lets you go from "checkout is slow" to "checkout is slow because the inventory service's database query is slow" in minutes instead of hours of manual log correlation.
The actual test of observability
Here's a useful test: pick your last real incident, and ask whether your current dashboards and alerts would have told you the root cause within five minutes, without anyone needing to SSH into a box or grep through logs by hand. If the honest answer is no, the gap isn't more dashboards — it's the wrong signals, or no way to correlate them across services. That's a design problem, not a tooling problem, and it's worth treating it as one.
Related service