Observability stack
Metrics, logs and traces wired together well enough to answer 'why' during an incident, not just 'something is wrong'.
The problem
Metrics, logs and traces that live in separate tools with no shared context force engineers to manually correlate timestamps during an incident. This lab wires them together around a consistent set of labels so a slow request can be traced end to end.
Architecture
- Prometheus for metrics with a consistent labeling convention across services
- OpenTelemetry instrumentation exporting traces correlated to request IDs
- Loki for structured logs, queryable by the same labels used in metrics and traces
- Grafana dashboards built around SLO burn rate, not raw infrastructure metrics
Technologies
Design decisions
SLO burn-rate alerting instead of static threshold alerts
Static thresholds either fire too often or miss slow degradations. Burn-rate alerting reflects actual user impact.
Consistent label schema enforced across metrics, logs and traces
Without shared labels, correlation during an incident becomes manual and slow.
Lessons learned
- The hardest part of observability isn't the tools — it's agreeing on a labeling convention early and enforcing it
- Dashboards built around SLOs get used during incidents far more than infrastructure-metric dashboards do
Working through something similar?
This lab reflects our own reference work, not a client engagement — but the approach transfers directly.
Start a technical conversation