What Good Observability Actually Looks Like (And Why Most Teams
Don't Have It)

The most common way an engineering team discovers their system is broken is through a customer support ticket. The second most common way is a spike in error rate. Both of these are lagging indicators — they tell you the failure already happened. The damage is done.

Most teams call this 'monitoring.' Some of it is. But monitoring that only tells you what broke after it broke isn't observability — it's an incident log with a notification system bolted on.

Observability, in the sense that actually matters operationally, is the ability to ask arbitrary questions about system behaviour without knowing in advance what you're going to ask. It requires the right data, at the right granularity, available in the right shape. Most teams have dashboards. Very few have observability.

Lagging vs leading indicators

A lagging indicator measures the outcome of something that already happened. Error rate, p99 latency, failed payment count — these are all lagging. They're useful for understanding what happened and why. They're not useful for preventing the next one.

A leading indicator measures something that predicts a future state. Database connection pool saturation predicts query timeout spikes. Memory growth rate predicts OOM crashes. Queue depth predicts worker processing lag. These metrics are harder to instrument and harder to threshold correctly — but they're the ones that let you intervene before the user-visible failure.

What most teams get wrong about the three pillars

Logs, metrics, and traces are the standard framework for observability. Most teams have all three. Most teams are still flying partially blind. The problem isn't the presence of the pillars — it's how they're used.

1. 01Logs: The common failure is logging too much unstructured noise and too little structured signal. A log line that says 'processing request' is noise. A log line that says { event: 'payment.initiated', userId, amount, provider, correlationId } is signal. Structured logs can be queried. Log lines cannot.
2. 02Metrics: The common failure is tracking counts when you should be tracking distributions. 'Average response time' is almost always misleading — a system with p50 of 50ms and p99 of 8,000ms looks fine on average and is broken in practice. Histogram percentiles tell you what your slowest users are experiencing.
3. 03Traces: The common failure is instrumenting the happy path and forgetting the error paths. A trace that only shows successful requests tells you nothing about why requests are failing. Trace the failures with the same granularity as the successes.

What on-call engineers actually need at 2am

The test of an observability setup is not how good it looks in a post-mortem. It's how useful it is to an engineer who was woken up fifteen minutes ago, doesn't have full context on the change that just deployed, and needs to diagnose whether to roll back or keep investigating.

That engineer needs three things: a clear signal of what is wrong (not a wall of amber alerts, all of which are 'elevated'), a way to narrow down where in the system the problem is, and enough granularity to tell the difference between a symptom and a cause.

This means your alert fatigue problem is an observability problem. When everything alerts, nothing is actionable. When you have to open five dashboards to correlate what's happening, the system isn't observable — it's just instrumented. The signal has to be clear enough that the on-call engineer can distinguish 'this is serious, roll back now' from 'this is elevated, monitor and investigate' from 'this is noise, can wait until morning.'

Building towards proactive observability

The path from reactive monitoring to proactive observability is not a tool migration. You can do it with Datadog or with open-source Prometheus and Grafana or with any combination in between. The tools matter less than the discipline.

Start by auditing your existing alerts. For each one, ask: the last three times this fired, what did the on-call engineer do? If the answer is 'checked a few things and dismissed it,' that alert is noise — either its threshold is wrong or it's measuring something that isn't actionable. Remove it or fix it.

1. 01Identify the two or three failure modes that have caused your worst incidents. For each one, find or instrument the leading indicator — the metric that was trending in the wrong direction before the failure became visible.
2. 02Move from logging events to logging state transitions. 'Payment initiated' is an event. 'Payment state: initiated → processing → failed, reason: provider_timeout, attempt: 2/3' is a state transition. State transitions are debuggable.
3. 03Instrument your dependencies separately from your application. If your database is slow, you want to know that the database is slow — not just that your API is slow. Dependency-level tracing makes this distinction clear.
4. 04Define what 'healthy' looks like in numerical terms for your most critical flows. If you can't define it, you can't measure it, and you can't alert on deviations from it.

The last step — and the one most teams skip — is validating your observability setup before you need it. Run a chaos experiment in staging. Kill a dependency. Saturate the connection pool. Watch whether your dashboards show you what's wrong before the errors appear. If they don't, you know what to fix before you find out in production.