"The three pillars of observability" is a marketing frame that got repeated until it became a design principle. It describes three data types somebody can sell you storage for. It describes no property of a system whatsoever.
The real property is narrower and a lot harder: can you answer a question you did not think to ask in advance?
Why the distinction bites¶
A metric is a pre-aggregated answer to a question somebody already had. It is cheap, it is fast, and it will tell you nothing about a dimension nobody thought to label.
Here at AxisOps we see the same failure repeatedly. Your p99 latency is up. The metrics tell you that, and stop. They cannot tell you it is customers on one shard, in one region, on one API version, unless somebody anticipated that combination and added those labels. Had they anticipated it, they would probably have fixed it before you noticed.
Cardinality is why you cannot label everything and have done. High-cardinality labels (user IDs, request IDs) blow metric storage out, so the dimensions most useful for debugging are the ones metrics structurally cannot hold. Rather like keeping a tally chart of how many cars left the car park and then being asked which one had the dent in it.
What answers unanticipated questions¶
Wide structured events. One event per unit of work, carrying every dimension you have as structured fields: user, region, shard, version, feature flags, timings of each phase. Then you query it however you like, after the fact, without deploying anything.
That isn't a third pillar to stand alongside the others; it swallows most of them. Gauges, hardware counters and other continuous state are not per-request events and keep their own path. Metrics are aggregates over events, and traces are events with causal links between them.
What it costs you is storage and ingestion, and sampling is how you manage that. Tail-based for preference, so you keep every slow and failed request and throw away the boring ones.
Where kernel tracing changed the economics¶
The newer in-kernel programmable tracing (eBPF programs attached to tracepoints, kprobes and the networking hooks) made it possible to collect a good deal of this without instrumenting the application at all.
Syscall latency, TCP retransmits, block I/O queue depth, page faults, scheduler run-queue delay: all of it observable from outside the process, with no code change, at an overhead low enough to leave running in production.
That matters most for the things you can't instrument; a vendor binary, a runtime you don't control or a problem in the layer below your code. It's also the answer when somebody says "we can't add instrumentation to that".
What it won't give you is application semantics. The kernel can tell you a syscall was slow. It can't tell you whose invoice run it was part of. The two are complementary, and the sales pitch for each tends to leave that bit out.
What this makes possible¶
Treat telemetry as queryable events rather than three pre-bought answers, and the shape of an incident changes.
The investigation finishes in the same session. Today's debugging loop for anything unanticipated is: form a hypothesis, add a log line, deploy, wait for the problem to recur, read the log, repeat. Each turn is hours at best and days if the fault is intermittent. Query a wide event store and each turn is twenty seconds. An engineer who can test forty hypotheses in an afternoon solves problems that are out of reach when they can test three in a week.
Debugging stops requiring the person who wrote it. When the data is in fields rather than prose log lines, anybody can ask the question. That breaks the pattern where every hard incident waits for one named individual who knows which grep to run.
Performance work becomes evidence-led. With per-request timings across every phase, the slow thing is a query result rather than a committee opinion. We expect most of the commercial value to land here: saying which customers, on which path, are paying for which bit of latency, and therefore which engineering week is worth spending. Energy and hosting cost are on the same curve; you cannot cut what you cannot attribute.
The frontier bit: a system that can be interrogated after the fact without redeployment is one step away from one that can explain itself. Given wide enough events and a way to correlate them, "what is different about the failing requests" is a statistical question over structured data rather than a human reading dashboards. We expect that to be automated within a few years, and the teams who get it will be the ones already storing events rather than pre-aggregated counters.
The protocol does not trap you here. The pricing model does.
Wide events mean volume, and the dominant commercial model charges on ingest and retention: per gigabyte, per host, per million events, per custom metric. Your bill scales with your traffic, so the day you most need the data is the day it costs the most. Teams respond by sampling harder, dropping fields and shortening retention to thirty days, then discover the intermittent fault they are chasing has a six-week period. The pricing has quietly specified the engineering.
Worse, telemetry you paid to ingest frequently cannot be got back out. Query languages are proprietary, dashboards are not portable, export is per-query or capped or billed again, and historical data sits in a store you have no bulk read access to. You have paid, repeatedly, for a dataset you do not possess, and switching vendor means starting your history from zero. Which is why nobody switches.
Then consider what that dataset contains: user identifiers, IP addresses, request paths, feature flags, query parameters. Your observability store is the most detailed behavioural record of your users anywhere in your estate, shipped continuously to a third party, usually with no reviewed retention policy and often to a jurisdiction nobody checked. It is a data-protection surface and almost nobody treats it as one. Scrub at the point of emission, because after that it has already left.
The open side is in better shape than in most of these stories. Columnar storage on object storage, OpenTelemetry for the wire format, Prometheus for metrics and a handful of credible open query engines make a self-hosted wide-event store a real option for a mid-sized estate; the gap is operational polish, not capability. Whatever you buy, insist on an open wire format at emission so the agent is not the lock-in. Insist too on raw bulk export as a contract term you have exercised and a copy of your own events landing in storage you own on the way past.
Here at AxisOps we keep the originals. Telemetry is a record of how your systems and your customers behave, and renting the only copy of it is a strange thing to do with the most useful dataset you own.
The practical test¶
Instead of auditing whether you've collected all three pillars, try this. Take a real incident from the last quarter and ask whether the data you already hold would have answered it without anybody deploying anything.
If the answer is "we added some logging and waited for it to happen again", you haven't got observability. You've got monitoring, which is a different thing and also a useful one.