Drive caddies pulled out of a chassis used to be a weekly sight round here. While we were moving them, the numbers underneath the storage hierarchy changed.
The gap that shaped everything¶
For about thirty years the memory hierarchy had a cliff in the middle of it. RAM access was tens of nanoseconds. A spinning disk seek was somewhere around ten milliseconds, roughly five orders of magnitude slower. The difference between reaching for something on your desk and driving to another city for it.
Essentially every piece of systems design from that era is a response to that cliff:
- Buffer pools and page caches, to avoid going to disk at all.
- Log-structured writes, because sequential disk access was tolerable and random access was catastrophic.
- B-trees tuned to a block size, to minimise seeks per lookup.
- Query planners that count I/Os, because I/Os were the only thing that mattered.
None of that was wrong. It was correct engineering against a real constraint.
What NVMe did¶
An NVMe SSD hasn't got the cliff. Latency drops to tens of microseconds, a few hundred times slower than RAM rather than a hundred thousand times. And crucially, random access stops being special; the seek penalty that justified half the designs above simply isn't there any more.
That moves where the cost sits. When the device answers in 20µs, the kernel block layer, the syscall boundary and the filesystem stop being rounding errors and become a meaningful fraction of the total. Software written to hide a 10ms device is now the slow part. You've optimised the drive to another city and left the car on the drive with a flat tyre.
Which is why SPDK and userspace drivers exist, and why "just add another cache layer" has stopped being automatically correct. Our expectation is that the kernel grows a submission path built for this rather than ceding the fast case to userspace entirely. The current interfaces were designed when a syscall was free relative to the device, and that premise is gone.
Existing systems get faster; more importantly, designs that were never worth building become practical.
Databases are the clearest case. Much of what makes a serious database hard to operate is machinery for keeping the working set in RAM: sizing the buffer pool, the hours tuning it, the cliff when your data outgrows it, the capacity planning that exists because exceeding memory meant falling off a hundred-thousand-fold performance edge. With storage a few hundred times slower than RAM instead, that edge becomes a slope, and "does it fit in memory" stops being the question your whole architecture is organised around.
What that changes: a mid-sized organisation gets to hold its full history online. The analysis that today requires a separate warehouse, a nightly export and a specialist team can run against the operational store, because scanning cold data is no longer something you must avoid. A hospital trust can query ten years of records on one box. That is a different set of questions being answerable at all by people without a data engineering budget.
For engineering, the toil that goes is cache tuning, and with it an entire family of staleness and invalidation bugs that only existed to work around the disk. Less code, fewer failure modes, a system you can reason about.
The energy argument is substantial too. A rack of spinning disks spends real power keeping platters moving so that a seek is merely slow rather than impossible. Flash idles at nearly nothing and answers faster. Collapsing the hierarchy means fewer machines holding fewer redundant copies of hot data to compensate for a slow tier: less hardware, less cooling, a smaller bill.
Betting on the next tier down¶
Persistent memory is the one to watch and the one to be careful about. Byte-addressable, non-volatile, sitting between DRAM and flash: it would collapse the hierarchy further still, and whole research programmes are assuming it. The first products are announced and due this year, and the research is real.
Our view is that the engineering will be fine and the economics are the risk. This tier currently comes from essentially one vendor on one roadmap, needs specific memory controllers to use properly and asks you to rewrite your durability logic around a programming model nobody else implements. Any one of those is survivable. Together they mean a product whose core design assumes persistent memory is a product betting its architecture on one company's continued enthusiasm for a line it can discontinue without consulting you.
So: build the research, use the hardware and keep your durability model able to fall back to flash. The lesson we would write down now rather than learn later is that a technically excellent thing can lose on economics and ecosystem, and that would be no verdict on the engineering. The ideas transfer; a lot of this work is already pointing at coherent interconnects over PCIe, which is the version with more than one vendor in it.
Who controls the roadmap¶
Storage looks like a commodity and is not.
NAND and DRAM come from a handful of manufacturers, and the fast-interconnect and persistent-memory specifications are steered by a smaller handful still. Expect the capability to arrive first as a cloud instance type: the latency you want, available to rent, months or years before you can buy the part for your own chassis, with the interesting features exposed only through one provider's API. That is how a hardware advantage becomes a hosting advantage. The hyperscalers already buy flash at prices and volumes nobody else is offered, so the cost-per-IOPS gap between renting and owning widens even when the technology is open.
The privacy consequence of cheap, fast, dense storage is that nothing gets deleted. Retention policies have always been enforced in practice by storage costing money. When a decade of logs costs less than the meeting to discuss deleting them, the default becomes keep everything, and every one of those stores is a subpoena target and a breach waiting for a configuration error. The decision to make scans cheap is also a decision to make total recall affordable, and somebody should be choosing that on purpose.
Here at AxisOps we buy the drives and own the box. The collapse of the hierarchy is the best argument for self-hosting in a decade: the hardware that used to require a storage array and a consultant now fits in a 1U server.
The practical check¶
If you maintain anything with a cache in it, the question is whether that cache is still earning its keep or whether it's a monument to a device you no longer own. The answer is measurable, and it's frequently no.
Caching costs you something: invalidation, memory and a whole family of staleness bugs. Against a 10ms seek it was obviously worth paying. Against 20µs it's sometimes a straight loss, and you're maintaining the complexity for nothing.