Five Pis, a desktop switch and an unreasonable quantity of green patch cable, photographed at some unflattering hour.
The compute is negligible by any 2016 standard; about as much total RAM as one modest server. That was never the point.
What a real cluster teaches¶
Running a distributed system on one machine with processes on different ports teaches you the happy path and nothing else. It won't teach you:
- Partition. Pull a cable out and the system has to have an opinion. Pulling a cable out is very easy; simulating it on one host is fiddly enough that people quietly skip it.
- Asymmetric failure. A node that can receive but not send is a different animal from a node that's down, and almost every naive implementation treats them as the same thing.
- Clock behaviour. Five machines with no PTP and indifferent NTP drift relative to one another in a way a single host simply doesn't.
- Boot order. Nothing ever comes up in the order the diagram implies.
Those are the properties that make distributed systems hard, and they're the ones a single-host test quietly elides.
The ARM part, which is going to matter¶
The Pi is a toy for this purpose and the ARM cores are slow. What they are not is x86, and anything that only builds on x86 announces itself on the first attempt. Endianness assumptions, inline assembly, unaligned access that x86 forgives and some ARM instruction sequences still fault on, a binary dependency with no port: all of it surfaces in the first hour on a £30 board rather than in the first week of a migration.
We will take the bet that this is the useful part. ARM in the server room is coming, and the argument is performance per watt rather than performance. The Pi 2, the size of a credit card, is doing real work on a phone charger, and a rack full of cores built to that constraint changes the electricity bill, which is the bill that limits a datacentre in the end. Somebody will ship a serious ARM server part and sell it on energy rather than single-thread speed, and the people who can take it will be the ones whose software already builds clean on a second architecture.
That is a cheap option to hold. Keeping your build working across two architectures costs a CI runner and some discipline; acquiring the capability after the hardware arrives costs a project.
The wider benefit is access, and it is larger than the hobbyist framing suggests.
A cluster used to be a thing you needed an employer to see. Partition behaviour, quorum loss, split brain: the engineers who understood those were the ones who had worked somewhere with a datacentre. £150 of boards puts the same education in a bedroom, which means the next generation of distributed systems engineers will not be drawn only from people who got the right graduate job. We think that shows up in the quality of the software within a decade.
The same arithmetic travels. A school, a clinic or a field station in a place with expensive power and bad connectivity can run real local services on hardware that costs less than the annual bill for a hosted equivalent, on a solar panel if it has to, and keep working when the link drops. That is not a compromise version of computing; for a lot of the world it is the only version that is dependable.
That accessibility depends on supply rather than software.
The Pi is affordable because one foundation negotiated one SoC deal with one vendor and sells at near cost. There is no second source. The boards are already hard to get in volume at points in the year, and a board you cannot buy is not a platform. If ARM in the server room arrives the way we expect, the parts that matter will be designed by a handful of companies and, more to the point, fabbed by two or three. The first customers in the queue for any constrained process node are the companies buying a hundred thousand units. Everyone else waits. Cheap, inspectable, generally available ARM hardware is a market condition rather than a law of nature, and market conditions revert.
The architecture licence is the other chokepoint. ARM is one company's instruction set, licensed on its terms, and a world where the energy-efficient future runs on it is a world where that company sets a price for participating. The open instruction sets being worked on now are years behind and worth following for exactly that reason.
The privacy angle is the small nodes themselves. The same £30 computer that lets a clinic run its own records is the one that makes a camera with a network stack cost nothing. The cheap-sensor build-out arriving over the next few years will put far more surveillance capability in far more hands than any policy process is prepared for. The technology is neutral and the deployment will not be.
Here at AxisOps we buy hardware we can own, keep two architectures in the build and prefer the version of any system that runs on a board we can replace from a different supplier. A platform with one source is a dependency wearing a friendly face.
Would we do it again¶
Yes, and we have, with better hardware. A physical cluster costs so little that the alternative (reasoning about partition behaviour off a diagram) is a false economy. You can hire a van and move house, or you can draw the van on a whiteboard and hope the sofa fits.
The photograph is here because the cabling is the real part. The whiteboard version of this has five neat boxes and five neat lines.