Zero to ping, and the decade of doing it by hand

The same jobs keep coming back. Network boot, standing up a backup, getting remote access into a site, building a fleet of machines against a deadline: years apart, each time from scratch, each time with somebody in the room.

That repetition is why there is a control plane.

The jobs that kept coming back¶

Network boot. A machine arrives, it needs an operating system, and the way you do that at any scale is PXE. Setting it up means a DHCP option, a TFTP server, an image and somebody in the room when it does not work.

Backups, on site. Not the running of them, which automates well. The standing up of them: a box, a disk, a network path, a retention policy and a restore somebody has tried.

Remote access. By 2021 we were building our own, out of Raspberry Pis, because the alternative was a vendor appliance per site with a subscription attached.

A Raspberry Pi board wired into a patch panel, lit green, with coloured patch cables around it — A remote-access device built rather than bought, 2021: the commercial parts were in short supply and did not quite do what was needed. It worked. It was also a device per site, each one a thing to patch, key, inventory and eventually explain to whoever inherited it.

And then everybody went home at once.

An office with laptops and monitors laid out across several desks mid-deployment, boxes stacked behind — March 2020, Covid. Machines laid out across a room because that was the only way to get them all built in the time available. The constraint was never the imaging; it was that a person had to be standing in front of each one, and there was no time to write the automation that would have taken them out of it.

March 2020 is the clearest case because the deadline was absolute. The work was not hard. It was that the throughput of the whole operation was one engineer multiplied by the number of hours in a fortnight, and no amount of competence changes that number.

What the repetition is telling you¶

The obvious reading is that we should have automated sooner. That is the wrong lesson, and we had in fact automated: these jobs were Ansible by 2017.

The problem is one layer up. A playbook needs to know what it is running against, and that knowledge lived in exactly the places it always lives: a spreadsheet, a wiki page, a naming convention, somebody's memory. Which meant every one of these jobs started with the same unpaid work: establish what is out there, then act on it. That part had no artefact at the end. Next time, you did it again.

So the bottleneck was never the task. It was that the estate could not describe itself. We have made that argument elsewhere about security, where the defender's advantage is knowing their own ground and almost nobody can claim it. The operational version is the same shape and costs money every week rather than once in an incident.

What we built instead¶

A control plane whose job is to hold the declaration: what exists, what it depends on and what state it is supposed to be in. That control plane is micro-substrate. Three properties fall out of that, and they are what we were buying.

One place to look. Cloud providers and on-premises hardware behind one abstracted driver model, so a virtual machine, a firewall rule, a network and a backup are all resources of the same kind. Not because a unified API is elegant, but because an engineer jumping between four vendor consoles is doing the estate-discovery work again every time.

Zero to ping. One engineer, a laptop, the control plane as a single container image and access to the backups is enough to rebuild an estate from nothing to responding. That is not reasoned from the design: we have run the rebuild, held to the same standard we demand of a backup above, a restore somebody has tried. That is the property the 2015 data-recovery photographs are really about. We got the client's data back off failed arrays using a salvaged donor server, and the reason it was possible is that the state was on the disks rather than in the hypervisor. The control-plane version makes that the normal case rather than a good day.

Everything derives from a base. Hypervisors, virtual machines and container images inherit from common images, so a patch is applied once at the right layer, tested once and cascades. The alternative is the one the mitigation work keeps running into: a fix is available, and finding out which of your hosts needs it takes longer than applying it.

What this does not fix¶

It does not make the first description free. Something has to establish what is out there, and on an estate nobody declared, that is still archaeology.

It moves the dependency rather than removing it. An estate that reconciles against a control plane depends on the control plane. That is why it ships as a container you can run on a laptop and why the backups are the thing that has to be portable. A control plane you cannot run yourself is a vendor console with better marketing.

And it earns nothing on a small estate. Four machines a person understands do not need a declaration; they need a person. The economics turn somewhere around the point where the people who built a thing are no longer the people running it.

The part worth keeping¶

A stack of single-board computers in an open frame, wired into a desktop switch with green patch cables, photographed on a desk at night — Half past three in the morning, February 2016, prototyping the edge platform. The dev boards in this photograph are doing by hand what the edge tier does now, and the only reason we know what that tier should do is that somebody sat up wiring this one, building it from scratch from a concept.

None of the capability in the control plane was designed in the abstract. Each piece of it is a job somebody did manually enough times to learn what it required. That is the opposite of how infrastructure products are usually specified.

That is the argument for keeping your hands on the work. A team that has never built the thing by hand does not know which parts of it matter, and will automate the easy half.