Immutable infrastructure: don't modify a running system. Build a new image, deploy it, throw the old one away.
The technical mechanism is simple enough. The benefit comes from somewhere else entirely.
The problem it solves¶
Configuration drift. A server that's been patched, tuned and fixed by several people over three years is in a state nobody has written down and nobody can reproduce.
That matters most at the worst possible moment. The machine dies, and you discover that the thing you can rebuild isn't the thing that was running. The difference is all the accumulated undocumented fixes that made it work in the first place.
Configuration management (Puppet, Chef, Ansible) attacked this by continuously converging a machine towards a declared state. That helps a great deal, but it leaves a residual problem: convergence only covers what the manifests mention. Anything outside them persists invisibly, and so does anything a person did by hand between runs.
Immutability removes the residue by removing the machine's history. If the machine never changes after boot, there's no drift to accumulate.
Why the benefit is social¶
The technical half (build an image, deploy it) is the easy half, and plenty of organisations have it and still have drift.
The property that delivers the value is a commitment: nobody logs in and changes anything, ever, including at three in the morning during an incident.
That's a hard commitment to keep, because at three in the morning the fastest route back to service is frequently a manual change. Keeping the discipline means accepting a slower fix in the moment to preserve a property that pays later. It's the same bargain as not taking the shortcut across the lawn; one person doing it once is nothing, and that's how the path gets worn.
Where organisations fail at immutable infrastructure, it's nearly always here
and not in the tooling. One emergency sudo and the guarantee is gone, with
nothing to announce it. It is the same failure mode as the declarative model
generally.
What it costs you¶
- Deployment has to be fast and boring, or the discipline won't survive. If shipping a one-line fix takes forty minutes, people will log in.
- Debugging changes shape. You can't leave a debugger attached to something you're about to replace, so the data has to come out of the system while it runs. Plan for that before you need it, not during.
- State has to live somewhere. Immutability applies to compute. Databases and storage are as mutable as they ever were, and pretending otherwise is how people end up with an immutable web tier and an undocumented pet database in the corner.
- Image sprawl is real and wants its own lifecycle management.
What it makes possible¶
Once a running system is a build artefact rather than a history, things that were previously projects become routine.
Recovery stops being a rehearsal nobody has run. The disaster-recovery plan and the ordinary deploy become the same procedure, executed daily. That is the only version of DR that works, because the only tested path is the one you use all the time. A business that can rebuild its entire serving tier from source in twenty minutes does not need a site-failover runbook with eleven manual steps. It needs the pipeline it already has, pointed somewhere else.
Patching becomes cheap enough to do constantly. The reason estates run months behind on security updates is that patching a long-lived machine carries real risk of breaking it in a way nobody can undo. Rebuild instead and the risk moves to a place you can test, with a rollback that is "deploy the previous image". Teams that do this patch weekly rather than quarterly, which is worth more than most security products you could buy with the same money.
The audit question gets a real answer. "What is running in production, and what source produced it?" is answerable by looking at an image digest and a commit. On a drifted estate that question takes a week and the answer is a guess. For anyone in a regulated sector this converts a recurring compliance ordeal into a query.
And the one engineers feel immediately: the class of bug that only exists on one machine disappears. No more "it fails on web-03 and nobody knows why", no more three-year-old kernel module somebody installed by hand, no more afternoons diffing two servers that are supposed to be identical. That toil goes to zero, and nobody misses it.
All of those gains now depend on the image pipeline.
Immutability means your production estate is defined by an artefact built somewhere. Ask where, and the answer for most organisations adopting this pattern is: a hosted build service, from a base image pulled from a public registry, stored in a cloud provider's own image format, deployed by that provider's own machine-image API, through a managed pipeline product.
Now ask the only question that matters about any dependency: if that account were closed tomorrow, could you still produce a bootable machine? For a great many estates the answer is no, and the reason is not the application code. It is that nobody holds the base image, nobody has the build running anywhere else and the image format is the provider's.
This is a worse lock-in than the mutable estate it replaces, in one specific way. A hand-tuned server was undocumented, but it was yours, and you could at least take the disk. An estate whose every machine is derived from a pipeline you rent is a tenancy. The provider has done nothing wrong; you simply no longer possess the means of production.
Treat the pipeline the way you would treat a single-sourced component. That means a base image you build yourself from a distribution's published packages, mirrored locally. A build that runs on a machine you own as well as in the hosted service, and that is proven to run there, not assumed to. An output format that boots under plain KVM and not only in one cloud. And the whole thing exercised on a schedule, because an untested rebuild path is a promise, not a capability. It is the same reason you test the fire alarm rather than admiring the box on the wall.
Here at AxisOps we build images from distribution packages, in a container we control, and we boot the result on our own hardware before it goes anywhere else. Rebuilding without the vendor is a practical requirement rather than a preference. It marks the difference between owning infrastructure and renting the right to run it, and immutability is the technique that makes the distinction concrete: your estate is now exactly as portable as your build is.
Keep the promise or keep the complexity¶
It's a good idea whose benefit is roughly proportional to how strictly you keep it, and it degrades to ordinary infrastructure plus extra build steps the moment you don't.
Adopt it if you'll hold the line, and build the pipeline somewhere you can still reach it when the invoice changes.