On 29 March 2024 Andres Freund reported to the Openwall security list that xz-utils 5.6.0 and 5.6.1 contained a backdoor (mjtsai's roundup collects the original disclosures). Red Hat rated it Critical and scored it 10.0.
Freund is a PostgreSQL developer at Microsoft. He wasn't auditing xz. He noticed that SSH logins on a Debian testing box were taking about half a second longer than they ought to, and he followed it, which is the engineering equivalent of pulling over because the engine note changed.
Why this one is different¶
Most supply-chain incidents are opportunistic: a typosquatted package, a compromised credential, a dependency confusion trick. Crimes of convenience against automated systems.
This was a multi-year social engineering campaign against a person.
The pattern, as reconstructed: a project that felt unmaintained with one exhausted maintainer, sustained pressure from apparently-helpful new contributors complaining about the pace of releases, a gradual handover of trust and eventually commit and release-signing authority. The backdoored tarballs were signed legitimately, by the account that had been given the right to sign them.
No part of that is a software vulnerability. Every control that failed was a social one.
What it says about the dependency graph¶
xz is compression. It's in the boot path of most Linux distributions, in kernel module loading, in package management. It's also, functionally, one person's unpaid evening project.
That combination (critical to everything, maintained by nearly nobody) is the ordinary condition of a large fraction of the dependency graph, not an exception. The RIPE open-source working group thread titled "xz incident shows the need for structural change" is a fair summary of the response from people who think about this seriously.
What would have helped¶
Which controls would and wouldn't have caught this:
Would not have caught it. SBOM, because the component was legitimate and the version was current. Signature verification, because it was correctly signed. Dependency pinning, because pinning to the backdoored release leaves you pinned to a backdoor. Most CVE scanning, because there was no CVE until after the discovery.
Might have caught it. Reproducible builds, because the backdoor was in the release tarball and not in the git tree. Closing that gap is what reproducibility is for, and it's the single highest-value control to come out of the whole incident.
Did catch it. One engineer with the habit of chasing a performance anomaly instead of shrugging at it. Nobody can mandate that.
What we shipped this morning¶
The detection problem is the one we could write code for, so we did:
an Ansible role at
github.com/axisops/CVE-2024-3094,
MIT, and you are welcome to it. It reads the installed liblzma5 and
xz-utils versions, scans the library directories for liblzma.so.5*,
resolves symlinks so each real file is counted once, reports any 5.6.0 or
5.6.1 and says whether sshd links liblzma at all. Dry run by default;
nothing changes without --apply.
Two details in it are worth more than the rest of the code. Debian's fixed
package is versioned 5.6.1+really5.4.5-1, which a naive version comparison
reads as 5.6.1 and flags as backdoored. So the parser has to understand
+really or it cries wolf across an entire Debian estate. And after the upgrade
the script names any affected file the package manager does not own, built from
source into /usr/local or shipped inside an application bundle, rather than
reporting the host clean. An upgrade fixes what the package manager controls,
and the honest output is a list of what it does not.
The backups it takes are copies of a backdoored library, kept as evidence, and the README says in bold not to restore them. Scanning a host also says nothing about container images, chroots or application bundles, whose fix is a rebuild rather than a package upgrade.
What the role cannot do is the part this incident was about. It finds a known-bad version of a known-bad library, because by this morning somebody had told us which one to look for. Freund's half-second is not a thing a script can be taught.
What this incident makes possible¶
An odd thing to say about a 10.0, but a disaster that gets fully reconstructed in public is worth a great deal, and this one will be.
The whole attack is visible. The mailing list pressure, the commits, the build system trick, the signed tarballs: all of it is on the record, which means every distribution, every vendor and every engineer gets to study a complete professional, patient supply-chain compromise from the inside, for free. To this day it is publicly unattributed. Nobody gets that from a breach at a commercial vendor, where the incident report is three paragraphs and a lawyer. Open source just paid a large bill and received the full post-mortem in exchange. Use it.
The practical consequence we expect within a couple of years is that building from a release tarball stops being normal. The tooling to build distribution packages straight from a signed git tag, reproducibly, is nearly all in place; what was missing was a reason urgent enough to finish it. There is now one. The engineer's day changes concretely: a discrepancy between the published binary and an independent rebuild becomes an alert rather than an unanswerable question, and the class of attack that hides in the gap between the repository and the tarball closes for good.
The wider benefit is that this is the rare security improvement that helps everybody equally. A small company, a hospital and a government all get the same verifiable artefact, because the verification happens in public and costs nothing to consume.
The chokepoint for xz was a single unpaid maintainer with no colleague, no deputy and no capacity to say no to help. Not a registry, not a signing key, not a cloud. A person.
And notice who caught it. Not a vendor's threat intelligence team, not a scanner, not any of the paid supply-chain security products whose marketing will be rewritten around this incident by Friday. One engineer, working on something else entirely, noticing 500 milliseconds. Every expensive control in the industry had the backdoored package and said it was fine.
That matters for where this goes next. The response from the large platforms will be to offer you a curated, scanned, vendor-blessed package feed: their registry, their attestations, their judgement about what you may install. It will be better than nothing, and it is also the mechanism by which a handful of companies end up deciding which open source exists in practice. A curated feed is a gate, and a gate has an owner. If a maintainer cannot get through it, their work stops shipping, whatever its licence says.
The open, self-hostable version of this control is in decent shape: you can run your own mirror, do your own rebuilds and verify independently. It is losing on convenience, which is how these things are always lost.
Fund the dependency or inherit the exhaustion¶
The economics here don't work, and no amount of tooling fixes economics. An attacker willing to spend two years building trust will beat a volunteer maintainer who has no time, and the industry consuming that work for free has been externalising the cost of maintaining it for thirty years.
Paying maintainers is the only control on this list that addresses the failure that happened; it is not charity. We treat it as a line in the budget rather than a gesture, because the alternative is trusting somebody else's exhaustion.