The xz backdoor had one property that most of the discussion sailed past: the malicious code was never in the git repository.
The piece that activates it travelled only in the release tarballs. The obfuscated payload itself sat in the git tree as binary test files, where nobody thought to look. Anybody auditing the build scripts on GitHub would have found nothing wrong, because the part that wired it together was never in the tree.
The gap¶
Almost nobody builds from source. Distributions build from release tarballs, and users install the packages the distributions built. There's a chain of transformations between "the source a human reviewed" and "the binary a machine runs", and every link in it is an opportunity for somebody.
Reproducible builds close the gap by making that transformation verifiable: the same source, built with the same declared toolchain, has to produce bit-identical output (Reproducible Builds). If it does, anyone can rebuild independently and compare against what was published, and a discrepancy becomes evidence rather than a shrug. The input matters as much as the determinism: rebuilding the release tarball reproducibly would have reproduced the backdoor faithfully. The gap only closes when the thing rebuilt is the tree a human reviewed, with the tarball checked against it or abandoned.
That converts "trust the build infrastructure" into "verify the build infrastructure", which is the move the whole supply-chain argument keeps making in one costume or another.
Why it is hard¶
The obstacles are mundane and there are a great many of them, which is why the work took years:
- Timestamps baked into archives and binaries.
- Build paths embedded in debug info and
__FILE__. - Filesystem ordering, which differs between machines and changes link order.
- Locale and timezone affecting sorting and formatting.
- Parallelism making output order non-deterministic.
- Compiler non-determinism, including ASLR-influenced hash ordering.
- Embedded build IDs and signing timestamps.
Not one of those is conceptually hard. All of them are spread across thousands of build systems, which makes the job a social one rather than a technical one. It is the same shape as the architecture portability cleanup.
What it does and does not give you¶
Does: establishes that a published binary corresponds to published source, so reviewing the source means something, and a compromised build host becomes detectable by anybody who rebuilds.
Does not: tell you the source is any good. A reproducible build of backdoored source gives you a reliably backdoored binary. Reproducibility makes source review worthwhile; it doesn't perform the review for you.
That distinction matters, because reproducibility gets sold now and again as a supply-chain answer all by itself. It's one necessary link in a chain.
Once the property holds across a distribution, several things that are currently impossible become routine, and they are worth more than the security headline.
Trust stops being a relationship and becomes a calculation. A bank, a hospital or a defence supplier can run its own rebuild farm and verify every package it installs against the published artefact, without asking the vendor for anything and without trusting the vendor's build estate. No contract, no audit visit, no questionnaire. You rebuild and you compare. For an organisation in a country that does not trust the country the software was built in, this is the difference between adopting open source and being told not to.
A compromised build server stops being an invisible catastrophe. Right now, if someone owns a distribution's build host, the first sign is a researcher noticing half a second of SSH latency. With independent rebuilds the first sign is a mismatch, reported automatically, by anyone. The attack does not get quieter; the detection gets cheaper, and it moves from one lucky engineer to a cron job anybody can run.
The toil that disappears is incident archaeology. When something does go wrong, "which source produced this binary" becomes a question with an answer rather than three days of forensics across a build farm's logs. Every deployed artefact has a verifiable provenance to a reviewable tree. That also turns the SBOM from a best-effort inventory into a derivation you can check, which is the input the licence-compliance question needed rather than the answer to it.
And it is cheap. Unlike most security controls, this one has no per-seat price, no appliance and no vendor. It is engineering effort spent once per build system, after which the benefit accrues to everybody who ever installs the package, forever. There is not much else in security with that shape.
The compiler is not the chokepoint. The independent build infrastructure is.
Reproducibility means anyone can verify. Whether anyone does is a question about who runs the rebuild infrastructure, and rebuilding a distribution is not free: thousands of packages, repeatedly, on multiple architectures. That is a compute bill. The obvious places for it to land are the three clouds and the two large platform vendors, and if verification consolidates there then we have rebuilt the original problem one level up. "Trust the vendor's build" becomes "trust the vendor's verifier", and a signature from a rebuild service you cannot audit is just a longer way of taking somebody's word.
The practical version of that risk is already visible in how build attestation is being packaged: as a feature of a particular hosted CI product, with the provenance signed by the platform and verifiable chiefly by tools that the platform supplies. Useful, and also a lock-in surface. If your proof that your software is what you think it is only validates inside one company's ecosystem, you have bought a dependency rather than a property.
The open side here is in good shape and that deserves saying. The Reproducible
Builds project's work is public, the diffoscope tooling is excellent,
SOURCE_DATE_EPOCH is a de facto standard, and Debian's percentage is high
enough to be an existence proof rather than an aspiration. The remaining gap has
nothing technical in it: almost nobody runs a rebuild and compares, because it
is nobody's job.
Which is the whole opportunity. A rebuild-and-compare is within reach of a single competent engineer and a modest machine, and it is one of the few security controls where a small organisation can independently reach the same assurance as a large one. Here at AxisOps we rebuild what we deploy and we publish what we built it from, because the alternative is trusting a build estate we have never seen. Verification you run yourself is the only kind that cannot be withdrawn.
From build-time evidence to runtime evidence¶
Reproducibility swaps a trust relationship for a property you can check. It does it for the path from source to binary, and we expect the same move to arrive next for the running environment and for loaded code, by mechanisms nobody has settled on yet. The pattern is more durable than any of the implementations will be.
Of the controls to come out of the xz incident this is the least technically interesting and the best value per unit of effort. The condition bears restating, because it decides everything: the rebuild must start from the tree a human reviewed, with the release source verified against it. Rebuild the tarball and you reproduce the backdoor faithfully. Rebuild the reviewed tree, and the attack that has demonstrably been used in anger has nowhere left to live.