Over the last few years we have written one Ansible role per vulnerability: find the affected thing on a host, back it up, apply the vendor's fix or documented workaround, verify. Several of those entries link to their role already. Seven of them are now public in one shape, along with the shared library they all use.
The fixes are the boring part. A script that strips JndiLookup.class from a
JAR is a morning's work. The part worth writing about is what has to be true
before anyone, including us, lets one of these run against a fleet.
The problem with a remediation script¶
A mitigation role is the most dangerous kind of automation in an estate. It runs as root, by design. It modifies the thing you depend on, by design. It runs during an incident, when nobody is reading carefully. And it runs everywhere at once, because that is what it is for.
The usual review process does not fit. "Someone read the diff" is weak when the reviewer is tired and the role is four hundred lines of YAML calling a Python script that shells out to a package manager. What you want to know is narrow and specific: what will this write, what will it execute, what will it restart and does the code match those claims?
So each pack carries a manifest that answers those questions, and nothing is supposed to run on the strength of the README.
What a pack declares about itself¶
The manifest sits at the repository root and states the CVEs, which hosts the pack applies to, the entry points for detection and for applying, whether each needs root, every option with its default, every path an apply writes, every program it runs and the units it may restart.
Writing that down changes the review. A reviewer reads the declaration first and then checks the code against it, instead of reading four hundred lines hoping to notice an absence. An undeclared write is a defect whether or not it is harmful, because the declaration is the thing being reviewed.
Two rules make the declaration worth having.
Detection writes nothing of its own. The detect path
stages its script in a private temporary directory, reads the host, reports
and removes the directory. No installed script, no templated config, no backup
directory, no log file, no package installed. It runs under
ansible-playbook --check because there is nothing in it that could fail a dry
run. That property is testable, and we test it the dull way: run detection in a
throwaway container with no network, then list every path whose timestamps
changed outside the usual noise. The expected output is nothing.
The one exception we found is worth stating because it looks like a failure and
is not: on Red Hat family hosts, a bare rpm -q writes its own SQLite
write-ahead index. Every pack that asks rpm a question produces those two
files. A tool that reports "detection wrote something" without knowing that
will cry wolf on every Rocky host in the estate.
Nothing is restarted or reverted by default. Every restart is behind an option that defaults to false, and no pack reverts anything automatically. A downgrade reinstates the vulnerability, so an automatic rollback is a mitigation that un-mitigates under failure conditions. If the fix broke something, a human needs to be in that decision.
One copy of the version comparison¶
Judging whether a host is vulnerable means comparing package versions, and the comparison is where quiet errors live. Distributions backport fixes, so the upstream version number tells you very little; what matters is whether the installed package is at or past the distribution's fixed version. That requires implementing dpkg, rpm and apk ordering semantics, including the parts people forget. Tilde sorts before everything. Caret sorts after. Epochs beat everything.
Each pack used to carry its own copy of that, and the copies had drifted. Some reimplemented the algorithms, some incompletely and some shelled out to the distribution's own tool. That only works when you are judging the same distribution you are running on. One host inspecting another distribution's container cannot shell out to a tool it does not have.
There is now one implementation, pure Python, standard library only, checked
against the real dpkg --compare-versions, rpm.labelCompare and
apk version -t in containers. Packs vendor it by copy rather than depending on
a package, its hash is recorded in each manifest, and each pack has a test that
fails if its copy differs. Vendoring by copy is not elegant. It means a change
to the library is a change to every pack. It also means a pack is a single
directory you can read end to end with no resolution step, which matters more
for something that runs as root during an incident.
Signed releases, and what a signature is for¶
Releases are SSH-signed annotated tags, signed with a hardware key that cannot export its private half. The platform imports a signed tag and never a branch, and pins the commit that tag names.
Two details turned out to matter more than expected.
The first is that git verify-tag prints Good "git" signature on its own
error output even in runs where it then refuses the signature. A verifier
that greps for that string will accept a tag signed by a key it has never heard
of. The exit code and the matched principal are the only things worth reading,
which we know because we wrote a verifier the wrong way first and then tried to
make it fail. It did not fail, and the reason it did not fail was that string.
The second is about dates. Each pack's first commit is dated to its vulnerability's disclosure window, so the set reads in the order the work happened; the repositories themselves were created this month. The commit date is therefore an editorial choice about ordering and carries no proof of anything. The signature is the part that attests, and it is dated when the attestation happened. Nobody should read a commit timestamp as provenance: it is a field the author sets to whatever they like, and the only reason to trust one is that something signed it.
There is a third, smaller lesson that cost us the most time. A release tag is not a label you can slide along a branch. The importer refuses a tag that has moved, permanently, by name, because a release that changes what it points at is the beginning of a rollback attack. We moved one during a rework and the development platform refused every subsequent import of that pack, correctly. The remedy is a new version, never a moved tag.
How the platform takes one in¶
A pack that runs as a standalone role is a file on disk and a decision by whoever ran it. The reason for the manifest, the signature and the single library is that the platform treats the pack as untrusted input and has to establish each property for itself. That process is worth describing, because it is the real answer to "why does a four-hundred-line Ansible role need all this ceremony".
There is no URL anywhere. A pack is not fetched from an address written in a config file. There is a declared source list, and each row names a minted identifier, a reference to a separately declared network source the fetch goes through and a repository path that must sit directly under an allowed origin. A literal URL in that file is a configuration error with its own name and is refused. The point is that adding a new place code may come from is a change to reviewed policy, not a line someone adds in passing.
The rows carry no version at all. No tag, no branch, no commit, no revision. This surprised us when we specified it and now seems obviously right: if the source list pinned a commit, then updating a pack would mean editing policy, and policy edits get waved through. Instead the pin is whichever commit a signed release tag names at import time. The only thing the row records about branches is where release tags are cut from.
Unpacking happens somewhere that is assumed to be compromised. Git itself does the unpacking, in a container with no network, running as nobody, on a read-only filesystem. Nothing that container says is believed. Outside it, every object's hash is recomputed from the raw bytes, the tag is parsed, the signature is checked, the commit and its tree are walked and the snapshot is rebuilt using only blobs that verified. The rule being applied is that the worst a compromised program inside can do is lie about its own output, and the caller checks the output.
The signature is checked twice, by different code. Once in the platform's
own implementation, once by shelling out to ssh-keygen -Y verify, and the
import is refused unless both agree. That is not thoroughness for its own
sake. It is there because of the Good "git" signature behaviour described
above: having been caught once by a verifier that printed success while
refusing, the design now assumes either implementation may be wrong and makes
disagreement fail closed rather than open.
The signer list is governance, not configuration. It is an OpenSSH allowed-signers file, every line landed by a reviewed commit, every key required to be hardware-backed and scoped to git signing only. A software key is refused by name. While the file holds no key, every import refuses, failing closed on an empty policy rather than accepting anything. Revoking a key means deleting its line, after which packs signed only by that key need re-signing. That is a deliberate cost: it makes key rotation visible work rather than a silent change in what the platform trusts.
The refusals have names. Unsigned tag, unknown signer, lightweight tag, moved tag, tag naming the wrong object, invalid signature, the two verifiers disagreeing, missing manifest, manifest at an unrecognised schema version, vendored library not matching its recorded hash, a submodule where none should be. A refusal that has a name is one an operator can look up, and one that appears in a report rather than as a stack trace.
The moved-tag refusal is the one we have already triggered in anger, as described above. The importer refuses that pack by name from then on, and there is no argument to be had with it, which is the behaviour you want.
What exists today, and what does not¶
Being specific, because the design reads as though all of it is running.
Import and verification work, as described above. Scanning a pack's contents has landed too, with its own refusals, and a scan that cannot complete records as unverified rather than clean. Grey, not green, which is the right colour for "we do not know".
The approval gate is not finished. The record has fields for who approved a pack and when, and today the importer writes both as empty and marks the pack imported. So a pack can arrive, be verified and be scanned, and still nothing has said it may run anywhere.
Running is not built either. The manifest's host selectors are not yet matched against live inventory, so the platform cannot currently tell you which of your machines a given pack applies to. Today these run as standalone Ansible roles, the way they always have, with a human choosing the inventory.
That gap is the honest state of it. The packs are public now because the shape is settled and the review properties are real, not because the platform around them is finished.
Why publish them¶
Three reasons, in order of how much they matter.
A role that claims to detect a vulnerability is a claim about your estate, and you should be able to read how the claim is made before you believe it. The version-comparison code is the part most likely to be quietly wrong, and it is the part you can now check.
The declaration-then-check review pattern is worth copying whether or not you use any of our code. Most remediation automation does not declare what it touches, and the cost of adding that is a day per role.
And some of them will be useful directly. If you have Log4j in archives on disk or an unpatched polkit on a host that cannot take the upgrade, the job of working out which hosts and what to do about it is already done.
Most of what we write is not public, because most of it is client work on client systems. These generalise: the problem without the client attached.