Log4Shell went public on 9 December 2021. Six days later we published an Ansible role that walks application directories, finds every archive containing the vulnerable class, backs it up and rewrites the archive without it. It is at github.com/axisops/CVE-2021-44228, MIT, and you are welcome to it.
The mitigation was right and the timing was ahead of the official guidance. The six days are the interesting number, because almost none of them went on the fix. The fix is a single deleted entry in a zip file. They went on answering "where is Log4j running?", and that is a question about your own estate, not about Apache's.
Why strip a class instead of upgrading¶
The decision that looks obvious now was nothing like obvious that week.
Apache's first fix, 2.15.0, shipped on 10 December. It was incomplete: in non-default configurations an attacker controlling Thread Context Map input could still reach the lookup, which became CVE-2021-45046. 2.16.0 removed message lookups entirely. Two releases in a week, each the definitive answer until it wasn't, and nobody tonight would bet against a third.
Meanwhile an upgrade means a dependency bump, a rebuild, a test cycle and a release per application, across every application you own, several of which are maintained by people who left. Removing the class is a change to a file on disk. It needs no rebuild, no rerelease and no cooperation from anyone, and it works identically on an application whose build system has rotted.
Deleting the class is taking the handle off a door you cannot lock. It is not the repair. It stops anyone walking through while you arrange one.
We had it on the 15th, which is worth exactly as much as it sounds: we read the same mailing lists as everybody else and the right answer was visible to anyone looking.
One side effect our README did not name: removing JndiLookup breaks
anything that legitimately performs JNDI lookups, JndiContextSelector and
JMSAppender among them. If an application used either, this role broke it. "Confirm your applications still
start" is not the same as telling an operator what specifically might not.
The shape of the tool¶
The code is a hundred lines and the interesting thing about it is its structure, not its cleverness.
def strip_jndi(jar_path: Path, backup_dir: Path) -> bool:
backup_path = backup_path_for(jar_path, backup_dir) # mirrors the full path
shutil.copy2(jar_path, backup_path)
rewritten = jar_path.with_suffix(jar_path.suffix + ".mitigating")
with zipfile.ZipFile(jar_path) as source:
with zipfile.ZipFile(rewritten, "w", zipfile.ZIP_DEFLATED) as target:
for item in source.infolist():
if item.filename == JNDI_CLASS:
continue
target.writestr(item, source.read(item.filename))
rewritten.replace(jar_path)
Back up before mutating, and mirror the source path into the backup directory:
an estate has log4j-core-2.14.1.jar in a dozen places, one per service, and
keying the backup on the basename sends every one of them to the same
destination. Write to a new file and rename over the original, so a process
killed mid-run leaves a working JAR rather than a truncated one. Log every
action to a file, so the operator ends up with a record of which archives on
which host were modified: what you need the following week when somebody asks.
Catch the per-archive failure and carry on, because one unreadable file must not
stop the other two hundred.
Signed archives are the awkward case. This rewrite leaves the signature metadata in place, and a verifier may still accept the signatures on the entries that remain; verification is not an inventory check (Oracle's JAR specification).
Then the part that matters most, and the one easiest to get wrong under
pressure: say what you did not check. A WAR or an EAR with JARs inside it,
or a Spring Boot fat JAR with BOOT-INF/lib/log4j-core.jar, has no matching
entry at the top level. Walk the filesystem for archives and those are skipped,
on exactly the packaging formats a Java estate is most likely to use. A log
full of "could not inspect" invites investigation. A log that quietly omits a
format reads as a clean bill of health. The role names every nested archive it
cannot open and prints a count, because a tool's confidence has to be
calibrated to its coverage, and at three in the morning nobody reads the source
to find out what it skipped.
The question underneath¶
The reason this tool is a directory walker is the thing really worth taking away.
If you can answer "where is Log4j running?" from a declaration, you do not go looking for JARs at all. You patch the declaration. The entire design here, walk the filesystem, inspect what you find, hope the directory list is complete, is an admission that in December 2021 that answer was discoverable but not declared. We went round the house trying every door because nobody had drawn the floor plan.
The useful prediction from that December is how cheap the hard part becomes once the estate can answer questions about itself.
A small team responds as fast as a large one. A team with a declared estate patches faster than a much larger one working from a wiki, because the work is a query and an apply rather than an archaeology project. That inverts the usual assumption that incident response is something only a big security team does properly.
The out-of-hours call changes shape. The expense of a disclosure sits in
the hours before the fix, during which nobody can tell a customer whether
they are affected. The zip -d itself takes a second. Remove the
uncertainty and you remove most of the cost, including a team's weekend spent
producing an answer a declaration would have given in a minute.
A whole category of tooling stops being necessary. This role exists because the estate could not be asked. Every directory walker, every scanner, every inventory reconciliation script is work that evaporates when the deployment description is the inventory. Not a faster search. No search.
Our bet is that this is where the next few years of operations work goes, and that it arrives driven by incidents rather than by architecture reviews. The estate that can be queried will stop being a sophistication and start being a baseline, in the same way that version control did.
Who was actually holding the ceiling up¶
The uncomfortable part of Log4Shell was who had been carrying the bug, not the bug itself.
Log4j is a small Apache project. The people who shipped 2.15.0 and then 2.16.0 within the week, under worldwide scrutiny and a great deal of abuse, were volunteers. The library sat underneath a very large share of enterprise Java: banks, insurers, governments and most of the Fortune 500. None of them were paying for it. Somebody's unpaid weekend was holding up the world's payment infrastructure, and nobody involved had agreed to that arrangement.
Follow the money and the asymmetry is plain. The organisations that profited most from Log4j built products on it at zero marginal cost. The organisations profiting most from Log4Shell are the vendors selling scanning, SBOM generation, SCA licences and managed detection into the panic. Both are large companies. Neither is the maintainer. A disclosure like this transfers money from everyone to the security industry and nothing at all to the people who wrote and then fixed the code.
Watch what happens next, because it has a predictable shape. The response to a supply-chain crisis is a compliance regime, and a compliance regime is a fixed cost. A large vendor absorbs it as a line item and markets the badge; a four-person software house and an unpaid Apache committer cannot. If the industry's answer to Log4Shell is paperwork, the effect is to push small and independent software out of regulated supply chains and consolidate them around whoever can afford the audit. That makes the next one worse, because it concentrates the dependency further.
Here at AxisOps our position is that you do not get to depend on something for free and then be outraged when it breaks. If a library is in your critical path: fund it, mirror it, read it and know where it runs. The first three are cheap. The fourth is the one you can solve with code, and it is the one we spent six days discovering we could not answer.
What we know is still wrong with it¶
Two things, published alongside the code rather than discovered later by somebody else.
The README says to confirm your applications still start. That is not the same
as telling you what might not: removing JndiLookup breaks anything that
legitimately performs JNDI lookups, JndiContextSelector and JMSAppender
among them, so this role will break such an application. That belongs in the README and will be in it.
The repository has one commit, because the working history was squashed before it went public. There is a mild irony in a supply-chain tool arriving with no reviewable history, and we would rather say so than let you notice. Defensible for something shipped in a week; a bad habit to carry into anything that outlives the incident.
Not the exploit, not the mitigation, not even the six days: the hard problem here is not knowing where the vulnerable thing runs. That is the problem we are going to spend the next few years solving, and it will not be solved with a better scanner.