<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>AxisOps Labs</title><link href="https://labs.axisops.com/" rel="alternate"/><link href="https://labs.axisops.com/feeds/all.atom.xml" rel="self"/><id>https://labs.axisops.com/</id><updated>2026-10-29T09:00:00+00:00</updated><subtitle>How systems work, what they cost, and who controls them</subtitle><entry><title>CVE-2026-98432: a use-after-free in every supported Ubuntu release back to 16.04</title><link href="https://labs.axisops.com/insights/every-ubuntu-kernel-since-2016/" rel="alternate"/><published>2026-10-29T09:00:00+00:00</published><updated>2026-10-29T09:00:00+00:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-29:/insights/every-ubuntu-kernel-since-2016/</id><summary type="html">&lt;p&gt;A 9.8 in the kernel's IPv4 TCP path, no fixed kernel for any Ubuntu release and a tracker page that lists no mitigation. The fix you cannot install yet still tells you what the bug needs, and what a bug needs can be taken away.&lt;/p&gt;</summary><content type="html">&lt;p&gt;&lt;strong&gt;Status, 29 October 2026.&lt;/strong&gt; No fixed kernel is released for any Ubuntu
series. What is published below is a mitigation, not a fix: it denies the
bug its trigger, and the bug stays in the kernel until a fixed one is
running. When Ubuntu releases the fixed kernels we will update the
published mitigation to recognise them. Detection will then report against the
real versions, the apply will remove the rule from hosts that reboot into one
and we will update this note to say so.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://ubuntu.com/security/CVE-2026-98432"&gt;CVE-2026-98432&lt;/a&gt; is a
use-after-free in the Linux kernel's IPv4 TCP code, published on 15 October
and carrying a CVSS of 9.8. When the kernel handles an incoming SYN it saves
any IPv4 options the packet carried, and ownership of that saved copy later
passes to the new connection. Two paths can free the copy while another CPU
is still reading it to build the SYN+ACK. The upstream fix,
&lt;a href="https://git.kernel.org/linus/e3f33b0a1d89d738e2b913dab519226eaf76e15b"&gt;e3f33b0a1d89&lt;/a&gt;,
is small: free the options after an RCU grace period instead of immediately.&lt;/p&gt;
&lt;p&gt;Ubuntu's page, as of yesterday's update, is an unusual read. Every supported
release is listed as affected, back to 16.04. The fixed kernel versions are
on the page, release by release and flavour by flavour, each one marked
"work in progress". Nothing is released. Only 14.04 is unaffected. That is odd for the oldest kernel on the page, and a
reminder
that this class of race arrived with a performance improvement: request
sockets have been processed without the listener lock since 2015, and the
freeing discipline did not keep up with the sharing.&lt;/p&gt;
&lt;p&gt;So on 29 October a use-after-free that CVSS scores 9.8,
Critical, and that
&lt;a href="https://ubuntu.com/security/CVE-2026-98432"&gt;Canonical's tracker&lt;/a&gt;
rates priority High, reachable by anything that can send the first packet
of a TCP handshake, is present on every supported Ubuntu release back to 16.04, and the updates
channel has nothing to install. The page's mitigation section
says none.&lt;/p&gt;
&lt;h2 id="what-the-fix-tells-you"&gt;What the fix tells you&lt;a class="headerlink" href="#what-the-fix-tells-you" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A patch you cannot install is still a specification of the bug. Both racing
paths in the fix read the same object: the IPv4 options saved off the
incoming SYN. The SYN+ACK builder reads them while connection setup can free
them, and request-socket migration shares then frees the same copy. When the
incoming SYN carries no IPv4 options, the kernel saves nothing, the pointer
is null and there is nothing to free or to race over. We read the commit to
check that precondition before trusting it; it holds for both halves.&lt;/p&gt;
&lt;p&gt;That is the mitigation the tracker does not list: take the trigger away.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;samp_cve_2026_98432&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nx"&gt;chain&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;prerouting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;filter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hook&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;prerouting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;priority&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;ip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hdrlength&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;syn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ack&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;syn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;counter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;comment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;"CVE-2026-98432: drop inbound IPv4 SYN carrying IP options"&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;One rule, in a table of its own so it cannot collide with Docker's or
anything else's, dropping inbound IPv4 SYNs whose header is longer than
twenty bytes. TCP almost never carries IP options; several option types are
dropped by many networks as standing policy
(&lt;a href="https://www.rfc-editor.org/rfc/rfc7126"&gt;RFC 7126&lt;/a&gt;), and a connection that
needs them is rarer than the counter on that rule will ever show. We put a
packet generator behind it before trusting it: a SYN carrying options is
dropped and counted, a plain SYN passes untouched.&lt;/p&gt;
&lt;h2 id="what-it-does-not-do"&gt;What it does not do&lt;a class="headerlink" href="#what-it-does-not-do" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;It does not drop fragmented SYNs on a host with no connection tracking,
because nothing reassembles them at that hook. It does nothing for IPv6,
which this CVE does not reach. It does not make the kernel fixed: the race
is still there, waiting for anyone who removes the rule before rebooting
into a fixed kernel that does not yet exist. And a client that genuinely sends optioned SYNs stops connecting, which is
the trade, stated.&lt;/p&gt;
&lt;p&gt;It also does not survive a reboot, on purpose. The rule lives in kernel
memory and is never written to disk, because a mitigation pack that
installs boot-time configuration becomes a persistence mechanism of its
own. After a reboot the rule is gone, the next detection says so
(&lt;code&gt;rule_live=0&lt;/code&gt; on the result line) and the next apply puts it back.
Re-applying after every reboot is the operating model, not a failure of
it; anyone running the role by hand instead of under automation needs to
know the protection does not outlive the uptime.&lt;/p&gt;
&lt;p&gt;To be precise about what it is: the repository below is a mitigation, not
a fix. The fix is a kernel, and nobody can ship you one tonight. The rule,
the detection that reads a running kernel against the tracker's versions
and the removal path for when fixed kernels land are published at
&lt;a href="https://github.com/axisops/CVE-2026-98432"&gt;github.com/axisops/CVE-2026-98432&lt;/a&gt;,
runnable as a standalone Ansible role. The work-in-progress kernel versions
from the tracker sit in the defaults as a comment, ready to become the
fixed-version table the day they ship.&lt;/p&gt;
&lt;p&gt;"No mitigation" on a tracker page means no patch-shaped mitigation. The fix
names what the bug needs to fire. Needs can be denied.&lt;/p&gt;</content><category term="insights"/><category term="security"/><category term="kernel"/><category term="networking"/><category term="mitigation"/></entry><entry><title>What a mitigation pack has to prove before it runs</title><link href="https://labs.axisops.com/insights/what-a-mitigation-pack-has-to-prove/" rel="alternate"/><published>2026-10-28T09:00:00+00:00</published><updated>2026-10-28T09:00:00+00:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-28:/insights/what-a-mitigation-pack-has-to-prove/</id><summary type="html">&lt;p&gt;Seven of our remediation roles are public, with the shared library they depend on. The fixes themselves are a morning's work each. What took the time was deciding what a pack must declare and demonstrate before anything will run it as root across a fleet, and only some of those gates exist today.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Over the last few years we have written one Ansible role per vulnerability:
find the affected thing on a host, back it up, apply the vendor's fix or
documented workaround, verify. Several of those entries link to their role
already. Seven of them are now public in one shape, along with the shared
library they all use.&lt;/p&gt;
&lt;p&gt;The fixes are the boring part. A script that strips &lt;code&gt;JndiLookup.class&lt;/code&gt; from a
JAR is a morning's work. The part worth writing about is what has to be true
before anyone, including us, lets one of these run against a fleet.&lt;/p&gt;
&lt;h2 id="the-problem-with-a-remediation-script"&gt;The problem with a remediation script&lt;a class="headerlink" href="#the-problem-with-a-remediation-script" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A mitigation role is the most dangerous kind of automation in an estate. It
runs as root, by design. It modifies the thing you depend on, by design. It
runs during an incident, when nobody is reading carefully. And it runs
everywhere at once, because that is what it is for.&lt;/p&gt;
&lt;p&gt;The usual review process does not fit. "Someone read the diff" is weak when the
reviewer is tired and the role is four hundred lines of YAML calling a Python
script that shells out to a package manager. What you want to know is narrow
and specific: what will this write, what will it execute, what will it restart
and does the code match those claims?&lt;/p&gt;
&lt;p&gt;So each pack carries a manifest that answers those questions, and nothing is
supposed to run on the strength of the README.&lt;/p&gt;
&lt;h2 id="what-a-pack-declares-about-itself"&gt;What a pack declares about itself&lt;a class="headerlink" href="#what-a-pack-declares-about-itself" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The manifest sits at the repository root and states the CVEs, which hosts the
pack applies to, the entry points for detection and for applying, whether each
needs root, every option with its default, every path an apply writes, every
program it runs and the units it may restart.&lt;/p&gt;
&lt;p&gt;Writing that down changes the review. A reviewer reads the declaration first
and then checks the code against it, instead of reading four hundred lines
hoping to notice an absence. An undeclared write is a defect whether or not it
is harmful, because the declaration is the thing being reviewed.&lt;/p&gt;
&lt;p&gt;Two rules make the declaration worth having.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Detection writes nothing of its own.&lt;/strong&gt; The detect path
stages its script in a private temporary directory, reads the host, reports
and removes the directory. No installed script, no templated config, no backup
directory, no log file, no package installed. It runs under
&lt;code&gt;ansible-playbook --check&lt;/code&gt; because there is nothing in it that could fail a dry
run. That property is testable, and we test it the dull way: run detection in a
throwaway container with no network, then list every path whose timestamps
changed outside the usual noise. The expected output is nothing.&lt;/p&gt;
&lt;p&gt;The one exception we found is worth stating because it looks like a failure and
is not: on Red Hat family hosts, a bare &lt;code&gt;rpm -q&lt;/code&gt; writes its own SQLite
write-ahead index. Every pack that asks rpm a question produces those two
files. A tool that reports "detection wrote something" without knowing that
will cry wolf on every Rocky host in the estate.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Nothing is restarted or reverted by default.&lt;/strong&gt; Every restart is behind an
option that defaults to false, and no pack reverts anything automatically. A
downgrade reinstates the vulnerability, so an automatic rollback is a mitigation
that un-mitigates under failure conditions. If the fix broke something, a human
needs to be in that decision.&lt;/p&gt;
&lt;h2 id="one-copy-of-the-version-comparison"&gt;One copy of the version comparison&lt;a class="headerlink" href="#one-copy-of-the-version-comparison" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Judging whether a host is vulnerable means comparing package versions, and the
comparison is where quiet errors live. Distributions backport fixes, so the
upstream version number tells you very little; what matters is whether the
installed package is at or past the distribution's fixed version. That requires
implementing dpkg, rpm and apk ordering semantics, including the parts people
forget. Tilde sorts before everything. Caret sorts after. Epochs beat
everything.&lt;/p&gt;
&lt;p&gt;Each pack used to carry its own copy of that, and the copies had drifted. Some
reimplemented the algorithms, some incompletely and some shelled out to the
distribution's own tool. That only works when you are judging the same
distribution you are running on. One host inspecting another distribution's
container cannot shell out to a tool it does not have.&lt;/p&gt;
&lt;p&gt;There is now one implementation, pure Python, standard library only, checked
against the real &lt;code&gt;dpkg --compare-versions&lt;/code&gt;, &lt;code&gt;rpm.labelCompare&lt;/code&gt; and
&lt;code&gt;apk version -t&lt;/code&gt; in containers. Packs vendor it by copy rather than depending on
a package, its hash is recorded in each manifest, and each pack has a test that
fails if its copy differs. Vendoring by copy is not elegant. It means a change
to the library is a change to every pack. It also means a pack is a single
directory you can read end to end with no resolution step, which matters more
for something that runs as root during an incident.&lt;/p&gt;
&lt;h2 id="signed-releases-and-what-a-signature-is-for"&gt;Signed releases, and what a signature is for&lt;a class="headerlink" href="#signed-releases-and-what-a-signature-is-for" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Releases are SSH-signed annotated tags, signed with a hardware key that cannot
export its private half. The platform imports a signed tag and never a branch,
and pins the commit that tag names.&lt;/p&gt;
&lt;p&gt;Two details turned out to matter more than expected.&lt;/p&gt;
&lt;p&gt;The first is that &lt;code&gt;git verify-tag&lt;/code&gt; prints &lt;code&gt;Good "git" signature&lt;/code&gt; on its own
error output &lt;strong&gt;even in runs where it then refuses the signature&lt;/strong&gt;. A verifier
that greps for that string will accept a tag signed by a key it has never heard
of. The exit code and the matched principal are the only things worth reading,
which we know because we wrote a verifier the wrong way first and then tried to
make it fail. It did not fail, and the reason it did not fail was that string.&lt;/p&gt;
&lt;p&gt;The second is about dates. Each pack's first commit is dated to its
vulnerability's disclosure window, so the set reads in the order the work
happened; the repositories themselves were created this month. The commit date
is therefore an editorial choice about ordering and carries no proof of
anything. The signature is the part that attests, and it is dated when the
attestation happened. Nobody should read a commit timestamp as provenance: it is
a field the author sets to whatever they like, and the only reason to trust one
is that something signed it.&lt;/p&gt;
&lt;p&gt;There is a third, smaller lesson that cost us the most time. A release tag is
not a label you can slide along a branch. The importer refuses a tag that has
moved, permanently, by name, because a release that changes what it points at is
the beginning of a rollback attack. We moved one during a rework and the
development platform refused every subsequent import of that pack, correctly.
The remedy is a new version, never a moved tag.&lt;/p&gt;
&lt;h2 id="how-the-platform-takes-one-in"&gt;How the platform takes one in&lt;a class="headerlink" href="#how-the-platform-takes-one-in" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A pack that runs as a standalone role is a file on disk and a decision by
whoever ran it. The reason for the manifest, the signature and the single
library is that the platform treats the pack as untrusted input and has to
establish each property for itself. That process is worth describing, because it
is the real answer to "why does a four-hundred-line Ansible role need all this
ceremony".&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;There is no URL anywhere.&lt;/strong&gt; A pack is not fetched from an address written in
a config file. There is a declared source list, and each row names a minted
identifier, a reference to a separately declared network source the fetch goes
through and a repository path that must sit directly under an allowed origin.
A literal URL in that file is a configuration error with its own name and is
refused. The point is that adding a new place code may come from is a change
to reviewed policy, not a line someone adds in passing.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The rows carry no version at all.&lt;/strong&gt; No tag, no branch, no commit, no
revision. This surprised us when we specified it and now seems obviously
right: if the source list pinned a commit, then updating a pack would mean
editing policy, and policy edits get waved through. Instead the pin &lt;em&gt;is&lt;/em&gt;
whichever commit a signed release tag names at import time. The only thing the
row records about branches is where release tags are cut from.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Unpacking happens somewhere that is assumed to be compromised.&lt;/strong&gt; Git itself
does the unpacking, in a container with no network, running as nobody, on a
read-only filesystem. Nothing that container says is believed. Outside it,
every object's hash is recomputed from the raw bytes, the tag is parsed, the
signature is checked, the commit and its tree are walked and the snapshot is
rebuilt using only blobs that verified. The rule being applied is that the
worst a compromised program inside can do is lie about its own output, and the
caller checks the output.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The signature is checked twice, by different code.&lt;/strong&gt; Once in the platform's
own implementation, once by shelling out to &lt;code&gt;ssh-keygen -Y verify&lt;/code&gt;, and the
import is refused unless both agree. That is not thoroughness for its own
sake. It is there because of the &lt;code&gt;Good "git" signature&lt;/code&gt; behaviour described
above: having been caught once by a verifier that printed success while
refusing, the design now assumes either implementation may be wrong and makes
disagreement fail closed rather than open.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The signer list is governance, not configuration.&lt;/strong&gt; It is an OpenSSH
allowed-signers file, every line landed by a reviewed commit, every key
required to be hardware-backed and scoped to git signing only. A software key
is refused by name. While the file holds no key, every import refuses,
failing closed on an empty policy rather than accepting anything. Revoking a
key means deleting its line, after which packs signed only by that key need
re-signing. That is a deliberate cost: it makes key rotation visible work
rather than a silent change in what the platform trusts.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The refusals have names.&lt;/strong&gt; Unsigned tag, unknown signer, lightweight tag,
moved tag, tag naming the wrong object, invalid signature, the two verifiers
disagreeing, missing manifest, manifest at an unrecognised schema version,
vendored library not matching its recorded hash, a submodule where none should
be. A refusal that has a name is one an operator can look up, and one that
appears in a report rather than as a stack trace.&lt;/p&gt;
&lt;p&gt;The moved-tag refusal is the one we have already triggered in anger, as
described above. The importer refuses that pack by name from then on, and there
is no argument to be had with it, which is the behaviour you want.&lt;/p&gt;
&lt;h2 id="what-exists-today-and-what-does-not"&gt;What exists today, and what does not&lt;a class="headerlink" href="#what-exists-today-and-what-does-not" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Being specific, because the design reads as though all of it is running.&lt;/p&gt;
&lt;p&gt;Import and verification work, as described above. Scanning a pack's contents
has landed too, with its own refusals, and a scan that cannot complete records
as unverified rather than clean. Grey, not green, which is the right colour
for "we do not know".&lt;/p&gt;
&lt;p&gt;The approval gate is not finished. The record has fields for who approved a
pack and when, and today the importer writes both as empty and marks the pack
imported. So a pack can arrive, be verified and be scanned, and still nothing
has said it may run anywhere.&lt;/p&gt;
&lt;p&gt;Running is not built either. The manifest's host selectors are not yet matched
against live inventory, so the platform cannot currently tell you which of your
machines a given pack applies to. Today these run as standalone Ansible roles,
the way they always have, with a human choosing the inventory.&lt;/p&gt;
&lt;p&gt;That gap is the honest state of it. The packs are public now because the shape
is settled and the review properties are real, not because the platform around
them is finished.&lt;/p&gt;
&lt;h2 id="why-publish-them"&gt;Why publish them&lt;a class="headerlink" href="#why-publish-them" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Three reasons, in order of how much they matter.&lt;/p&gt;
&lt;p&gt;A role that claims to detect a vulnerability is a claim about your estate, and
you should be able to read how the claim is made before you believe it. The
version-comparison code is the part most likely to be quietly wrong, and it is
the part you can now check.&lt;/p&gt;
&lt;p&gt;The declaration-then-check review pattern is worth copying whether or not you
use any of our code. Most remediation automation does not declare what it
touches, and the cost of adding that is a day per role.&lt;/p&gt;
&lt;p&gt;And some of them will be useful directly. If you have Log4j in archives on disk
or an unpatched polkit on a host that cannot take the upgrade, the job of
working out which hosts and what to do about it is already done.&lt;/p&gt;
&lt;p&gt;Most of what we write is not public, because most of it is client work on
client systems. These generalise: the problem without the client attached.&lt;/p&gt;</content><category term="insights"/><category term="security"/><category term="ansible"/><category term="supply-chain"/><category term="mitigation"/><category term="substrate"/></entry><entry><title>The one that did not get the meltdown</title><link href="https://labs.axisops.com/insights/brotli-version-is-not-a-verdict/" rel="alternate"/><published>2026-10-27T09:00:00+00:00</published><updated>2026-10-27T09:00:00+00:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-27:/insights/brotli-version-is-not-a-verdict/</id><summary type="html">&lt;p&gt;PHP2Shell and the Brotli decoder bug were patched the same day. One got the emergency response, the other did not, and the quiet one is harder to find on your estate because the obvious version strings lie about it. There is a role you can run to work out what you have.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Six days ago the industry stopped what it was doing for PHP. CVE-2026-109618
and CVE-2026-109619, unauthenticated remote code execution on anything serving
web requests, CVSS 9.2. CISA told people to take unpatched systems offline
(&lt;a href="https://www.cisa.gov/news-events/alerts/2026/10/21/critical-vulnerabilities-php-require-immediate-patching"&gt;CISA alert, 21 October&lt;/a&gt;),
which is not language it uses often, and added both to the Known Exploited
Vulnerabilities catalogue on the 26th. That was the right response.&lt;/p&gt;
&lt;p&gt;The same day PHP was patched, Google shipped Brotli 1.0.10, 1.1.1 and 1.2.1,
and Node.js shipped 22.23.4, 24.22.1 and 26.12.1
(&lt;a href="https://nodejs.org/en/blog/vulnerability/october-2026-security-releases"&gt;Node.js October 2026 security releases&lt;/a&gt;).
CVE-2026-111497: an out-of-bounds write in the Brotli decoder, CVSS 9.2 from
Google's CNA, reachable by anything that decompresses Brotli it did not
produce. It got a fraction of the attention.&lt;/p&gt;
&lt;p&gt;We think the gap is a mistake. The two bugs are not equivalent, and the
argument here does not need them to be. The problem is that this one is
materially harder to find on an estate, because the version numbers you would
naturally check give the wrong answer.&lt;/p&gt;
&lt;h2 id="the-bug-briefly"&gt;The bug, briefly&lt;a class="headerlink" href="#the-bug-briefly" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The decoder treated an invalid block-type switch as "need more input" rather
than a format error. Decoding carried on, appending attacker-supplied bytes to
an eight-byte internal buffer and writing past the end of it
(&lt;a href="https://access.redhat.com/security/vulnerabilities/RHSB-2026-015"&gt;Red Hat RHSB-2026-015&lt;/a&gt;).
The fixed decoder returns a hard error.&lt;/p&gt;
&lt;p&gt;Exposure is wider than the Node.js headline suggests: HTTP clients that accept
&lt;code&gt;br&lt;/code&gt; responses, servers that decode &lt;code&gt;br&lt;/code&gt; request bodies, anything pulling
Brotli out of an archive, a cache or a queue. On the Python side &lt;code&gt;requests&lt;/code&gt;,
&lt;code&gt;urllib3&lt;/code&gt;, &lt;code&gt;httpx&lt;/code&gt; and &lt;code&gt;aiohttp&lt;/code&gt; all decode Brotli automatically if a backend
is installed, which it often is as a transitive dependency nobody chose.&lt;/p&gt;
&lt;p&gt;Public proof-of-concept code exists and reaches code execution under narrow
lab conditions, which is where this class of bug usually sits a week in.&lt;/p&gt;
&lt;h2 id="why-it-is-harder-to-find-than-php"&gt;Why it is harder to find than PHP&lt;a class="headerlink" href="#why-it-is-harder-to-find-than-php" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;With PHP the question is nearly answerable. You know where your PHP is and
&lt;code&gt;php -v&lt;/code&gt; tells you the truth about it.&lt;/p&gt;
&lt;p&gt;Brotli does not work like that, and three separate things are wrong with the
obvious check.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The upstream version is close to useless.&lt;/strong&gt; Distributions backport the patch without changing the version. Ubuntu 24.04's &lt;code&gt;1.1.0-2build2&lt;/code&gt; is vulnerable and
&lt;code&gt;1.1.0-2ubuntu0.1&lt;/code&gt; is fixed, and both are upstream 1.1.0. A scanner comparing
against the published affected range of "1.1.0 before 1.1.1" marks the patched
host as vulnerable, forever, because 1.1.0 is where it will stay.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Patched Node.js reports the version it fixed.&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;docker&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;node:22.23.4-slim&lt;span class="w"&gt; &lt;/span&gt;node&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s1"&gt;'process.version + " brotli=" + process.versions.brotli'&lt;/span&gt;
v22.23.4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;brotli&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.1.0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every scanner reading &lt;code&gt;process.versions.brotli&lt;/code&gt; for a verdict is wrong on
patched hosts and will stay wrong. The only thing that answers the question is
the Node.js release number against the published fixed release per line.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Whether Node.js even uses the system library depends on where it came from.&lt;/strong&gt;
The nodejs.org builds and official &lt;code&gt;node&lt;/code&gt; images bundle their own copy, so
&lt;code&gt;libbrotli&lt;/code&gt; on the host is irrelevant to them. Distribution packages are the
other way around: Ubuntu 24.04's &lt;code&gt;node&lt;/code&gt; reports &lt;code&gt;node_shared_brotli: true&lt;/code&gt;, and
Ubuntu marks its own nodejs "not affected" because fixing the system library
fixes it. One host can have both kinds installed.&lt;/p&gt;
&lt;p&gt;None of that is visible in a version string, which is why "we patched" settles
much less here than it does for PHP.&lt;/p&gt;
&lt;h2 id="so-there-is-a-role"&gt;So there is a role&lt;a class="headerlink" href="#so-there-is-a-role" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;&lt;a href="https://github.com/axisops/CVE-2026-111497"&gt;github.com/axisops/CVE-2026-111497&lt;/a&gt;,
same shape as the others. Point it at a substrate deployment and it reports
what is affected before it changes anything.&lt;/p&gt;
&lt;p&gt;It compares the installed package against the distribution's fixed version,
with dpkg, rpm and apk comparison implemented in Python so one host can judge a
container running a different distribution. It reads &lt;code&gt;node_shared_brotli&lt;/code&gt; per
binary and judges each by whichever thing determines it. Dry run by default;
&lt;code&gt;--apply&lt;/code&gt; backs up, upgrades and verifies.&lt;/p&gt;
&lt;p&gt;Four verdicts, and the two unsatisfying ones carry most of the value. &lt;strong&gt;no_fix&lt;/strong&gt;
is the vendor saying affected with nothing published: that is Red Hat today,
advisory up on 26 October, status ongoing, no errata, RHEL 8 through 10
affected, with Rocky and Alma inheriting it. &lt;strong&gt;unknown&lt;/strong&gt; is a release with no
published verdict at all, such as Alpine 3.22. Reporting those plainly beats
inferring from the upstream version, which is the thing we just established does
not work.&lt;/p&gt;
&lt;p&gt;It will not see a decoder compiled into another binary: Electron apps, &lt;code&gt;pkg&lt;/code&gt;
single binaries, Go and Rust programs linking it statically, browsers with their
own copy. Python wheels are their own problem, since &lt;code&gt;pip install Brotli&lt;/code&gt; puts a
copy in a virtualenv no package manager owns. Containers get reported, never
changed, because a package upgraded inside a running container is gone when it
is recreated. For our own images that means &lt;code&gt;mg docker build --refresh&lt;/code&gt;, since a
plain rebuild reuses the cached layer and the old base.&lt;/p&gt;
&lt;p&gt;And the upgrade is not the end. The library is loaded by long-running
processes, and replacing the file on disk leaves every one of them running the
old code until it restarts. The role reads &lt;code&gt;/proc/*/maps&lt;/code&gt; for the &lt;code&gt;(deleted)&lt;/code&gt;
marker, names the service each process belongs to and restarts nothing unless
asked.&lt;/p&gt;
&lt;h2 id="the-part-that-keeps-recurring"&gt;The part that keeps recurring&lt;a class="headerlink" href="#the-part-that-keeps-recurring" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The workaround people reach for is rejecting &lt;code&gt;Content-Encoding: br&lt;/code&gt; at the
proxy. It covers inbound request bodies and nothing else: not a client fetching
a &lt;code&gt;br&lt;/code&gt; response from somewhere hostile, not Brotli read out of a queue.&lt;/p&gt;
&lt;p&gt;Which is the same conclusion as OpenSSH in 2024 and Log4j before it. The
advisory tells you the affected versions. Working out which of your hosts are in
that set needs every distribution and release you run, every vendor's fixed
version for each and which vendors had shipped when you looked. This time it
also needs the knowledge that two of the obvious version strings are actively
misleading.&lt;/p&gt;
&lt;p&gt;PHP got a week of everyone's attention and it still took CISA telling people to
pull systems offline. This one got a fraction of that, and is harder to answer.
An estate that can say "what package, from what release, built how" from a
declaration answers it on day one. We went looking, again.&lt;/p&gt;</content><category term="insights"/><category term="security"/><category term="ansible"/><category term="linux"/><category term="python"/><category term="mitigation"/></entry><entry><title>What led us here</title><link href="https://labs.axisops.com/insights/what-led-us-here/" rel="alternate"/><published>2026-10-24T09:00:00+01:00</published><updated>2026-10-24T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-24:/insights/what-led-us-here/</id><summary type="html">&lt;p&gt;Microkernels, eBPF, WebAssembly, LoRa mesh and confidential computing are usually filed as five unrelated hobbies. They are one argument, running for about a decade, about where you put a boundary and what it costs to cross it.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Five things have been going on in parallel for about a decade, and they get
filed in five different drawers. Microkernels are an OS-research curiosity.
eBPF is a networking tool. WebAssembly is a browser thing that escaped. LoRa
mesh is for hikers. Confidential computing is a slide in a cloud vendor's deck.&lt;/p&gt;
&lt;p&gt;Here at AxisOps we think they are all the same argument. Every one of them
takes a position on a single question: &lt;strong&gt;where does the boundary go, and what
does it cost to cross it?&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;That question is as old as the industry. What changed over the last ten years
is that the answer stopped being "wherever the hardware happened to put it".&lt;/p&gt;
&lt;h2 id="the-boundary-we-inherited"&gt;The boundary we inherited&lt;a class="headerlink" href="#the-boundary-we-inherited" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;For most of the industry's working life there were two boundaries worth the
name, and the hardware chose both of them for us.&lt;/p&gt;
&lt;p&gt;There's the process boundary, enforced by the MMU; one ring transition to
cross, a page table per side, and a fault on one side doesn't take the other
down with it. It's a good boundary and it's still doing most of the work in
production today.&lt;/p&gt;
&lt;p&gt;Then there's the machine boundary, enforced by a hypervisor. Much stronger,
and much more expensive, because you're paying for a whole kernel per side.&lt;/p&gt;
&lt;p&gt;Everything we built, we built by picking one of those two. Something that
needed isolating got a process, or it got a VM. Think of it like a building
where the only two options are a desk or a separate office block; most of the
interesting failures of the last twenty years happened when neither size fitted,
so nobody isolated anything at all.&lt;/p&gt;
&lt;p&gt;Each of the five strands below is an attempt to build a boundary that the
hardware never handed us.&lt;/p&gt;
&lt;h2 id="1-microkernels-the-boundary-moved-down"&gt;1. Microkernels: the boundary moved down&lt;a class="headerlink" href="#1-microkernels-the-boundary-moved-down" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The microkernel argument is old enough to have gone out of fashion twice. The
architecture has not changed. What changed is that the proof turned up.&lt;/p&gt;
&lt;p&gt;seL4's functional-correctness proof means something quite specific, and it's
worth being careful here because it gets quoted loosely. The kernel's C
implementation is proven to implement its abstract specification, with no
unproven gap in between. That claim now holds across Arm 32-bit, AArch64,
x86-64 and RISC-V 64-bit, and the AArch64 port has reached parity with the
others on functional correctness
(&lt;a href="https://sel4.systems/Summit/2024/abstracts2024.html"&gt;seL4 Summit 2024&lt;/a&gt;). The
proof for the MCS configuration has since been completed on RISC-V
(&lt;a href="https://www.sel4.systems/news/"&gt;seL4 news&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;The usual objection is IPC cost, and it's a fair objection; the cost is real.
It's also the wrong place to stop the conversation, because what you're buying
is a fault boundary around a driver, and a monolithic kernel can't sell you one
of those at any price. A driver bug in Linux is a kernel bug. That is a
structural problem, not a code quality one.&lt;/p&gt;
&lt;p&gt;What a microkernel actually costs you
&lt;a href="/insights/microkernel-isolation/"&gt;has its own entry&lt;/a&gt;, which is honest about
which numbers it still owes.&lt;/p&gt;
&lt;h2 id="2-ebpf-the-boundary-became-programmable"&gt;2. eBPF: the boundary became programmable&lt;a class="headerlink" href="#2-ebpf-the-boundary-became-programmable" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;eBPF is the strand people underrate most, and we think that's because it turned
up dressed as a networking tool.&lt;/p&gt;
&lt;p&gt;The original extended-BPF patch landed in Linux 3.15: 64-bit registers, ten of
them rather than two, and an instruction set mapped closely enough to real
hardware to JIT. LWN reported it as "up to four times faster on x86-64" than
classic BPF on some network-filter microbenchmarks
(&lt;a href="https://lwn.net/Articles/740157/"&gt;LWN, 2017&lt;/a&gt;). The &lt;code&gt;bpf()&lt;/code&gt; system call that
exposed the VM to user space went up for review in July 2014
(&lt;a href="https://lwn.net/Articles/603983/"&gt;LWN, 2014&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;Described as a faster packet filter, that's a footnote. Described properly,
it's the first time the Linux kernel took code from user space and ran it in
kernel context on the strength of a proof.&lt;/p&gt;
&lt;p&gt;The verifier is the whole story. It walks the control-flow graph depth-first to
establish that the program terminates and rejects unreachable instructions. Then
it simulates execution instruction by instruction, tracking register and stack
state and refusing out-of-bounds jumps, out-of-range access and reads of
uninitialised registers. In secure mode, for a loader without &lt;code&gt;CAP_SYS_ADMIN&lt;/code&gt;,
pointer arithmetic is prohibited outright so that kernel addresses can't leak
(&lt;a href="https://lwn.net/Articles/740157/"&gt;LWN, 2017&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;So here's a boundary made out of static analysis rather than hardware. No ring
transition, no page table, no context switch. You pay for the boundary once,
at load time, in the verifier; after that the crossing is free, though the
program still costs whatever its instructions and helper calls cost.&lt;/p&gt;
&lt;p&gt;Kernel-side validation was not new: classic BPF had it, and proof-carrying
code is older still. The scope was new, arbitrary programs against kernel
data structures rather than packet filters.&lt;/p&gt;
&lt;h2 id="3-webassembly-the-boundary-became-portable"&gt;3. WebAssembly: the boundary became portable&lt;a class="headerlink" href="#3-webassembly-the-boundary-became-portable" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;WASM is the same trick as eBPF, aimed at a different problem and approached
from the opposite end. eBPF is a verified sandbox that got pushed down into the
kernel. WASM is a verified sandbox that got pulled out of the browser.&lt;/p&gt;
&lt;p&gt;Both of them give you a restricted instruction set, a validation pass that has
to succeed before anything runs, linear memory the guest can't escape and no
ambient authority; the guest gets the capabilities the host hands it and
nothing more.&lt;/p&gt;
&lt;p&gt;The capability model is the part we care about, and it's also where the
reporting gets difficult. WASI's design is explicitly capability-based, so a
module receives handles rather than a filesystem. The ecosystem timeline,
though, is in a muddle, and most of what's written about it isn't reliable.
We cover the component model separately in
&lt;a href="/insights/wasi-capabilities/"&gt;WASI: capabilities instead of containers&lt;/a&gt;,
working from the specification and the Bytecode Alliance's own records rather
than from secondary write-ups. A survey of the blog coverage gave us three
contradictory dates for the same release. That's a comment on the state of the
commentary, not on the technology.&lt;/p&gt;
&lt;p&gt;What matters architecturally is that this boundary is portable. An eBPF program
runs in a Linux kernel. A WASM module runs anywhere there's a runtime, with the
same isolation properties, which makes it the first sandbox whose guarantees
survive the trip between a server, an edge node and a browser tab.&lt;/p&gt;
&lt;h2 id="4-confidential-computing-the-boundary-turned-inwards"&gt;4. Confidential computing: the boundary turned inwards&lt;a class="headerlink" href="#4-confidential-computing-the-boundary-turned-inwards" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The three strands above all assume the host is trustworthy and the guest is
suspect. Confidential computing turns that on its head and protects the guest
from the host: memory encrypted with a key the hypervisor doesn't hold, and an
attestation the workload can check before it will unseal anything.&lt;/p&gt;
&lt;p&gt;This is the strand with the widest gap between the marketing and the
engineering. The deployment reality has
&lt;a href="/insights/confidential-computing-counted/"&gt;its own entry&lt;/a&gt;: what attestation
really proves, what's inside the trusted computing base once you've counted
the firmware and what encrypted memory costs you.
Until we've measured it, treat anything here as a vendor claim.&lt;/p&gt;
&lt;p&gt;It belongs on this list for a structural reason. It's the first of the five
where crossing the boundary means proving who you are, rather than simply being
permitted.&lt;/p&gt;
&lt;h2 id="5-lora-mesh-the-boundary-lost-its-centre"&gt;5. LoRa mesh: the boundary lost its centre&lt;a class="headerlink" href="#5-lora-mesh-the-boundary-lost-its-centre" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The odd one out, and the one that gets dismissed as a hobby more reliably than
any of the others.&lt;/p&gt;
&lt;p&gt;Meshtastic is LoRa radios running a mesh protocol with no infrastructure
underneath: no base station, no carrier, no DNS, no CA. Nodes relay for each
other, and the topology is whatever happens to be in range.&lt;/p&gt;
&lt;p&gt;The engineering is a good deal more serious than "hikers with walkie-talkies"
suggests, and the 2.8 firmware line is where you can see it. Packet signing via
XEdDSA. Node IDs derived from a node's public-key identity rather than its
hardware MAC address. A traffic-management module doing deduplication and rate
limiting, and hop limits that vary with observed mesh activity
(&lt;a href="https://github.com/meshtastic/firmware/releases"&gt;Meshtastic firmware releases&lt;/a&gt;,
&lt;a href="https://meshtastic.org/blog/"&gt;Meshtastic blog&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;Read that list again as an identity architecture. A node &lt;em&gt;is&lt;/em&gt; its key. There's
no registry to ask because there's nothing to ask; authority lives in the
keypair, so trust gets established peer to peer or not at all. The 2.7.15
release disallowed the legacy direct-message path, protected only by the
shared channel key, after a spoofing advisory; DMs broke for nodes on older
firmware while relaying and channels kept working. That's a compatibility
break made for a security fix, in favour of the stronger property.&lt;/p&gt;
&lt;p&gt;One note on currency, because it's the sort of thing a reader will check: the
2.8 line is &lt;strong&gt;alpha&lt;/strong&gt; as we write this. 2.8.0 was revoked over a defect in the
packet-signing code and replaced by 2.8.1, and the project's own advice is that
most people should stay on 2.7 for the time being
(&lt;a href="https://meshtastic.org/blog/"&gt;Meshtastic blog&lt;/a&gt;). The stable base is 2.7; 2.8
is where the interesting protocol work is happening.&lt;/p&gt;
&lt;p&gt;Same question as the other four, asked in a place with no centre at all: who is
allowed to cross, and what proves it?&lt;/p&gt;
&lt;h2 id="what-actually-led-us-here"&gt;What actually led us here&lt;a class="headerlink" href="#what-actually-led-us-here" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Put the five side by side and the pattern is hard to miss.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Strand&lt;/th&gt;
&lt;th&gt;Boundary enforced by&lt;/th&gt;
&lt;th&gt;Paid when&lt;/th&gt;
&lt;th&gt;Protects&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Process / MMU&lt;/td&gt;
&lt;td&gt;Hardware page tables&lt;/td&gt;
&lt;td&gt;Every crossing&lt;/td&gt;
&lt;td&gt;Host from guest&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Microkernel&lt;/td&gt;
&lt;td&gt;Hardware + proof of the kernel&lt;/td&gt;
&lt;td&gt;Every IPC&lt;/td&gt;
&lt;td&gt;Everything from drivers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;eBPF&lt;/td&gt;
&lt;td&gt;Static analysis (verifier)&lt;/td&gt;
&lt;td&gt;Once, at load&lt;/td&gt;
&lt;td&gt;Kernel from user code&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WASM&lt;/td&gt;
&lt;td&gt;Validation + capabilities&lt;/td&gt;
&lt;td&gt;Once, at validate&lt;/td&gt;
&lt;td&gt;Host from module, portably&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Confidential computing&lt;/td&gt;
&lt;td&gt;Hardware + attestation&lt;/td&gt;
&lt;td&gt;Continuously&lt;/td&gt;
&lt;td&gt;Guest from host&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LoRa mesh&lt;/td&gt;
&lt;td&gt;Keypairs&lt;/td&gt;
&lt;td&gt;Per packet&lt;/td&gt;
&lt;td&gt;Peers from each other&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The direction of travel is consistent: &lt;strong&gt;away from boundaries the hardware
chose for us, and towards boundaries we specify, verify and place where we
want them&lt;/strong&gt;. The cost model keeps shifting too, from per-crossing to
paid-once-up-front, and that's what makes fine-grained isolation affordable for
the first time.&lt;/p&gt;
&lt;p&gt;That's why "security" is the wrong single label for any
of this. Security is a consequence. What's being built is the ability to decide
where a boundary goes, instead of accepting wherever the MMU put it in 1985.&lt;/p&gt;
&lt;h2 id="the-strand-that-explains-the-other-five"&gt;The strand that explains the other five&lt;a class="headerlink" href="#the-strand-that-explains-the-other-five" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;One thing is missing from that table, and it's the thing that made all of it
urgent.&lt;/p&gt;
&lt;p&gt;In January 2018, Spectre and Meltdown made speculative execution everybody's
problem. The hardware boundary, the one we'd all been treating as the solid
floor underneath every design, turned out to leak across it by construction, in
every high-performance processor shipped for twenty years.&lt;/p&gt;
&lt;p&gt;Spectre did not start those five histories. It made their common boundary
problem urgent and put them on the same table. Before it, "the MMU
will handle it" was a perfectly reasonable engineering position. After it, all
five strands above read differently; not as researchers being clever, but as
five teams independently working out that a boundary you can't inspect is a
boundary you can't rely on.&lt;/p&gt;
&lt;p&gt;It has &lt;a href="/insights/spectre-meltdown-the-floor-moved/"&gt;its own entry&lt;/a&gt;, because
it deserves rather more than a paragraph.&lt;/p&gt;
&lt;h2 id="what-else-belongs-on-this-map"&gt;What else belongs on this map&lt;a class="headerlink" href="#what-else-belongs-on-this-map" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Five strands is the clean version of the story, and it's too clean. The fuller
list of what's been worth watching since roughly 2012 is longer and messier:
containers winning on ergonomics rather than isolation, the programmable
dataplane, ARM reaching the server room, the storage hierarchy collapsing, Rust
arriving in systems work, WireGuard, the supply chain turning into an attack
surface and formal methods becoming practical. Most of it feeds one of the
five.&lt;/p&gt;
&lt;p&gt;Several of those have their own entries already, and they are the ones to read
next if this map is useful:
&lt;a href="/insights/post-quantum-the-hard-half/"&gt;post-quantum migration&lt;/a&gt;, where key
exchange is largely solved and certificates aren't;
&lt;a href="/insights/cheri-pointers-with-bounds/"&gt;CHERI&lt;/a&gt;, putting bounds on pointers in
silicon, the hardware answer to the problem eBPF and WASM solve in software;
&lt;a href="/insights/riscv-production/"&gt;RISC-V reaching production&lt;/a&gt;;
&lt;a href="/insights/rust-reaches-systems/"&gt;Rust arriving in systems work&lt;/a&gt;;
&lt;a href="/insights/energy-first-class/"&gt;energy as a design axis&lt;/a&gt; rather than an
afterthought; and
&lt;a href="/insights/local-inference-on-the-bench/"&gt;running inference on the bench&lt;/a&gt;
rather than in somebody else's datacentre.&lt;/p&gt;
&lt;p&gt;What is not covered here and should be: the supply chain as a first-class
attack surface in its own right, and what happens to any of this when the
verification key sits with somebody else.&lt;/p&gt;
&lt;h2 id="where-we-think-this-goes"&gt;Where we think this goes&lt;a class="headerlink" href="#where-we-think-this-goes" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The strands are converging rather than competing, and the interesting systems
of the next few years will use several at once: a verified kernel hosting WASM
workloads, attested, with eBPF doing the observability that a sandbox otherwise
makes opaque. Each solves the granularity problem at a different altitude, and
none is a general answer alone.&lt;/p&gt;
&lt;p&gt;The part we would bet on is the cost model rather than any particular
technology. Once you pay for a boundary at load time instead of at every
crossing, the economics of "isolate this one function" change completely, and
most of the architectural decisions the industry made under the old economics
deserve another look.&lt;/p&gt;
&lt;h2 id="what-it-buys-everyone-else"&gt;What it buys everyone else&lt;a class="headerlink" href="#what-it-buys-everyone-else" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Cheap, fine-grained, verified boundaries are not an operating-systems hobby.
They are the precondition for a handful of things people have wanted for years
and could not afford:&lt;/p&gt;
&lt;p&gt;A hospital analysing patient data on hardware it does not own, where the
operator holds ciphertext and the privacy guarantee is arithmetic rather than a
contract clause. A fifteen-year-old infusion pump getting a security patch,
because a verified driver boundary means the vendor fixes one component instead
of recertifying a whole device. A flood-warning mesh in a valley no telco will
ever wire, running on nodes that hold their own keys and need no operator at
all. An engineer running a capable model on the machine under their desk, where
no prompt leaves the building. A small firm able to answer "what exactly is
running on this box" with evidence.&lt;/p&gt;
&lt;p&gt;Every one of those is a boundary problem dressed up as a policy problem. That
is the argument for the whole decade of work above.&lt;/p&gt;
&lt;h2 id="who-ends-up-holding-the-keys"&gt;Who ends up holding the keys&lt;a class="headerlink" href="#who-ends-up-holding-the-keys" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The uncomfortable half, and the reason we bother writing any of this down.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A boundary is only as good as the key that attests it, and somebody holds
that key.&lt;/strong&gt; Attestation proves what is running to whoever holds the
verification root, and today that root belongs to a silicon vendor. The cloud
operator genuinely cannot read your memory; you have swapped trusting Amazon
for trusting AMD. Better, and some distance short of ownership.&lt;/p&gt;
&lt;p&gt;The same shape turns up at every layer of this map: the public registry
everybody defaults to, the handful of certificate authorities, three clouds,
two app stores, the model weights, the accelerators. A technology that could
have gone either way acquires a chokepoint, and the chokepoint is where the
rent gets collected and where the surveillance capability sits, whether or not
anyone currently means to use it.&lt;/p&gt;
&lt;p&gt;Worse, these primitives cut both ways by design. The attestation that proves to
you that your workload is untampered proves to a platform that your device is
unmodified, which is the mechanism behind locked bootloaders and signed-only
firmware. That is a computer you own and cannot change, built from the tools
these five strands spent a decade perfecting. The technology has no opinion.&lt;/p&gt;
&lt;p&gt;Ours: &lt;strong&gt;we build infrastructure people can own, inspect and rebuild&lt;/strong&gt;, and we
are against the version of each technology above that works only while you keep
paying for it. That preference is why a mesh node holding its own key gets a
warmer write-up here than a cloud that will mind your secrets for a fee. The
open, self-hostable version of most of this stack exists; some of it is keeping
pace and some of it is quietly losing, and saying which is most of what this
site is for.&lt;/p&gt;
&lt;p&gt;Each strand has its own entry. This one is the map.&lt;/p&gt;</content><category term="insights"/><category term="microkernels"/><category term="wasm"/><category term="ebpf"/><category term="lorawan"/><category term="security"/></entry><entry><title>The patch took an afternoon. The inventory will take the week</title><link href="https://labs.axisops.com/insights/php2shell-the-inventory-problem/" rel="alternate"/><published>2026-10-21T21:00:00+01:00</published><updated>2026-10-21T21:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-21:/insights/php2shell-the-inventory-problem/</id><summary type="html">&lt;p&gt;Two unauthenticated remote code execution bugs in PHP core, CVSS 9.2, with upstream releases out the same afternoon. The patch is the easy part. CISA's first instruction is to inventory every system running any PHP, and on most estates nobody can do that tonight.&lt;/p&gt;</summary><content type="html">&lt;p&gt;PHP released 8.5.12, 8.4.27, 8.3.36 and 8.2.35 at 14:00 UTC today, alongside
the disclosure.&lt;/p&gt;
&lt;p&gt;CVE-2026-109618 and CVE-2026-109619, collectively php2shell, are
unauthenticated remote code execution on anything serving web requests, CVSS
9.2 from PHP's own security team. CISA published an alert this afternoon
telling organisations that cannot update immediately, or that are still on PHP
8.1 or earlier, to take their PHP systems offline until they can
(&lt;a href="https://www.cisa.gov/news-events/alerts/2026/10/21/critical-vulnerabilities-php-require-immediate-patching"&gt;CISA alert, 21 October&lt;/a&gt;).
That is not language CISA reaches for often. Read the alert in order, though,
and the instruction before it is the one that will cost people their week:
inventory every system running any version of PHP, including PHP embedded in
other software. Nobody is going to do that by the end of the day.&lt;/p&gt;
&lt;h2 id="what-the-bugs-are"&gt;What the bugs are&lt;a class="headerlink" href="#what-the-bugs-are" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Both live in request handling, and both need an earlier failure to get going.&lt;/p&gt;
&lt;p&gt;CVE-2026-109618 is memory corruption reached through an out-of-memory
condition. A request crosses &lt;code&gt;memory_limit&lt;/code&gt; partway through setup, PHP bails
out of native code before it has finished building its internal state, and the
cleanup path then treats a half-initialised structure as a valid one. Getting a
request to run out of memory on demand is the easy half: a multipart body with
no &lt;code&gt;Content-Length&lt;/code&gt; can slip past &lt;code&gt;post_max_size&lt;/code&gt;, the header count is
unbounded and &lt;code&gt;max_file_uploads&lt;/code&gt; does not count empty files
(&lt;a href="https://isc.sans.edu/diary/33440"&gt;SANS ISC, 21 October&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;CVE-2026-109619 is a use-after-free in the same neighbourhood. Per-request
global state is not always reset before the memory behind it is released, so a
worker that handled a failed request can act on the remains of it when it
handles the next one. It wants an error first, then a warning during request
initialisation, which a per-directory override such as an &lt;code&gt;.htaccess&lt;/code&gt; will
happily supply.&lt;/p&gt;
&lt;p&gt;So an attacker who can make your application fail gets to influence what the
worker does on somebody else's request afterwards. Every PHP serving requests is
in scope: php-fpm, Apache with mod_php, php-cgi, LiteSpeed, FrankenPHP. PHP
rates mod_php worst, because there the corruption can land before any of your
own code runs. An ordinary urlencoded form post reaches it, so the obvious
"block multipart at the proxy" workaround does not cover that case.&lt;/p&gt;
&lt;p&gt;Patched releases exist for 8.2 through 8.5 and that is all the good news. There
is no release for 8.1, 8.0 or 7.4. Upstream has published minimal source
backports as git branches for people who build their own, and everyone else is
waiting on a vendor. Below 7.4 there is nothing, so a great deal of the PHP on
the public internet has no fix tonight at any price.&lt;/p&gt;
&lt;h2 id="the-workarounds-all-have-a-sting"&gt;The workarounds all have a sting&lt;a class="headerlink" href="#the-workarounds-all-have-a-sting" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;session.upload_progress.enabled = 0&lt;/code&gt; closes the confirmed session path and
leaves the rest open. It has to be set somewhere that applies before request
parsing: php.ini, an FPM pool or the server config. An application calling
&lt;code&gt;ini_set()&lt;/code&gt; is far too late, and so is &lt;code&gt;.user.ini&lt;/code&gt;. That is the sort of detail
that produces a host recorded as mitigated that isn't.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;memory_limit = -1&lt;/code&gt; is the strange one. Removing the cap removes the
predictable failure the working exploit depends on, so upstream suggests it,
administratively, where the application cannot override it. You are deleting
your only guard against a single runaway request eating the box, so it needs a
cgroup limit underneath it before you touch it, and an allocation failure can
still reach the vulnerable path anyway.&lt;/p&gt;
&lt;p&gt;Both are the trade we keep making. With regreSSHion in 2024 the workaround
swapped remote root for a denial of service. Here you are choosing between an
availability risk you have measured and a remote code execution you have not.
Take the trade, then get off it.&lt;/p&gt;
&lt;h2 id="the-question-that-has-no-quick-answer"&gt;The question that has no quick answer&lt;a class="headerlink" href="#the-question-that-has-no-quick-answer" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;You cannot patch what you cannot enumerate, and PHP is close to the worst case.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;php -v&lt;/code&gt; answers for one binary on the PATH. A web host runs several PHP
versions at once by design, one per customer; a server runs php-fpm for the
application and a CLI for cron, usually different builds; containers carry
their own, invisible to anything reading the host's packages. Behind that sits
the category CISA names and nobody tracks: PHP embedded in other software,
appliances and vendor bundles with a copy nobody chose.&lt;/p&gt;
&lt;p&gt;Then the version string lies to you. Distributions backport the patch without
touching the upstream number, so a Debian or Ubuntu host with the fix installed
still prints the old version. The only thing that decides is the package
revision against that vendor's fixed version for that release, and for 8.1 and
older there is no vendor answer yet to compare against.&lt;/p&gt;
&lt;p&gt;So the verdict on a lot of hosts tonight is unknown, and that is the verdict
worth reporting. A false "vulnerable" sends somebody to remove a memory limit
on a box that was already safe. A false "fixed" is worse.&lt;/p&gt;
&lt;h2 id="there-is-a-role-and-it-mostly-counts-things"&gt;There is a role, and it mostly counts things&lt;a class="headerlink" href="#there-is-a-role-and-it-mostly-counts-things" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;We are writing &lt;code&gt;php2shell_mitigation&lt;/code&gt; tonight, same shape as the five before it,
and it will go up at
&lt;a href="https://github.com/axisops/CVE-2026-109618-109619"&gt;github.com/axisops/CVE-2026-109618-109619&lt;/a&gt;
like the rest. It is an Ansible role that deploys one Python detector, runs it
read-only and runs it again with &lt;code&gt;--apply&lt;/code&gt; only where a published fix exists and
is not installed. The fixed-version table is the artefact that takes the time.
It lives in the role's &lt;code&gt;defaults/main.yml&lt;/code&gt;, keyed by os-release ID and codename,
one row per release carrying the vendor's first fixed package and where it was
read from. Tonight most of its 8.1 rows will be &lt;code&gt;null&lt;/code&gt;: affected, nothing
shipped.&lt;/p&gt;
&lt;p&gt;Discovery leads, because that is where the work is. Every SAPI rather than one:
php-fpm masters and their pools, mod_php inside Apache, php-cgi, the CLI and
the versioned binaries beside it. For each, the owning package against that
release's fixed version, with dpkg, rpm and apk comparison implemented in
Python so one host can judge a container running another distribution. Then
&lt;code&gt;memory_limit&lt;/code&gt;, &lt;code&gt;session.upload_progress.enabled&lt;/code&gt; and &lt;code&gt;fatal_error_backtraces&lt;/code&gt;
read per SAPI through &lt;code&gt;php-fpm -i&lt;/code&gt; and the pool config rather than from
php.ini, because a value set in the wrong place is the difference between
mitigated and recorded as mitigated.&lt;/p&gt;
&lt;p&gt;Containers get a shell script piped through &lt;code&gt;docker exec&lt;/code&gt;, and are reported
rather than changed, because a package upgraded inside a running container dies
with it. That probe fails usefully: an image with no shell comes back
&lt;strong&gt;unknown&lt;/strong&gt; and named instead of counting as clean. &lt;strong&gt;no_fix&lt;/strong&gt; and &lt;strong&gt;unknown&lt;/strong&gt;
are the verdicts carrying the value here, and refusing to infer either from the upstream version is what makes them worth
having.&lt;/p&gt;
&lt;p&gt;Three things it will not do. A PHP compiled into &lt;code&gt;/usr/local&lt;/code&gt; or baked into an
appliance has no package to judge, so it is reported rather than guessed at. A
patched package on disk is not a patched service: php-fpm masters and Apache
carry the old code until they restart, so the role names every process still
mapping a replaced binary and restarts nothing unless asked, because a restart
is an outage and the operator picks the hour. And there is no rollback, because
a downgrade here reinstalls an unauthenticated remote code execution. The
backup is a record of what was replaced, not an undo button.&lt;/p&gt;
&lt;p&gt;Which is the uncomfortable part. Most of the people patching tonight will
spend it working out where PHP is, for the third time in five years against a
different dependency. Applying the fix is an afternoon. Knowing what PHP you
have, from what package, on what release, in what container, is the week, and
it is the same week every time, because nobody writes the answer down in a
form that survives to the next disclosure. Everything else is archaeology
with a deadline. We are going looking, again.&lt;/p&gt;
&lt;p&gt;Ours is declared, because Microcelium holds the declaration, so tonight was
the afternoon's work without the week in front of it. The week was spent
years ago by somebody who wrote the answer down in a form that survived, and
that is the whole of the advantage. There is a
&lt;a href="https://www.axisops.com/insights/php-cve-26-minutes"&gt;write-up of the run&lt;/a&gt; on
the company site.&lt;/p&gt;</content><category term="insights"/><category term="security"/><category term="php"/><category term="ansible"/><category term="python"/><category term="operations"/><category term="mitigation"/></entry><entry><title>CHERI: putting the bounds in the pointer</title><link href="https://labs.axisops.com/insights/cheri-pointers-with-bounds/" rel="alternate"/><published>2026-10-20T09:00:00+01:00</published><updated>2026-10-20T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-20:/insights/cheri-pointers-with-bounds/</id><summary type="html">&lt;p&gt;The hardware answer to the problem eBPF and WASM solve in software: a pointer that carries its own bounds and permissions, enforced in silicon. Still an unratified RISC-V extension, whatever the coverage implies.&lt;/p&gt;</summary><content type="html">&lt;p&gt;On every mainstream machine, a C pointer is represented as a bare integer
address. That single fact is at the root of an enormous share of the security
vulnerabilities ever shipped: nothing in the representation says what the
pointer is allowed to reach.&lt;/p&gt;
&lt;p&gt;CHERI (Capability Hardware Enhanced RISC Instructions) makes a pointer a
&lt;strong&gt;capability&lt;/strong&gt; instead: an address, plus bounds, plus permissions, plus a
validity tag, with the hardware enforcing all of it on every dereference.&lt;/p&gt;
&lt;h2 id="what-that-buys-you"&gt;What that buys you&lt;a class="headerlink" href="#what-that-buys-you" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A buffer overflow stops being a memory-safety bug and becomes a trap. Not
spotted afterwards by a sanitiser, not caught now and then by a canary;
refused in hardware, because the processor won't perform an access
outside the bounds.&lt;/p&gt;
&lt;p&gt;The tag is the clever part. It's a bit held outside the addressable data and
maintained by the hardware, and it drops to zero if the capability is
manipulated with ordinary arithmetic. You can't forge a capability by
constructing an integer. Provenance is therefore unforgeable: a capability can
only be derived from another with equal or lesser rights.&lt;/p&gt;
&lt;p&gt;That property, &lt;strong&gt;monotonic non-increase of authority&lt;/strong&gt;, is what makes CHERI a
capability system rather than bounds-checked pointers with better marketing.&lt;/p&gt;
&lt;h2 id="where-it-is-today"&gt;Where it is today&lt;a class="headerlink" href="#where-it-is-today" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The RISC-V CHERI specification is &lt;strong&gt;not ratified&lt;/strong&gt;. The most recent version we
can find is an intermediate release, v0.9.10.1, dated 2026-10-14
(&lt;a href="https://github.com/riscv/riscv-cheri/releases"&gt;riscv/riscv-cheri releases&lt;/a&gt;).
Hot Chips 2026 coverage lists CHERI among work-in-progress items. The base is
designated RV64Y.&lt;/p&gt;
&lt;p&gt;There's also a compatibility wrinkle: Zicfiss is incompatible at present,
because its shadow-stack push and pop instructions would need modifying for
CHERI.&lt;/p&gt;
&lt;p&gt;Arm's Morello was a research prototype of the same ideas on ARMv8 and produced
real evaluation results. It was a research platform, not a product line.&lt;/p&gt;
&lt;p&gt;So: promising, extensively researched, demonstrably implementable and not
something you can buy and deploy.&lt;/p&gt;
&lt;h2 id="what-it-would-cost"&gt;What it would cost&lt;a class="headerlink" href="#what-it-would-cost" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Pointers get bigger.&lt;/strong&gt; A 128-bit capability where a 64-bit pointer used to
  be. On pointer-dense workloads that's cache pressure you'll feel.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Source changes are needed.&lt;/strong&gt; Not a recompile. Code that stores pointers in
  integers, does its own tagging in spare bits or reconstructs pointers from
  arithmetic (all common in systems code) has to change.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The whole stack has to agree.&lt;/strong&gt; A capability-aware kernel, a
  capability-aware libc and a capability-aware compiler. You don't get to do
  this one layer at a time.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The prize is larger than the coverage suggests, and it is not only a security
prize. Roughly two thirds of serious vulnerabilities in systems software are
memory safety. The industry's answer so far is to rewrite in a safe language,
which works and which will take thirty years, because the C and C++ in service
is measured in billions of lines and most of it has no owner. CHERI is the
other route: keep the code, recompile it and have the hardware refuse the
access.&lt;/p&gt;
&lt;p&gt;Be precise about which half that is. A recompile buys &lt;strong&gt;spatial&lt;/strong&gt; safety:
every pointer carries its bounds and its permissions, so a read or write
outside the object it was derived from is refused by the hardware. That is
the buffer-overflow family, and it is most of the historical catalogue.&lt;/p&gt;
&lt;p&gt;It does not, on its own, buy &lt;strong&gt;temporal&lt;/strong&gt; safety. A pointer to freed memory
is still a valid capability until something invalidates it, and use-after-free
is half of Chromium's high-severity memory bugs. The answer is capability
revocation: sweeping memory for capabilities to freed regions, as CHERIvoke
and Cornucopia do. That is a real cost, paid at runtime, on top of the
recompile. It is the unfinished half, and anyone selling CHERI as the end of
memory unsafety is selling the easy part.&lt;/p&gt;
&lt;p&gt;The second-order effect is the one we would watch. A crash is a bug report; a
breach is an incident, a disclosure, a regulator and a fortnight. The economics
of maintaining old software change completely once its worst failure mode is
downtime. A great deal of working equipment then gets to keep working safely
instead of being replaced for security reasons. That is a repairability and a
waste argument as much as a security one, and nobody makes it.&lt;/p&gt;
&lt;p&gt;For engineering itself the win is debugging. A capability violation traps at
the instruction that did it, with bounds attached. Compare the current
experience: memory corrupted at ten in the morning, crash at four in the
afternoon, somewhere else entirely, in innocent code. The bug class that
disappears is the one where symptom and cause have nothing to do with each
other, and it is the most expensive class there is.&lt;/p&gt;
&lt;p&gt;Compartmentalisation is what the research community cares about most, and it is
furthest from deployment. Cheap unforgeable capabilities let you split a large
program into mutually distrusting parts without processes or IPC costs: the
image decoder in a browser, the parser in a mail server, each holding only the
memory it was handed.&lt;/p&gt;
&lt;p&gt;The new boundary moves the chokepoint to intellectual property.&lt;/p&gt;
&lt;p&gt;CHERI came out of Cambridge and SRI, with substantial public funding, and the
research is published. The question that decides whether it becomes
infrastructure or a feature: can anybody implement it, or only those who can do
a deal with Arm?&lt;/p&gt;
&lt;p&gt;Both routes are live at once. The RISC-V extension, if it ratifies, is
implementable by anyone who can make a chip; that is the version where a small
vendor, a university or a state-funded programme can build capability hardware
without asking permission. The Arm route reaches far more devices far sooner,
because Arm is in everything, and on Arm's licensing terms and schedule.
Morello was a research prototype; whether the ideas reach a shipping Arm
architecture, in what form and under what terms, is Arm's commercial decision
rather than a technical one.&lt;/p&gt;
&lt;p&gt;Watch for the version that is worst for everybody: memory safety as a premium
architectural feature. Available in the high-margin parts, absent from the cheap
ones, so the devices hardest to patch and longest-lived in the field (the meter,
the camera, the controller in a building) keep shipping the unsafe pointer while
phones and servers get the protection. Security stratified by unit price, with
the exposure concentrated where nobody is maintaining the software. The cost is
not nothing: doubling pointer width pushes on cache and memory bandwidth, as
above, and the tag bits have to be carried through the whole hierarchy. But it
is a one-off silicon and bandwidth cost rather than a per-unit licence. It will
land in the premium tier anyway if the only licensable implementation is a
premium one.&lt;/p&gt;
&lt;p&gt;The specification work is happening in the open in RISC-V, and the compiler and
kernel work exists as open source from the research programme. Here at
AxisOps our position is that publicly funded memory safety
ought to be universally implementable, and the test is simple: if a chip
designer with no Arm relationship cannot build it, the public money bought a
competitive advantage rather than a public good.&lt;/p&gt;
&lt;h2 id="where-it-sits-among-the-strands"&gt;Where it sits among the strands&lt;a class="headerlink" href="#where-it-sits-among-the-strands" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;CHERI is the hardware answer to the same problem eBPF solves with a verifier
and WASM solves with validation.&lt;/p&gt;
&lt;p&gt;The software approaches check once at load time, after which the crossing costs
nothing even though the checked code still costs what it costs. They are limited
to code they can analyse. CHERI checks on every dereference, in hardware, with
no analysis needed and no language restriction; the price is new silicon and a
recompiled world. Neither is strictly better: they're the same instinct (make
authority explicit and unforgeable) applied at different points in the stack.&lt;/p&gt;</content><category term="insights"/><category term="cheri"/><category term="hardware"/><category term="memory-safety"/><category term="riscv"/></entry><entry><title>What a microkernel actually costs you</title><link href="https://labs.axisops.com/insights/microkernel-isolation/" rel="alternate"/><published>2026-10-16T09:00:00+01:00</published><updated>2026-10-16T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-16:/insights/microkernel-isolation/</id><summary type="html">&lt;p&gt;The IPC objection to microkernels is fair, and it is the wrong place to stop the conversation, because what it buys you is a fault boundary that a monolithic kernel cannot sell at any price.&lt;/p&gt;</summary><content type="html">&lt;p&gt;The standard objection to microkernels is IPC cost. The cost is real, and it is
where the discussion usually stops. That is the problem.&lt;/p&gt;
&lt;p&gt;What you're buying is a fault boundary around a driver, and a monolithic kernel
can't sell you one at any price, because a driver bug &lt;em&gt;is&lt;/em&gt; a kernel bug by
construction.&lt;/p&gt;
&lt;h2 id="the-structural-claim"&gt;The structural claim&lt;a class="headerlink" href="#the-structural-claim" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;In Linux, a device driver runs in kernel address space with kernel privileges.
There's no boundary to cross because there's no boundary there at all. A null
dereference in a driver is a kernel oops; memory corruption can corrupt anything
it likes.&lt;/p&gt;
&lt;p&gt;This isn't an argument about code quality. You can write excellent driver code
and the structural property doesn't budge. Drivers are also where the largest
volume of kernel code lives and where the least-reviewed code lives, an
uncomfortable combination on anyone's reading.&lt;/p&gt;
&lt;p&gt;Put the driver in its own address space and a fault takes down the driver. A
supervisor restarts it; the system carries on.&lt;/p&gt;
&lt;h2 id="what-the-proof-says"&gt;What the proof says&lt;a class="headerlink" href="#what-the-proof-says" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;seL4's functional correctness proof gets cited loosely, so to be precise about
it: the C implementation is proven to refine an abstract specification, with no
unproven gap between the two. It doesn't mean "no bugs", because a specification
can specify the wrong thing. It means the implementation and the specification
provably agree.&lt;/p&gt;
&lt;p&gt;That holds across Arm 32-bit, AArch64, x86-64 and RISC-V 64-bit, AArch64 having
reached parity on functional correctness
(&lt;a href="https://sel4.systems/Summit/2024/abstracts2024.html"&gt;seL4 Summit 2024&lt;/a&gt;). The
MCS configuration now has a functional correctness proof on 64-bit RISC-V,
completed by Proofcraft in June 2026 and shipped in seL4 16.0.0
(&lt;a href="https://www.sel4.systems/news/"&gt;seL4 news&lt;/a&gt;); the AArch64 port of that proof is
in progress.&lt;/p&gt;
&lt;p&gt;On the 32-bit Arm configuration the proof also covers integrity and
confidentiality above functional correctness, which is the part that lifts it
out of academic curiosity. Confidentiality there means non-interference
between the domain scheduler's domains, not between arbitrary threads. The
other architectures have functional correctness alone; the stronger
properties are not verified there.&lt;/p&gt;
&lt;h2 id="where-the-cost-lands"&gt;Where the cost lands&lt;a class="headerlink" href="#where-the-cost-lands" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The useful question about IPC isn't what one round trip costs. It's &lt;strong&gt;how often
you hit the path&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;A few thousand calls a second against a microsecond-scale IPC is noise. A
million calls a second is the design. So the engineering work in a microkernel
system is almost all batching: making the interface coarse enough that you
rarely cross the boundary. That's the same optimisation as any RPC boundary, and
a design constraint rather than a tax.&lt;/p&gt;
&lt;p&gt;The open question is the sizing one, and it is answerable: on a named part, at a
named kernel version, what does the IPC path cost against the equivalent
syscall, and at what call rate does the batching argument stop holding? That
measurement decides the architecture for a whole class of embedded system and we
have not taken it on our own hardware. It is on the bench list, with the part
number and the command, and no numbers appear here until it is.&lt;/p&gt;
&lt;h2 id="what-a-verified-boundary-makes-possible"&gt;What a verified boundary makes possible&lt;a class="headerlink" href="#what-a-verified-boundary-makes-possible" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The structural property changes what can be built and certified.&lt;/p&gt;
&lt;p&gt;Take a car. A modern vehicle carries dozens of controllers because the
infotainment stack cannot be allowed near the braking logic, so they get
separate silicon, wiring and suppliers. With a verified separation kernel
underneath, one processor can host the instrument cluster, the entertainment
system and a safety function, isolation proven rather than asserted. That
removes boxes, weight, wiring and cost from every vehicle built, and the same
reasoning applies to avionics, industrial controllers and medical devices.&lt;/p&gt;
&lt;p&gt;The certification effect is bigger than the hardware saving. Today, changing
anything in a safety-certified system means re-arguing the safety case for the
whole box, because nothing structurally prevents one component affecting
another. With proven isolation you argue about one partition at a time. That is
the difference between a device that gets a security update and one that does
not, which is why so much safety-critical equipment in the field runs software
nobody has touched in a decade. Proven separation is how you patch an infusion
pump.&lt;/p&gt;
&lt;p&gt;It also buys survivability where there is no staff and no second chance. A
driver fault a supervisor can restart is a system that can sit in a substation
or a satellite and keep going, which spacecraft software has historically bought
with redundant hardware instead.&lt;/p&gt;
&lt;h2 id="where-it-falls-down"&gt;Where it falls down&lt;a class="headerlink" href="#where-it-falls-down" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Driver availability.&lt;/strong&gt; Linux's enormous driver tree is its real moat, and a
  microkernel hasn't got one. For most hardware that settles the argument.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Shared-memory-heavy workloads&lt;/strong&gt;, where the batching argument doesn't apply
  because the data is fine-grained.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Developer familiarity.&lt;/strong&gt; The pool of people who can debug a microkernel
  system is far smaller.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Those explain why Linux won the general case. None touches the structural claim,
which is why microkernels keep winning where a driver fault is unacceptable:
avionics, automotive, secure enclaves.&lt;/p&gt;
&lt;p&gt;In the end, verified systems come down to the licence on the proof. seL4 is open
source, and so are its proofs. That combination is unusual and the thing to
protect. A published proof can be inspected, built on, ported and argued with by
anybody, so a four-person company can ship on a formally verified base without
employing a proof engineer: the expensive artefact has already been made and
given away. That redistribution happened because the work was publicly funded
and published.&lt;/p&gt;
&lt;p&gt;The alternative model is already common here. Commercial separation kernels are
certified to the relevant standards and sold per unit with the evidence package
under NDA: you get a certificate and a binary. You cannot read the proof, port
it or verify the claim yourself, and your safety argument depends on a document
you are not allowed to show anyone. Same technology, opposite ownership.&lt;/p&gt;
&lt;p&gt;Watch where the funding points. Much of the serious verification work in this
field runs on defence research money. That is how it gets done at all, and it
comes with a preference for the results reaching some people rather than
everyone. The optimistic case is that proofs are artefacts and artefacts get
published. The pessimistic one is a verified trusted base that is
export-controlled while the open one stays a research kernel. The difference
between those futures is licensing decisions being taken now by people not
thinking about it hard.&lt;/p&gt;
&lt;p&gt;Here at AxisOps we would build on the verified base we can read, and not for
ideology. A safety case resting on a proof you are forbidden from examining has
swapped one trust relationship for another and charged you for it. Verification
converts a claim into something checkable; a proof under NDA converts it back.&lt;/p&gt;
&lt;h2 id="the-price-of-enforcing-the-boundary"&gt;The price of enforcing the boundary&lt;a class="headerlink" href="#the-price-of-enforcing-the-boundary" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The microkernel is the oldest approach to isolation and the only one where the
boundary is enforced by hardware. Everything newer, eBPF and WASM included, is
an attempt at a comparable guarantee without paying for it on every crossing.&lt;/p&gt;</content><category term="insights"/><category term="microkernels"/><category term="kernel"/><category term="formal-methods"/></entry><entry><title>LoRaWAN and mesh are not the same argument</title><link href="https://labs.axisops.com/insights/lorawan-vs-mesh/" rel="alternate"/><published>2026-10-12T09:00:00+01:00</published><updated>2026-10-12T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-12:/insights/lorawan-vs-mesh/</id><summary type="html">&lt;p&gt;They share a radio layer and almost nothing else. One has a network server and a join procedure; the other has neither, by design. Conflating them makes both sound worse than they are.&lt;/p&gt;</summary><content type="html">&lt;p&gt;LoRa is a modulation: Semtech's chirp spread spectrum, operating in sub-GHz ISM
bands, trading data rate for range and power.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;LoRaWAN&lt;/strong&gt; and &lt;strong&gt;Meshtastic&lt;/strong&gt; both use it, and they're architecturally
opposite. They get discussed interchangeably, which does neither of them any
favours. Both run on tarmac, and so do a bus route and a rally stage.&lt;/p&gt;
&lt;h2 id="lorawan-is-a-star-with-a-server"&gt;LoRaWAN is a star with a server&lt;a class="headerlink" href="#lorawan-is-a-star-with-a-server" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Devices talk to gateways. Gateways forward to a network server. The network server handles deduplication (several gateways often hear the same
packet) and adaptive data rate, and in the 1.0.x deployments that dominate it also manages session keys.
LoRaWAN 1.1 moves the join to a separate join server,
which changes who holds the keys rather than whether somebody does. An
application server sits behind that.&lt;/p&gt;
&lt;p&gt;Devices join via OTAA, deriving session keys from a root key, with the
operator's infrastructure as the authority. There's a &lt;code&gt;DevEUI&lt;/code&gt;, a &lt;code&gt;JoinEUI&lt;/code&gt; and an entity whose
job is deciding whether you're allowed on.&lt;/p&gt;
&lt;p&gt;The properties that follow:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Range is gateway-to-device&lt;/strong&gt;, which can be kilometres, and your coverage is
  whatever your gateways cover.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Device power consumption is excellent.&lt;/strong&gt; Class A devices transmit and then
  open two short receive windows, sleeping the rest of the time. Years on a
  battery is real.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;There's an operator&lt;/strong&gt;, whether that's you or a public network.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Downlink is constrained&lt;/strong&gt;, which surprises people. You can't generally
  reach a Class A device on demand; you wait for it to transmit.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="meshtastic-is-peers-with-no-server"&gt;Meshtastic is peers with no server&lt;a class="headerlink" href="#meshtastic-is-peers-with-no-server" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;No gateway, no network server, no join procedure. Nodes relay for each other,
and your reach is a function of how many nodes sit between you and the
destination.&lt;/p&gt;
&lt;p&gt;The properties that follow:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Coverage is emergent.&lt;/strong&gt; It's whatever the node density gives you, and it
  changes as people move about.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Power consumption is worse&lt;/strong&gt;, because a relaying node has to listen.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;There's no operator and no authority.&lt;/strong&gt;
  &lt;a href="/insights/meshtastic-node-is-its-key/"&gt;A node is its key&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Capacity is low and shared.&lt;/strong&gt; Every relay costs airtime, and duty-cycle
  limits in ISM bands are a hard constraint. A busy mesh congests.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="when-each-is-right"&gt;When each is right&lt;a class="headerlink" href="#when-each-is-right" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;LoRaWAN&lt;/strong&gt; suits fixed sensors reporting small amounts of data where you
control the coverage or can buy it: utility metering, agricultural sensing,
building monitoring. The star topology and the sleeping devices are right for
that job.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Meshtastic&lt;/strong&gt; suits places where there's no infrastructure and none is coming,
and where the participants move: remote terrain, event coordination, resilience
when the other networks are down.&lt;/p&gt;
&lt;p&gt;They overlap far less than the shared radio suggests.&lt;/p&gt;
&lt;h2 id="the-duty-cycle-constraint-both-share"&gt;The duty cycle constraint both share&lt;a class="headerlink" href="#the-duty-cycle-constraint-both-share" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;This one is routinely discovered late, so know it early. ISM band regulation
limits transmit duty cycle; in Europe commonly 1% in the relevant sub-bands.&lt;/p&gt;
&lt;p&gt;That's a hard ceiling on how much either system can say, and it's why LoRa is
for telemetry and short messages rather than anything resembling bulk data. A
design that assumes it can send whenever it likes will work beautifully on a
bench and fail a regulatory check.&lt;/p&gt;
&lt;p&gt;The price of coverage is where the radio argument becomes a human one.&lt;/p&gt;
&lt;figure&gt;
  &lt;img src="/images/p-radio-mast.jpg"
       srcset="/images/p-radio-mast-600.webp 600w,
               /images/p-radio-mast-1000.webp 1000w,
               /images/p-radio-mast-1600.webp 1600w"
       sizes="(max-width: 1180px) 100vw, 1180px"
       alt="A lattice radio mast with sector antennas, rising above a treeline against grey cloud" loading="lazy" decoding="async"&gt;
  &lt;figcaption&gt;Somebody paid for this one. A LoRaWAN gateway on a pole costs a few hundred pounds; the tower behind the national networks is a different kind of commitment, and it is the reason coverage is a commercial question rather than a technical one.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;p&gt;A LoRaWAN gateway costs a few hundred pounds and covers a few kilometres. That
means a farm, a university campus, a canal trust, a parish council can own its
own network outright, with no monthly bill and no contract, and put two hundred
sensors on it. The thing that becomes cheap is measuring a place. River levels
across a catchment, soil moisture across a holding, temperature in every flat in
a block to find the ones with damp, bin levels, bridge strain, bat activity.
These are all projects that previously needed either a cellular subscription per
device or a cable, and the subscription is what killed them. The same hardware
in a lower-income country does the same job without requiring a mobile operator
to have found the area worth covering.&lt;/p&gt;
&lt;p&gt;Meshtastic answers a different question: what works when nothing works. A
network that comes into being because people turned up with nodes, with no
operator to bill and no infrastructure to fail, is the right shape for a flooded
valley, a mountain rescue, a festival, a power cut lasting a week. Here at
AxisOps we think the resilience case is badly undervalued because it is only needed occasionally. The plans most
organisations hold for the loss of
mobile coverage amount to hoping it does not happen.&lt;/p&gt;
&lt;p&gt;For engineering, the lesson that generalises beyond radio is in the duty cycle.
A hard regulatory ceiling on transmission forces you to decide what is worth
saying, and designs built under that constraint are better: event-driven rather
than polling, deltas rather than full state, local decisions rather than a round
trip to ask permission. Plenty of systems with no radio at all would benefit
from being written as though airtime cost money.&lt;/p&gt;
&lt;p&gt;Ownership follows the architectural split, which is why these two are worth
contrasting rather than comparing.&lt;/p&gt;
&lt;p&gt;For LoRaWAN it is the network server. Whoever runs it holds the routing,
the network session keys, every device's traffic metadata and the decision
about who is allowed on the network. Under 1.0 it also holds the join
authority and the application keys. One reason 1.1 exists is to split those
off, into a join server and application keys the network server never sees,
wherever a deployment runs 1.1. Use a public network and you have a telecoms
relationship with everything that implies: coverage you do not control, a price
that can change, a per-device fee and a company that knows where your sensors
are and what they report. The devices are cheap and the dependency is not. The
good news, and the reason we keep recommending it, is that the network server is
software you can run yourself on a small machine, and the open implementations
are mature. A LoRaWAN estate can be entirely yours, gateway to application,
with nobody to pay. That is a choice most deployments do not realise they are
making, because the hosted option is the one in the quick-start guide.&lt;/p&gt;
&lt;p&gt;The pattern to watch is the quiet one: hosted network servers are free at the
scale where you evaluate and priced per device at the scale where you depend on
them. By then the join credentials, the device provisioning and the integrations
all live in somebody's console, and the migration is a site visit to every
sensor. Decide this on day one, because it is cheap on day one and expensive
forever after.&lt;/p&gt;
&lt;p&gt;Meshtastic has no such chokepoint, and that is its reason to exist. No
operator means nobody to charge you, nobody to shut it down, nobody to serve an
order on and nobody holding a list of participants. The cost is that you get no
guarantees at all: no coverage commitment, no capacity, no delivery, and the
network exists only while people keep their nodes on. That is an engineering trade, not an ideological one. What we would say is that the
option of a network with no owner is a thing a society should keep, in the same
way as footpaths, and it survives because the hardware is cheap and the firmware
is open rather than because anyone is defending it.&lt;/p&gt;
&lt;h2 id="the-limit-of-both"&gt;The limit of both&lt;a class="headerlink" href="#the-limit-of-both" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Bandwidth. These are kilobits, and often hundreds of bits per second at long
range and high spreading factors. Any design that drifts towards "and then we
send the image" has left the technology somewhere behind it.&lt;/p&gt;</content><category term="insights"/><category term="lorawan"/><category term="mesh"/><category term="networking"/><category term="iot"/></entry><entry><title>Meshtastic: a node is its key</title><link href="https://labs.axisops.com/insights/meshtastic-node-is-its-key/" rel="alternate"/><published>2026-10-10T09:00:00+01:00</published><updated>2026-10-10T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-10-10:/insights/meshtastic-node-is-its-key/</id><summary type="html">&lt;p&gt;Dismissed as hikers with walkie-talkies. Read the 2.8 release notes as an identity architecture instead: node IDs derived from public keys, packets signed and no registry to ask because there's nothing there to ask.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Meshtastic is LoRa radios running a mesh protocol with no infrastructure
underneath it. No base station, no carrier, no DNS, no certificate authority.
Nodes relay for each other and the topology is whatever happens to be in range.
The "hobbyist" framing does a lot of work to stop people looking at the
engineering.&lt;/p&gt;
&lt;h2 id="read-the-release-notes-as-an-identity-system"&gt;Read the release notes as an identity system&lt;a class="headerlink" href="#read-the-release-notes-as-an-identity-system" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The 2.8 line's changes, from the
&lt;a href="https://github.com/meshtastic/firmware/releases"&gt;firmware releases&lt;/a&gt; and the
&lt;a href="https://meshtastic.org/blog/"&gt;project blog&lt;/a&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Packet signing via XEdDSA.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Node IDs derived from the node's public-key identity&lt;/strong&gt; rather than from its
  hardware MAC address.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A traffic management module&lt;/strong&gt; doing deduplication and rate limiting.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automatic variable hop limits&lt;/strong&gt; responding to observed mesh activity.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The second is the architecturally interesting one. A node &lt;em&gt;is&lt;/em&gt; its keypair.
Nothing assigns the identity; it's derived from a secret the node holds. So
there's no registry you can ask whether a node is who it claims to be, because
there's nothing to ask. Trust gets established peer to peer or not at all.
That does not make signatures a mesh-wide invariant: the
&lt;a href="https://meshtastic.org/docs/overview/encryption/#signed-broadcasts"&gt;default packet-signature policy&lt;/a&gt;
accepts unsigned traffic so that older nodes remain usable. Strict mode is what
turns rejection of unsigned traffic into a local rule.&lt;/p&gt;
&lt;h2 id="why-that-is-unusual"&gt;Why that is unusual&lt;a class="headerlink" href="#why-that-is-unusual" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Nearly every identity system we build has an authority somewhere: a CA, a
directory, an identity provider, a registrar. The authority is what makes names
mean anything, and it's also a central point of compulsion, failure and cost.
Self-certifying identifiers do away with it: name and key are the same thing, so
verification happens locally.&lt;/p&gt;
&lt;p&gt;The trade is real: &lt;strong&gt;you lose human-meaningful names.&lt;/strong&gt; A public key is not a
name anyone can remember or type. Every system that has tried this, from PGP
fingerprints to onion addresses, walks into the same wall, and the usual answer
is a petname layer: locally-assigned nicknames mapped to keys, like your phone's
contacts list, where "Mum" means something to you and nothing to the network.&lt;/p&gt;
&lt;p&gt;It's the same trade
&lt;a href="/insights/crypto-wars-round-two/"&gt;the crypto wars entry&lt;/a&gt; touches from the
other side. A system with no registry cannot be served an order for its
registry.&lt;/p&gt;
&lt;h2 id="a-deliberate-compatibility-break"&gt;A deliberate compatibility break&lt;a class="headerlink" href="#a-deliberate-compatibility-break" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Firmware 2.7.15 (November 2025) disallowed the legacy direct-message path,
which protected a DM with the shared channel key rather than the recipient's
own, after CVE-2025-53627 showed it let a sender be spoofed. Nodes on older
firmware could still relay traffic and use channels, and their DMs broke
until they updated.&lt;/p&gt;
&lt;p&gt;That's a security fix willing to break compatibility, with a cost attached. The opposite choice, keeping the insecure path working indefinitely,
is the one almost everybody makes, and it's why so many protocols carry
downgrade attacks.&lt;/p&gt;
&lt;h2 id="what-a-network-with-no-operator-makes-possible"&gt;What a network with no operator makes possible&lt;a class="headerlink" href="#what-a-network-with-no-operator-makes-possible" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A group of people can have communications that nobody provides. Twenty pounds of
hardware per person, no SIM, no account, no monthly bill, no coverage map, no
terms of service and no company that can be acquired, go bust, raise its prices
or be ordered to hand over a participant list. For most of the history of
telecommunications that was not available at any price.&lt;/p&gt;
&lt;p&gt;That matters where infrastructure reaches badly. A valley with no mobile
coverage, where the network exists because the farms each have a node on a barn
roof. A week-long power cut, when the mobile network's batteries have gone and
the mesh is still running because each node draws less than a light bulb. The
network comes into being from the participants rather than being supplied to
them.&lt;/p&gt;
&lt;p&gt;Where the local infrastructure exists but cannot be trusted, the same property
matters most. A network with no operator cannot be asked for records it
does not keep, and a direct message encrypted to a key held on the receiving
device is not readable by the relay nodes that carry it. The relaying itself
is visible, headers included, and anything on a shared channel is readable by
whoever holds that channel's key, so the property is confidentiality of
content, not invisibility. That is a real capability for a
journalist, a trade union or a family in a country where the carrier is an arm
of the state.&lt;/p&gt;
&lt;p&gt;The engineering lesson generalises beyond radio. Self-certifying identity removes
an entire category of system: the provisioning service, the registration flow,
the credential database, the password reset, the breach of that database. A node
generates its own identity and is immediately addressable. Anybody who has built
a device fleet knows provisioning is most of the work and most of the risk, and
here it is absent.&lt;/p&gt;
&lt;h2 id="the-key-is-on-the-node"&gt;The key is on the node&lt;a class="headerlink" href="#the-key-is-on-the-node" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;There is no chokepoint here, by design. There is no key to hold over
you: it is generated on the node, never escrowed, and the identity is derived
from it rather than assigned to it. No registry, so no participant list to
subpoena, leak or sell. No operator, so no company to acquire, no account to
suspend, nobody to serve a notice on. The firmware is open and the hardware is
commodity, so no supply chain chokepoint beyond a radio module several vendors
make.&lt;/p&gt;
&lt;p&gt;Meshtastic is the clearest evidence that the ownable version is not a compromise
forced by poverty: it beats the commercial alternative on resistance to
compulsion, privacy and resilience, and loses only on bandwidth and convenience.&lt;/p&gt;
&lt;p&gt;Two caveats. The absence of an authority means the absence of guarantees: no
delivery, no capacity, no coverage commitment and a mesh that congests under
load. And self-certifying identity moves the trust problem to key exchange,
done in person or over some other channel, which is where users of every such
system have always made their mistakes.&lt;/p&gt;
&lt;p&gt;One direction of attack is worth watching, because it is where this kind of
project usually loses: the regulation and the supply, not the protocol. ISM band
rules, device certification requirements and the availability of cheap radio
modules are decided by people with no attachment to unlicensed peer networks,
and a licensing change could do what no technical attack can. If you want this
capability to exist in ten years, defend the unlicensed spectrum and the right
to build your own radio, not the software.&lt;/p&gt;
&lt;h2 id="current-state"&gt;Current state&lt;a class="headerlink" href="#current-state" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The 2.8 line is &lt;strong&gt;alpha&lt;/strong&gt; as we write this. 2.8.0 was revoked over a defect in
the packet-signing code and replaced by 2.8.1, and the project's own advice is
that most people should stay on 2.7.&lt;/p&gt;
&lt;p&gt;That's the project being straight about its own release.&lt;/p&gt;
&lt;p&gt;We have not run a mesh of our own at any scale; everything above is read from
the protocol and the release notes. The experiment that would settle it is a
handful of nodes, measured range against terrain, observed behaviour as nodes
drop out and whether the variable hop limit holds up under congestion. That is
buildable for a few hundred pounds, and it is on the list.&lt;/p&gt;</content><category term="insights"/><category term="lorawan"/><category term="meshtastic"/><category term="mesh"/><category term="cryptography"/><category term="identity"/></entry><entry><title>Post-quantum: key exchange was the easy half</title><link href="https://labs.axisops.com/insights/post-quantum-the-hard-half/" rel="alternate"/><published>2026-09-30T09:00:00+01:00</published><updated>2026-09-30T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-09-30:/insights/post-quantum-the-hard-half/</id><summary type="html">&lt;p&gt;Hybrid key exchange is largely deployed; certificates and signatures are nowhere near. The real bottleneck is crypto-agility, because anything that hard-codes an algorithm or needs a firmware update can't pivot in time.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Post-quantum migration gets discussed as one project. It's two, and they're in
very different states of health.&lt;/p&gt;
&lt;h2 id="the-half-that-is-going-well"&gt;The half that is going well&lt;a class="headerlink" href="#the-half-that-is-going-well" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Key exchange. The hybrid construction, classical X25519 combined with ML-KEM so
the result holds if &lt;em&gt;either&lt;/em&gt; component does, is deployed and deployable today,
and standardised as X25519MLKEM768 in
&lt;a href="https://www.rfc-editor.org/rfc/rfc10024.html"&gt;RFC 10024&lt;/a&gt;. ML-KEM itself is
&lt;a href="https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.203.pdf"&gt;FIPS 203&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Deployment figures vary wildly depending on who you ask and when. DigiCert's
guidance cites Cloudflare reporting more than two-thirds of human-generated TLS
protected by hybrid ML-KEM; an arXiv paper citing an earlier Cloudflare report
gives 1.8% of TLS 1.3 connections. You can't reconcile those without knowing the
measurement and the date, so treat any single number as a snapshot. A 2026
measurement study found roughly half of the domains it analysed with no
post-quantum readiness at all, and about half with partial readiness via hybrid
key exchange.&lt;/p&gt;
&lt;p&gt;AWS has deployed hybrid ML-KEM for KMS, ACM and Secrets Manager, opt-in and
client-dependent rather than on by default
(&lt;a href="https://aws.amazon.com/blogs/security/ml-kem-post-quantum-tls-now-supported-in-aws-kms-acm-and-secrets-manager/"&gt;AWS&lt;/a&gt;);
Google Cloud has quantum-safe KEMs in Cloud KMS. The handshake overhead runs to about a
kilobyte extra in each direction, with minimal latency impact. If you run a TLS
endpoint, this is mostly a matter of upgrading the stack and turning it on.&lt;/p&gt;
&lt;h2 id="the-half-that-is-not"&gt;The half that is not&lt;a class="headerlink" href="#the-half-that-is-not" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Authentication. Certificates and signatures (ML-DSA, SLH-DSA) are a materially
harder migration, and the reasons are structural rather than cryptographic:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Signatures and keys are much larger.&lt;/strong&gt; Certificate chains grow, and some
  protocols have size assumptions baked in.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The PKI is a multi-party system.&lt;/strong&gt; Changing it means CAs, browsers, OSes and
  every validating client moving roughly in step.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Long-lived roots.&lt;/strong&gt; A root certificate issued now may have twenty years
  ahead of it.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="the-actual-bottleneck"&gt;The actual bottleneck&lt;a class="headerlink" href="#the-actual-bottleneck" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The 2026 measurement study frames it correctly: the barrier is the operational
and compatibility work of migration, no longer the absence of standards.&lt;/p&gt;
&lt;p&gt;More specifically, the barrier is &lt;strong&gt;crypto-agility&lt;/strong&gt;: whether a system can
change algorithm at all. Anything that hard-codes a cipher suite, pins a
certificate by key or needs a firmware update to change its cryptography cannot
pivot inside the relevant timescale. Industrial control systems, IoT fleets and
embedded devices are the named problem cases.&lt;/p&gt;
&lt;p&gt;That changes what the engineering work is. The thing to build now is &lt;strong&gt;the
ability to change algorithms without a redeployment&lt;/strong&gt;: negotiated suites,
rotatable trust anchors, no algorithm identifiers compiled into firmware.&lt;/p&gt;
&lt;p&gt;A system with crypto-agility can adopt whatever wins later. A system without it
needs replacing, and that bill won't land on a cryptography budget line.&lt;/p&gt;
&lt;p&gt;The cheerful reading is that the industry is being forced to build something it
should have had anyway.&lt;/p&gt;
&lt;p&gt;Crypto-agility is the ability to respond when any algorithm breaks, not a
post-quantum feature. Algorithms break on a schedule nobody controls. MD5,
SHA-1, RC4, DES: each took years to retire, and the delay was never
mathematical. Somebody had compiled an identifier into something they could not
change. A fleet that can rotate its algorithms by configuration can also rotate
a compromised key in an afternoon, respond to a CA incident the same day and
shorten certificate lifetimes without a project. Post-quantum is the business
case that finally pays for those capabilities.&lt;/p&gt;
&lt;p&gt;For an engineer the toil that disappears is the emergency crypto migration: the
fortnight of finding every place a cipher suite was named, in configuration
files, in firmware, in a vendor appliance nobody has the password for. Do the
work once and it never happens again, for any future algorithm.&lt;/p&gt;
&lt;p&gt;Rotatable trust anchors also mean a device deployed in a water treatment plant
for twenty-five years can accept a new root without a truck roll. The
alternative is that it either keeps trusting a root long past its credibility,
or stops working. Both happen now, and the second is how you discover the
first.&lt;/p&gt;
&lt;h2 id="who-set-the-standards-and-who-pays-for-the-migration"&gt;Who set the standards, and who pays for the migration&lt;a class="headerlink" href="#who-set-the-standards-and-who-pays-for-the-migration" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The chokepoint here is distributional: &lt;strong&gt;the standards were set by a few, and
the migration cost falls on everybody.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The NIST process was open, competitive and better run than it had to be; ML-KEM,
ML-DSA and SLH-DSA have had more scrutiny than their predecessors. Hold that and
still notice the shape of it. A standards process led by one country's agency,
contributed to by a small academic community and a handful of large companies
has produced a mandatory change to the cryptography of every connected device on
earth. Nobody voted, and the bill is itemised on somebody else's timetable.&lt;/p&gt;
&lt;p&gt;Where that bill lands is the part to watch. The hyperscalers have already
migrated their key exchange, because for them it is a configuration change
across infrastructure they control and a line in a compliance document they can
sell. For everybody else it is a hardware refresh. A hospital with infusion
pumps, a utility with meters, a manufacturer with controllers on a twenty-year
cycle: crypto-agility means replacement, and the deadline becomes an unfunded
capital programme with a cryptographic justification. Expect that to move more
on-premises infrastructure into managed cloud services, because there the
cryptography is somebody else's problem and priced monthly.&lt;/p&gt;
&lt;p&gt;Watch the certificate authorities too. Larger keys, shorter lifetimes and more
frequent rotation all increase the operational dependency on the CA, and
automation-only issuance makes every TLS endpoint dependent on a live service
relationship rather than a file.&lt;/p&gt;
&lt;p&gt;The open version is in good shape, for once. OpenSSL and the common libraries
have the algorithms, and you can run your own post-quantum PKI on your own
hardware today. What is scarce is people who have done the migration, which
argues for doing it early rather than buying your way out.&lt;/p&gt;
&lt;p&gt;Here at AxisOps the position is that cryptographic agility is an ownership
property. A system whose algorithms you can change is a system you control; one
that needs a vendor's firmware release is a system you are renting. Treat any
device that cannot have its cryptography changed by configuration as having a
disposal date rather than a lifespan.&lt;/p&gt;
&lt;h2 id="harvest-now-decrypt-later"&gt;Harvest-now-decrypt-later&lt;a class="headerlink" href="#harvest-now-decrypt-later" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The case for urgency is that traffic recorded today can be decrypted once a
capable machine exists. A real threat model, wheeled out as a scare tactic more
often than it's assessed.&lt;/p&gt;
&lt;p&gt;The sober version is a question whose answer differs per organisation: &lt;strong&gt;what of
today's traffic is still sensitive in fifteen years?&lt;/strong&gt; For most session traffic,
nothing at all. For long-lived secrets, medical records, state communications
and anything with a statutory retention period, quite a lot.&lt;/p&gt;
&lt;p&gt;Do that analysis before you buy anything. The answer decides whether this is
urgent for you or merely eventual, and the two have very different price tags.&lt;/p&gt;
&lt;p&gt;Related, on where the threat comes from:
&lt;a href="/insights/quantum-error-correction/"&gt;error correction is the whole game&lt;/a&gt;.&lt;/p&gt;</content><category term="insights"/><category term="cryptography"/><category term="post-quantum"/><category term="security"/></entry><entry><title>Error correction is the whole game</title><link href="https://labs.axisops.com/insights/quantum-error-correction/" rel="alternate"/><published>2026-09-02T09:00:00+01:00</published><updated>2026-09-02T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-09-02:/insights/quantum-error-correction/</id><summary type="html">&lt;p&gt;Qubit counts make the headlines. The physical-to-logical ratio is the engineering problem, and nearly everything else in the field sits downstream of it.&lt;/p&gt;</summary><content type="html">&lt;p&gt;A quantum computer with a thousand physical qubits and no error correction is
not a thousandth of a useful quantum computer. It's a different kind of object
altogether, in the way that a thousand bricks is a different kind of object from
a thousandth of a house.&lt;/p&gt;
&lt;p&gt;The number that matters is &lt;strong&gt;logical&lt;/strong&gt; qubits, and the ratio between physical
and logical is where the field really lives.&lt;/p&gt;
&lt;h2 id="why-the-ratio-is-so-brutal"&gt;Why the ratio is so brutal&lt;a class="headerlink" href="#why-the-ratio-is-so-brutal" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Quantum states decohere. A physical qubit holds its state for anything from
microseconds to milliseconds depending on the technology, and every gate
operation carries an error rate of its own.&lt;/p&gt;
&lt;p&gt;Classical error correction copies bits. Quantum error correction can't, because
the no-cloning theorem forbids it. Instead you encode one logical qubit across
many physical ones and measure &lt;em&gt;stabilisers&lt;/em&gt;: operators that tell you whether an
error happened without telling you (and so collapsing) the encoded state.&lt;/p&gt;
&lt;p&gt;Surface codes are the leading family. The physical cost per logical qubit is
large, and it scales with how good your physical qubits are in the first place.&lt;/p&gt;
&lt;h2 id="the-threshold-theorem-stated-carefully"&gt;The threshold theorem, stated carefully&lt;a class="headerlink" href="#the-threshold-theorem-stated-carefully" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The threshold theorem says arbitrary-length quantum computation is possible
&lt;strong&gt;if&lt;/strong&gt; the physical error rate sits below some threshold. Above it, adding more
physical qubits makes matters worse, because the correction machinery brings in
more errors than it clears up.&lt;/p&gt;
&lt;p&gt;Two things follow, and both get glossed over:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The threshold isn't a single universal number. It depends on the code, the
  noise model and how well connected the architecture is.&lt;/li&gt;
&lt;li&gt;Being below threshold is necessary and not sufficient. How far below sets the
  overhead, and the overhead decides whether a useful machine is the size of a
  rack or the size of a small town.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Which is why "we have N qubits" is close to meaningless with no error rates
beside it, and why the field's real milestones are the demonstrations where
increasing the code distance &lt;em&gt;decreases&lt;/em&gt; the logical error rate. That's the sign
correction is working rather than merely being performed.&lt;/p&gt;
&lt;h2 id="the-engineering-not-the-physics"&gt;The engineering, not the physics&lt;a class="headerlink" href="#the-engineering-not-the-physics" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The part that gets least attention, and is arguably the hardest:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Measuring stabilisers fast enough.&lt;/strong&gt; Syndrome extraction has to run far
  faster than decoherence, which means real-time classical processing sitting in
  the loop with the quantum hardware.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Decoding in real time.&lt;/strong&gt; The classical decoder turning syndromes into
  corrections has a latency budget in microseconds. That's a hard classical
  computing problem that happens to live inside a quantum computer.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Wiring.&lt;/strong&gt; Routing control lines per qubit into a dilution refrigerator is a physical scaling problem and no amount of clever software will answer it.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That last one is thoroughly unglamorous and may well turn out to be the binding
constraint.&lt;/p&gt;
&lt;p&gt;The cryptographic story dominates the coverage and it is the least interesting
application. Breaking RSA is a one-off event with a known mitigation that is
already being deployed. The reasons to want one of these machines are elsewhere
and they are substantial.&lt;/p&gt;
&lt;p&gt;Simulating quantum systems is the obvious one, and it is obvious because it is
the thing a quantum computer is natively for. Chemistry at useful accuracy:
catalysts, battery electrolytes, nitrogen fixation, enzyme behaviour. Those are
problems where the current method is to approximate, synthesise and test, which
is why developing a new catalyst takes a decade and a laboratory. A machine that
can model the electronic structure directly turns part of that loop into
computation. The payoff, if it lands, looks like cheaper fertiliser and better
batteries rather than anything to do with cryptography. The energy implications
of either are larger than anything else in this field.&lt;/p&gt;
&lt;p&gt;Optimisation is the claim to be most sceptical about, since the speedups are
modest where they exist at all and classical methods keep improving. We would
not plan a business on it.&lt;/p&gt;
&lt;p&gt;The quiet benefit is to classical engineering, and it is already being paid out.
Building error-corrected quantum hardware requires real-time classical decoding
in microseconds, cryogenic control electronics and absurdly precise timing. That
work produces instrumentation, low-noise electronics and real-time computing
techniques that are useful whether or not the quantum machine ever works. A good
deal of the field's actual output so far has been classical.&lt;/p&gt;
&lt;p&gt;Code is not the scarce part. Capital is.&lt;/p&gt;
&lt;p&gt;An error-corrected machine needs a dilution refrigerator, a fabrication
capability, a real-time control stack and several hundred people, sustained for
a decade with no product. That is a short list of organisations: a few very
large technology companies, a few state programmes and a small number of
well-funded specialists who will probably end up owned by the first group. There
is no garage version of this today, and our bet is that there will not be one:
the cryogenics and the error-correction overhead look structural, not early. If
that holds, whatever comes out will be rented by the hour, and it will be the
most concentrated computing capability ever built, by physics and finance rather
than by a business strategy anyone chose. That is a bet about engineering
economics, and we have marked it as one.&lt;/p&gt;
&lt;p&gt;The consequence worth planning for is about archives rather than access. The
asymmetry lies in the archive, not in three companies having a fast computer.
The encrypted traffic recorded over the last twenty years is held by
organisations with the budget to record it, and a capable machine arriving in
that environment retroactively opens all of it. State communications, dissident
networks, journalists' sources, medical records, anything with a long
confidentiality horizon. The people with the most to lose from that are in the
worst position to do anything about it, and the people who will get first access
are the ones who have been collecting. That is the actual privacy story in
quantum computing and it has very little to do with whether anyone can factor a
number next year.&lt;/p&gt;
&lt;p&gt;Which makes migration an equity question as much as a technical one. Those with
a budget are already rotating to post-quantum key exchange. The clinic, the
charity, the local authority, the small operator with embedded devices in the
field: they are not, and the capability that decrypts their archives will be
rented by somebody else. The useful response is unglamorous and available now:
move to post-quantum key exchange where you terminate the connection
yourself, because on a current TLS stack that part is close to a
configuration change. Then start the inventory for everything else, because
the appliances, the embedded kit and the long-lived certificates are a
programme, not a setting.
The cost later is an archive you cannot unpublish.&lt;/p&gt;
&lt;p&gt;Where is the open version? Not in hardware, and it never will be. It is in the
software and the standards, and that is where it matters: the algorithms are
public, the implementations are open and nobody needs permission to protect
their own traffic. Here at AxisOps that is the lever we can pull and the one we
recommend pulling. You cannot build the machine and you do not have to; you can
make the archive worthless, and the window for doing that cheaply is open now.&lt;/p&gt;
&lt;h2 id="what-a-non-specialist-should-take-from-it"&gt;What a non-specialist should take from it&lt;a class="headerlink" href="#what-a-non-specialist-should-take-from-it" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The summary for anyone outside the field: the progress is real, the timelines in
the press coverage aren't reliable, and the things to watch are logical qubit
counts and error rates rather than physical qubit counts.&lt;/p&gt;
&lt;p&gt;We are reporting here, not measuring. We don't run quantum hardware, and the
people who do are not publishing the numbers that would settle the overhead
question.&lt;/p&gt;</content><category term="insights"/><category term="quantum"/><category term="hardware"/><category term="error-correction"/></entry><entry><title>micro-zero: own the harness</title><link href="https://labs.axisops.com/insights/micro-zero-own-the-harness/" rel="alternate"/><published>2026-07-29T09:00:00+01:00</published><updated>2026-07-29T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-07-29:/insights/micro-zero-own-the-harness/</id><summary type="html">&lt;p&gt;Agentic sessions are where our features get built now, which makes the harness they run in a piece of infrastructure. We are building our own, called micro-zero, and connecting it to the organisation's memory. The zero is where every session starts.&lt;/p&gt;</summary><content type="html">&lt;p&gt;A growing share of our engineering happens inside an agentic session: a model
with a shell, a checkout and a test suite, working on one feature in that
feature's workspace, with a person directing it. Whatever this week's opinion
says about the ceiling of these tools, the floor has moved. By February,
Spotify's co-chief executive was &lt;a href="https://techcrunch.com/2026/02/12/spotify-says-its-best-developers-havent-written-a-line-of-code-since-december-thanks-to-ai/"&gt;telling an earnings
call&lt;/a&gt;
that their best engineers had not written a line of code since December. The
longer argument is written up in &lt;a href="https://www.richardstubbs.io/insights/spec-driven-development"&gt;Spec-driven development, at
richardstubbs.io&lt;/a&gt;.
The session is where the work happens. That sentence has an operational
consequence most of the commentary skips: &lt;strong&gt;the thing hosting the session is
infrastructure now&lt;/strong&gt;, and it gets the questions we ask of any infrastructure.
What does it make possible, what does it cost and who can inspect it, change it
or switch it off?&lt;/p&gt;
&lt;p&gt;The harness decides what a session can reach. Which filesystem, which
credentials, which network, which memory of previous work, what gets logged
and where the log lives. That is a boundary in the exact sense the rest of
these insights use the word: a rule somebody specifies and a machine
enforces. Run your sessions in a hosted harness and the boundary is
specified elsewhere, logged elsewhere, retained on somebody else's terms.
For exploratory work that trade is fine. For the sessions that build our
features, on our estates, against our customers' systems, it fails the
ownership test the same way a hosted control plane does: possession of the
output, control of nothing underneath it.&lt;/p&gt;
&lt;p&gt;So we are building our own.&lt;/p&gt;
&lt;p class="mz-lockup" style="display:flex;align-items:center;gap:0.8rem;margin:1.8rem 0;"&gt;
&lt;svg role="img" viewBox="0 0 24 24" width="42" height="42" aria-hidden="true"&gt;&lt;ellipse cx="12" cy="12" rx="7.2" ry="9.4" fill="none" stroke="currentColor" stroke-width="2.6"/&gt;&lt;rect x="10.4" y="9.6" width="3.2" height="5.2" fill="currentColor"/&gt;&lt;/svg&gt;
&lt;span style="font-size:1.25em;"&gt;Introducing &lt;strong&gt;micro-zero&lt;/strong&gt;&lt;/span&gt;
&lt;/p&gt;

&lt;p&gt;The design comes
first, written down before any code, each decision recorded with the
failure the alternative produces; the code will be held to the document,
not the other way round.&lt;/p&gt;
&lt;h2 id="what-it-is"&gt;What it is&lt;a class="headerlink" href="#what-it-is" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A workspace session, reachable from a browser, belonging to a real Linux
user. Authentication is the machine's own, through PAM: no parallel account
system, no identity invented for the occasion. One feature being built gets
one session in that feature's workspace, with the shell, the checkout and
the tests, and the person steering it from whatever device is to hand.&lt;/p&gt;
&lt;p&gt;The design rule underneath it is the one these insights keep arriving at:
the part that faces the network holds no privilege, and the part that holds
privilege accepts no network. A browser talks to an unprivileged gateway,
which runs with an all-zero capability set. The privileged broker that can
spawn a shell listens on no TCP port at all, only a filesystem-permissioned
local socket. And the shell that appears runs as the authenticated user,
never as the broker that spawned it.&lt;/p&gt;
&lt;p&gt;Be precise about what that buys. A compromised gateway is not harmless: it relays terminal
streams and file transfers, so whoever owns it sees what the sessions it
carries see. What the design bounds is escalation. Owning the gateway
confers neither root nor the host's filesystem, because the gateway can ask
the privileged side for only a short, enumerated list of things:
authenticate a user, spawn a shell, open a path, signal a process. The
broker refuses each unless the request belongs to the authenticated user
who owns the target, and adding a verb to that list is a design change,
not a feature. Paths are resolved by the kernel itself, confined beneath the
session's root with symlink escapes refused, so path traversal is removed
from the broker's interface rather than defended against in application
code. Passwords are held transiently and
zeroed; the broker's privilege is dropped between fork and exec, before any
user code runs.&lt;/p&gt;
&lt;p&gt;Those are claims a test can check, so the build will gate on tests that check
them. The assertions worth gating on are the ones that state the split is
intact: that a shell spawned by the privileged broker runs as the target user,
that the broker accepts no TCP connection, that traversal attempts are refused
end to end. A security design that is never restated as a failing test is a
press release.&lt;/p&gt;
&lt;h2 id="which-way-up"&gt;Which way up&lt;a class="headerlink" href="#which-way-up" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Most harnesses run inside a general-purpose desktop operating system, alongside
years of accumulated packages, services, credentials, mounts and reachable
vulnerabilities. The OS is the world and the harness is a guest in it,
inheriting whatever that world can reach. An agent running as a process on your
desktop can turn any reachable weakness into a route beyond its intended
workspace, and tidying up after it means believing you found everything. We
think that is upside down.&lt;/p&gt;
&lt;p&gt;In micro-zero the harness is the outer layer and the operating environment
runs inside it. The harness will create a minimal operating space from a
&lt;a href="/insights/why-we-built-substrate/"&gt;micro-substrate&lt;/a&gt;-validated
image, its provenance and scan state checked before the session starts,
then grant into it only the tools, files, network reach and credentials the
session requires. Control is added to a declared boundary rather than
subtracted from a general-purpose desktop. Shell, desktop and workspace
live inside that boundary rather than loose among the operator's own
processes, and cleanup is deletion of the session, not a search for
everything it touched.&lt;/p&gt;
&lt;p&gt;The flip is not only for the agents. A person will be able to take a
desktop inside the harness on the same terms: the same boundary, the same
validated start, with the agent working alongside them in the same bounded
space. One limit, stated plainly: the host kernel remains shared and must
be secured separately. The image boundary removes accumulated userland
exposure; it does not remove kernel escape risk.&lt;/p&gt;
&lt;figure&gt;
  &lt;img src="/images/p-micro-zero-desk.jpg"
       srcset="/images/p-micro-zero-desk-600.webp 600w,
               /images/p-micro-zero-desk-1000.webp 1000w,
               /images/p-micro-zero-desk-1600.webp 1600w"
       sizes="(max-width: 1180px) 100vw, 1180px"
       alt="The micro-zero web desk: session list and panels on the left, and a Display pane showing a connected container desktop running an xterm and xeyes" loading="lazy" decoding="async"&gt;
  &lt;figcaption&gt;The desk, with a container desktop connected through the Display
pane: our internal dev build, running a basic code-defined desktop.&lt;/figcaption&gt;
&lt;/figure&gt;

&lt;h2 id="the-memory-and-whose-it-is"&gt;The memory, and whose it is&lt;a class="headerlink" href="#the-memory-and-whose-it-is" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A session that starts blank re-derives the organisation every morning. The
longer the work runs, the more the context is the work: long-running context
belongs with the agent, not in a scrollback. So micro-zero will connect to
&lt;a href="https://www.microcelium.com/loom/hive-mind/"&gt;&lt;strong&gt;hive-mind&lt;/strong&gt;&lt;/a&gt;, the search over
our vault and shared corpora. The read path lets a session ask what the
organisation already knows, what was decided and why, and what earlier work
discovered. Retrieved material remains evidence, not authority: appearing in
memory does not turn text into an instruction the agent may execute.&lt;/p&gt;
&lt;p&gt;The write path is separate, and it needs more scrutiny than the read,
because an agent can be wrong, can argue from incomplete evidence or can be
led by something it retrieved. These sessions are designed to run long,
days and weeks on one feature, and what a session learns will be recorded
as it goes in a provenance-carrying feature journal: which session produced
it, when, from what evidence, against which feature. Promotion into shared
organisational memory will be a separate act with a review in it. An agent
may propose what the organisation knows; it does not silently decide it.
That is also what makes the disposability above safe to mean: deleting a
session loses no declared durable state once its repository, its journal
and its accepted contributions are current, and uncommitted state stays
disposable by design.&lt;/p&gt;
&lt;p&gt;The connection will have an identity of its own: a desk token issued to
the service, its read scope and its contribution scope defined separately. Never a person's
credential copied into a config, because a service reading with a human's
token acquires that human's reach and makes the audit trail say a person
did something a machine did. Context is becoming the asset in this kind of
engineering. An asset gets an owner, a scope and an audit trail, or it
leaks.&lt;/p&gt;
&lt;h2 id="why-zero"&gt;Why "zero"&lt;a class="headerlink" href="#why-zero" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The zero is where every session starts. A clean workspace, a clean shell
and no authority beyond what the authenticated user already holds. That is
a narrower claim than "no authority", on purpose: a session runs as a real
Linux user and inherits exactly that user's files, groups and network
reach, which is a plain, inspectable boundary. What the harness adds on
top of it is nothing: no standing credentials, no service account riding
along, no privilege the user did not bring. What a session accumulates that
is worth keeping goes back to the estate: committed to the repo, written to
the vault, recorded against the feature. The session itself remains
disposable. A harness built on any other assumption turns the chat
transcript into the system of record, and a chat transcript is the worst
database anyone has ever operated.&lt;/p&gt;
&lt;h2 id="what-it-costs"&gt;What it costs&lt;a class="headerlink" href="#what-it-costs" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Building it, when a dozen hosted products would love to do it for us and
self-hosted pieces already solve parts of the problem: browser terminals,
isolated workspaces, agent desktops. Those components exist. The reason to build
is that the privilege boundary, the machine identity and the memory path are the
product here, and adopting a system that combines them behind one privileged
service would inherit the decision micro-zero exists to control. Running it:
another service with an attack surface precisely where it is most useful, which
is why the privilege split above is the one decision every other decision leans
on. Ownership also means maintaining and rescanning the validated images,
rotating service identities, expiring abandoned sessions, defining what the
audit trail retains and recovering when repository or journal write-back fails:
a disposable session is cheap only while the durable paths around it are
explicit and tested. And where it is allowed to listen will be a named decision
rather than a default. On a lab network this is an ordinary admin tool; on the
public internet it is a remote-shell endpoint. Until that decision has a named
owner the gateway does not run in production at all. Then the usual cost of
owning anything: when it breaks, the pager is ours.&lt;/p&gt;
&lt;p&gt;What it buys is the other side of every entry here about rented control
planes. The boundary around the sessions that build our systems is ours to
specify, inspect, change and switch off. The work those sessions produce
lands in repositories and a vault we hold, read back through a memory
service we run, under identities we issue. The tools changed this year. The
questions did not.&lt;/p&gt;</content><category term="insights"/><category term="ai"/><category term="tooling"/><category term="operations"/><category term="infrastructure"/></entry><entry><title>RISC-V: an ISA you are allowed to read</title><link href="https://labs.axisops.com/insights/riscv-production/" rel="alternate"/><published>2026-07-22T09:00:00+01:00</published><updated>2026-07-22T09:00:00+01:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-07-22:/insights/riscv-production/</id><summary type="html">&lt;p&gt;The Server SoC, Boot and Server Platform specifications are ratified. What makes it interesting is the specification being nobody's property, rather than the instruction set itself.&lt;/p&gt;</summary><content type="html">&lt;p&gt;An instruction set architecture is a specification. There's no technical reason
one has to be proprietary, and yet for forty years nearly every commercially
significant one was. SPARC's published specification and POWER's late opening
were real exceptions, but neither broke the x86-Arm duopoly's grip.&lt;/p&gt;
&lt;p&gt;RISC-V's contribution is that its specification is open and its extensions are
modular. Both halves matter, and the second half causes most of the real-world
grief.&lt;/p&gt;
&lt;h2 id="where-it-is"&gt;Where it is&lt;a class="headerlink" href="#where-it-is" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;We'll report this carefully, because coverage of RISC-V runs well ahead of
ratification and conference material gets cited as though it were a standard.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Server SoC and Boot specifications are ratified.&lt;/strong&gt; RISC-V's own 2025 annual
  report says so.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The Server Platform specification is ratified too.&lt;/strong&gt; The
  &lt;a href="https://docs.riscv.org/reference/home/index.html"&gt;ratified specifications library&lt;/a&gt;
  lists Server Platform v1.0, dated May 2026. The 2025 annual report had forecast ratification by the end of 2026. A forecast and the record of the event it forecast are not in conflict, and the library is the record.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Recently ratified extensions&lt;/strong&gt; include Instruction/Data Coherence (Ziccid,
  June 2026), per the
  &lt;a href="https://riscv.atlassian.net/wiki/spaces/HOME/pages/16154732/Ratified+Extensions"&gt;ratified extensions table&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you want a current answer, go and read the ratified-extensions table. Don't
trust a summary, this one included, past its date.&lt;/p&gt;
&lt;h2 id="the-fragmentation-problem-is-the-real-one"&gt;The fragmentation problem is the real one&lt;a class="headerlink" href="#the-fragmentation-problem-is-the-real-one" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Modularity is the selling point and the liability in one. An ISA assembled out of
optional extensions means that "RISC-V" on its own tells you nothing about what
a chip can do, and two conforming parts can fail to run the same binary. It's a
kit car; everyone's is road-legal and no two take the same spare.&lt;/p&gt;
&lt;p&gt;Platform specifications exist to fix this: a profile that declares &lt;em&gt;these&lt;/em&gt;
extensions mandatory, so software has a baseline to target. The work is
thankless and it matters more to adoption than any individual extension, because
it's the difference between an ecosystem and a pile of incompatible chips.&lt;/p&gt;
&lt;p&gt;ARM solved the same problem by being one company with the authority to say no.
An open standard has to do it by committee, which is slower and produces
documents like the ones above.&lt;/p&gt;
&lt;p&gt;It matters because of what happens downstream when nobody owns the document:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Verification.&lt;/strong&gt; You can't formally verify an implementation of a
  specification you aren't allowed to read. The seL4 proofs reached RISC-V for a
  reason.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Capability extensions.&lt;/strong&gt; Memory-safety research that adds new semantics to
  pointers is being specified as a RISC-V extension, which on a proprietary ISA
  would need the owner's agreement and the owner's commercial interest in the
  outcome.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Research silicon.&lt;/strong&gt; Academic and small-vendor parts exist that would never
  have cleared a licensing conversation.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The ARM port already did the hard cultural work; see
&lt;a href="/insights/arm-server-room/"&gt;ARM reaching the server room&lt;/a&gt;. RISC-V arrives in a
world where "does it build on another architecture" is a normal question rather
than an exotic one, which is why its software story starts from a far better
place than ARM's did.&lt;/p&gt;
&lt;p&gt;Look a little further and the prize is a different shape of hardware industry.
If the specification is free, then designing a chip stops requiring a
negotiation with a competitor. That matters most for the parts nobody large
wants to make: a controller for agricultural equipment in a country without a
semiconductor industry, a processor for a medical device with a thirty-year
service life, a repairable design whose documentation outlives the company that
sold it. The thing that becomes cheap is the low-volume part, and low-volume
parts are where repairability, longevity and local manufacturing live. A
specification with no owner cannot be withdrawn when a product line stops being
interesting, which is the failure mode that strands working hardware today.&lt;/p&gt;
&lt;p&gt;For engineering the win is more immediate: a target you can read is a target
you can simulate, model and prove things about. Expect the serious verification
work of the next decade to happen on RISC-V, for no better reason than that the
documents are available and the licence does not have to be renegotiated for
each new academic.&lt;/p&gt;
&lt;p&gt;The profile, rather than the specification, is the chokepoint.&lt;/p&gt;
&lt;p&gt;An open ISA with optional extensions does not guarantee an open ecosystem. It
guarantees an open &lt;em&gt;document&lt;/em&gt;. What software targets is a profile, and whoever
decides which extensions are mandatory in the profile that matters decides what
the architecture effectively is. Today that is a committee of members, which is
slow and visible. The risk is that it stops being the thing that matters.&lt;/p&gt;
&lt;p&gt;The failure mode to watch for is vendor dialects. A large chip vendor ships a
part with its own custom extensions, writes a toolchain that targets them, gets
enough software built against it, and the practical baseline becomes "their
RISC-V" rather than the ratified profile. Nothing about this breaks the licence
or the specification. It does not have to. The specification stays open while
the ecosystem splits, and you end up with three incompatible families, each
open, each useless to the others. That is the worst of both worlds: the
coordination cost of a committee and the lock-in of a proprietary ISA. Anybody
who lived through the Unix wars has seen this film, and the lesson from it is
that an open standard with no enforcement is a suggestion.&lt;/p&gt;
&lt;p&gt;The other pressure is geopolitical and it is specific. An ISA designed to be
unlicensable is attractive to whoever is worried about losing a licence, which
means significant RISC-V investment now comes from states with an interest in
not being dependent on American architecture licences. That is a real strength
for the architecture and a real risk for its coherence, because the incentive
for a state-backed programme is a profile it controls rather than one it shares.
Export control pressure applied to the specification process would accelerate
that split. The specification being public does nothing to stop the community
that maintains it fragmenting along the same lines.&lt;/p&gt;
&lt;p&gt;And the thing an open ISA never addressed: fabrication. You can read the
specification, design the chip, verify it formally and still need one of a very
small number of fabs to make it. Open silicon design with closed manufacturing
is a real improvement over closed both, and it is not sovereignty.&lt;/p&gt;
&lt;p&gt;Where is the open side? Winning on the document, holding its own on toolchains
and vulnerable on profiles. The practical defence is dull and effective: target
a ratified profile, treat vendor extensions as the proprietary dependency they
are and refuse to ship software that only runs on one company's dialect. Here
at AxisOps that is how we will buy RISC-V hardware when we buy it. If a board's
value proposition requires their fork of the toolchain, it is not an open
architecture, whatever the marketing says; it is a proprietary chip with a
fashionable name on the box.&lt;/p&gt;
&lt;h2 id="where-the-value-turns-up"&gt;Where the value turns up&lt;a class="headerlink" href="#where-the-value-turns-up" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;RISC-V's significance is unlikely to be displacing x86 or ARM in a datacentre
this decade. It's that the &lt;em&gt;specification&lt;/em&gt; is a public good, and the second and
third order effects of that (verification, research, capability hardware) are
where the value turns up.&lt;/p&gt;</content><category term="insights"/><category term="riscv"/><category term="hardware"/><category term="open-source"/><category term="architecture"/></entry><entry><title>Schedulers, and why the default is usually right</title><link href="https://labs.axisops.com/insights/kernel-scheduler-notes/" rel="alternate"/><published>2026-02-10T09:00:00+00:00</published><updated>2026-02-10T09:00:00+00:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-02-10:/insights/kernel-scheduler-notes/</id><summary type="html">&lt;p&gt;CFS gave way to EEVDF over 6.6 to 6.12. Most people who tune a scheduler are solving a problem that isn't a scheduling problem, and the cases where it really is are narrower than the tuning guides let on.&lt;/p&gt;</summary><content type="html">&lt;p&gt;EEVDF became the default fair-class scheduler in 6.6, October 2023
(&lt;a href="https://kernelnewbies.org/Linux_6.6"&gt;kernel.org&lt;/a&gt;), and the transition off CFS completed in 6.12, with the remaining pieces those LWN
catalogued that spring (&lt;a href="https://lwn.net/Articles/969062/"&gt;LWN&lt;/a&gt;). That's the
first change of that magnitude to the fair scheduler since CFS turned up in
2.6.23 back in 2007 (&lt;a href="https://kernelnewbies.org/Linux_2_6_23"&gt;kernel.org&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;Most people will never notice, and that's the interesting part.&lt;/p&gt;
&lt;h2 id="what-changed"&gt;What changed&lt;a class="headerlink" href="#what-changed" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;CFS tracked virtual runtime and picked the task with the lowest; fairness by
accumulated service. Latency was managed indirectly, through heuristics and
tunables like &lt;code&gt;sched_latency_ns&lt;/code&gt; and wakeup preemption rules that accreted over
fifteen years.&lt;/p&gt;
&lt;p&gt;EEVDF (Earliest Eligible Virtual Deadline First) makes latency explicit. Each
task gets a virtual deadline derived from its requested time slice, and the
scheduler picks the eligible task with the earliest deadline. A task that wants
low latency asks for a shorter slice, and gets scheduled more often for less
time each go.&lt;/p&gt;
&lt;p&gt;The practical difference is that latency becomes a &lt;strong&gt;declared property&lt;/strong&gt;
instead of an emergent consequence of heuristics. That's the same move being
made all over systems work right now: replacing a pile of accumulated special
cases with a model.&lt;/p&gt;
&lt;h2 id="why-tuning-usually-fails"&gt;Why tuning usually fails&lt;a class="headerlink" href="#why-tuning-usually-fails" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The common failure is that the symptom is scheduling-shaped and the cause
isn't. Before you touch a scheduler knob, here are the far more likely
explanations:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Lock contention.&lt;/strong&gt; Threads are runnable but serialised. The scheduler is
  doing its job; your application is the bottleneck.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;I/O blocking misread as CPU starvation.&lt;/strong&gt; A task waiting on a device
  isn't waiting on the scheduler.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;NUMA placement.&lt;/strong&gt; Memory on the wrong node costs you more than any
  scheduling decision will ever win back.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;cgroup limits.&lt;/strong&gt; CPU throttling via quota produces latency spikes that
  look just like scheduler unfairness and aren't.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Frequency scaling.&lt;/strong&gt; That's the governor, not the scheduler.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The diagnostic order matters. Establish whether tasks are
runnable-but-not-running (&lt;code&gt;sched:sched_wakeup&lt;/code&gt; to &lt;code&gt;sched:sched_switch&lt;/code&gt; latency)
before you assume the scheduler is choosing wrongly. If the tasks aren't
runnable, scheduling isn't your issue. Tuning before that check is like
replacing the gearbox because the car won't move, when nobody has looked in the
fuel tank.&lt;/p&gt;
&lt;h2 id="where-it-really-is-a-scheduling-problem"&gt;Where it really is a scheduling problem&lt;a class="headerlink" href="#where-it-really-is-a-scheduling-problem" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Narrow, and real:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Mixed latency-critical and batch workloads on shared cores&lt;/strong&gt;, which is
  what EEVDF's explicit slice request is there to address.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Heterogeneous cores.&lt;/strong&gt; big.LITTLE and performance/efficiency core
  asymmetry is a hard scheduling problem and still not fully solved; see
  &lt;a href="/insights/heterogeneous-default/"&gt;heterogeneous by default&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Hard real-time&lt;/strong&gt;, which wants &lt;code&gt;SCHED_DEADLINE&lt;/code&gt; or a different kernel
  altogether rather than any amount of tuning the fair scheduler.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The more significant development is &lt;code&gt;sched_ext&lt;/code&gt;: loading a scheduler written as
a BPF program. That turns scheduling policy into something you can iterate on
in minutes rather than across kernel releases.&lt;/p&gt;
&lt;p&gt;What that opens up is worth spelling out, because it is a change in who is
allowed to have an opinion. Scheduling policy has always been decided by a
small number of kernel developers arbitrating between workloads they cannot all
test, and the result is a good compromise that is nobody's optimum. With
sched_ext, the person who knows the workload writes the policy. A game engine
developer can ship a scheduler tuned for frame deadlines. A database vendor can
ship one that understands which of its threads must not be preempted mid-latch.
A build farm can run one that knows a compile job is throughput work and a test
runner is latency work, and stop pretending they are the same shape.&lt;/p&gt;
&lt;p&gt;The engineering win underneath is iteration speed. Today, trying a scheduling
idea means patching the kernel, building it, rebooting a machine into it and
hoping the box comes back. That is an afternoon per experiment and it is why
scheduler research moves slowly. Make it a program you load and unload while
the system runs, and the experiment loop goes from hours to seconds. The
safety is two mechanisms, and it is worth being precise about which does
what: the verifier bounds what the program can touch, and a watchdog ejects a
scheduler that stalls runnable tasks or misbehaves and falls back to the
default. The experiment is recoverable, not proven harmless. A loaded policy
can still be slow, unfair or starvation-prone right up until the kernel pulls
it. Our expectation is that
more scheduling progress happens in the five years after this lands than in the
fifteen before it, and for the dull reason that people can finally try things.&lt;/p&gt;
&lt;p&gt;There is an energy argument too. Correct placement on heterogeneous cores is
worth real power, which matters on a phone, on a laptop and across a fleet
where the electricity bill is the second-largest line of spend. A policy that knows the
workload beats a general heuristic, and now the policy can know.&lt;/p&gt;
&lt;h2 id="whose-tuning-is-it"&gt;Whose tuning is it&lt;a class="headerlink" href="#whose-tuning-is-it" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The chokepoint here is unglamorous: &lt;strong&gt;whether your workload's scheduling policy
is yours or your distribution's.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Right now it is the distribution's, and nobody minds much because the default is
good. sched_ext changes the stakes. Once policy is a loadable program, it
becomes something that can be shipped, bundled, licensed and withheld. The
plausible near future is that the people with the resources to write and
maintain a good one are the hyperscalers tuning their own fleets, the large
database and game vendors and the distributions. Everyone else takes what they
are given, as now.&lt;/p&gt;
&lt;p&gt;That is tolerable. What would not be is the version where loading your own
policy requires a signature the platform owner controls, or where the managed
platform you rent runs its scheduler and declines to run yours. The capability
then exists and belongs to the operator rather than the workload owner, and the
fact that it is BPF rather than a kernel patch has bought you nothing. Watch
what managed Kubernetes providers permit here; that is where the answer gets
set in practice, long before anybody writes it down.&lt;/p&gt;
&lt;p&gt;The open side is in a strong position and should stay there. sched_ext is in
the mainline kernel, the schedulers people are writing are published, and
running your own on your own metal needs permission from nobody. Here at AxisOps
that is the whole argument for owning the machine: the day the tuning matters,
you want to be the one who can change it. A workload whose scheduling you
cannot influence is a workload running on somebody else's terms, and you will
discover that at the worst possible moment.&lt;/p&gt;
&lt;h2 id="start-from-the-default"&gt;Start from the default&lt;a class="headerlink" href="#start-from-the-default" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The default scheduler is the product of enormous amounts of tuning against an
enormous variety of workloads, and your prior should be that it's right.
Measure first, and expect the answer to be somewhere other than the scheduler.&lt;/p&gt;</content><category term="insights"/><category term="linux"/><category term="kernel"/><category term="performance"/><category term="scheduling"/></entry><entry><title>The context problem is a retrieval problem</title><link href="https://labs.axisops.com/insights/context-is-a-retrieval-problem/" rel="alternate"/><published>2026-01-20T09:00:00+00:00</published><updated>2026-01-20T09:00:00+00:00</updated><author><name>AxisOps Labs</name></author><id>tag:labs.axisops.com,2026-01-20:/insights/context-is-a-retrieval-problem/</id><summary type="html">&lt;p&gt;An assistant session starts knowing nothing about the organisation, and no context window is ever going to hold fourteen years of decisions. We pointed retrieval at our own vault instead. The idea is a weekend's work to try; whether it holds up in daily use is what we intend to find out.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Every assistant session starts from amnesia. It does not know which client
runs which stack, what was decided about backups in 2019 and reversed in
2023, why the monitoring alert thresholds are what they are or that the
one server with the odd disk layout is that way on purpose. The person
driving the session knows, and spends the first ten minutes pasting context
in by hand, session after session, like briefing a brilliant contractor
with no memory on their four-hundredth first day.&lt;/p&gt;
&lt;p&gt;The industry's answer is a bigger context window. That answer misreads the
problem. An organisation's memory is fourteen years of decisions, designs,
client history and hard-won operational folklore, most of it irrelevant to any
given question and all of it essential to some question eventually. It is not a
document you load. You do not want it all in the window. You want &lt;strong&gt;the right
ten paragraphs, found on demand&lt;/strong&gt;. The name for that is retrieval, and retrieval
is a solved- enough problem to go and build with.&lt;/p&gt;
&lt;h2 id="what-we-have-and-what-we-built"&gt;What we have, and what we built&lt;a class="headerlink" href="#what-we-have-and-what-we-built" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;We have an unusual asset, built by accident of discipline: years of the
company written down as plain markdown in one vault. Decisions, designs,
runbooks, meeting notes, client records. No wiki product, no SaaS
knowledge base, no export problem. Files in folders, under sync, on
machines we own.&lt;/p&gt;
&lt;p&gt;So the build is unglamorous, and the first version of it went together over the
holidays. The vault lands on a small machine of its own. An embedding model
running locally turns each note into vectors, in crude line-range slices.
Markdown's headings would make the better boundary, and a disposable index is
allowed to start crude. The vectors sit in SQLite next to a full-text index of
the same notes, and a query runs both: semantic search for "what do we know
about this shape of problem", keyword search for the exact error string, the two
result lists fused by rank. An assistant session asks over a standard tool
interface and gets back passages, with paths, from the organisation's own
memory.&lt;/p&gt;
&lt;p&gt;Nothing in that paragraph is novel, and nothing needed to be. The components are
a local embedding model, SQLite with a vector extension and a sync tool we
already ran. No new database server, no managed vector store, no per-query bill,
and nothing about the company's internal history leaves the premises to be
embedded on somebody else's machines. Whether this is possible was never in
doubt. The question is whether the answers are good enough to change how the
sessions run.&lt;/p&gt;
&lt;h2 id="early-returns"&gt;Early returns&lt;a class="headerlink" href="#early-returns" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Three weeks in: better than expected, with sharp edges. "Better than expected"
rests on three weeks of daily use, not on any measurement. The case we keep
retelling is a single one: the session that asks "have we dealt with this
vendor's licensing before" gets the 2021 note that a person would have taken
twenty minutes to remember existed. Keyword search rescues the queries where
semantic similarity goes vague. The failure mode that matters is silent absence:
a search that returns nothing proves nothing, because the note may exist under a
heading no query reaches. The index is a view of the vault, never the record;
when it matters, somebody still greps the files.&lt;/p&gt;
&lt;p&gt;The design rules we are holding onto, stated now so we can be judged
against them later:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The vault stays the source of truth.&lt;/strong&gt; The index can be rebuilt from
  it at any time, and deleted without losing a word.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Read-only.&lt;/strong&gt; The retrieval service answers questions; it does not
  write memory. What goes into the vault goes through a person.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Local models, and an honest line about where that stops.&lt;/strong&gt; The
  embedding runs on our hardware: the vault is never shipped out wholesale
  to be vectorised. But the passages a query retrieves go into the prompt
  of whatever model the session runs on, and when that model is hosted,
  the retrieved text leaves the premises at answer time. Local embedding
  bounds the indexing; it does not bound the withdrawal.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;One reader, for now.&lt;/strong&gt; The vault holds client records alongside our
  own, and today the only thing that queries the index is us. The moment
  it gains a second reader, human or service, retrieval needs an
  authorisation story of its own: a scope on who may withdraw what. That
  control is not built, and until it is, the reader count staying at one
  is the control.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Plain files underneath.&lt;/strong&gt; The day this experiment dies, the vault is
  still markdown in folders, worth what it always was.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="what-it-changes-if-it-works"&gt;What it changes if it works&lt;a class="headerlink" href="#what-it-changes-if-it-works" title="Permanent link"&gt;&amp;para;&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A session that can ask the organisation what it knows stops being a
clever tool you brief and starts being something closer to a colleague
with access to the filing. The ten minutes of pasted context goes. The
answers improve in the specific way that matters: grounded in what this
company actually decided, not in what the internet thinks companies
generally do.&lt;/p&gt;
&lt;p&gt;And the three questions we ask of everything have tidy answers for once. What it
makes possible: sessions that know the estate. What it costs: one VM, one
embedding model, the sync, indexing and backups that keep an index honest,
vigilance about the silent-absence failure and an authorisation story it does
not yet have. Who controls it: we do, end to end, because it is files we own,
indexed by models we run, on hardware in the building. Memory is becoming the
asset in this kind of engineering, and this one is ours.&lt;/p&gt;</content><category term="insights"/><category term="ai"/><category term="tooling"/><category term="operations"/></entry></feed>