[{"b": "Status, 29 October 2026. No fixed kernel is released for any Ubuntu series. What is published below is a mitigation, not a fix: it denies the bug its trigger, and the bug stays in the kernel until a fixed one is running. When Ubuntu releases the fixed kernels we will update the published mitigation to recognise them. Detection will then report against the real versions, the apply will remove the rule from hosts that reboot into one and we will update this note to say so. CVE-2026-98432 is a use-after-free in the Linux kernel\u0027s IPv4 TCP code, published on 15 October and carrying a CVSS of 9.8. When the kernel handles an incoming SYN it saves any IPv4 options the packet carried, and ownership of that saved copy later passes to the new connection. Two paths can free the copy while another CPU is still reading it to build the SYN+ACK. The upstream fix, e3f33b0a1d89, is small: free the options after an RCU grace period instead of immediately. Ubuntu\u0027s page, as of yesterday\u0027s update, is an unusual read. Every supported release is listed as affected, back to 16.04. The fixed kernel versions are on the page, release by release and flavour by flavour, each one marked \"work in progress\". Nothing is released. Only 14.04 is unaffected. That is odd for the oldest kernel on the page, and a reminder that this class of race arrived with a performance improvement: request sockets have been processed without the listener lock since 2015, and the freeing discipline did not keep up with the sharing. So on 29 October a use-after-free that CVSS scores 9.8, Critical, and that Canonical\u0027s tracker rates priority High, reachable by anything that can send the first packet of a TCP handshake, is present on every supported Ubuntu release back to 16.04, and the updates channel has nothing to install. The page\u0027s mitigation section says none. What the fix tells you\u00b6 A patch you cannot install is still a specification of the bug. Both racing paths in the fix read the same object: the IPv4 options saved off the incoming SYN. The SYN+ACK builder reads them while connection setup can free them, and request-socket migration shares then frees the same copy. When the incoming SYN carries no IPv4 options, the kernel saves nothing, the pointer is null and there is nothing to free or to race over. We read the commit to check that precondition before trusting it; it holds for both halves. That is the mitigation the tracker does not list: take the trigger away. table inet samp_cve_2026_98432 { chain prerouting { type filter hook prerouting priority -300; policy accept; ip hdrlength \u003e 5 tcp flags \u0026 (syn | ack) == syn counter drop \\ comment \"CVE-2026-98432: drop inbound IPv4 SYN carrying IP options\" } } One rule, in a table of its own so it cannot collide with Docker\u0027s or anything else\u0027s, dropping inbound IPv4 SYNs whose header is longer than twenty bytes. TCP almost never carries IP options; several option types are dropped by many networks as standing policy (RFC 7126), and a connection that needs them is rarer than the counter on that rule will ever show. We put a packet generator behind it before trusting it: a SYN carrying options is dropped and counted, a plain SYN passes untouched. What it does not do\u00b6 It does not drop fragmented SYNs on a host with no connection tracking, because nothing reassembles them at that hook. It does nothing for IPv6, which this CVE does not reach. It does not make the kernel fixed: the race is still there, waiting for anyone who removes the rule before rebooting into a fixed kernel that does not yet exist. And a client that genuinely sends optioned SYNs stops connecting, which is the trade, stated. It also does not survive a reboot, on purpose. The rule lives in kernel memory and is never written to disk, because a mitigation pack that installs boot-time configuration becomes a persistence mechanism of its own. After a reboot the rule is gone, the next detection says so (rule_live=0 on the result line) and the next apply puts it back. Re-applying after every reboot is the operating model, not a failure of it; anyone running the role by hand instead of under automation needs to know the protection does not outlive the uptime. To be precise about what it is: the repository below is a mitigation, not a fix. The fix is a kernel, and nobody can ship you one tonight. The rule, the detection that reads a running kernel against the tracker\u0027s versions and the removal path for when fixed kernels land are published at github.com/axisops/CVE-2026-98432, runnable as a standalone Ansible role. The work-in-progress kernel versions from the tracker sit in the defaults as a comment, ready to become the fixed-version table the day they ship. \"No mitigation\" on a tracker page means no patch-shaped mitigation. The fix names what the bug needs to fire. Needs can be denied.", "d": "2026-10-29", "g": ["security", "kernel", "networking", "mitigation"], "t": "CVE-2026-98432: a use-after-free in every supported Ubuntu release back to 16.04", "u": "insights/every-ubuntu-kernel-since-2016/"},{"b": "Over the last few years we have written one Ansible role per vulnerability: find the affected thing on a host, back it up, apply the vendor\u0027s fix or documented workaround, verify. Several of those entries link to their role already. Seven of them are now public in one shape, along with the shared library they all use. The fixes are the boring part. A script that strips JndiLookup.class from a JAR is a morning\u0027s work. The part worth writing about is what has to be true before anyone, including us, lets one of these run against a fleet. The problem with a remediation script\u00b6 A mitigation role is the most dangerous kind of automation in an estate. It runs as root, by design. It modifies the thing you depend on, by design. It runs during an incident, when nobody is reading carefully. And it runs everywhere at once, because that is what it is for. The usual review process does not fit. \"Someone read the diff\" is weak when the reviewer is tired and the role is four hundred lines of YAML calling a Python script that shells out to a package manager. What you want to know is narrow and specific: what will this write, what will it execute, what will it restart and does the code match those claims? So each pack carries a manifest that answers those questions, and nothing is supposed to run on the strength of the README. What a pack declares about itself\u00b6 The manifest sits at the repository root and states the CVEs, which hosts the pack applies to, the entry points for detection and for applying, whether each needs root, every option with its default, every path an apply writes, every program it runs and the units it may restart. Writing that down changes the review. A reviewer reads the declaration first and then checks the code against it, instead of reading four hundred lines hoping to notice an absence. An undeclared write is a defect whether or not it is harmful, because the declaration is the thing being reviewed. Two rules make the declaration worth having. Detection writes nothing of its own. The detect path stages its script in a private temporary directory, reads the host, reports and removes the directory. No installed script, no templated config, no backup directory, no log file, no package installed. It runs under ansible-playbook --check because there is nothing in it that could fail a dry run. That property is testable, and we test it the dull way: run detection in a throwaway container with no network, then list every path whose timestamps changed outside the usual noise. The expected output is nothing. The one exception we found is worth stating because it looks like a failure and is not: on Red Hat family hosts, a bare rpm -q writes its own SQLite write-ahead index. Every pack that asks rpm a question produces those two files. A tool that reports \"detection wrote something\" without knowing that will cry wolf on every Rocky host in the estate. Nothing is restarted or reverted by default. Every restart is behind an option that defaults to false, and no pack reverts anything automatically. A downgrade reinstates the vulnerability, so an automatic rollback is a mitigation that un-mitigates under failure conditions. If the fix broke something, a human needs to be in that decision. One copy of the version comparison\u00b6 Judging whether a host is vulnerable means comparing package versions, and the comparison is where quiet errors live. Distributions backport fixes, so the upstream version number tells you very little; what matters is whether the installed package is at or past the distribution\u0027s fixed version. That requires implementing dpkg, rpm and apk ordering semantics, including the parts people forget. Tilde sorts before everything. Caret sorts after. Epochs beat everything. Each pack used to carry its own copy of that, and the copies had drifted. Some reimplemented the algorithms, some incompletely and some shelled out to the distribution\u0027s own tool. That only works when you are judging the same distribution you are running on. One host inspecting another distribution\u0027s container cannot shell out to a tool it does not have. There is now one implementation, pure Python, standard library only, checked against the real dpkg --compare-versions, rpm.labelCompare and apk version -t in containers. Packs vendor it by copy rather than depending on a package, its hash is recorded in each manifest, and each pack has a test that fails if its copy differs. Vendoring by copy is not elegant. It means a change to the library is a change to every pack. It also means a pack is a single directory you can read end to end with no resolution step, which matters more for something that runs as root during an incident. Signed releases, and what a signature is for\u00b6 Releases are SSH-signed annotated tags, signed with a hardware key that cannot export its private half. The platform imports a signed tag and never a branch, and pins the commit that tag names. Two details turned out to matter more than expected. The first is that git verify-tag prints Good \"git\" signature on its own error output even in runs where it then refuses the signature. A verifier that greps for that string will accept a tag signed by a key it has never heard of. The exit code and the matched principal are the only things worth reading, which we know because we wrote a verifier the wrong way first and then tried to make it fail. It did not fail, and the reason it did not fail was that string. The second is about dates. Each pack\u0027s first commit is dated to its vulnerability\u0027s disclosure window, so the set reads in the order the work happened; the repositories themselves were created this month. The commit date is therefore an editorial choice about ordering and carries no proof of anything. The signature is the part that attests, and it is dated when the attestation happened. Nobody should read a commit timestamp as provenance: it is a field the author sets to whatever they like, and the only reason to trust one is that something signed it. There is a third, smaller lesson that cost us the most time. A release tag is not a label you can slide along a branch. The importer refuses a tag that has moved, permanently, by name, because a release that changes what it points at is the beginning of a rollback attack. We moved one during a rework and the development platform refused every subsequent import of that pack, correctly. The remedy is a new version, never a moved tag. How the platform takes one in\u00b6 A pack that runs as a standalone role is a file on disk and a decision by whoever ran it. The reason for the manifest, the signature and the single library is that the platform treats the pack as untrusted input and has to establish each property for itself. That process is worth describing, because it is the real answer to \"why does a four-hundred-line Ansible role need all this ceremony\". There is no URL anywhere. A pack is not fetched from an address written in a config file. There is a declared source list, and each row names a minted identifier, a reference to a separately declared network source the fetch goes through and a repository path that must sit directly under an allowed origin. A literal URL in that file is a configuration error with its own name and is refused. The point is that adding a new place code may come from is a change to reviewed policy, not a line someone adds in passing. The rows carry no version at all. No tag, no branch, no commit, no revision. This surprised us when we specified it and now seems obviously right: if the source list pinned a commit, then updating a pack would mean editing policy, and policy edits get waved through. Instead the pin is whichever commit a signed release tag names at import time. The only thing the row records about branches is where release tags are cut from. Unpacking happens somewhere that is assumed to be compromised. Git itself does the unpacking, in a container with no network, running as nobody, on a read-only filesystem. Nothing that container says is believed. Outside it, every object\u0027s hash is recomputed from the raw bytes, the tag is parsed, the signature is checked, the commit and its tree are walked and the snapshot is rebuilt using only blobs that verified. The rule being applied is that the worst a compromised program inside can do is lie about its own output, and the caller checks the output. The signature is checked twice, by different code. Once in the platform\u0027s own implementation, once by shelling out to ssh-keygen -Y verify, and the import is refused unless both agree. That is not thoroughness for its own sake. It is there because of the Good \"git\" signature behaviour described above: having been caught once by a verifier that printed success while refusing, the design now assumes either implementation may be wrong and makes disagreement fail closed rather than open. The signer list is governance, not configuration. It is an OpenSSH allowed-signers file, every line landed by a reviewed commit, every key required to be hardware-backed and scoped to git signing only. A software key is refused by name. While the file holds no key, every import refuses, failing closed on an empty policy rather than accepting anything. Revoking a key means deleting its line, after which packs signed only by that key need re-signing. That is a deliberate cost: it makes key rotation visible work rather than a silent change in what the platform trusts. The refusals have names. Unsigned tag, unknown signer, lightweight tag, moved tag, tag naming the wrong object, invalid signature, the two verifiers disagreeing, missing manifest, manifest at an unrecognised schema version, vendored library not matching its recorded hash, a submodule where none should be. A refusal that has a name is one an operator can look up, and one that appears in a report rather than as a stack trace. The moved-tag refusal is the one we have already triggered in anger, as described above. The importer refuses that pack by name from then on, and there is no argument to be had with it, which is the behaviour you want. What exists today, and what does not\u00b6 Being specific, because the design reads as though all of it is running. Import and verification work, as described above. Scanning a pack\u0027s contents has landed too, with its own refusals, and a scan that cannot complete records as unverified rather than clean. Grey, not green, which is the right colour for \"we do not know\". The approval gate is not finished. The record has fields for who approved a pack and when, and today the importer writes both as empty and marks the pack imported. So a pack can arrive, be verified and be scanned, and still nothing has said it may run anywhere. Running is not built either. The manifest\u0027s host selectors are not yet matched against live inventory, so the platform cannot currently tell you which of your machines a given pack applies to. Today these run as standalone Ansible roles, the way they always have, with a human choosing the inventory. That gap is the honest state of it. The packs are public now because the shape is settled and the review properties are real, not because the platform around them is finished. Why publish them\u00b6 Three reasons, in order of how much they matter. A role that claims to detect a vulnerability is a claim about your estate, and you should be able to read how the claim is made before you believe it. The version-comparison code is the part most likely to be quietly wrong, and it is the part you can now check. The declaration-then-check review pattern is worth copying whether or not you use any of our code. Most remediation automation does not declare what it touches, and the cost of adding that is a day per role. And some of them will be useful directly. If you have Log4j in archives on disk or an unpatched polkit on a host that cannot take the upgrade, the job of working out which hosts and what to do about it is already done. Most of what we write is not public, because most of it is client work on client systems. These generalise: the problem without the client attached.", "d": "2026-10-28", "g": ["security", "ansible", "supply-chain", "mitigation", "substrate"], "t": "What a mitigation pack has to prove before it runs", "u": "insights/what-a-mitigation-pack-has-to-prove/"},{"b": "Six days ago the industry stopped what it was doing for PHP. CVE-2026-109618 and CVE-2026-109619, unauthenticated remote code execution on anything serving web requests, CVSS 9.2. CISA told people to take unpatched systems offline (CISA alert, 21 October), which is not language it uses often, and added both to the Known Exploited Vulnerabilities catalogue on the 26th. That was the right response. The same day PHP was patched, Google shipped Brotli 1.0.10, 1.1.1 and 1.2.1, and Node.js shipped 22.23.4, 24.22.1 and 26.12.1 (Node.js October 2026 security releases). CVE-2026-111497: an out-of-bounds write in the Brotli decoder, CVSS 9.2 from Google\u0027s CNA, reachable by anything that decompresses Brotli it did not produce. It got a fraction of the attention. We think the gap is a mistake. The two bugs are not equivalent, and the argument here does not need them to be. The problem is that this one is materially harder to find on an estate, because the version numbers you would naturally check give the wrong answer. The bug, briefly\u00b6 The decoder treated an invalid block-type switch as \"need more input\" rather than a format error. Decoding carried on, appending attacker-supplied bytes to an eight-byte internal buffer and writing past the end of it (Red Hat RHSB-2026-015). The fixed decoder returns a hard error. Exposure is wider than the Node.js headline suggests: HTTP clients that accept br responses, servers that decode br request bodies, anything pulling Brotli out of an archive, a cache or a queue. On the Python side requests, urllib3, httpx and aiohttp all decode Brotli automatically if a backend is installed, which it often is as a transitive dependency nobody chose. Public proof-of-concept code exists and reaches code execution under narrow lab conditions, which is where this class of bug usually sits a week in. Why it is harder to find than PHP\u00b6 With PHP the question is nearly answerable. You know where your PHP is and php -v tells you the truth about it. Brotli does not work like that, and three separate things are wrong with the obvious check. The upstream version is close to useless. Distributions backport the patch without changing the version. Ubuntu 24.04\u0027s 1.1.0-2build2 is vulnerable and 1.1.0-2ubuntu0.1 is fixed, and both are upstream 1.1.0. A scanner comparing against the published affected range of \"1.1.0 before 1.1.1\" marks the patched host as vulnerable, forever, because 1.1.0 is where it will stay. Patched Node.js reports the version it fixed. $ docker run --rm node:22.23.4-slim node -p \\ \u0027process.version + \" brotli=\" + process.versions.brotli\u0027 v22.23.4 brotli=1.1.0 Every scanner reading process.versions.brotli for a verdict is wrong on patched hosts and will stay wrong. The only thing that answers the question is the Node.js release number against the published fixed release per line. Whether Node.js even uses the system library depends on where it came from. The nodejs.org builds and official node images bundle their own copy, so libbrotli on the host is irrelevant to them. Distribution packages are the other way around: Ubuntu 24.04\u0027s node reports node_shared_brotli: true, and Ubuntu marks its own nodejs \"not affected\" because fixing the system library fixes it. One host can have both kinds installed. None of that is visible in a version string, which is why \"we patched\" settles much less here than it does for PHP. So there is a role\u00b6 github.com/axisops/CVE-2026-111497, same shape as the others. Point it at a substrate deployment and it reports what is affected before it changes anything. It compares the installed package against the distribution\u0027s fixed version, with dpkg, rpm and apk comparison implemented in Python so one host can judge a container running a different distribution. It reads node_shared_brotli per binary and judges each by whichever thing determines it. Dry run by default; --apply backs up, upgrades and verifies. Four verdicts, and the two unsatisfying ones carry most of the value. no_fix is the vendor saying affected with nothing published: that is Red Hat today, advisory up on 26 October, status ongoing, no errata, RHEL 8 through 10 affected, with Rocky and Alma inheriting it. unknown is a release with no published verdict at all, such as Alpine 3.22. Reporting those plainly beats inferring from the upstream version, which is the thing we just established does not work. It will not see a decoder compiled into another binary: Electron apps, pkg single binaries, Go and Rust programs linking it statically, browsers with their own copy. Python wheels are their own problem, since pip install Brotli puts a copy in a virtualenv no package manager owns. Containers get reported, never changed, because a package upgraded inside a running container is gone when it is recreated. For our own images that means mg docker build --refresh, since a plain rebuild reuses the cached layer and the old base. And the upgrade is not the end. The library is loaded by long-running processes, and replacing the file on disk leaves every one of them running the old code until it restarts. The role reads /proc/*/maps for the (deleted) marker, names the service each process belongs to and restarts nothing unless asked. The part that keeps recurring\u00b6 The workaround people reach for is rejecting Content-Encoding: br at the proxy. It covers inbound request bodies and nothing else: not a client fetching a br response from somewhere hostile, not Brotli read out of a queue. Which is the same conclusion as OpenSSH in 2024 and Log4j before it. The advisory tells you the affected versions. Working out which of your hosts are in that set needs every distribution and release you run, every vendor\u0027s fixed version for each and which vendors had shipped when you looked. This time it also needs the knowledge that two of the obvious version strings are actively misleading. PHP got a week of everyone\u0027s attention and it still took CISA telling people to pull systems offline. This one got a fraction of that, and is harder to answer. An estate that can say \"what package, from what release, built how\" from a declaration answers it on day one. We went looking, again.", "d": "2026-10-27", "g": ["security", "ansible", "linux", "python", "mitigation"], "t": "The one that did not get the meltdown", "u": "insights/brotli-version-is-not-a-verdict/"},{"b": "Five things have been going on in parallel for about a decade, and they get filed in five different drawers. Microkernels are an OS-research curiosity. eBPF is a networking tool. WebAssembly is a browser thing that escaped. LoRa mesh is for hikers. Confidential computing is a slide in a cloud vendor\u0027s deck. Here at AxisOps we think they are all the same argument. Every one of them takes a position on a single question: where does the boundary go, and what does it cost to cross it? That question is as old as the industry. What changed over the last ten years is that the answer stopped being \"wherever the hardware happened to put it\". The boundary we inherited\u00b6 For most of the industry\u0027s working life there were two boundaries worth the name, and the hardware chose both of them for us. There\u0027s the process boundary, enforced by the MMU; one ring transition to cross, a page table per side, and a fault on one side doesn\u0027t take the other down with it. It\u0027s a good boundary and it\u0027s still doing most of the work in production today. Then there\u0027s the machine boundary, enforced by a hypervisor. Much stronger, and much more expensive, because you\u0027re paying for a whole kernel per side. Everything we built, we built by picking one of those two. Something that needed isolating got a process, or it got a VM. Think of it like a building where the only two options are a desk or a separate office block; most of the interesting failures of the last twenty years happened when neither size fitted, so nobody isolated anything at all. Each of the five strands below is an attempt to build a boundary that the hardware never handed us. 1. Microkernels: the boundary moved down\u00b6 The microkernel argument is old enough to have gone out of fashion twice. The architecture has not changed. What changed is that the proof turned up. seL4\u0027s functional-correctness proof means something quite specific, and it\u0027s worth being careful here because it gets quoted loosely. The kernel\u0027s C implementation is proven to implement its abstract specification, with no unproven gap in between. That claim now holds across Arm 32-bit, AArch64, x86-64 and RISC-V 64-bit, and the AArch64 port has reached parity with the others on functional correctness (seL4 Summit 2024). The proof for the MCS configuration has since been completed on RISC-V (seL4 news). The usual objection is IPC cost, and it\u0027s a fair objection; the cost is real. It\u0027s also the wrong place to stop the conversation, because what you\u0027re buying is a fault boundary around a driver, and a monolithic kernel can\u0027t sell you one of those at any price. A driver bug in Linux is a kernel bug. That is a structural problem, not a code quality one. What a microkernel actually costs you has its own entry, which is honest about which numbers it still owes. 2. eBPF: the boundary became programmable\u00b6 eBPF is the strand people underrate most, and we think that\u0027s because it turned up dressed as a networking tool. The original extended-BPF patch landed in Linux 3.15: 64-bit registers, ten of them rather than two, and an instruction set mapped closely enough to real hardware to JIT. LWN reported it as \"up to four times faster on x86-64\" than classic BPF on some network-filter microbenchmarks (LWN, 2017). The bpf() system call that exposed the VM to user space went up for review in July 2014 (LWN, 2014). Described as a faster packet filter, that\u0027s a footnote. Described properly, it\u0027s the first time the Linux kernel took code from user space and ran it in kernel context on the strength of a proof. The verifier is the whole story. It walks the control-flow graph depth-first to establish that the program terminates and rejects unreachable instructions. Then it simulates execution instruction by instruction, tracking register and stack state and refusing out-of-bounds jumps, out-of-range access and reads of uninitialised registers. In secure mode, for a loader without CAP_SYS_ADMIN, pointer arithmetic is prohibited outright so that kernel addresses can\u0027t leak (LWN, 2017). So here\u0027s a boundary made out of static analysis rather than hardware. No ring transition, no page table, no context switch. You pay for the boundary once, at load time, in the verifier; after that the crossing is free, though the program still costs whatever its instructions and helper calls cost. Kernel-side validation was not new: classic BPF had it, and proof-carrying code is older still. The scope was new, arbitrary programs against kernel data structures rather than packet filters. 3. WebAssembly: the boundary became portable\u00b6 WASM is the same trick as eBPF, aimed at a different problem and approached from the opposite end. eBPF is a verified sandbox that got pushed down into the kernel. WASM is a verified sandbox that got pulled out of the browser. Both of them give you a restricted instruction set, a validation pass that has to succeed before anything runs, linear memory the guest can\u0027t escape and no ambient authority; the guest gets the capabilities the host hands it and nothing more. The capability model is the part we care about, and it\u0027s also where the reporting gets difficult. WASI\u0027s design is explicitly capability-based, so a module receives handles rather than a filesystem. The ecosystem timeline, though, is in a muddle, and most of what\u0027s written about it isn\u0027t reliable. We cover the component model separately in WASI: capabilities instead of containers, working from the specification and the Bytecode Alliance\u0027s own records rather than from secondary write-ups. A survey of the blog coverage gave us three contradictory dates for the same release. That\u0027s a comment on the state of the commentary, not on the technology. What matters architecturally is that this boundary is portable. An eBPF program runs in a Linux kernel. A WASM module runs anywhere there\u0027s a runtime, with the same isolation properties, which makes it the first sandbox whose guarantees survive the trip between a server, an edge node and a browser tab. 4. Confidential computing: the boundary turned inwards\u00b6 The three strands above all assume the host is trustworthy and the guest is suspect. Confidential computing turns that on its head and protects the guest from the host: memory encrypted with a key the hypervisor doesn\u0027t hold, and an attestation the workload can check before it will unseal anything. This is the strand with the widest gap between the marketing and the engineering. The deployment reality has its own entry: what attestation really proves, what\u0027s inside the trusted computing base once you\u0027ve counted the firmware and what encrypted memory costs you. Until we\u0027ve measured it, treat anything here as a vendor claim. It belongs on this list for a structural reason. It\u0027s the first of the five where crossing the boundary means proving who you are, rather than simply being permitted. 5. LoRa mesh: the boundary lost its centre\u00b6 The odd one out, and the one that gets dismissed as a hobby more reliably than any of the others. Meshtastic is LoRa radios running a mesh protocol with no infrastructure underneath: no base station, no carrier, no DNS, no CA. Nodes relay for each other, and the topology is whatever happens to be in range. The engineering is a good deal more serious than \"hikers with walkie-talkies\" suggests, and the 2.8 firmware line is where you can see it. Packet signing via XEdDSA. Node IDs derived from a node\u0027s public-key identity rather than its hardware MAC address. A traffic-management module doing deduplication and rate limiting, and hop limits that vary with observed mesh activity (Meshtastic firmware releases, Meshtastic blog). Read that list again as an identity architecture. A node is its key. There\u0027s no registry to ask because there\u0027s nothing to ask; authority lives in the keypair, so trust gets established peer to peer or not at all. The 2.7.15 release disallowed the legacy direct-message path, protected only by the shared channel key, after a spoofing advisory; DMs broke for nodes on older firmware while relaying and channels kept working. That\u0027s a compatibility break made for a security fix, in favour of the stronger property. One note on currency, because it\u0027s the sort of thing a reader will check: the 2.8 line is alpha as we write this. 2.8.0 was revoked over a defect in the packet-signing code and replaced by 2.8.1, and the project\u0027s own advice is that most people should stay on 2.7 for the time being (Meshtastic blog). The stable base is 2.7; 2.8 is where the interesting protocol work is happening. Same question as the other four, asked in a place with no centre at all: who is allowed to cross, and what proves it? What actually led us here\u00b6 Put the five side by side and the pattern is hard to miss. Strand Boundary enforced by Paid when Protects Process / MMU Hardware page tables Every crossing Host from guest Microkernel Hardware + proof of the kernel Every IPC Everything from drivers eBPF Static analysis (verifier) Once, at load Kernel from user code WASM Validation + capabilities Once, at validate Host from module, portably Confidential computing Hardware + attestation Continuously Guest from host LoRa mesh Keypairs Per packet Peers from each other The direction of travel is consistent: away from boundaries the hardware chose for us, and towards boundaries we specify, verify and place where we want them. The cost model keeps shifting too, from per-crossing to paid-once-up-front, and that\u0027s what makes fine-grained isolation affordable for the first time. That\u0027s why \"security\" is the wrong single label for any of this. Security is a consequence. What\u0027s being built is the ability to decide where a boundary goes, instead of accepting wherever the MMU put it in 1985. The strand that explains the other five\u00b6 One thing is missing from that table, and it\u0027s the thing that made all of it urgent. In January 2018, Spectre and Meltdown made speculative execution everybody\u0027s problem. The hardware boundary, the one we\u0027d all been treating as the solid floor underneath every design, turned out to leak across it by construction, in every high-performance processor shipped for twenty years. Spectre did not start those five histories. It made their common boundary problem urgent and put them on the same table. Before it, \"the MMU will handle it\" was a perfectly reasonable engineering position. After it, all five strands above read differently; not as researchers being clever, but as five teams independently working out that a boundary you can\u0027t inspect is a boundary you can\u0027t rely on. It has its own entry, because it deserves rather more than a paragraph. What else belongs on this map\u00b6 Five strands is the clean version of the story, and it\u0027s too clean. The fuller list of what\u0027s been worth watching since roughly 2012 is longer and messier: containers winning on ergonomics rather than isolation, the programmable dataplane, ARM reaching the server room, the storage hierarchy collapsing, Rust arriving in systems work, WireGuard, the supply chain turning into an attack surface and formal methods becoming practical. Most of it feeds one of the five. Several of those have their own entries already, and they are the ones to read next if this map is useful: post-quantum migration, where key exchange is largely solved and certificates aren\u0027t; CHERI, putting bounds on pointers in silicon, the hardware answer to the problem eBPF and WASM solve in software; RISC-V reaching production; Rust arriving in systems work; energy as a design axis rather than an afterthought; and running inference on the bench rather than in somebody else\u0027s datacentre. What is not covered here and should be: the supply chain as a first-class attack surface in its own right, and what happens to any of this when the verification key sits with somebody else. Where we think this goes\u00b6 The strands are converging rather than competing, and the interesting systems of the next few years will use several at once: a verified kernel hosting WASM workloads, attested, with eBPF doing the observability that a sandbox otherwise makes opaque. Each solves the granularity problem at a different altitude, and none is a general answer alone. The part we would bet on is the cost model rather than any particular technology. Once you pay for a boundary at load time instead of at every crossing, the economics of \"isolate this one function\" change completely, and most of the architectural decisions the industry made under the old economics deserve another look. What it buys everyone else\u00b6 Cheap, fine-grained, verified boundaries are not an operating-systems hobby. They are the precondition for a handful of things people have wanted for years and could not afford: A hospital analysing patient data on hardware it does not own, where the operator holds ciphertext and the privacy guarantee is arithmetic rather than a contract clause. A fifteen-year-old infusion pump getting a security patch, because a verified driver boundary means the vendor fixes one component instead of recertifying a whole device. A flood-warning mesh in a valley no telco will ever wire, running on nodes that hold their own keys and need no operator at all. An engineer running a capable model on the machine under their desk, where no prompt leaves the building. A small firm able to answer \"what exactly is running on this box\" with evidence. Every one of those is a boundary problem dressed up as a policy problem. That is the argument for the whole decade of work above. Who ends up holding the keys\u00b6 The uncomfortable half, and the reason we bother writing any of this down. A boundary is only as good as the key that attests it, and somebody holds that key. Attestation proves what is running to whoever holds the verification root, and today that root belongs to a silicon vendor. The cloud operator genuinely cannot read your memory; you have swapped trusting Amazon for trusting AMD. Better, and some distance short of ownership. The same shape turns up at every layer of this map: the public registry everybody defaults to, the handful of certificate authorities, three clouds, two app stores, the model weights, the accelerators. A technology that could have gone either way acquires a chokepoint, and the chokepoint is where the rent gets collected and where the surveillance capability sits, whether or not anyone currently means to use it. Worse, these primitives cut both ways by design. The attestation that proves to you that your workload is untampered proves to a platform that your device is unmodified, which is the mechanism behind locked bootloaders and signed-only firmware. That is a computer you own and cannot change, built from the tools these five strands spent a decade perfecting. The technology has no opinion. Ours: we build infrastructure people can own, inspect and rebuild, and we are against the version of each technology above that works only while you keep paying for it. That preference is why a mesh node holding its own key gets a warmer write-up here than a cloud that will mind your secrets for a fee. The open, self-hostable version of most of this stack exists; some of it is keeping pace and some of it is quietly losing, and saying which is most of what this site is for. Each strand has its own entry. This one is the map.", "d": "2026-10-24", "g": ["microkernels", "wasm", "ebpf", "lorawan", "security"], "t": "What led us here", "u": "insights/what-led-us-here/"},{"b": "PHP released 8.5.12, 8.4.27, 8.3.36 and 8.2.35 at 14:00 UTC today, alongside the disclosure. CVE-2026-109618 and CVE-2026-109619, collectively php2shell, are unauthenticated remote code execution on anything serving web requests, CVSS 9.2 from PHP\u0027s own security team. CISA published an alert this afternoon telling organisations that cannot update immediately, or that are still on PHP 8.1 or earlier, to take their PHP systems offline until they can (CISA alert, 21 October). That is not language CISA reaches for often. Read the alert in order, though, and the instruction before it is the one that will cost people their week: inventory every system running any version of PHP, including PHP embedded in other software. Nobody is going to do that by the end of the day. What the bugs are\u00b6 Both live in request handling, and both need an earlier failure to get going. CVE-2026-109618 is memory corruption reached through an out-of-memory condition. A request crosses memory_limit partway through setup, PHP bails out of native code before it has finished building its internal state, and the cleanup path then treats a half-initialised structure as a valid one. Getting a request to run out of memory on demand is the easy half: a multipart body with no Content-Length can slip past post_max_size, the header count is unbounded and max_file_uploads does not count empty files (SANS ISC, 21 October). CVE-2026-109619 is a use-after-free in the same neighbourhood. Per-request global state is not always reset before the memory behind it is released, so a worker that handled a failed request can act on the remains of it when it handles the next one. It wants an error first, then a warning during request initialisation, which a per-directory override such as an .htaccess will happily supply. So an attacker who can make your application fail gets to influence what the worker does on somebody else\u0027s request afterwards. Every PHP serving requests is in scope: php-fpm, Apache with mod_php, php-cgi, LiteSpeed, FrankenPHP. PHP rates mod_php worst, because there the corruption can land before any of your own code runs. An ordinary urlencoded form post reaches it, so the obvious \"block multipart at the proxy\" workaround does not cover that case. Patched releases exist for 8.2 through 8.5 and that is all the good news. There is no release for 8.1, 8.0 or 7.4. Upstream has published minimal source backports as git branches for people who build their own, and everyone else is waiting on a vendor. Below 7.4 there is nothing, so a great deal of the PHP on the public internet has no fix tonight at any price. The workarounds all have a sting\u00b6 session.upload_progress.enabled = 0 closes the confirmed session path and leaves the rest open. It has to be set somewhere that applies before request parsing: php.ini, an FPM pool or the server config. An application calling ini_set() is far too late, and so is .user.ini. That is the sort of detail that produces a host recorded as mitigated that isn\u0027t. memory_limit = -1 is the strange one. Removing the cap removes the predictable failure the working exploit depends on, so upstream suggests it, administratively, where the application cannot override it. You are deleting your only guard against a single runaway request eating the box, so it needs a cgroup limit underneath it before you touch it, and an allocation failure can still reach the vulnerable path anyway. Both are the trade we keep making. With regreSSHion in 2024 the workaround swapped remote root for a denial of service. Here you are choosing between an availability risk you have measured and a remote code execution you have not. Take the trade, then get off it. The question that has no quick answer\u00b6 You cannot patch what you cannot enumerate, and PHP is close to the worst case. php -v answers for one binary on the PATH. A web host runs several PHP versions at once by design, one per customer; a server runs php-fpm for the application and a CLI for cron, usually different builds; containers carry their own, invisible to anything reading the host\u0027s packages. Behind that sits the category CISA names and nobody tracks: PHP embedded in other software, appliances and vendor bundles with a copy nobody chose. Then the version string lies to you. Distributions backport the patch without touching the upstream number, so a Debian or Ubuntu host with the fix installed still prints the old version. The only thing that decides is the package revision against that vendor\u0027s fixed version for that release, and for 8.1 and older there is no vendor answer yet to compare against. So the verdict on a lot of hosts tonight is unknown, and that is the verdict worth reporting. A false \"vulnerable\" sends somebody to remove a memory limit on a box that was already safe. A false \"fixed\" is worse. There is a role, and it mostly counts things\u00b6 We are writing php2shell_mitigation tonight, same shape as the five before it, and it will go up at github.com/axisops/CVE-2026-109618-109619 like the rest. It is an Ansible role that deploys one Python detector, runs it read-only and runs it again with --apply only where a published fix exists and is not installed. The fixed-version table is the artefact that takes the time. It lives in the role\u0027s defaults/main.yml, keyed by os-release ID and codename, one row per release carrying the vendor\u0027s first fixed package and where it was read from. Tonight most of its 8.1 rows will be null: affected, nothing shipped. Discovery leads, because that is where the work is. Every SAPI rather than one: php-fpm masters and their pools, mod_php inside Apache, php-cgi, the CLI and the versioned binaries beside it. For each, the owning package against that release\u0027s fixed version, with dpkg, rpm and apk comparison implemented in Python so one host can judge a container running another distribution. Then memory_limit, session.upload_progress.enabled and fatal_error_backtraces read per SAPI through php-fpm -i and the pool config rather than from php.ini, because a value set in the wrong place is the difference between mitigated and recorded as mitigated. Containers get a shell script piped through docker exec, and are reported rather than changed, because a package upgraded inside a running container dies with it. That probe fails usefully: an image with no shell comes back unknown and named instead of counting as clean. no_fix and unknown are the verdicts carrying the value here, and refusing to infer either from the upstream version is what makes them worth having. Three things it will not do. A PHP compiled into /usr/local or baked into an appliance has no package to judge, so it is reported rather than guessed at. A patched package on disk is not a patched service: php-fpm masters and Apache carry the old code until they restart, so the role names every process still mapping a replaced binary and restarts nothing unless asked, because a restart is an outage and the operator picks the hour. And there is no rollback, because a downgrade here reinstalls an unauthenticated remote code execution. The backup is a record of what was replaced, not an undo button. Which is the uncomfortable part. Most of the people patching tonight will spend it working out where PHP is, for the third time in five years against a different dependency. Applying the fix is an afternoon. Knowing what PHP you have, from what package, on what release, in what container, is the week, and it is the same week every time, because nobody writes the answer down in a form that survives to the next disclosure. Everything else is archaeology with a deadline. We are going looking, again. Ours is declared, because Microcelium holds the declaration, so tonight was the afternoon\u0027s work without the week in front of it. The week was spent years ago by somebody who wrote the answer down in a form that survived, and that is the whole of the advantage. There is a write-up of the run on the company site.", "d": "2026-10-21", "g": ["security", "php", "ansible", "python", "operations", "mitigation"], "t": "The patch took an afternoon. The inventory will take the week", "u": "insights/php2shell-the-inventory-problem/"},{"b": "On every mainstream machine, a C pointer is represented as a bare integer address. That single fact is at the root of an enormous share of the security vulnerabilities ever shipped: nothing in the representation says what the pointer is allowed to reach. CHERI (Capability Hardware Enhanced RISC Instructions) makes a pointer a capability instead: an address, plus bounds, plus permissions, plus a validity tag, with the hardware enforcing all of it on every dereference. What that buys you\u00b6 A buffer overflow stops being a memory-safety bug and becomes a trap. Not spotted afterwards by a sanitiser, not caught now and then by a canary; refused in hardware, because the processor won\u0027t perform an access outside the bounds. The tag is the clever part. It\u0027s a bit held outside the addressable data and maintained by the hardware, and it drops to zero if the capability is manipulated with ordinary arithmetic. You can\u0027t forge a capability by constructing an integer. Provenance is therefore unforgeable: a capability can only be derived from another with equal or lesser rights. That property, monotonic non-increase of authority, is what makes CHERI a capability system rather than bounds-checked pointers with better marketing. Where it is today\u00b6 The RISC-V CHERI specification is not ratified. The most recent version we can find is an intermediate release, v0.9.10.1, dated 2026-10-14 (riscv/riscv-cheri releases). Hot Chips 2026 coverage lists CHERI among work-in-progress items. The base is designated RV64Y. There\u0027s also a compatibility wrinkle: Zicfiss is incompatible at present, because its shadow-stack push and pop instructions would need modifying for CHERI. Arm\u0027s Morello was a research prototype of the same ideas on ARMv8 and produced real evaluation results. It was a research platform, not a product line. So: promising, extensively researched, demonstrably implementable and not something you can buy and deploy. What it would cost\u00b6 Pointers get bigger. A 128-bit capability where a 64-bit pointer used to be. On pointer-dense workloads that\u0027s cache pressure you\u0027ll feel. Source changes are needed. Not a recompile. Code that stores pointers in integers, does its own tagging in spare bits or reconstructs pointers from arithmetic (all common in systems code) has to change. The whole stack has to agree. A capability-aware kernel, a capability-aware libc and a capability-aware compiler. You don\u0027t get to do this one layer at a time. The prize is larger than the coverage suggests, and it is not only a security prize. Roughly two thirds of serious vulnerabilities in systems software are memory safety. The industry\u0027s answer so far is to rewrite in a safe language, which works and which will take thirty years, because the C and C++ in service is measured in billions of lines and most of it has no owner. CHERI is the other route: keep the code, recompile it and have the hardware refuse the access. Be precise about which half that is. A recompile buys spatial safety: every pointer carries its bounds and its permissions, so a read or write outside the object it was derived from is refused by the hardware. That is the buffer-overflow family, and it is most of the historical catalogue. It does not, on its own, buy temporal safety. A pointer to freed memory is still a valid capability until something invalidates it, and use-after-free is half of Chromium\u0027s high-severity memory bugs. The answer is capability revocation: sweeping memory for capabilities to freed regions, as CHERIvoke and Cornucopia do. That is a real cost, paid at runtime, on top of the recompile. It is the unfinished half, and anyone selling CHERI as the end of memory unsafety is selling the easy part. The second-order effect is the one we would watch. A crash is a bug report; a breach is an incident, a disclosure, a regulator and a fortnight. The economics of maintaining old software change completely once its worst failure mode is downtime. A great deal of working equipment then gets to keep working safely instead of being replaced for security reasons. That is a repairability and a waste argument as much as a security one, and nobody makes it. For engineering itself the win is debugging. A capability violation traps at the instruction that did it, with bounds attached. Compare the current experience: memory corrupted at ten in the morning, crash at four in the afternoon, somewhere else entirely, in innocent code. The bug class that disappears is the one where symptom and cause have nothing to do with each other, and it is the most expensive class there is. Compartmentalisation is what the research community cares about most, and it is furthest from deployment. Cheap unforgeable capabilities let you split a large program into mutually distrusting parts without processes or IPC costs: the image decoder in a browser, the parser in a mail server, each holding only the memory it was handed. The new boundary moves the chokepoint to intellectual property. CHERI came out of Cambridge and SRI, with substantial public funding, and the research is published. The question that decides whether it becomes infrastructure or a feature: can anybody implement it, or only those who can do a deal with Arm? Both routes are live at once. The RISC-V extension, if it ratifies, is implementable by anyone who can make a chip; that is the version where a small vendor, a university or a state-funded programme can build capability hardware without asking permission. The Arm route reaches far more devices far sooner, because Arm is in everything, and on Arm\u0027s licensing terms and schedule. Morello was a research prototype; whether the ideas reach a shipping Arm architecture, in what form and under what terms, is Arm\u0027s commercial decision rather than a technical one. Watch for the version that is worst for everybody: memory safety as a premium architectural feature. Available in the high-margin parts, absent from the cheap ones, so the devices hardest to patch and longest-lived in the field (the meter, the camera, the controller in a building) keep shipping the unsafe pointer while phones and servers get the protection. Security stratified by unit price, with the exposure concentrated where nobody is maintaining the software. The cost is not nothing: doubling pointer width pushes on cache and memory bandwidth, as above, and the tag bits have to be carried through the whole hierarchy. But it is a one-off silicon and bandwidth cost rather than a per-unit licence. It will land in the premium tier anyway if the only licensable implementation is a premium one. The specification work is happening in the open in RISC-V, and the compiler and kernel work exists as open source from the research programme. Here at AxisOps our position is that publicly funded memory safety ought to be universally implementable, and the test is simple: if a chip designer with no Arm relationship cannot build it, the public money bought a competitive advantage rather than a public good. Where it sits among the strands\u00b6 CHERI is the hardware answer to the same problem eBPF solves with a verifier and WASM solves with validation. The software approaches check once at load time, after which the crossing costs nothing even though the checked code still costs what it costs. They are limited to code they can analyse. CHERI checks on every dereference, in hardware, with no analysis needed and no language restriction; the price is new silicon and a recompiled world. Neither is strictly better: they\u0027re the same instinct (make authority explicit and unforgeable) applied at different points in the stack.", "d": "2026-10-20", "g": ["cheri", "hardware", "memory-safety", "riscv"], "t": "CHERI: putting the bounds in the pointer", "u": "insights/cheri-pointers-with-bounds/"},{"b": "The standard objection to microkernels is IPC cost. The cost is real, and it is where the discussion usually stops. That is the problem. What you\u0027re buying is a fault boundary around a driver, and a monolithic kernel can\u0027t sell you one at any price, because a driver bug is a kernel bug by construction. The structural claim\u00b6 In Linux, a device driver runs in kernel address space with kernel privileges. There\u0027s no boundary to cross because there\u0027s no boundary there at all. A null dereference in a driver is a kernel oops; memory corruption can corrupt anything it likes. This isn\u0027t an argument about code quality. You can write excellent driver code and the structural property doesn\u0027t budge. Drivers are also where the largest volume of kernel code lives and where the least-reviewed code lives, an uncomfortable combination on anyone\u0027s reading. Put the driver in its own address space and a fault takes down the driver. A supervisor restarts it; the system carries on. What the proof says\u00b6 seL4\u0027s functional correctness proof gets cited loosely, so to be precise about it: the C implementation is proven to refine an abstract specification, with no unproven gap between the two. It doesn\u0027t mean \"no bugs\", because a specification can specify the wrong thing. It means the implementation and the specification provably agree. That holds across Arm 32-bit, AArch64, x86-64 and RISC-V 64-bit, AArch64 having reached parity on functional correctness (seL4 Summit 2024). The MCS configuration now has a functional correctness proof on 64-bit RISC-V, completed by Proofcraft in June 2026 and shipped in seL4 16.0.0 (seL4 news); the AArch64 port of that proof is in progress. On the 32-bit Arm configuration the proof also covers integrity and confidentiality above functional correctness, which is the part that lifts it out of academic curiosity. Confidentiality there means non-interference between the domain scheduler\u0027s domains, not between arbitrary threads. The other architectures have functional correctness alone; the stronger properties are not verified there. Where the cost lands\u00b6 The useful question about IPC isn\u0027t what one round trip costs. It\u0027s how often you hit the path. A few thousand calls a second against a microsecond-scale IPC is noise. A million calls a second is the design. So the engineering work in a microkernel system is almost all batching: making the interface coarse enough that you rarely cross the boundary. That\u0027s the same optimisation as any RPC boundary, and a design constraint rather than a tax. The open question is the sizing one, and it is answerable: on a named part, at a named kernel version, what does the IPC path cost against the equivalent syscall, and at what call rate does the batching argument stop holding? That measurement decides the architecture for a whole class of embedded system and we have not taken it on our own hardware. It is on the bench list, with the part number and the command, and no numbers appear here until it is. What a verified boundary makes possible\u00b6 The structural property changes what can be built and certified. Take a car. A modern vehicle carries dozens of controllers because the infotainment stack cannot be allowed near the braking logic, so they get separate silicon, wiring and suppliers. With a verified separation kernel underneath, one processor can host the instrument cluster, the entertainment system and a safety function, isolation proven rather than asserted. That removes boxes, weight, wiring and cost from every vehicle built, and the same reasoning applies to avionics, industrial controllers and medical devices. The certification effect is bigger than the hardware saving. Today, changing anything in a safety-certified system means re-arguing the safety case for the whole box, because nothing structurally prevents one component affecting another. With proven isolation you argue about one partition at a time. That is the difference between a device that gets a security update and one that does not, which is why so much safety-critical equipment in the field runs software nobody has touched in a decade. Proven separation is how you patch an infusion pump. It also buys survivability where there is no staff and no second chance. A driver fault a supervisor can restart is a system that can sit in a substation or a satellite and keep going, which spacecraft software has historically bought with redundant hardware instead. Where it falls down\u00b6 Driver availability. Linux\u0027s enormous driver tree is its real moat, and a microkernel hasn\u0027t got one. For most hardware that settles the argument. Shared-memory-heavy workloads, where the batching argument doesn\u0027t apply because the data is fine-grained. Developer familiarity. The pool of people who can debug a microkernel system is far smaller. Those explain why Linux won the general case. None touches the structural claim, which is why microkernels keep winning where a driver fault is unacceptable: avionics, automotive, secure enclaves. In the end, verified systems come down to the licence on the proof. seL4 is open source, and so are its proofs. That combination is unusual and the thing to protect. A published proof can be inspected, built on, ported and argued with by anybody, so a four-person company can ship on a formally verified base without employing a proof engineer: the expensive artefact has already been made and given away. That redistribution happened because the work was publicly funded and published. The alternative model is already common here. Commercial separation kernels are certified to the relevant standards and sold per unit with the evidence package under NDA: you get a certificate and a binary. You cannot read the proof, port it or verify the claim yourself, and your safety argument depends on a document you are not allowed to show anyone. Same technology, opposite ownership. Watch where the funding points. Much of the serious verification work in this field runs on defence research money. That is how it gets done at all, and it comes with a preference for the results reaching some people rather than everyone. The optimistic case is that proofs are artefacts and artefacts get published. The pessimistic one is a verified trusted base that is export-controlled while the open one stays a research kernel. The difference between those futures is licensing decisions being taken now by people not thinking about it hard. Here at AxisOps we would build on the verified base we can read, and not for ideology. A safety case resting on a proof you are forbidden from examining has swapped one trust relationship for another and charged you for it. Verification converts a claim into something checkable; a proof under NDA converts it back. The price of enforcing the boundary\u00b6 The microkernel is the oldest approach to isolation and the only one where the boundary is enforced by hardware. Everything newer, eBPF and WASM included, is an attempt at a comparable guarantee without paying for it on every crossing.", "d": "2026-10-16", "g": ["microkernels", "kernel", "formal-methods"], "t": "What a microkernel actually costs you", "u": "insights/microkernel-isolation/"},{"b": "LoRa is a modulation: Semtech\u0027s chirp spread spectrum, operating in sub-GHz ISM bands, trading data rate for range and power. LoRaWAN and Meshtastic both use it, and they\u0027re architecturally opposite. They get discussed interchangeably, which does neither of them any favours. Both run on tarmac, and so do a bus route and a rally stage. LoRaWAN is a star with a server\u00b6 Devices talk to gateways. Gateways forward to a network server. The network server handles deduplication (several gateways often hear the same packet) and adaptive data rate, and in the 1.0.x deployments that dominate it also manages session keys. LoRaWAN 1.1 moves the join to a separate join server, which changes who holds the keys rather than whether somebody does. An application server sits behind that. Devices join via OTAA, deriving session keys from a root key, with the operator\u0027s infrastructure as the authority. There\u0027s a DevEUI, a JoinEUI and an entity whose job is deciding whether you\u0027re allowed on. The properties that follow: Range is gateway-to-device, which can be kilometres, and your coverage is whatever your gateways cover. Device power consumption is excellent. Class A devices transmit and then open two short receive windows, sleeping the rest of the time. Years on a battery is real. There\u0027s an operator, whether that\u0027s you or a public network. Downlink is constrained, which surprises people. You can\u0027t generally reach a Class A device on demand; you wait for it to transmit. Meshtastic is peers with no server\u00b6 No gateway, no network server, no join procedure. Nodes relay for each other, and your reach is a function of how many nodes sit between you and the destination. The properties that follow: Coverage is emergent. It\u0027s whatever the node density gives you, and it changes as people move about. Power consumption is worse, because a relaying node has to listen. There\u0027s no operator and no authority. A node is its key. Capacity is low and shared. Every relay costs airtime, and duty-cycle limits in ISM bands are a hard constraint. A busy mesh congests. When each is right\u00b6 LoRaWAN suits fixed sensors reporting small amounts of data where you control the coverage or can buy it: utility metering, agricultural sensing, building monitoring. The star topology and the sleeping devices are right for that job. Meshtastic suits places where there\u0027s no infrastructure and none is coming, and where the participants move: remote terrain, event coordination, resilience when the other networks are down. They overlap far less than the shared radio suggests. The duty cycle constraint both share\u00b6 This one is routinely discovered late, so know it early. ISM band regulation limits transmit duty cycle; in Europe commonly 1% in the relevant sub-bands. That\u0027s a hard ceiling on how much either system can say, and it\u0027s why LoRa is for telemetry and short messages rather than anything resembling bulk data. A design that assumes it can send whenever it likes will work beautifully on a bench and fail a regulatory check. The price of coverage is where the radio argument becomes a human one. Somebody paid for this one. A LoRaWAN gateway on a pole costs a few hundred pounds; the tower behind the national networks is a different kind of commitment, and it is the reason coverage is a commercial question rather than a technical one. A LoRaWAN gateway costs a few hundred pounds and covers a few kilometres. That means a farm, a university campus, a canal trust, a parish council can own its own network outright, with no monthly bill and no contract, and put two hundred sensors on it. The thing that becomes cheap is measuring a place. River levels across a catchment, soil moisture across a holding, temperature in every flat in a block to find the ones with damp, bin levels, bridge strain, bat activity. These are all projects that previously needed either a cellular subscription per device or a cable, and the subscription is what killed them. The same hardware in a lower-income country does the same job without requiring a mobile operator to have found the area worth covering. Meshtastic answers a different question: what works when nothing works. A network that comes into being because people turned up with nodes, with no operator to bill and no infrastructure to fail, is the right shape for a flooded valley, a mountain rescue, a festival, a power cut lasting a week. Here at AxisOps we think the resilience case is badly undervalued because it is only needed occasionally. The plans most organisations hold for the loss of mobile coverage amount to hoping it does not happen. For engineering, the lesson that generalises beyond radio is in the duty cycle. A hard regulatory ceiling on transmission forces you to decide what is worth saying, and designs built under that constraint are better: event-driven rather than polling, deltas rather than full state, local decisions rather than a round trip to ask permission. Plenty of systems with no radio at all would benefit from being written as though airtime cost money. Ownership follows the architectural split, which is why these two are worth contrasting rather than comparing. For LoRaWAN it is the network server. Whoever runs it holds the routing, the network session keys, every device\u0027s traffic metadata and the decision about who is allowed on the network. Under 1.0 it also holds the join authority and the application keys. One reason 1.1 exists is to split those off, into a join server and application keys the network server never sees, wherever a deployment runs 1.1. Use a public network and you have a telecoms relationship with everything that implies: coverage you do not control, a price that can change, a per-device fee and a company that knows where your sensors are and what they report. The devices are cheap and the dependency is not. The good news, and the reason we keep recommending it, is that the network server is software you can run yourself on a small machine, and the open implementations are mature. A LoRaWAN estate can be entirely yours, gateway to application, with nobody to pay. That is a choice most deployments do not realise they are making, because the hosted option is the one in the quick-start guide. The pattern to watch is the quiet one: hosted network servers are free at the scale where you evaluate and priced per device at the scale where you depend on them. By then the join credentials, the device provisioning and the integrations all live in somebody\u0027s console, and the migration is a site visit to every sensor. Decide this on day one, because it is cheap on day one and expensive forever after. Meshtastic has no such chokepoint, and that is its reason to exist. No operator means nobody to charge you, nobody to shut it down, nobody to serve an order on and nobody holding a list of participants. The cost is that you get no guarantees at all: no coverage commitment, no capacity, no delivery, and the network exists only while people keep their nodes on. That is an engineering trade, not an ideological one. What we would say is that the option of a network with no owner is a thing a society should keep, in the same way as footpaths, and it survives because the hardware is cheap and the firmware is open rather than because anyone is defending it. The limit of both\u00b6 Bandwidth. These are kilobits, and often hundreds of bits per second at long range and high spreading factors. Any design that drifts towards \"and then we send the image\" has left the technology somewhere behind it.", "d": "2026-10-12", "g": ["lorawan", "mesh", "networking", "iot"], "t": "LoRaWAN and mesh are not the same argument", "u": "insights/lorawan-vs-mesh/"},{"b": "Meshtastic is LoRa radios running a mesh protocol with no infrastructure underneath it. No base station, no carrier, no DNS, no certificate authority. Nodes relay for each other and the topology is whatever happens to be in range. The \"hobbyist\" framing does a lot of work to stop people looking at the engineering. Read the release notes as an identity system\u00b6 The 2.8 line\u0027s changes, from the firmware releases and the project blog: Packet signing via XEdDSA. Node IDs derived from the node\u0027s public-key identity rather than from its hardware MAC address. A traffic management module doing deduplication and rate limiting. Automatic variable hop limits responding to observed mesh activity. The second is the architecturally interesting one. A node is its keypair. Nothing assigns the identity; it\u0027s derived from a secret the node holds. So there\u0027s no registry you can ask whether a node is who it claims to be, because there\u0027s nothing to ask. Trust gets established peer to peer or not at all. That does not make signatures a mesh-wide invariant: the default packet-signature policy accepts unsigned traffic so that older nodes remain usable. Strict mode is what turns rejection of unsigned traffic into a local rule. Why that is unusual\u00b6 Nearly every identity system we build has an authority somewhere: a CA, a directory, an identity provider, a registrar. The authority is what makes names mean anything, and it\u0027s also a central point of compulsion, failure and cost. Self-certifying identifiers do away with it: name and key are the same thing, so verification happens locally. The trade is real: you lose human-meaningful names. A public key is not a name anyone can remember or type. Every system that has tried this, from PGP fingerprints to onion addresses, walks into the same wall, and the usual answer is a petname layer: locally-assigned nicknames mapped to keys, like your phone\u0027s contacts list, where \"Mum\" means something to you and nothing to the network. It\u0027s the same trade the crypto wars entry touches from the other side. A system with no registry cannot be served an order for its registry. A deliberate compatibility break\u00b6 Firmware 2.7.15 (November 2025) disallowed the legacy direct-message path, which protected a DM with the shared channel key rather than the recipient\u0027s own, after CVE-2025-53627 showed it let a sender be spoofed. Nodes on older firmware could still relay traffic and use channels, and their DMs broke until they updated. That\u0027s a security fix willing to break compatibility, with a cost attached. The opposite choice, keeping the insecure path working indefinitely, is the one almost everybody makes, and it\u0027s why so many protocols carry downgrade attacks. What a network with no operator makes possible\u00b6 A group of people can have communications that nobody provides. Twenty pounds of hardware per person, no SIM, no account, no monthly bill, no coverage map, no terms of service and no company that can be acquired, go bust, raise its prices or be ordered to hand over a participant list. For most of the history of telecommunications that was not available at any price. That matters where infrastructure reaches badly. A valley with no mobile coverage, where the network exists because the farms each have a node on a barn roof. A week-long power cut, when the mobile network\u0027s batteries have gone and the mesh is still running because each node draws less than a light bulb. The network comes into being from the participants rather than being supplied to them. Where the local infrastructure exists but cannot be trusted, the same property matters most. A network with no operator cannot be asked for records it does not keep, and a direct message encrypted to a key held on the receiving device is not readable by the relay nodes that carry it. The relaying itself is visible, headers included, and anything on a shared channel is readable by whoever holds that channel\u0027s key, so the property is confidentiality of content, not invisibility. That is a real capability for a journalist, a trade union or a family in a country where the carrier is an arm of the state. The engineering lesson generalises beyond radio. Self-certifying identity removes an entire category of system: the provisioning service, the registration flow, the credential database, the password reset, the breach of that database. A node generates its own identity and is immediately addressable. Anybody who has built a device fleet knows provisioning is most of the work and most of the risk, and here it is absent. The key is on the node\u00b6 There is no chokepoint here, by design. There is no key to hold over you: it is generated on the node, never escrowed, and the identity is derived from it rather than assigned to it. No registry, so no participant list to subpoena, leak or sell. No operator, so no company to acquire, no account to suspend, nobody to serve a notice on. The firmware is open and the hardware is commodity, so no supply chain chokepoint beyond a radio module several vendors make. Meshtastic is the clearest evidence that the ownable version is not a compromise forced by poverty: it beats the commercial alternative on resistance to compulsion, privacy and resilience, and loses only on bandwidth and convenience. Two caveats. The absence of an authority means the absence of guarantees: no delivery, no capacity, no coverage commitment and a mesh that congests under load. And self-certifying identity moves the trust problem to key exchange, done in person or over some other channel, which is where users of every such system have always made their mistakes. One direction of attack is worth watching, because it is where this kind of project usually loses: the regulation and the supply, not the protocol. ISM band rules, device certification requirements and the availability of cheap radio modules are decided by people with no attachment to unlicensed peer networks, and a licensing change could do what no technical attack can. If you want this capability to exist in ten years, defend the unlicensed spectrum and the right to build your own radio, not the software. Current state\u00b6 The 2.8 line is alpha as we write this. 2.8.0 was revoked over a defect in the packet-signing code and replaced by 2.8.1, and the project\u0027s own advice is that most people should stay on 2.7. That\u0027s the project being straight about its own release. We have not run a mesh of our own at any scale; everything above is read from the protocol and the release notes. The experiment that would settle it is a handful of nodes, measured range against terrain, observed behaviour as nodes drop out and whether the variable hop limit holds up under congestion. That is buildable for a few hundred pounds, and it is on the list.", "d": "2026-10-10", "g": ["lorawan", "meshtastic", "mesh", "cryptography", "identity"], "t": "Meshtastic: a node is its key", "u": "insights/meshtastic-node-is-its-key/"},{"b": "Post-quantum migration gets discussed as one project. It\u0027s two, and they\u0027re in very different states of health. The half that is going well\u00b6 Key exchange. The hybrid construction, classical X25519 combined with ML-KEM so the result holds if either component does, is deployed and deployable today, and standardised as X25519MLKEM768 in RFC 10024. ML-KEM itself is FIPS 203. Deployment figures vary wildly depending on who you ask and when. DigiCert\u0027s guidance cites Cloudflare reporting more than two-thirds of human-generated TLS protected by hybrid ML-KEM; an arXiv paper citing an earlier Cloudflare report gives 1.8% of TLS 1.3 connections. You can\u0027t reconcile those without knowing the measurement and the date, so treat any single number as a snapshot. A 2026 measurement study found roughly half of the domains it analysed with no post-quantum readiness at all, and about half with partial readiness via hybrid key exchange. AWS has deployed hybrid ML-KEM for KMS, ACM and Secrets Manager, opt-in and client-dependent rather than on by default (AWS); Google Cloud has quantum-safe KEMs in Cloud KMS. The handshake overhead runs to about a kilobyte extra in each direction, with minimal latency impact. If you run a TLS endpoint, this is mostly a matter of upgrading the stack and turning it on. The half that is not\u00b6 Authentication. Certificates and signatures (ML-DSA, SLH-DSA) are a materially harder migration, and the reasons are structural rather than cryptographic: Signatures and keys are much larger. Certificate chains grow, and some protocols have size assumptions baked in. The PKI is a multi-party system. Changing it means CAs, browsers, OSes and every validating client moving roughly in step. Long-lived roots. A root certificate issued now may have twenty years ahead of it. The actual bottleneck\u00b6 The 2026 measurement study frames it correctly: the barrier is the operational and compatibility work of migration, no longer the absence of standards. More specifically, the barrier is crypto-agility: whether a system can change algorithm at all. Anything that hard-codes a cipher suite, pins a certificate by key or needs a firmware update to change its cryptography cannot pivot inside the relevant timescale. Industrial control systems, IoT fleets and embedded devices are the named problem cases. That changes what the engineering work is. The thing to build now is the ability to change algorithms without a redeployment: negotiated suites, rotatable trust anchors, no algorithm identifiers compiled into firmware. A system with crypto-agility can adopt whatever wins later. A system without it needs replacing, and that bill won\u0027t land on a cryptography budget line. The cheerful reading is that the industry is being forced to build something it should have had anyway. Crypto-agility is the ability to respond when any algorithm breaks, not a post-quantum feature. Algorithms break on a schedule nobody controls. MD5, SHA-1, RC4, DES: each took years to retire, and the delay was never mathematical. Somebody had compiled an identifier into something they could not change. A fleet that can rotate its algorithms by configuration can also rotate a compromised key in an afternoon, respond to a CA incident the same day and shorten certificate lifetimes without a project. Post-quantum is the business case that finally pays for those capabilities. For an engineer the toil that disappears is the emergency crypto migration: the fortnight of finding every place a cipher suite was named, in configuration files, in firmware, in a vendor appliance nobody has the password for. Do the work once and it never happens again, for any future algorithm. Rotatable trust anchors also mean a device deployed in a water treatment plant for twenty-five years can accept a new root without a truck roll. The alternative is that it either keeps trusting a root long past its credibility, or stops working. Both happen now, and the second is how you discover the first. Who set the standards, and who pays for the migration\u00b6 The chokepoint here is distributional: the standards were set by a few, and the migration cost falls on everybody. The NIST process was open, competitive and better run than it had to be; ML-KEM, ML-DSA and SLH-DSA have had more scrutiny than their predecessors. Hold that and still notice the shape of it. A standards process led by one country\u0027s agency, contributed to by a small academic community and a handful of large companies has produced a mandatory change to the cryptography of every connected device on earth. Nobody voted, and the bill is itemised on somebody else\u0027s timetable. Where that bill lands is the part to watch. The hyperscalers have already migrated their key exchange, because for them it is a configuration change across infrastructure they control and a line in a compliance document they can sell. For everybody else it is a hardware refresh. A hospital with infusion pumps, a utility with meters, a manufacturer with controllers on a twenty-year cycle: crypto-agility means replacement, and the deadline becomes an unfunded capital programme with a cryptographic justification. Expect that to move more on-premises infrastructure into managed cloud services, because there the cryptography is somebody else\u0027s problem and priced monthly. Watch the certificate authorities too. Larger keys, shorter lifetimes and more frequent rotation all increase the operational dependency on the CA, and automation-only issuance makes every TLS endpoint dependent on a live service relationship rather than a file. The open version is in good shape, for once. OpenSSL and the common libraries have the algorithms, and you can run your own post-quantum PKI on your own hardware today. What is scarce is people who have done the migration, which argues for doing it early rather than buying your way out. Here at AxisOps the position is that cryptographic agility is an ownership property. A system whose algorithms you can change is a system you control; one that needs a vendor\u0027s firmware release is a system you are renting. Treat any device that cannot have its cryptography changed by configuration as having a disposal date rather than a lifespan. Harvest-now-decrypt-later\u00b6 The case for urgency is that traffic recorded today can be decrypted once a capable machine exists. A real threat model, wheeled out as a scare tactic more often than it\u0027s assessed. The sober version is a question whose answer differs per organisation: what of today\u0027s traffic is still sensitive in fifteen years? For most session traffic, nothing at all. For long-lived secrets, medical records, state communications and anything with a statutory retention period, quite a lot. Do that analysis before you buy anything. The answer decides whether this is urgent for you or merely eventual, and the two have very different price tags. Related, on where the threat comes from: error correction is the whole game.", "d": "2026-09-30", "g": ["cryptography", "post-quantum", "security"], "t": "Post-quantum: key exchange was the easy half", "u": "insights/post-quantum-the-hard-half/"},{"b": "A quantum computer with a thousand physical qubits and no error correction is not a thousandth of a useful quantum computer. It\u0027s a different kind of object altogether, in the way that a thousand bricks is a different kind of object from a thousandth of a house. The number that matters is logical qubits, and the ratio between physical and logical is where the field really lives. Why the ratio is so brutal\u00b6 Quantum states decohere. A physical qubit holds its state for anything from microseconds to milliseconds depending on the technology, and every gate operation carries an error rate of its own. Classical error correction copies bits. Quantum error correction can\u0027t, because the no-cloning theorem forbids it. Instead you encode one logical qubit across many physical ones and measure stabilisers: operators that tell you whether an error happened without telling you (and so collapsing) the encoded state. Surface codes are the leading family. The physical cost per logical qubit is large, and it scales with how good your physical qubits are in the first place. The threshold theorem, stated carefully\u00b6 The threshold theorem says arbitrary-length quantum computation is possible if the physical error rate sits below some threshold. Above it, adding more physical qubits makes matters worse, because the correction machinery brings in more errors than it clears up. Two things follow, and both get glossed over: The threshold isn\u0027t a single universal number. It depends on the code, the noise model and how well connected the architecture is. Being below threshold is necessary and not sufficient. How far below sets the overhead, and the overhead decides whether a useful machine is the size of a rack or the size of a small town. Which is why \"we have N qubits\" is close to meaningless with no error rates beside it, and why the field\u0027s real milestones are the demonstrations where increasing the code distance decreases the logical error rate. That\u0027s the sign correction is working rather than merely being performed. The engineering, not the physics\u00b6 The part that gets least attention, and is arguably the hardest: Measuring stabilisers fast enough. Syndrome extraction has to run far faster than decoherence, which means real-time classical processing sitting in the loop with the quantum hardware. Decoding in real time. The classical decoder turning syndromes into corrections has a latency budget in microseconds. That\u0027s a hard classical computing problem that happens to live inside a quantum computer. Wiring. Routing control lines per qubit into a dilution refrigerator is a physical scaling problem and no amount of clever software will answer it. That last one is thoroughly unglamorous and may well turn out to be the binding constraint. The cryptographic story dominates the coverage and it is the least interesting application. Breaking RSA is a one-off event with a known mitigation that is already being deployed. The reasons to want one of these machines are elsewhere and they are substantial. Simulating quantum systems is the obvious one, and it is obvious because it is the thing a quantum computer is natively for. Chemistry at useful accuracy: catalysts, battery electrolytes, nitrogen fixation, enzyme behaviour. Those are problems where the current method is to approximate, synthesise and test, which is why developing a new catalyst takes a decade and a laboratory. A machine that can model the electronic structure directly turns part of that loop into computation. The payoff, if it lands, looks like cheaper fertiliser and better batteries rather than anything to do with cryptography. The energy implications of either are larger than anything else in this field. Optimisation is the claim to be most sceptical about, since the speedups are modest where they exist at all and classical methods keep improving. We would not plan a business on it. The quiet benefit is to classical engineering, and it is already being paid out. Building error-corrected quantum hardware requires real-time classical decoding in microseconds, cryogenic control electronics and absurdly precise timing. That work produces instrumentation, low-noise electronics and real-time computing techniques that are useful whether or not the quantum machine ever works. A good deal of the field\u0027s actual output so far has been classical. Code is not the scarce part. Capital is. An error-corrected machine needs a dilution refrigerator, a fabrication capability, a real-time control stack and several hundred people, sustained for a decade with no product. That is a short list of organisations: a few very large technology companies, a few state programmes and a small number of well-funded specialists who will probably end up owned by the first group. There is no garage version of this today, and our bet is that there will not be one: the cryogenics and the error-correction overhead look structural, not early. If that holds, whatever comes out will be rented by the hour, and it will be the most concentrated computing capability ever built, by physics and finance rather than by a business strategy anyone chose. That is a bet about engineering economics, and we have marked it as one. The consequence worth planning for is about archives rather than access. The asymmetry lies in the archive, not in three companies having a fast computer. The encrypted traffic recorded over the last twenty years is held by organisations with the budget to record it, and a capable machine arriving in that environment retroactively opens all of it. State communications, dissident networks, journalists\u0027 sources, medical records, anything with a long confidentiality horizon. The people with the most to lose from that are in the worst position to do anything about it, and the people who will get first access are the ones who have been collecting. That is the actual privacy story in quantum computing and it has very little to do with whether anyone can factor a number next year. Which makes migration an equity question as much as a technical one. Those with a budget are already rotating to post-quantum key exchange. The clinic, the charity, the local authority, the small operator with embedded devices in the field: they are not, and the capability that decrypts their archives will be rented by somebody else. The useful response is unglamorous and available now: move to post-quantum key exchange where you terminate the connection yourself, because on a current TLS stack that part is close to a configuration change. Then start the inventory for everything else, because the appliances, the embedded kit and the long-lived certificates are a programme, not a setting. The cost later is an archive you cannot unpublish. Where is the open version? Not in hardware, and it never will be. It is in the software and the standards, and that is where it matters: the algorithms are public, the implementations are open and nobody needs permission to protect their own traffic. Here at AxisOps that is the lever we can pull and the one we recommend pulling. You cannot build the machine and you do not have to; you can make the archive worthless, and the window for doing that cheaply is open now. What a non-specialist should take from it\u00b6 The summary for anyone outside the field: the progress is real, the timelines in the press coverage aren\u0027t reliable, and the things to watch are logical qubit counts and error rates rather than physical qubit counts. We are reporting here, not measuring. We don\u0027t run quantum hardware, and the people who do are not publishing the numbers that would settle the overhead question.", "d": "2026-09-02", "g": ["quantum", "hardware", "error-correction"], "t": "Error correction is the whole game", "u": "insights/quantum-error-correction/"},{"b": "A growing share of our engineering happens inside an agentic session: a model with a shell, a checkout and a test suite, working on one feature in that feature\u0027s workspace, with a person directing it. Whatever this week\u0027s opinion says about the ceiling of these tools, the floor has moved. By February, Spotify\u0027s co-chief executive was telling an earnings call that their best engineers had not written a line of code since December. The longer argument is written up in Spec-driven development, at richardstubbs.io. The session is where the work happens. That sentence has an operational consequence most of the commentary skips: the thing hosting the session is infrastructure now, and it gets the questions we ask of any infrastructure. What does it make possible, what does it cost and who can inspect it, change it or switch it off? The harness decides what a session can reach. Which filesystem, which credentials, which network, which memory of previous work, what gets logged and where the log lives. That is a boundary in the exact sense the rest of these insights use the word: a rule somebody specifies and a machine enforces. Run your sessions in a hosted harness and the boundary is specified elsewhere, logged elsewhere, retained on somebody else\u0027s terms. For exploratory work that trade is fine. For the sessions that build our features, on our estates, against our customers\u0027 systems, it fails the ownership test the same way a hosted control plane does: possession of the output, control of nothing underneath it. So we are building our own. Introducing micro-zero The design comes first, written down before any code, each decision recorded with the failure the alternative produces; the code will be held to the document, not the other way round. What it is\u00b6 A workspace session, reachable from a browser, belonging to a real Linux user. Authentication is the machine\u0027s own, through PAM: no parallel account system, no identity invented for the occasion. One feature being built gets one session in that feature\u0027s workspace, with the shell, the checkout and the tests, and the person steering it from whatever device is to hand. The design rule underneath it is the one these insights keep arriving at: the part that faces the network holds no privilege, and the part that holds privilege accepts no network. A browser talks to an unprivileged gateway, which runs with an all-zero capability set. The privileged broker that can spawn a shell listens on no TCP port at all, only a filesystem-permissioned local socket. And the shell that appears runs as the authenticated user, never as the broker that spawned it. Be precise about what that buys. A compromised gateway is not harmless: it relays terminal streams and file transfers, so whoever owns it sees what the sessions it carries see. What the design bounds is escalation. Owning the gateway confers neither root nor the host\u0027s filesystem, because the gateway can ask the privileged side for only a short, enumerated list of things: authenticate a user, spawn a shell, open a path, signal a process. The broker refuses each unless the request belongs to the authenticated user who owns the target, and adding a verb to that list is a design change, not a feature. Paths are resolved by the kernel itself, confined beneath the session\u0027s root with symlink escapes refused, so path traversal is removed from the broker\u0027s interface rather than defended against in application code. Passwords are held transiently and zeroed; the broker\u0027s privilege is dropped between fork and exec, before any user code runs. Those are claims a test can check, so the build will gate on tests that check them. The assertions worth gating on are the ones that state the split is intact: that a shell spawned by the privileged broker runs as the target user, that the broker accepts no TCP connection, that traversal attempts are refused end to end. A security design that is never restated as a failing test is a press release. Which way up\u00b6 Most harnesses run inside a general-purpose desktop operating system, alongside years of accumulated packages, services, credentials, mounts and reachable vulnerabilities. The OS is the world and the harness is a guest in it, inheriting whatever that world can reach. An agent running as a process on your desktop can turn any reachable weakness into a route beyond its intended workspace, and tidying up after it means believing you found everything. We think that is upside down. In micro-zero the harness is the outer layer and the operating environment runs inside it. The harness will create a minimal operating space from a micro-substrate-validated image, its provenance and scan state checked before the session starts, then grant into it only the tools, files, network reach and credentials the session requires. Control is added to a declared boundary rather than subtracted from a general-purpose desktop. Shell, desktop and workspace live inside that boundary rather than loose among the operator\u0027s own processes, and cleanup is deletion of the session, not a search for everything it touched. The flip is not only for the agents. A person will be able to take a desktop inside the harness on the same terms: the same boundary, the same validated start, with the agent working alongside them in the same bounded space. One limit, stated plainly: the host kernel remains shared and must be secured separately. The image boundary removes accumulated userland exposure; it does not remove kernel escape risk. The desk, with a container desktop connected through the Display pane: our internal dev build, running a basic code-defined desktop. The memory, and whose it is\u00b6 A session that starts blank re-derives the organisation every morning. The longer the work runs, the more the context is the work: long-running context belongs with the agent, not in a scrollback. So micro-zero will connect to hive-mind, the search over our vault and shared corpora. The read path lets a session ask what the organisation already knows, what was decided and why, and what earlier work discovered. Retrieved material remains evidence, not authority: appearing in memory does not turn text into an instruction the agent may execute. The write path is separate, and it needs more scrutiny than the read, because an agent can be wrong, can argue from incomplete evidence or can be led by something it retrieved. These sessions are designed to run long, days and weeks on one feature, and what a session learns will be recorded as it goes in a provenance-carrying feature journal: which session produced it, when, from what evidence, against which feature. Promotion into shared organisational memory will be a separate act with a review in it. An agent may propose what the organisation knows; it does not silently decide it. That is also what makes the disposability above safe to mean: deleting a session loses no declared durable state once its repository, its journal and its accepted contributions are current, and uncommitted state stays disposable by design. The connection will have an identity of its own: a desk token issued to the service, its read scope and its contribution scope defined separately. Never a person\u0027s credential copied into a config, because a service reading with a human\u0027s token acquires that human\u0027s reach and makes the audit trail say a person did something a machine did. Context is becoming the asset in this kind of engineering. An asset gets an owner, a scope and an audit trail, or it leaks. Why \"zero\"\u00b6 The zero is where every session starts. A clean workspace, a clean shell and no authority beyond what the authenticated user already holds. That is a narrower claim than \"no authority\", on purpose: a session runs as a real Linux user and inherits exactly that user\u0027s files, groups and network reach, which is a plain, inspectable boundary. What the harness adds on top of it is nothing: no standing credentials, no service account riding along, no privilege the user did not bring. What a session accumulates that is worth keeping goes back to the estate: committed to the repo, written to the vault, recorded against the feature. The session itself remains disposable. A harness built on any other assumption turns the chat transcript into the system of record, and a chat transcript is the worst database anyone has ever operated. What it costs\u00b6 Building it, when a dozen hosted products would love to do it for us and self-hosted pieces already solve parts of the problem: browser terminals, isolated workspaces, agent desktops. Those components exist. The reason to build is that the privilege boundary, the machine identity and the memory path are the product here, and adopting a system that combines them behind one privileged service would inherit the decision micro-zero exists to control. Running it: another service with an attack surface precisely where it is most useful, which is why the privilege split above is the one decision every other decision leans on. Ownership also means maintaining and rescanning the validated images, rotating service identities, expiring abandoned sessions, defining what the audit trail retains and recovering when repository or journal write-back fails: a disposable session is cheap only while the durable paths around it are explicit and tested. And where it is allowed to listen will be a named decision rather than a default. On a lab network this is an ordinary admin tool; on the public internet it is a remote-shell endpoint. Until that decision has a named owner the gateway does not run in production at all. Then the usual cost of owning anything: when it breaks, the pager is ours. What it buys is the other side of every entry here about rented control planes. The boundary around the sessions that build our systems is ours to specify, inspect, change and switch off. The work those sessions produce lands in repositories and a vault we hold, read back through a memory service we run, under identities we issue. The tools changed this year. The questions did not.", "d": "2026-07-29", "g": ["ai", "tooling", "operations", "infrastructure"], "t": "micro-zero: own the harness", "u": "insights/micro-zero-own-the-harness/"},{"b": "An instruction set architecture is a specification. There\u0027s no technical reason one has to be proprietary, and yet for forty years nearly every commercially significant one was. SPARC\u0027s published specification and POWER\u0027s late opening were real exceptions, but neither broke the x86-Arm duopoly\u0027s grip. RISC-V\u0027s contribution is that its specification is open and its extensions are modular. Both halves matter, and the second half causes most of the real-world grief. Where it is\u00b6 We\u0027ll report this carefully, because coverage of RISC-V runs well ahead of ratification and conference material gets cited as though it were a standard. Server SoC and Boot specifications are ratified. RISC-V\u0027s own 2025 annual report says so. The Server Platform specification is ratified too. The ratified specifications library lists Server Platform v1.0, dated May 2026. The 2025 annual report had forecast ratification by the end of 2026. A forecast and the record of the event it forecast are not in conflict, and the library is the record. Recently ratified extensions include Instruction/Data Coherence (Ziccid, June 2026), per the ratified extensions table. If you want a current answer, go and read the ratified-extensions table. Don\u0027t trust a summary, this one included, past its date. The fragmentation problem is the real one\u00b6 Modularity is the selling point and the liability in one. An ISA assembled out of optional extensions means that \"RISC-V\" on its own tells you nothing about what a chip can do, and two conforming parts can fail to run the same binary. It\u0027s a kit car; everyone\u0027s is road-legal and no two take the same spare. Platform specifications exist to fix this: a profile that declares these extensions mandatory, so software has a baseline to target. The work is thankless and it matters more to adoption than any individual extension, because it\u0027s the difference between an ecosystem and a pile of incompatible chips. ARM solved the same problem by being one company with the authority to say no. An open standard has to do it by committee, which is slower and produces documents like the ones above. It matters because of what happens downstream when nobody owns the document: Verification. You can\u0027t formally verify an implementation of a specification you aren\u0027t allowed to read. The seL4 proofs reached RISC-V for a reason. Capability extensions. Memory-safety research that adds new semantics to pointers is being specified as a RISC-V extension, which on a proprietary ISA would need the owner\u0027s agreement and the owner\u0027s commercial interest in the outcome. Research silicon. Academic and small-vendor parts exist that would never have cleared a licensing conversation. The ARM port already did the hard cultural work; see ARM reaching the server room. RISC-V arrives in a world where \"does it build on another architecture\" is a normal question rather than an exotic one, which is why its software story starts from a far better place than ARM\u0027s did. Look a little further and the prize is a different shape of hardware industry. If the specification is free, then designing a chip stops requiring a negotiation with a competitor. That matters most for the parts nobody large wants to make: a controller for agricultural equipment in a country without a semiconductor industry, a processor for a medical device with a thirty-year service life, a repairable design whose documentation outlives the company that sold it. The thing that becomes cheap is the low-volume part, and low-volume parts are where repairability, longevity and local manufacturing live. A specification with no owner cannot be withdrawn when a product line stops being interesting, which is the failure mode that strands working hardware today. For engineering the win is more immediate: a target you can read is a target you can simulate, model and prove things about. Expect the serious verification work of the next decade to happen on RISC-V, for no better reason than that the documents are available and the licence does not have to be renegotiated for each new academic. The profile, rather than the specification, is the chokepoint. An open ISA with optional extensions does not guarantee an open ecosystem. It guarantees an open document. What software targets is a profile, and whoever decides which extensions are mandatory in the profile that matters decides what the architecture effectively is. Today that is a committee of members, which is slow and visible. The risk is that it stops being the thing that matters. The failure mode to watch for is vendor dialects. A large chip vendor ships a part with its own custom extensions, writes a toolchain that targets them, gets enough software built against it, and the practical baseline becomes \"their RISC-V\" rather than the ratified profile. Nothing about this breaks the licence or the specification. It does not have to. The specification stays open while the ecosystem splits, and you end up with three incompatible families, each open, each useless to the others. That is the worst of both worlds: the coordination cost of a committee and the lock-in of a proprietary ISA. Anybody who lived through the Unix wars has seen this film, and the lesson from it is that an open standard with no enforcement is a suggestion. The other pressure is geopolitical and it is specific. An ISA designed to be unlicensable is attractive to whoever is worried about losing a licence, which means significant RISC-V investment now comes from states with an interest in not being dependent on American architecture licences. That is a real strength for the architecture and a real risk for its coherence, because the incentive for a state-backed programme is a profile it controls rather than one it shares. Export control pressure applied to the specification process would accelerate that split. The specification being public does nothing to stop the community that maintains it fragmenting along the same lines. And the thing an open ISA never addressed: fabrication. You can read the specification, design the chip, verify it formally and still need one of a very small number of fabs to make it. Open silicon design with closed manufacturing is a real improvement over closed both, and it is not sovereignty. Where is the open side? Winning on the document, holding its own on toolchains and vulnerable on profiles. The practical defence is dull and effective: target a ratified profile, treat vendor extensions as the proprietary dependency they are and refuse to ship software that only runs on one company\u0027s dialect. Here at AxisOps that is how we will buy RISC-V hardware when we buy it. If a board\u0027s value proposition requires their fork of the toolchain, it is not an open architecture, whatever the marketing says; it is a proprietary chip with a fashionable name on the box. Where the value turns up\u00b6 RISC-V\u0027s significance is unlikely to be displacing x86 or ARM in a datacentre this decade. It\u0027s that the specification is a public good, and the second and third order effects of that (verification, research, capability hardware) are where the value turns up.", "d": "2026-07-22", "g": ["riscv", "hardware", "open-source", "architecture"], "t": "RISC-V: an ISA you are allowed to read", "u": "insights/riscv-production/"},{"b": "EEVDF became the default fair-class scheduler in 6.6, October 2023 (kernel.org), and the transition off CFS completed in 6.12, with the remaining pieces those LWN catalogued that spring (LWN). That\u0027s the first change of that magnitude to the fair scheduler since CFS turned up in 2.6.23 back in 2007 (kernel.org). Most people will never notice, and that\u0027s the interesting part. What changed\u00b6 CFS tracked virtual runtime and picked the task with the lowest; fairness by accumulated service. Latency was managed indirectly, through heuristics and tunables like sched_latency_ns and wakeup preemption rules that accreted over fifteen years. EEVDF (Earliest Eligible Virtual Deadline First) makes latency explicit. Each task gets a virtual deadline derived from its requested time slice, and the scheduler picks the eligible task with the earliest deadline. A task that wants low latency asks for a shorter slice, and gets scheduled more often for less time each go. The practical difference is that latency becomes a declared property instead of an emergent consequence of heuristics. That\u0027s the same move being made all over systems work right now: replacing a pile of accumulated special cases with a model. Why tuning usually fails\u00b6 The common failure is that the symptom is scheduling-shaped and the cause isn\u0027t. Before you touch a scheduler knob, here are the far more likely explanations: Lock contention. Threads are runnable but serialised. The scheduler is doing its job; your application is the bottleneck. I/O blocking misread as CPU starvation. A task waiting on a device isn\u0027t waiting on the scheduler. NUMA placement. Memory on the wrong node costs you more than any scheduling decision will ever win back. cgroup limits. CPU throttling via quota produces latency spikes that look just like scheduler unfairness and aren\u0027t. Frequency scaling. That\u0027s the governor, not the scheduler. The diagnostic order matters. Establish whether tasks are runnable-but-not-running (sched:sched_wakeup to sched:sched_switch latency) before you assume the scheduler is choosing wrongly. If the tasks aren\u0027t runnable, scheduling isn\u0027t your issue. Tuning before that check is like replacing the gearbox because the car won\u0027t move, when nobody has looked in the fuel tank. Where it really is a scheduling problem\u00b6 Narrow, and real: Mixed latency-critical and batch workloads on shared cores, which is what EEVDF\u0027s explicit slice request is there to address. Heterogeneous cores. big.LITTLE and performance/efficiency core asymmetry is a hard scheduling problem and still not fully solved; see heterogeneous by default. Hard real-time, which wants SCHED_DEADLINE or a different kernel altogether rather than any amount of tuning the fair scheduler. The more significant development is sched_ext: loading a scheduler written as a BPF program. That turns scheduling policy into something you can iterate on in minutes rather than across kernel releases. What that opens up is worth spelling out, because it is a change in who is allowed to have an opinion. Scheduling policy has always been decided by a small number of kernel developers arbitrating between workloads they cannot all test, and the result is a good compromise that is nobody\u0027s optimum. With sched_ext, the person who knows the workload writes the policy. A game engine developer can ship a scheduler tuned for frame deadlines. A database vendor can ship one that understands which of its threads must not be preempted mid-latch. A build farm can run one that knows a compile job is throughput work and a test runner is latency work, and stop pretending they are the same shape. The engineering win underneath is iteration speed. Today, trying a scheduling idea means patching the kernel, building it, rebooting a machine into it and hoping the box comes back. That is an afternoon per experiment and it is why scheduler research moves slowly. Make it a program you load and unload while the system runs, and the experiment loop goes from hours to seconds. The safety is two mechanisms, and it is worth being precise about which does what: the verifier bounds what the program can touch, and a watchdog ejects a scheduler that stalls runnable tasks or misbehaves and falls back to the default. The experiment is recoverable, not proven harmless. A loaded policy can still be slow, unfair or starvation-prone right up until the kernel pulls it. Our expectation is that more scheduling progress happens in the five years after this lands than in the fifteen before it, and for the dull reason that people can finally try things. There is an energy argument too. Correct placement on heterogeneous cores is worth real power, which matters on a phone, on a laptop and across a fleet where the electricity bill is the second-largest line of spend. A policy that knows the workload beats a general heuristic, and now the policy can know. Whose tuning is it\u00b6 The chokepoint here is unglamorous: whether your workload\u0027s scheduling policy is yours or your distribution\u0027s. Right now it is the distribution\u0027s, and nobody minds much because the default is good. sched_ext changes the stakes. Once policy is a loadable program, it becomes something that can be shipped, bundled, licensed and withheld. The plausible near future is that the people with the resources to write and maintain a good one are the hyperscalers tuning their own fleets, the large database and game vendors and the distributions. Everyone else takes what they are given, as now. That is tolerable. What would not be is the version where loading your own policy requires a signature the platform owner controls, or where the managed platform you rent runs its scheduler and declines to run yours. The capability then exists and belongs to the operator rather than the workload owner, and the fact that it is BPF rather than a kernel patch has bought you nothing. Watch what managed Kubernetes providers permit here; that is where the answer gets set in practice, long before anybody writes it down. The open side is in a strong position and should stay there. sched_ext is in the mainline kernel, the schedulers people are writing are published, and running your own on your own metal needs permission from nobody. Here at AxisOps that is the whole argument for owning the machine: the day the tuning matters, you want to be the one who can change it. A workload whose scheduling you cannot influence is a workload running on somebody else\u0027s terms, and you will discover that at the worst possible moment. Start from the default\u00b6 The default scheduler is the product of enormous amounts of tuning against an enormous variety of workloads, and your prior should be that it\u0027s right. Measure first, and expect the answer to be somewhere other than the scheduler.", "d": "2026-02-10", "g": ["linux", "kernel", "performance", "scheduling"], "t": "Schedulers, and why the default is usually right", "u": "insights/kernel-scheduler-notes/"},{"b": "Every assistant session starts from amnesia. It does not know which client runs which stack, what was decided about backups in 2019 and reversed in 2023, why the monitoring alert thresholds are what they are or that the one server with the odd disk layout is that way on purpose. The person driving the session knows, and spends the first ten minutes pasting context in by hand, session after session, like briefing a brilliant contractor with no memory on their four-hundredth first day. The industry\u0027s answer is a bigger context window. That answer misreads the problem. An organisation\u0027s memory is fourteen years of decisions, designs, client history and hard-won operational folklore, most of it irrelevant to any given question and all of it essential to some question eventually. It is not a document you load. You do not want it all in the window. You want the right ten paragraphs, found on demand. The name for that is retrieval, and retrieval is a solved- enough problem to go and build with. What we have, and what we built\u00b6 We have an unusual asset, built by accident of discipline: years of the company written down as plain markdown in one vault. Decisions, designs, runbooks, meeting notes, client records. No wiki product, no SaaS knowledge base, no export problem. Files in folders, under sync, on machines we own. So the build is unglamorous, and the first version of it went together over the holidays. The vault lands on a small machine of its own. An embedding model running locally turns each note into vectors, in crude line-range slices. Markdown\u0027s headings would make the better boundary, and a disposable index is allowed to start crude. The vectors sit in SQLite next to a full-text index of the same notes, and a query runs both: semantic search for \"what do we know about this shape of problem\", keyword search for the exact error string, the two result lists fused by rank. An assistant session asks over a standard tool interface and gets back passages, with paths, from the organisation\u0027s own memory. Nothing in that paragraph is novel, and nothing needed to be. The components are a local embedding model, SQLite with a vector extension and a sync tool we already ran. No new database server, no managed vector store, no per-query bill, and nothing about the company\u0027s internal history leaves the premises to be embedded on somebody else\u0027s machines. Whether this is possible was never in doubt. The question is whether the answers are good enough to change how the sessions run. Early returns\u00b6 Three weeks in: better than expected, with sharp edges. \"Better than expected\" rests on three weeks of daily use, not on any measurement. The case we keep retelling is a single one: the session that asks \"have we dealt with this vendor\u0027s licensing before\" gets the 2021 note that a person would have taken twenty minutes to remember existed. Keyword search rescues the queries where semantic similarity goes vague. The failure mode that matters is silent absence: a search that returns nothing proves nothing, because the note may exist under a heading no query reaches. The index is a view of the vault, never the record; when it matters, somebody still greps the files. The design rules we are holding onto, stated now so we can be judged against them later: The vault stays the source of truth. The index can be rebuilt from it at any time, and deleted without losing a word. Read-only. The retrieval service answers questions; it does not write memory. What goes into the vault goes through a person. Local models, and an honest line about where that stops. The embedding runs on our hardware: the vault is never shipped out wholesale to be vectorised. But the passages a query retrieves go into the prompt of whatever model the session runs on, and when that model is hosted, the retrieved text leaves the premises at answer time. Local embedding bounds the indexing; it does not bound the withdrawal. One reader, for now. The vault holds client records alongside our own, and today the only thing that queries the index is us. The moment it gains a second reader, human or service, retrieval needs an authorisation story of its own: a scope on who may withdraw what. That control is not built, and until it is, the reader count staying at one is the control. Plain files underneath. The day this experiment dies, the vault is still markdown in folders, worth what it always was. What it changes if it works\u00b6 A session that can ask the organisation what it knows stops being a clever tool you brief and starts being something closer to a colleague with access to the filing. The ten minutes of pasted context goes. The answers improve in the specific way that matters: grounded in what this company actually decided, not in what the internet thinks companies generally do. And the three questions we ask of everything have tidy answers for once. What it makes possible: sessions that know the estate. What it costs: one VM, one embedding model, the sync, indexing and backups that keep an index honest, vigilance about the silent-absence failure and an authorisation story it does not yet have. Who controls it: we do, end to end, because it is files we own, indexed by models we run, on hardware in the building. Memory is becoming the asset in this kind of engineering, and this one is ours.", "d": "2026-01-20", "g": ["ai", "tooling", "operations"], "t": "The context problem is a retrieval problem", "u": "insights/context-is-a-retrieval-problem/"},{"b": "\"Digital sovereignty\" is a phrase that attracts a great deal of posturing. Underneath it sits a legitimate engineering question with concrete answers, and here at AxisOps we\u0027d rather talk about that one. The question, stated usefully\u00b6 The question is not \"is it built here\". That\u0027s industrial policy, a different argument for a different room. The engineering question is: which capabilities that we depend on could be withdrawn or compelled by a party outside our control, and what is our position if that happens? Ask it that way and flag-planting becomes dependency analysis, which engineers can sit down and do on a Tuesday afternoon. The layers, by difficulty\u00b6 Data location. The easiest, the most discussed and nowhere near enough on its own. Data in a local region is still subject to the operator\u0027s home jurisdiction in various circumstances. Necessary; not sufficient. Operator access. Who can read it. This is where confidential computing makes a real contribution, though it moves the trust to the silicon vendor rather than removing it, and silicon vendors are concentrated in a handful of jurisdictions. Software supply. If a licence can be revoked or an update withheld, your software is a dependency on a continuing commercial relationship. A licence that can\u0027t be revoked, over source that can be forked, is a different risk class even if you never fork a line of it. Silicon. The hardest layer, with no near-term answer. Leading-edge fabrication is concentrated in very few places and the supply chain underneath it is more concentrated still. RISC-V addresses the specification layer, which matters, and does nothing whatever about fabrication. Operational capability. The layer everybody ignores. The right to run something is worthless without people who can run it. A sovereign stack nobody can operate is a worse position than a dependency you\u0027ve written down. Why open source does real work here\u00b6 Not for ideological reasons; think of it as who holds the keys to the garage. A proprietary dependency is a relationship with a company that can change its terms, get acquired, leave the market or find itself on the wrong end of an export control. Recovering from any of that needs their cooperation. An open-source dependency can be forked. You may never do it, but the capability changes both your negotiating position and your worst case, without a nation-state-scale programme. What a graded answer makes possible\u00b6 The useful version of sovereignty is the ability to survive a withdrawal. Independence does not come into it. That is achievable for one organisation, and the effects are larger than the phrase suggests. Take a hospital trust. Today a change in a software vendor\u0027s licensing, or a dispute between two governments, is an event it can only absorb. With the stack declared, the data in documented formats and the ability to run it on hardware in the building, the same event becomes a procurement decision taken at a time of its own choosing. What becomes cheap is continuing to work during an argument you are not a party to. Scale that up and it changes what a mid-sized country can plan. The sovereign posture most are reaching for is a hyperscaler region with a local flag on it, because the alternative looks like building everything. The achievable middle is an estate where the critical services are declared, the formats are open, the software is forkable, and the operational knowledge sits with people on the payroll: an engineering programme with a budget rather than a decade-long industrial fantasy. The benefit to engineering itself is the one we feel most directly. A stack rebuildable from source and configuration can be stood up again after a disaster, moved between providers and handed to a new team intact. The toil that disappears is archaeology: the week spent discovering how production was assembled because the people who assembled it have moved on and the console is the only record. Vendors blur the definition at the point where it becomes inconvenient. A hyperscaler region inside your borders, operated by a local subsidiary with local staff and a reassuring certification, is marketed as sovereign. Test it with one question: if the parent company were ordered by its home government to withdraw the service, could the region keep running? If the answer needs the parent\u0027s control plane, identity service, software updates or key management, you have a datacentre location rather than sovereignty. The dependency is the control plane, and it has never been in your jurisdiction. The two are priced similarly and sold interchangeably. A sovereign deployment means software licensed in a way that survives the relationship ending, operational knowledge held locally and somebody who can rebuild it from the artefacts on hand. That is harder to buy and easier to verify: ask for the rebuild to be demonstrated. Nobody demonstrates it, which tells you how often it has been tested. What concentrates power sits a layer above the datacentre. Identity above all: an estate where every employee signs in through one provider\u0027s directory has handed that provider the ability to stop the organisation working, and no amount of local hosting touches it. Then the egress charges that make leaving expensive at the moment you need to, and the certificate and DNS dependencies nobody listed. Each was a sensible decision, and collectively they are a sovereign commitment no minister ever signed. The open side is better placed on this question than on almost any other: the whole stack for a serious estate can be assembled from components nobody can revoke. What is scarce is operational capability, a training and staffing problem, and it is the layer that gets cut first. The real risk is that nobody is left who can run it, not that the open version loses on merit. Here at AxisOps this is the position we build from: infrastructure the customer owns, can inspect and can rebuild without us in the room. We would rather hand over something you could fork than something you have to keep paying for. The limits\u00b6 Full independence isn\u0027t achievable for any organisation, and it\u0027s achievable for very few countries. Anyone promising it to you is selling something. What you can have is a graded answer: know your dependencies, know which are replaceable and on what timescale, reduce the ones where the recovery story is \"we\u0027d be stuck\" and accept the rest as a decision rather than an oversight. It\u0027s the same discipline as knowing your dependency graph for availability. Same graph, read for a different failure mode.", "d": "2025-12-09", "g": ["policy", "infrastructure", "sovereignty", "open-source"], "t": "Digital sovereignty is a dependency question", "u": "insights/sovereignty-and-the-stack/"},{"b": "In February 2025 Apple withdrew Advanced Data Protection, its end-to-end encrypted iCloud option, for users in the United Kingdom. Apple didn\u0027t explain why in any detail, and the reason it couldn\u0027t is the interesting part. What is established\u00b6 Being careful here, because this case gets reported heavily and accurately rather less often. What\u0027s solid: ADP was withdrawn for UK users in February 2025. That\u0027s observable. The Investigatory Powers Act allows the Home Office to issue a technical capability notice, and a recipient can be prohibited from disclosing that one exists. The power is in the statute. Apple has gone to the Investigatory Powers Tribunal (The Register), and the government\u0027s position is neither to confirm nor deny that an order was issued. What\u0027s not established: whether a notice exists at all. The Home Office doesn\u0027t confirm or deny orders under the Act. Reporting that states flatly that one was issued is going beyond the record, however likely it may be. The engineering content\u00b6 Strip the politics out and you\u0027re left with a concrete technical claim, and it\u0027s the same one from the 1990s: that exceptional access can be added to an end-to-end encrypted system without weakening it for everyone. The argument against isn\u0027t ideological. The access mechanism is itself an asset (a key, a key-escrow service, a client-side scanning hook) and assets get compromised. You aren\u0027t adding a door for one party. You\u0027re adding a door, and then making a separate claim about who can open it that rests on operational security rather than on mathematics. Every hotel that ever fitted a master key will tell you how that goes. The second technical point is subtler and matters more for anyone building: end-to-end encryption is a property of a system rather than a feature of a product. Client-side scanning before encryption doesn\u0027t break the cryptography at all. It makes the cryptography irrelevant, which is why \"the encryption is untouched\" manages to be true and beside the point at once. The chokepoint is who gets to mandate the code on the device. Scanning on the device before encryption needs three things: code running on every handset, a list of what to look for and someone with the authority to update that list. Two companies ship the operating systems on which essentially all of the world\u0027s phones run. They already push code to those devices weekly, without asking, and the update mechanism is the most thoroughly engineered compulsion surface ever built. Nobody has to break encryption if they can instead ask Apple and Google to add a check in front of it. What makes this different from the 1990s fight is that the chokepoint has moved from the algorithm to the device, and the device is not yours. In 1995 the argument was about export controls on a mathematical technique, which was unwinnable because maths does not have a factory. Today\u0027s argument is about a build pipeline with two owners, and that is an entirely winnable fight for a government. That is why this version is quieter. You do not need a public debate about key escrow when you can have a private conversation about a feature flag. The second-order effect is worse. A scanning list is a general-purpose mechanism with one stated purpose. The infrastructure built to detect one category of material detects whatever is on the list. The list is not public, and the set of governments who will want their own entries on it is not small. Every operator who builds the capability has then to maintain a position on which states may use it. That is a political judgement a company is poorly placed to make and cannot decline to make once the code exists. Where is the open version? Running, and holding up better than you might expect. Signal has said it will leave a jurisdiction rather than comply, which is the only credible answer a centralised service can give. The structurally stronger answer is to hold no key and run no registry, so there is nothing to hand over and no server to serve. That is buildable today on commodity hardware. Here at AxisOps this is a design rule rather than an opinion. Customer data is encrypted under the customer\u0027s own keys, in vaults we do not hold the keys to, so we cannot read it even where we run the storage. The most reliable way to resist a demand for data is to be unable to satisfy it, and every key we do hold is a liability we are keeping warm for somebody else. Because this is a design constraint rather than a news story. If you build anything that holds other people\u0027s data in a jurisdiction with this kind of power, then \"what could we be compelled to do, and could we comply even if we wanted to?\" is an architecture question with real answers. An operator holding no key cannot be compelled to produce one. That\u0027s design, not defiance. The flip side, and the thing worth being cheerful about: the same shift that makes device-side compulsion effective also makes end-to-end encryption ordinary. Messaging that nobody in the middle can read is now the default for billions of people, which was a cypherpunk aspiration within living memory. The fight is over the edges because the centre was won. The prediction\u00b6 The pattern across three decades is that these powers are sought, resisted, partially granted and never repealed. The statute outlasts the news cycle. Build on the assumption that the legal surface available to a future government is the one written down now, not the one currently being used.", "d": "2025-11-18", "g": ["security", "policy", "cryptography"], "t": "The crypto wars never ended, they just got quieter", "u": "insights/crypto-wars-round-two/"},{"b": "The internet\u0027s founding property is that it has no centre. Packets route around damage, and no single operator can take the thing down. At the IP layer that\u0027s still broadly true. At every layer above it, it has substantially gone. Where the concentration sits\u00b6 Not one chokepoint but several, each significant on its own. CDN and edge, where a small number of providers terminate TLS for an enormous fraction of web traffic. Three cloud providers, with particular regions carrying wildly disproportionate load (us-east-1 has its own folklore for a reason). Authoritative DNS and, increasingly, resolution, where DNS over HTTPS is a real privacy improvement that also moved resolution from thousands of ISP resolvers to a handful of large operators. Certificate authorities, one of which issues a very large share of the web\u0027s certificates. One package registry per language ecosystem. And app distribution, where two companies decide what software can be installed on the device most people use to reach any of the above. Concentration turns faults into outages\u00b6 Correlated failure, which is the one distributed systems design handles worst. The design assumption is independent failures: node A falling over tells you nothing about node B. Concentration breaks that assumption. When a major CDN or DNS provider has an incident, thousands of unrelated organisations fail at the same moment, and so do their status pages, their incident tooling and sometimes their ability to log in and fix it. AxisOps wrote about one such event on the main site, from the business side. The engineering version: your availability is the product of dependencies you did not choose and mostly cannot see. Convenience built the centre\u00b6 Not villainy. Economics. Running your own authoritative DNS, your own CDN and your own certificate issuance is expensive, needs specialist skills and will almost certainly be done worse than a specialist does it. For any one organisation, using the big provider is the right call. Everyone making that right individual call adds up to a fragile system: the motorway at rush hour, where every driver picked the fastest route and that is why nobody is moving. A collective action problem, with no solution available to any single participant. Name the layer, because \"centralisation\" is too vague to act on\u00b6 Each chokepoint has a different owner, a different failure and a different remedy. Treating them as one problem is why the conversation goes nowhere. The CDN is the surveillance layer. A handful of companies hold the private keys for a large fraction of the web and decrypt the traffic as a condition of doing the job. Every request, from every visitor, in plaintext, at one operator, with nothing in the browser to indicate it. The outage risk gets the headlines. The part that should keep you up is the standing view of human behaviour, held under commercial terms and subject to whichever government\u0027s legal process applies. DNS is the compulsion layer. It is where blocking is implemented, because it is cheap and it works. A handful of encrypted resolvers can make a name cease to exist for most of a country\u0027s population. The certificate authority is the existence layer. Nobody suggests the big non-profit CA is badly run. The problem is that its continued existence is now a prerequisite for a large fraction of the web renewing its certificates on schedule. Its failure is a ninety-day countdown, not a preference. The three clouds are the capability layer. Identity, queueing, the managed database, the ingest pipeline. Build on them and your system is not portable, whatever the containers suggest. Egress pricing is the mechanism: data costs nothing to put in and a great deal to take out, so the cost of leaving rises with every byte you store. A pricing decision, not a technical constraint. The mobile app stores are the access layer, and the hardest. Two companies hold a veto over what software reaches the device through which most of the world reaches everything else. Not pricing, not performance: permission. There is no appeal to a technical standard, because the gate is not a protocol. Every other layer here has a self-hosted path. This one does not, on the dominant platform, by design. Limited, but not nothing: Know your dependency graph, transitive ones included. Most organisations cannot name what their availability depends on. Secondary DNS with a second provider. One of the few practical diversifications, because DNS was designed for multiple authoritative servers in the first place. Keep your incident tooling off your production dependencies. If your status page sits behind the CDN that is down, you have not got a status page. Terminate your own TLS where the traffic is sensitive. An origin with your own certificate and no third party in the middle removes the surveillance layer for the traffic you care most about. Price your exit. Know what it would cost in egress to leave, and treat that figure as a debt on the balance sheet, because it is one. What you cannot practically do is run your own everything. Anyone selling you that as the answer has not costed it. The optimistic case is specific, and more achievable than most people assume. The cost of running your own infrastructure has collapsed. A capable server costs less than a laptop, a residential connection carries more bandwidth than a business line did a decade ago, and the software (nginx, PostgreSQL, Pelican, WireGuard, a Linux distribution anyone can inspect) is free, excellent and boring. Most organisations rent instead out of habit and a scarcity of people who have done it recently. That is a skills problem, and skills problems are solvable. A self-hosted estate gets you the thing no provider can sell: your system fails when you make a mistake. Running your own origin, your own DNS and your own mail does not make you immune to outage; it makes your outage uncorrelated with ten thousand strangers. For a hospital, a council or a legal practice, being able to say where the data physically sits and who can be served an order for it is becoming a requirement rather than a preference. The prediction we will commit to is that the pendulum swings back, driven by cost rather than principle. Egress pricing and the size of the cloud bill are doing more for self-hosting than any argument about sovereignty managed. Expect tooling whose entire pitch is operating your own infrastructure with a small team, and expect the large providers to respond by deepening the lock-in rather than fixing the pricing. The useful work is making the self-hosted path cheap enough and dull enough that choosing it is the obvious commercial decision rather than an act of conviction.", "d": "2025-11-04", "g": ["policy", "infrastructure", "resilience", "dns"], "t": "Three companies and a protocol stack", "u": "insights/centralisation-of-the-internet/"},{"b": "An argument was made here about time before: that it is harder than it looks and duller than it sounds, and that distributed systems quietly assume a shared clock they have not earned. That entry treated drift and leap seconds as the adversaries. They are the benign case. Interference with satellite navigation over the Baltic, the eastern Mediterranean and the Gulf has become routine rather than exceptional, to the point that aviation authorities issue standing advisories and airlines plan around it. The reporting is about aircraft, because that is where the safety case is sharpest. The infrastructure consequence is broader and gets less attention. What actually depends on this\u00b6 For infrastructure, GNSS serves mainly as a time distribution system that happens to also give you position, and it is the cheapest source of traceable time that exists. A receiver costs very little and yields nanosecond-grade time anywhere with a view of the sky, which is why it ended up underneath so much. Mobile networks need tight synchronisation between base stations, and tighter still for the time-division modes. Electricity grids use synchronised measurement to work out what is happening across a region, and a phase-angle measurement is meaningless without a common clock. Financial venues are required to timestamp to traceable sources. Broadcast, industrial control and most of the measurement equipment in this trade carry the same dependency, usually undocumented. The failure mode that matters is not loss of signal. Loss is noisy and obvious, and a system that notices can act. Spoofing is the problem: a receiver presented with a stronger, coherent, wrong signal will report a confident, wrong answer. Downstream, a system that assumed its clock was correct begins making consistent decisions on false data, and nothing in the stack is instrumented to catch that, because the input was never previously hostile. What it costs\u00b6 Holdover, and holdover is a line item. The engineering answer is well understood: keep an oscillator good enough to coast when the reference is unavailable or untrusted. The choices run from ordinary oven-controlled crystals through rubidium up to caesium, and the difference between them is roughly the difference between coasting for hours and coasting for months. Price tracks that span steeply, which is why the capability is usually absent. The second cost is that you must now decide whether to believe a reference, which is a problem class the trade is better at than it realises. Compare multiple constellations. Compare against the local oscillator and reject a step that physics does not permit. Compare against a terrestrial source. Treat time as an input requiring corroboration rather than a fact, which is how a careful system treats every other input arriving over a network. None of this is novel engineering. It is standard practice in one or two industries and essentially absent everywhere else, and the reason is that it costs money to defend against something that has historically never happened. What it makes possible\u00b6 The constructive reading is that this forces an overdue correction, and the result is better systems than the ones we have. A system that treats time as an attested input rather than an assumption is more robust against every failure, not only deliberate ones. Receiver faults, antenna problems, firmware bugs and solar events all produce wrong time, and a design that corroborates catches the lot. The adversarial case is simply the one that finally justifies the work. It also pushes towards distributed timing that does not depend on a single provider. Terrestrial time distribution, time over fibre between known endpoints and mutual comparison between peers are all workable. A mesh of modest clocks that agree with each other is more resilient than a hundred excellent clocks all listening to the same satellite. That is the same architectural argument this notebook has made about registries and about routing, arriving in the time domain. The wider benefit is sovereignty in the practical sense rather than the rhetorical one. A country, a hospital or a grid operator that can keep accurate time without a foreign military signal has removed a dependency it probably did not know it had. The equipment to do that amounts to a decent oscillator and the discipline to specify it. Who owns the reference\u00b6 Every global constellation is a state asset operated by a military or a space agency: the American, Russian, Chinese and European systems, with regional systems besides. The civilian signal is a courtesy. It is unauthenticated by design in its basic form, it can be degraded regionally and the operators retain the ability to do so. That is the dependency in one sentence. A large share of civilian infrastructure takes its clock from a signal controlled by foreign militaries. The signal is free, unauthenticated and trivially overpowered by a transmitter costing very little. It has worked for decades, which is why nobody specified an alternative. Authenticated civil signals are arriving and are a real improvement, since they make spoofing substantially harder. They do not address jamming, they require receivers that support them and the receiver fleet is long-lived. The replacement cycle is measured in decades. The small-operator position is the usual one. A large bank or a national grid can afford caesium and a terrestrial backup. A regional hospital, a water utility or a four-person company running something that matters cannot, and will discover the dependency during the event rather than before it. Here at AxisOps our position is that time belongs on the dependency list with everything else. Write down where yours comes from, what happens when it is wrong rather than absent, how long you can run without it and how you would know. That is an afternoon of work and almost nobody has done it, including most organisations that would describe themselves as critical national infrastructure. The measurement we have not taken\u00b6 We do not have a figure for how long our own estate holds usable time with the sky removed. We know the answer is worse than we would like and better than nothing, and that it depends mostly on parts chosen for other reasons. It is a cheap experiment: disconnect the antenna and watch the offset. We have not run it, and the number would be worth more than the argument above.", "d": "2025-10-14", "g": ["networking", "hardware", "resilience", "time", "policy"], "t": "Time and position became contested inputs", "u": "insights/time-and-position-contested/"},{"b": "Running a capable language model on your own hardware stopped being exotic. The engineering underneath that shift is more interesting than the fact of it. Quantisation, and what it costs\u00b6 A model\u0027s weights are trained in 16- or 32-bit floating point. Quantisation stores them in fewer bits (8, 5, 4, sometimes fewer) with a scale factor per block to keep the range. Dropping from 16-bit to 4-bit cuts memory by roughly four and degrades quality far less than that ratio implies, because the information is spread redundantly across an enormous number of parameters. It isn\u0027t free. The degradation is uneven. It shows up first in long-context coherence and in tasks needing precise recall rather than in fluency, which means casual testing systematically understates the loss. A quantised model sounds just as good while being measurably worse at the things you\u0027d want to use it for. Rather like a cover band; fine from the back of the room, less convincing up close. Memory bandwidth is the constraint\u00b6 Generating one token means reading essentially every weight. A 7-billion parameter model at 4-bit is roughly 4GB, so each token needs around 4GB of memory traffic. At 50GB/s you can\u0027t exceed about 12 tokens per second however fast the arithmetic unit is. The ceiling for single-stream inference is memory bandwidth divided by model size, and the compute sits idle, waiting. Which is why: Unified memory architectures do unexpectedly well; high bandwidth to a large pool beats a fast GPU whose VRAM can\u0027t hold the model at all. A GPU that has to swap layers across PCIe performs dramatically worse than its specification implies. Batching helps throughput enormously (the weights get read once and many sequences advance) and does nothing for single-user latency. So for local single-user inference the amount of fast memory matters more than the arithmetic throughput, which is the opposite of the intuition most people bring over from gaming hardware. The open question, and the measurement that settles it\u00b6 The number everyone wants is tokens per second on a named part, and it is the one nobody should quote loosely: it moves with the model, the quantisation, the context length, the runtime and the build flags, so a figure without all five is noise. The bandwidth analysis above is arithmetic from published specifications and holds regardless. What it does not tell you is how far real runtimes fall short of that ceiling. The experiment is cheap: fix the part, the model, the quantisation, the context length and the runtime version, publish the command and report the measured rate against the bandwidth-derived bound. That ratio is the interesting figure, almost nobody publishes it and it is on our bench next. No prompt leaves the building. A GP surgery can run clinical summarisation over patient notes on a machine in the back office, and no record, no identifier and no query crosses the practice\u0027s firewall. A solicitor can run discovery across privileged material without the question of whether a third-party API waives privilege, because the question does not arise. The blocker in both cases is not capability; the models are good enough for a draft a professional then checks. It is that the only capable option means posting the most sensitive text the organisation holds to somebody else\u0027s server, under terms that can change, in a jurisdiction whose legal process does not answer to the data subject. Local inference removes that transfer, and with it the risks that ride on the transfer. The data protection story is substantially different. No processor agreement, no transfer mechanism, no third party in the chain, because nothing was transmitted. The duties that attach to holding the data stay exactly where they were: retention, access control and breach notification are the controller\u0027s whether the model ran locally or not, and the machine in the back office is now part of the attack surface. Anybody who has filled in a DPIA for an API-based AI feature knows how much of that document disappears, and which pages remain. For a regulated organisation it is a year of legal review against an afternoon. It works where the network does not, so a clinic with four hours of grid power and no uplink can run the same model a teaching hospital runs. And it does not change underneath you: no deprecation notice, no silent model swap, no rate limit, no price rise, no provider deciding your use case is off-policy. The weights on your disk will behave the same way in five years, which for anything in a clinical process beats the last few percent of benchmark performance. The toil that disappears with all of it is the apparatus of guarding an outbound connection: prompt sanitisation, token budgeting, API key management, reconciling a bill nobody can predict. The version of AI worth building toward is the one that still works when you stop paying somebody else to look at your most sensitive data. The chokepoints, and there are three\u00b6 The weights. You cannot run locally what you cannot obtain. Open-weight models are released at the discretion of a small number of large companies that also sell the hosted alternative. Licences are increasingly \"open\" with usage restrictions that would not pass as open source anywhere else, and the strongest models are not released at all. Open weights trail the frontier by something like a year and the gap is not obviously closing. If it widens into a permanent two-tier arrangement, local inference becomes the cheap option rather than the good one. This one is the hinge. The accelerator supply chain. High-bandwidth memory and leading-edge packaging are made in very few places, and the output is on multi-year allocation to the largest buyers. When the hardware that makes local inference viable is allocated to the companies selling hosted inference, the market is not functioning as a market, and export controls add political allocation on top of the commercial kind. The bandwidth analysis above is the optimistic note: local inference wants large fast memory, not exotic compute, and that is a much broader supply base. The distribution of what you run. If the practical way to get a model becomes a vendor\u0027s hub with its own terms, telemetry and acceptable-use enforcement, then \"running it locally\" means running a file whose availability somebody else controls. Keep your own copies, hash them and treat a model you cannot re-download as a dependency you have already lost. Prompt logs are a record of what an organisation is worried about and what an individual is thinking, retained by default, discoverable by legal process and valuable as training data. Hosted inference is the most intimate telemetry stream ever built; local inference makes that record not exist rather than promising it will be handled well. The open ecosystem is in better shape than the gloom suggests: excellent runtimes, first-rate community quantisation work and a private assistant you can stand up in an afternoon on hardware costing less than a year of API fees. It is losing on one axis only, access to the best weights, and that is a licensing fight rather than a technical one. Which means it can be won.", "d": "2025-09-16", "g": ["ai", "inference", "hardware", "quantisation"], "t": "The model fits; the memory bus decides", "u": "insights/local-inference-on-the-bench/"},{"b": "The same jobs keep coming back. Network boot, standing up a backup, getting remote access into a site, building a fleet of machines against a deadline: years apart, each time from scratch, each time with somebody in the room. That repetition is why there is a control plane. The jobs that kept coming back\u00b6 Network boot. A machine arrives, it needs an operating system, and the way you do that at any scale is PXE. Setting it up means a DHCP option, a TFTP server, an image and somebody in the room when it does not work. Backups, on site. Not the running of them, which automates well. The standing up of them: a box, a disk, a network path, a retention policy and a restore somebody has tried. Remote access. By 2021 we were building our own, out of Raspberry Pis, because the alternative was a vendor appliance per site with a subscription attached. A remote-access device built rather than bought, 2021: the commercial parts were in short supply and did not quite do what was needed. It worked. It was also a device per site, each one a thing to patch, key, inventory and eventually explain to whoever inherited it. And then everybody went home at once. March 2020, Covid. Machines laid out across a room because that was the only way to get them all built in the time available. The constraint was never the imaging; it was that a person had to be standing in front of each one, and there was no time to write the automation that would have taken them out of it. March 2020 is the clearest case because the deadline was absolute. The work was not hard. It was that the throughput of the whole operation was one engineer multiplied by the number of hours in a fortnight, and no amount of competence changes that number. What the repetition is telling you\u00b6 The obvious reading is that we should have automated sooner. That is the wrong lesson, and we had in fact automated: these jobs were Ansible by 2017. The problem is one layer up. A playbook needs to know what it is running against, and that knowledge lived in exactly the places it always lives: a spreadsheet, a wiki page, a naming convention, somebody\u0027s memory. Which meant every one of these jobs started with the same unpaid work: establish what is out there, then act on it. That part had no artefact at the end. Next time, you did it again. So the bottleneck was never the task. It was that the estate could not describe itself. We have made that argument elsewhere about security, where the defender\u0027s advantage is knowing their own ground and almost nobody can claim it. The operational version is the same shape and costs money every week rather than once in an incident. What we built instead\u00b6 A control plane whose job is to hold the declaration: what exists, what it depends on and what state it is supposed to be in. That control plane is micro-substrate. Three properties fall out of that, and they are what we were buying. One place to look. Cloud providers and on-premises hardware behind one abstracted driver model, so a virtual machine, a firewall rule, a network and a backup are all resources of the same kind. Not because a unified API is elegant, but because an engineer jumping between four vendor consoles is doing the estate-discovery work again every time. Zero to ping. One engineer, a laptop, the control plane as a single container image and access to the backups is enough to rebuild an estate from nothing to responding. That is not reasoned from the design: we have run the rebuild, held to the same standard we demand of a backup above, a restore somebody has tried. That is the property the 2015 data-recovery photographs are really about. We got the client\u0027s data back off failed arrays using a salvaged donor server, and the reason it was possible is that the state was on the disks rather than in the hypervisor. The control-plane version makes that the normal case rather than a good day. Everything derives from a base. Hypervisors, virtual machines and container images inherit from common images, so a patch is applied once at the right layer, tested once and cascades. The alternative is the one the mitigation work keeps running into: a fix is available, and finding out which of your hosts needs it takes longer than applying it. What this does not fix\u00b6 It does not make the first description free. Something has to establish what is out there, and on an estate nobody declared, that is still archaeology. It moves the dependency rather than removing it. An estate that reconciles against a control plane depends on the control plane. That is why it ships as a container you can run on a laptop and why the backups are the thing that has to be portable. A control plane you cannot run yourself is a vendor console with better marketing. And it earns nothing on a small estate. Four machines a person understands do not need a declaration; they need a person. The economics turn somewhere around the point where the people who built a thing are no longer the people running it. The part worth keeping\u00b6 Half past three in the morning, February 2016, prototyping the edge platform. The dev boards in this photograph are doing by hand what the edge tier does now, and the only reason we know what that tier should do is that somebody sat up wiring this one, building it from scratch from a concept. None of the capability in the control plane was designed in the abstract. Each piece of it is a job somebody did manually enough times to learn what it required. That is the opposite of how infrastructure products are usually specified. That is the argument for keeping your hands on the work. A team that has never built the thing by hand does not know which parts of it matter, and will automate the easy half.", "d": "2025-08-28", "g": ["infrastructure", "operations", "deployment", "substrate", "ansible"], "t": "Zero to ping, and the decade of doing it by hand", "u": "insights/why-we-built-substrate/"},{"b": "eBPF usually gets introduced as a fast packet filter that grew up. That framing buries the achievement, which is this: The Linux kernel accepts code from userspace and runs it in kernel context on the strength of a proof. Classic BPF validated kernel-side too, and proof-carrying code is older still. What is new is the scope: running arbitrary programs against kernel data structures rather than filtering packets. The verifier is the proof, and if you want to understand it, look at what it refuses rather than what it permits. What it checks\u00b6 Before a program loads, the verifier does the following (LWN\u0027s description is the clearest): Walks the control-flow graph depth-first to establish termination. Unreachable instructions are rejected outright. Loops were forbidden entirely to begin with; bounded loops arrived later, with the verifier proving the bound. Simulates execution instruction by instruction, tracking the type and value range of every register and stack slot. Out-of-bounds jumps and out-of-range access are rejected. Rejects reads of uninitialised registers. A register the program has not written is unreadable: R0 and R2 to R5 at entry, since only R1 arrives holding the context, and R1 to R5 again after a helper call, when R0 comes back carrying the return value. Prunes paths already covered by a previously verified state, which is what keeps the whole thing tractable. Applies secure mode for a loader without CAP_SYS_ADMIN: pointer arithmetic is prohibited so kernel addresses cannot leak. Restricts helper calls by program type. A tracing program and an XDP program can call different things. Why the refusals are the interesting part\u00b6 The verifier is a conservative static analyser, so it rejects programs that are correct but that it cannot prove correct. Working with eBPF is largely the experience of being told no about something you can see is fine. That is the real cost of the thing, and you learn from it. Take a pointer dereference the verifier can\u0027t bound, because the bound comes from a value it can\u0027t track; it gets rejected even though the invariant is sitting there in front of you. So you restructure the code until the invariant is visible to the analyser. It\u0027s the same discipline as satisfying Rust\u0027s borrow checker. The work is making the correctness legible to a machine, not making the code correct. A good exercise: write a program with an unbounded loop, or a map lookup dereferenced without a null check, and then read the verifier log. It prints the state it inferred at each instruction and where that state fell over. Once you learn to read it, it\u0027s one of the better error messages in systems software. The cost model\u00b6 A hardware boundary charges you on every crossing. The verifier charges once, at load time, and after that the crossing is free even though the work is not: no ring transition, no page table switch, no context switch. The program itself still costs what it costs, as JIT-compiled instructions, helper calls and map lookups on a hot path. What disappears is the price of the boundary, not the price of the check. It\u0027s a bit like paying for a season ticket instead of buying a ticket for every journey. That shift is why eBPF scaled out of packet filtering into tracing, security enforcement and load balancing. The economics of \"put a check in this hot path\" changed completely. A verifier willing to say no lets working engineers ask a running system a question it was not built to answer, without restarting it. That sounds modest. It is the end of a whole genre of outage. The classic shape is a production problem that only appears under real traffic, where the instrumentation you need was not compiled in. The choice is to deploy a debug build and wait for the problem to recur, or to guess. With eBPF you attach a probe to the function you care about, on the box that is currently broken, and you have the answer in a minute. The class of bug that disappears is the unreproducible one, and it disappears because reproduction stops being the price of observation. Where it goes next is scheduling and policy. Any kernel decision currently hard-coded into a subsystem, chosen by heuristics that accreted over fifteen years, is a candidate for being handed to a verified program that the operator supplies. Our bet is that this is the dominant direction for kernel development over the next five years: the kernel keeps the mechanism and the enforcement, and the policy moves out to code you can iterate on in minutes rather than across release cycles. From packet filtering to choosing which process runs next, with a verifier standing between user code and catastrophe. The wider benefit is efficiency. Load balancing and packet processing that used to need a dedicated appliance now run on the host that was already there. For anyone running infrastructure on a small budget, in a cupboard, on a connection that comes and goes, the ability to do serious network work on commodity hardware is the difference between a plan and a quotation. That access creates a quieter chokepoint. eBPF\u0027s security model rests on the verifier being in the kernel you can read, and on you being allowed to load a program into your own machine. Both halves are under gentle pressure, from the direction of signing. The argument for signed-only BPF is reasonable on its face. A privileged eBPF program is extremely powerful, it is a lovely place to hide, and a vendor shipping a locked-down appliance would rather only its own programs load. Follow that to its conclusion and you get a kernel that runs verified programs only from keys the platform owner trusts, where the platform owner is the hardware vendor or the cloud rather than you. The verifier still works. It just works on behalf of somebody else, and the general-purpose instrumentation capability quietly becomes a vendor feature. The asymmetry is already visible in managed Kubernetes. Load your own BPF program on your own laptop: trivial. Do it on a managed node pool: that depends on what the provider permits, and increasingly the answer is their agent rather than your program. The capability has not been removed, it has been resold, and what you get back is dashboards rather than access. This matters more than the average lock-in complaint because the kernel is now the last layer in a modern stack that an operator can inspect. The firmware is signed and closed. The management engine is closed. The accelerator firmware is closed. The hypervisor, if you are renting, is not yours. The Linux kernel is the one place left where you can read the code that is making the decision and, with eBPF, ask it what it is doing right now. That is a position worth defending specifically. Here at AxisOps we run on hardware where we can load our own programs, and we treat \"can I attach a probe to this\" as a procurement question rather than a technical one. A system you cannot instrument is a system you are trusting somebody\u0027s description of. What it does not give you\u00b6 No security boundary against a malicious loader holding CAP_SYS_ADMIN. Secure mode is there for unprivileged loading; privileged eBPF is powerful by design. No correctness proof of your logic. It proves memory safety and termination, and says nothing about whether your program does the right thing. No stability across kernels. Verifier behaviour changes, so a program that loads on one kernel can be rejected on another. That\u0027s a real operational cost and it routinely gets understated.", "d": "2025-06-10", "g": ["ebpf", "linux", "kernel", "verification"], "t": "The eBPF verifier, and what it refuses", "u": "insights/ebpf-verifier-says-no/"},{"b": "Earlier this month Redis 8 shipped with AGPLv3 restored as a licensing option, fourteen months after the project left open source for dual source-available licences. It is the second walk-back in nine months: Elasticsearch added AGPLv3 back last August, three and a half years after leaving. With both reversals on the record, the whole arc can now be read end to end, and it reads better as a supply-chain story than as the community drama it was covered as. The wave\u00b6 MongoDB moved to the Server Side Public License in October 2018, naming cloud providers who sold the database as a service without carrying development cost. Elastic followed in January 2021, and AWS responded within weeks by forking Elasticsearch as OpenSearch. HashiCorp adopted the Business Source License in August 2023 for Terraform, Vault, Consul and the rest of its line; the community response became OpenTofu, which reached general availability under the Linux Foundation in January 2024. Redis changed terms in March 2024 and the Linux Foundation announced the Valkey fork eight days later. Each vendor\u0027s stated reason was the same economic complaint, and it is not a frivolous one: the hyperscalers captured the operating margin on software whose development they did not fund. The relicensing wave was the vendors\u0027 answer to the funding problem, the one the xz incident showed at the individual-maintainer scale. The forks were the answer from the businesses built on the software: hyperscalers for OpenSearch and Valkey, the toolmakers around Terraform for OpenTofu. Nobody in the chain was behaving irrationally, which is what makes it a structure rather than a scandal. What made the forks viable\u00b6 The forks that stuck share three properties, and they are checkable in advance. The code at the fork point was under a permissive licence, so there was something lawful to fork. The parties doing the forking had operational revenue riding on the software, so the maintenance burden landed on organisations with an incentive to carry it indefinitely. And in each case a foundation took the trademark-adjacent naming problem away by hosting the fork under a new name, which is why the question \"who owns the name\" sits in the same governance territory as standards. The property that decides whether relicensing is possible at all is copyright concentration. A contributor licence agreement or copyright assignment that grants relicensing rights lets one party change the terms unilaterally. The agreements differ, so the answer lives in the text of the one your dependency uses. A project with distributed copyright, the Linux kernel being the canonical case, cannot be relicensed unilaterally by anyone: it takes consent or replacement, contributor by contributor, which the kernel\u0027s own documentation treats as practically unachievable. That single fact about a dependency is worth more than its licence text, because it bounds what the licence can become. The same move without any source code\u00b6 Broadcom closed its acquisition of VMware in November 2023 and within weeks had ended the sale of perpetual licences. VMware\u0027s own notice of the end of availability followed in January 2024: no new perpetual entitlements, no renewal of support on existing ones, subscription as the only path forward. The Broadcom knowledge-base article lists the affected products, which is to say most of the virtualisation estate of most enterprises. A perpetual licence survived the change in a narrow and instructive sense: the right to run the versions you already hold. Patches, updates and support run out with the last contract. For an operator that is the same event as a relicence. The terms of trade on an installed dependency changed and the installed base had no vote. The only engineering question that matters is the one that should have been asked at adoption time: what is our position on the day the terms change? The open-source cases and the proprietary case usually get discussed in different rooms, one as a licensing controversy and one as a procurement problem. They are the same risk. The BSL conversion clause, the SSPL service condition and the subscription cliff are all answers a supplier gave to the question of how to monetise an installed base, and the installed base found out when the announcement shipped. What to check before depending\u00b6 The list we now apply, written down so it survives the next wave: The licence of record, per version. A relicence binds future releases, never the copies you hold. BSL additionally converts each release to an open licence after its change date, which for BSL 1.1 is at most four years. Pin the version, record its licence, and the worst case is a frozen dependency rather than a surprise obligation. Who can change the terms. A CLA or copyright assignment that grants relicensing rights means one party can; distributed copyright means nobody can unilaterally. The agreements differ, so read the one in the repository, not the project\u0027s reputation. Whether a fork constituency exists. Large operators with revenue on the line, a foundation willing to host, a permissive licence at the fork point. Where those exist, \"we would move to the fork\" is a plan. Where they do not, it is a slogan. For proprietary dependencies, what perpetual means. Usually it means the binary keeps running and everything around it expires. Price the exit on the day you sign, because the vendor has already priced the lock-in. The walk-backs do not change the list. Elastic and Redis returning to copyleft licences shows the forks imposed a real cost. A dependency whose terms changed twice in two years has demonstrated the property that matters: its terms can change again.", "d": "2025-05-20", "g": ["policy", "open-source", "supply-chain", "infrastructure"], "t": "A licence is a term of trade", "u": "insights/a-licence-is-a-term-of-trade/"},{"b": "Read a long-lived standard closely and you\u0027ll find things no engineer would ever design: options nobody implements, two ways to express the same thing, extension points that go nowhere. Those aren\u0027t mistakes. They\u0027re the shape of an agreement, in the same way a planning application is the shape of an argument with the neighbours. What a standards body is\u00b6 A room containing competitors. Each one has shipping implementations, existing customers and commercial interests, and between them they have to produce one document. A feature in the final text usually means one of three things: Everybody agreed it was necessary. One participant had already shipped it and wouldn\u0027t take it out. Two participants disagreed, and the compromise was to make it optional. The third case is where interoperability goes to die. An optional feature is one half the implementations have, so every implementation has to handle both cases and the test matrix doubles. Stack up enough optionality and \"conforming\" stops implying \"interoperable\" at all. Why you should care as an engineer\u00b6 It changes how you read the document and what you then build. Expect the ambiguity to be on purpose. If a specification is vague about behaviour in some case, the most likely explanation is that the participants couldn\u0027t agree and vagueness was the settlement. Don\u0027t assume you\u0027ve misread it; you probably haven\u0027t. Implement the common subset, not the whole document. Conformance to the full standard is often neither achievable nor useful. What matters is the subset used by the implementations you have to talk to. Test against implementations, not the spec. The spec is what was agreed in the room. The implementations are what exists in the world, and where the two differ, the implementations win every time. The counter-example worth holding onto\u00b6 WireGuard shows the other path: one cipher suite, no negotiation, no options. The result is small, analysable and free of downgrade attacks. That was possible because one party designed it, outside a standards process, with the authority to say no. The trade is that it can\u0027t accommodate anybody who needs something different. Compare TLS, which has to serve everybody, and whose history of downgrade attacks is substantially a history of negotiation complexity. Neither model wins outright. A protocol that has to bridge competing interests across a whole industry can\u0027t be designed by one party. A protocol that can afford to be narrow should be. Reading a standard as politics is a practical skill, and it lets a small organisation punch above its weight in a room full of incumbents. You can participate. Standards bodies are startlingly open compared with their reputation. Mailing lists are public and most working groups will take a well-argued implementation report. An engineer who turns up with \"we built this and here is where your text is ambiguous\" carries weight no amount of market share confers. A two-person company can get a sentence changed in a document that will govern a billion devices. Interoperability is what makes competition possible at all. Every market where you can swap a supplier without rewriting your systems exists because somebody fought this fight: changing electricity provider, buying a phone that works on any network, sending mail between two companies\u0027 servers. The next decade\u0027s version is fought over the interfaces into cloud services, and winning it means a business can move its workload without rebuilding it. That is decided in committee rooms, not markets. What becomes cheap is entering. A published, testable baseline means a new implementation can be built by a few people and trusted on arrival, which is how you get competitors to the three or four firms that currently own a layer. Where that baseline is missing, the incumbent\u0027s behaviour is the standard, and the cost of entry is reverse-engineering it. For engineering itself, the toil a real baseline removes is compatibility shimming: the per-vendor quirks table, the user-agent sniffing, the \"works with provider X only\" integration code. Some of the most dispiriting work in the industry, and it exists because a negotiation failed. A standards body with a membership fee, weighted voting and a work programme set by whoever can afford to send people full-time offers no neutrality whatever. It is a venue where the three largest implementers reach an accommodation and publish it, and the other participants get a vote on the wording. The tell is a specification that is enormous, that nobody has implemented completely and whose only two complete-ish implementations belong to firms that sat on the committee. The standard has stopped lowering the barrier to entry and started defining it, because the cost of conformance is the moat. The sharper version is a standard that mandates a service. Any specification whose implementation requires calling a particular registry, obtaining a certificate from a recognised authority or attesting to a central validation endpoint has placed a company in the middle of a protocol, permanently. That is how a technical document becomes a toll booth, and it is usually introduced for good reasons: trust, revocation, abuse prevention. So the question to ask of a new standard is not about its design quality. Ask whether a competent engineer could implement it fully, from the document, without anyone\u0027s permission. If not, name who grants permission; that is the chokepoint, and there is always one. Privacy gets decided in these rooms too, mostly by default. Identifier formats, telemetry fields, what a conforming implementation must report and to whom: standards decisions with enormous downstream consequence, settled by people optimising for operational convenience, with nobody in the room representing the person the identifier belongs to. Here at AxisOps we build on standards we can implement from the document, and we treat a required call to somebody\u0027s endpoint as a design smell rather than a feature. Where the open implementation exists and is keeping pace, use it and say so. Where it is losing, go and read the mailing list, because the decision that matters was taken there by the people who turned up. RISC-V is testing the alternative\u00b6 RISC-V\u0027s modularity is this same tension in current dress. An open ISA assembled from optional extensions has enormous flexibility and a real fragmentation risk, and platform profiles exist to impose the \"no\" that a single vendor would simply have issued over lunch. Whether an open standards body can enforce a baseline as effectively as ARM\u0027s commercial authority is a more interesting question about RISC-V than any individual extension. My bet is that it can, but only if the large implementers find fragmentation more expensive than differentiation, and that will be decided by a handful of profile votes rather than by anything technical. Marked as a view.", "d": "2025-05-13", "g": ["standards", "policy", "interoperability"], "t": "A standard is a negotiated settlement", "u": "insights/standards-are-politics/"},{"b": "In October 2022 the United States restricted exports of advanced computing hardware to China using thresholds expressed in processing performance and interconnect bandwidth. In October 2023 those rules were revised: interconnect bandwidth was dropped as a criterion and a performance density threshold added, explicitly because vendors had designed parts that sat just underneath the original line. Read that sequence as an engineer rather than as a lawyer. A regulator published a performance specification. Industry designed to the spec, as industry always does. The regulator observed the result and amended the spec. That is a standards process, conducted between a government and a market, with the specification being a list of things you are not allowed to build. Why this is an architecture problem now\u00b6 Because a number in a control list propagates into every layer above it. The part you can buy determines the memory bandwidth you have, which determines the model size that is practical, which determines whether a workload runs locally or must go to a provider in another jurisdiction, which determines where your data lives and which law applies to it. A threshold in a regulation reaches all the way up to a data protection question, and it does so without anyone in the middle making a decision. It also fragments the hardware landscape in a way this trade has not had to plan for. Designing for one global compute market was the assumption underneath a decade of infrastructure work. The reasonable planning assumption now is at least two, diverging, with different parts available, different performance ceilings and different software ecosystems growing on each. The practical consequences are mundane and annoying. Capacity planning becomes jurisdictional. A multi-region deployment may not have the same silicon in every region. \"Buy the next generation\" stops being a reliable answer to a performance problem, because the next generation may not be sold to you. What it costs\u00b6 Optionality, which was the quiet asset nobody was pricing. Infrastructure built on the assumption that you can buy a faster part next year has a dependency on a political relationship, and that dependency is now explicit. The cost shows up as inventory, second sources and designs that deliberately leave performance on the table in exchange for using parts that are available everywhere. That is real money spent on resilience against a policy change rather than a technical one. There is also a straightforward efficiency loss. Parts designed to sit under a threshold are worse than the parts that would otherwise have been built. Engineering effort goes into gaming a line instead of into making things faster. Nobody involved thinks this is a good use of anybody\u0027s time. What it makes possible\u00b6 Not everything downstream of this is bad, and the second-order effects are more interesting than the restrictions. The clearest is that constraint has restarted architectural competition. When you cannot buy the fastest part, efficiency per unit becomes the thing worth optimising, and the work that follows is the sort this notebook has always found more interesting than brute scaling. Smaller models, better quantisation, sparsity and getting more out of memory bandwidth are all areas where the constraint is doing something that abundance was not. The second is that it has made open instruction sets a strategic question rather than an enthusiast one. An ISA that nobody can revoke is worth a great deal more in 2025 than it was in 2020, and the investment flowing into RISC-V is substantially a response to exactly this. A control list is a powerful argument for an architecture whose specification cannot be withdrawn, even though implementations, tooling and the chips themselves remain controllable. And the licensing landscape for conventional ISAs looks different once licensing is understood as a point of leverage. The third, and the one we would bet on, is a durable shift towards local inference. If access to frontier compute is politically contingent, then running what you can on hardware you own stops being a preference and becomes a continuity requirement. That suits smaller operators better than it suits large ones, and it pushes the whole field towards models that fit on affordable hardware. We think that is a good direction for reasons that have nothing to do with geopolitics. Who ends up owning it\u00b6 The uncomfortable structure is that very few actors have leverage here, and none of them is you. Fabrication at the leading edge is effectively one company in one country, with the lithography equipment coming from effectively one company in another. That is the narrowest chokepoint in modern industry and it exists for technical reasons rather than commercial ones. Everything in the compute supply chain passes through it. Export control turns that technical chokepoint into a policy instrument. The list is written by one government, revised unilaterally and has extraterritorial reach through the equipment and the design tools. A European or British company\u0027s architecture is being constrained by a process it has no vote in, and the current list is not the last one. For a small operator the position is worse than it looks, because they will not be the ones who get allocations when a part is scarce. Restriction plus scarcity concentrates frontier capability in whoever is largest. So export control intended to preserve a strategic advantage also has the domestic effect of advantaging incumbents. That is not what it was for, and it is what it does. Here at AxisOps our position follows from everything else in this notebook: build so the substrate can be swapped. Keep the build working on more than one architecture, prefer the model that runs on hardware you can actually buy and treat a dependency on a single part number as the risk it now provably is. We have argued for that on technical grounds for years. It turns out the strongest case for it is political. What would change our mind\u00b6 If the thresholds stabilise and the market re-converges, most of this becomes an expensive few years rather than a structural change, and the second-source work looks like wasted effort. The 2023 revision suggests otherwise, since the direction of travel there was tightening rather than settling, but a specification written by a government can be rewritten by the next one.", "d": "2025-04-08", "g": ["hardware", "policy", "sovereignty", "architecture", "standards"], "t": "A control list is a system requirement", "u": "insights/control-list-is-a-spec/"},{"b": "Next.js used x-middleware-subrequest to tell itself that a request was its own internal subrequest, so middleware should not run again. An external client could send the same header. A request carrying it skipped middleware, and any authorisation check living there was skipped with it (GitHub Advisory, published 21 March). Patched releases are 12.3.5, 13.5.9, 14.2.25 and 15.2.3. There is no fixed 11.x, so an application on 11.x has to move major version or live on the workaround. The bug is a trust boundary in the wrong place\u00b6 An internal marker is only internal while nothing outside can produce it. The moment a header travels over the same channel as user input, it is user input, whatever the framework calls it. That is a very old class of mistake wearing modern clothes. It keeps happening because the component drawing the boundary and the component enforcing it are the same piece of software. Middleware decides whether to run, based on a header the request carries. Nothing above it is adjudicating. Which is also why the mitigation works at all. Strip it at the proxy\u00b6 The advisory\u0027s guidance for anyone who cannot upgrade is to stop external requests carrying that header from reaching the application. At an nginx reverse proxy that is one directive: proxy_set_header x-middleware-subrequest \"\"; The proxy is above the application, so it can assert something the application cannot assert about itself. The header is now unforgeable from outside because the only party who can set it is the one component that was ever entitled to. Putting the check one layer up is the real repair; upgrading next is how you stop needing it. Our role is at github.com/axisops/CVE-2025-29927, MIT. It finds Next.js applications, classifies them against the advisory\u0027s ranges and with --apply writes that snippet into every server block of the nginx files you list. Two things about nginx that cost an afternoon\u00b6 proxy_set_header does not inherit into a block that defines its own. If a location sets any proxy_set_header of its own, it inherits none from the server level, so a snippet installed only at server silently does not apply there. That is standard nginx inheritance, and it is just the shape of failure that produces a mitigation everyone believes is deployed. The role adds the snippet to those location blocks too. Test before reload, and roll back on failure. The role runs nginx -t after writing and reverts every change if the test fails; the Ansible handler reloads only once the test has passed. A config error here does more than degrade one application: it takes down every site on the proxy, which is a worse outcome than the vulnerability being mitigated. Version detection reads node_modules/next/package.json first, then falls back to package-lock.json, yarn.lock or pnpm-lock.yaml. Canaries, anything below 11.1.4 and anything unparseable are reported unknown rather than guessed, for the same reason as every other tool in this family: a confident wrong answer is worse than an honest absent one. Where it does not apply\u00b6 Vercel\u0027s postmortem, published this morning, sets out the exposure by deployment model (Vercel). Self-hosted applications using next start or output: \u0027standalone\u0027 were affected. Deployments on Vercel, Netlify and Cloudflare Workers were not, because routing runs in a separate system from the application there. Static exports were never affected, since middleware needs a server runtime. So the population this role is for is the self-hosted estate, which is the population least likely to have been told. A managed platform fixed this for its customers without their involvement; everyone running their own next start behind their own proxy had to find out and act. That asymmetry matters, because it is the recurring cost of self-hosting and it is paid in attention rather than money. What the scan cannot tell you\u00b6 The advisory is about authorisation checks made in middleware. Our scan classifies by version, and does not examine whether an application uses middleware or what it does there. A vulnerable version whose middleware does nothing security-relevant is reported the same as one gating an admin route, and only a person reading the code can tell those apart. The nginx parser is minimal on purpose and that cuts both ways. It does not follow include directives, so a location block in an included file that sets its own proxy_set_header will not get the snippet, and will not be mentioned. It expects each block\u0027s { on the same line as server or location, and reports anything else for handling by hand. List only http-context files: a server block in a stream context does not accept proxy_set_header, nginx -t fails and the role rolls back. The snippet is also a stopgap that outlives its usefulness quietly. Upgrade next, then take it out. A proxy stripping a header that no longer means anything is harmless and misleading, and the next person to read that config will spend twenty minutes working out which incident it came from.", "d": "2025-03-25", "g": ["security", "ansible", "python", "web", "mitigation"], "t": "A header you trusted because it was yours", "u": "insights/middleware-was-the-boundary/"},{"b": "\"The attacker has to be right once, the defender every time.\" True, and usually deployed either as fatalism or as a reason to buy something. Read as an engineering constraint, it points somewhere a good deal more useful. The asymmetry, stated properly\u00b6 The clich\u00e9 undersells it. The accurate form: The attacker chooses the time, the target and the method. The attacker needs one path through; the defender has to cover all of them. The defender pays continuously; the attacker pays only when attacking. The defender\u0027s cost scales with the size of the estate; the attacker\u0027s doesn\u0027t. That last one matters most and gets the least attention, because what it says is that the problem gets harder as you grow, however well you\u0027re doing it. Which suggests the actual move\u00b6 If the cost of defending scales with surface, your leverage lies in having less to defend, not in defending better. Which reframes a fair amount of what gets sold as security work: One place to change a version. An estate where PHP is declared once has one thing to patch. An estate where it\u0027s installed forty times has forty. Small protocols. 4,000 lines is less to get wrong than 100,000. Capability systems. Authority that was never granted can\u0027t be abused, which is structurally different from authority that\u0027s granted and then policed. Deleting things. The most underrated security control going. A service that no longer exists has no vulnerabilities, needs no patching and will never appear in an incident report. None of those is a security product. They\u0027re all architecture. Where the asymmetry runs the other way\u00b6 The defender has advantages that are real and badly under-used: Home advantage. You know your own estate, or you could. The attacker is discovering it as they go. Detection asymmetry. The attacker has to act, and acting makes signal. The defender need only be watching the right thing. The attacker must be right about everything too, once inside. Their tooling has to work, against your specific versions, without crashing anything conspicuous. Attacks fail far more often than the vendor slides suggest. The reason that advantage stays theoretical for most organisations is the first one on the list: almost nobody can accurately describe their own estate. The defender\u0027s single biggest structural advantage is given away by not knowing what\u0027s being defended. It\u0027s defending a building when you\u0027ve no idea how many doors it has. Counting the doors. Drawn as the IP justification for a RIPE address application, after the ISP had rejected it several times: every embedded device that needed a public address, spelt out. The answer is the defender\u0027s advantage; not having it drawn anywhere is how the advantage gets given away. An estate that can describe itself changes the defender\u0027s side of the equation. An estate defined as data rather than discovered by scanning changes what a small team can do. Ask \"where does this library run\" and get an answer in seconds from the declaration rather than in days from an agent sweep that misses the box nobody remembered. Patch once and have it land everywhere, because everywhere is one list. That is the difference between an estate that patches a critical CVE the same morning and one that takes six weeks over it, and the gap is not talent. What that unlocks is a defensible small organisation. Today a serious security posture is priced like a department: a SOC, a detection vendor, an asset inventory project that is stale before it finishes. Most of that spend is buying knowledge about your own estate that you should never have lost. A four-person company that declares its infrastructure can answer a regulator\u0027s questions directly and patch faster than a bank, and we have watched it happen. The toil that disappears is the worst toil in the trade: the incident where most of the week goes on working out what you have. There is a wider benefit too. The single biggest cause of compromise in organisations that cannot afford specialists is an unpatched thing nobody knew was running. A GP surgery, a school, a council. None of them will ever buy detection engineering. All of them could run infrastructure that is declared in one place and updated in one action, and that would move the national picture more than any amount of threat intelligence. Then the pricing model restores the asymmetry: the defender pays per asset, and the attacker picks one. Look at how security tooling is sold. Per endpoint, per host, per seat, per gigabyte ingested. Your bill rises with the thing that was already your problem. The attacker\u0027s cost does not move at all; they needed one way in before and they need one way in now. The industry has built its commercial model directly on top of the asymmetry. The economics of the market therefore reward estate growth rather than estate reduction, and nobody is selling you less. Watch where that ends up. The large platforms now offer defence bundled with the infrastructure, which does work and which prices on consumption of their own services. Read the pitch carefully and it says you can be secure as long as everything you own is inside one company\u0027s estate and you keep paying. Log retention is the cleanest example: your own security history, the evidence you need to investigate your own incident, held in a store you rent by the gigabyte and lose when you stop. An organisation that cannot afford the ingest price gets a shorter memory, and attackers have read that pricing page too. The open side is in decent shape and under-resourced. You can run your own log pipeline, your own detection rules, your own asset declaration, on hardware you own, for the cost of the hardware. The capability is there. What is missing is the assumption that anyone will: the industry\u0027s default is now that telemetry leaves the building. Here at AxisOps we build the version you own. Declare the estate, hold your own logs and keep the ability to answer questions about your own infrastructure without a subscription. Security knowledge about your estate is the one asset you should never rent, because the day you most need it is the day the invoice is in dispute. The uncomfortable implication\u00b6 Most security spending goes on detecting and blocking attacks. Comparatively little goes on knowing what you have, reducing it and being able to change it quickly. The second category is harder to sell. There\u0027s no product called \"we deleted four hundred things and can now patch in one place\", no stand at the conference and no quadrant to be in. On the evidence of how incidents go, it\u0027s where the leverage is, and we would put money on that being the consensus within five years.", "d": "2025-02-18", "g": ["security", "cyber-defence", "policy", "operations"], "t": "The defender has to be right about everything", "u": "insights/cyber-defence-asymmetry/"},{"b": "For most of computing\u0027s history energy was someone else\u0027s problem. You wrote software to be fast; the electricity bill turned up in a different department. That stopped for an unglamorous reason: you cannot cool what you cannot cool. Rack density is limited by thermal capacity, so performance per watt decides how much compute fits in a building. The chain of consequences\u00b6 Once perf-per-watt is the binding constraint rather than raw throughput, much of the architecture follows: More cores at lower clocks. Dynamic power goes as capacitance times voltage squared times frequency, and holding a higher clock stable needs a higher voltage. So power rises with roughly the cube of frequency while throughput rises linearly. Two cores at half the clock therefore do the same parallel work for a fraction of the energy. That is most of why core counts went up instead of clocks. Efficiency cores. A core optimised for work-per-joule rather than latency. ARM\u0027s opening. ARM in the server room happened largely because the perf-per-watt case was strong long before the raw performance case was. Accelerators. Fixed-function silicon for a common operation is far more energy-efficient than general-purpose cores grinding through the same work. The open measurement\u00b6 The tooling to measure software energy consumption already exists: RAPL on Intel, equivalent counters elsewhere, PDU-level measurement at the rack. What does not exist is the practice. Hardly anybody puts a power figure in CI, so the numbers that circulate are modelled rather than measured, and \"this is more efficient\" is nearly always an argument from architecture rather than a reading. That includes most of what follows. The measurement that would settle it is cheap: a fixed workload, a named part, RAPL and PDU readings taken together, run per commit the way a latency benchmark is. We have not instrumented our own estate for this and it is next on the list, because whoever does it first gets to make claims nobody else can check and will find savings everybody else is paying for blind. Where the easy wins are\u00b6 Drawn from general principles rather than our own readings, and marked as such: Doing less work. The biggest energy saving available to most software is algorithmic. An unnecessary poll loop costs power continuously, like a tap left running. Letting the CPU idle properly. Timer coalescing and avoiding short-interval wakeups let a core reach deeper C-states. A process that wakes every 10ms to check nothing prevents that. Right-sizing. Over-provisioned infrastructure idles, and idle isn\u0027t free. None of those involve choosing a faster language, which is telling: the conversations that generate the most heat about efficiency are rarely the ones with the most energy in them. What treating energy as first-class makes possible\u00b6 Put a watt figure next to the latency figure in CI and the economics of software change shape. Efficiency becomes a feature you can sell. A service whose power draw per request is measured and falling is cheaper to run at every scale, and where capacity is thermally capped that is capacity you did not have to buy. The team that halves its watts per transaction has doubled what fits in the building: no procurement cycle, no planning application, no grid connection queue. Compute goes where the power is, not where the people are. Characterise workloads by energy as well as latency and it becomes practical to put the ones that tolerate delay next to intermittent, therefore cheap, generation. Batch work that follows the wind is a real architecture, and it turns a grid problem (what to do with surplus) into a compute opportunity. Small places get to host real compute. A rack that needs eight kilowatts can go in a town with a modest supply; one that needs forty cannot. Efficient silicon is what lets a hospital, a council or a manufacturer keep its data and processing on its own premises instead of shipping both to a hyperscale region three countries away. Perf-per-watt is a sovereignty and resilience story as much as a cost one. And the toil that disappears is capacity firefighting. The recurring emergency where a rack trips its thermal budget, or a deployment doubles a fleet\u0027s draw unnoticed for a quarter, becomes a failing build instead. Who the savings accrue to\u00b6 Efficiency gains do not go to the planet by default. They go to whoever owns the capacity, and historically get spent on more compute rather than less power: the building was thermally capped, you made the work cheaper, so you put more work in it. That is why a decade of remarkable perf-per-watt improvement has coincided with datacentre electricity consumption rising. Expect every efficiency claim from a large operator to be about intensity per unit of work, carefully not about total draw. The grid connection is the chokepoint here and is barely discussed as one. In a constrained network, the right to draw tens of megawatts at a given substation is a scarce, queued, politically-allocated asset. It is secured years ahead by a handful of companies with the balance sheet to pay for options on power they do not yet need. That is a harder barrier to entry than any technology: a competitor can buy the same silicon, but not a connection allocated to somebody else until 2032. So the cheapest compute in a country is available only as rent, from whoever queued earliest, while local operators and community projects wait. Watch also for efficiency being used as the argument for why compute must consolidate: the hyperscale PUE figures are real and better than yours, and they are also the best available case for making sure you never run anything yourself. The open side does well on the software (the kernel\u0027s power management work is public, RAPL is readable, the tooling is free) and poorly on transparency. Per-workload energy figures from the large providers are not published at a granularity that would let you compare them with your own estate, which makes the comparison unwinnable. Here at AxisOps we would rather run efficient hardware we own in a building we can point at, and publish the watts, than accept an intensity ratio from a company that picks its own numerator. The position\u00b6 Energy is a real engineering constraint and badly measured, which makes it fertile ground for confident claims with nothing behind them, sustainability marketing included. The discipline is the one we apply to every other number we quote: if you can\u0027t state what you measured, on what hardware, with what tool, you\u0027re asserting. The corollary is the opportunity. Measure it, and you will know something about your own software that almost nobody knows about theirs.", "d": "2025-02-11", "g": ["energy", "hardware", "performance", "datacentre"], "t": "Energy as a design axis, not an afterthought", "u": "insights/energy-first-class/"},{"b": "Attestation is a signed statement that something is in a known state. You\u0027ll find it in confidential computing, in measured boot, in build provenance and in device identity. It\u0027s useful, and it\u0027s routinely described in a way that hides what it does. The chain always terminates\u00b6 An attestation is a signature. Verifying it means checking it against a key. Trusting that key means either you hold it yourself, or it\u0027s signed by another key, which recurses. Every chain ends at a root you trust for reasons outside the system: A silicon vendor\u0027s provisioning key, trusted because you trust the vendor\u0027s key management and their supply chain. A CA root, trusted because it shipped in your OS and you never looked. Your own root, trusted because you generated it and you believe your own key management. There\u0027s no attestation without a root, and no root that justifies itself. The technology doesn\u0027t create trust; it propagates it from wherever you decided to put it. Systems designed without grasping that tend to get trusted a good deal more than they\u0027ve earned. The useful question\u00b6 Not \"is this attested\" but \"what does this attestation claim, who signed it and what would make that signature meaningless?\" Measured boot attests to what was loaded. Not that what was loaded is safe. A measurement matching a known-vulnerable version is a perfectly successful attestation of a bad state. Build provenance (SLSA and similar) attests that an artefact came from a particular source and a particular builder. Not that the source is any good; see xz, where the provenance would have been immaculate. Device identity attests that a key lives in particular hardware. Not that the device is uncompromised at the software level. The pattern holds: attestation establishes identity and integrity of a measurement, never virtue. Almost every overstatement you\u0027ll read comes from muddling those two. Revocation, which is where it gets hard\u00b6 If a root key is compromised, every attestation under it becomes worthless. Revocation at scale is one of the properly unsolved operational problems in this area. CRLs and OCSP both have well-documented difficulties, and in many implementations a hardware root cannot be rotated at all. Which makes a hardware root of trust a single point of catastrophic failure with no recovery path: a building held up by a single wall, with no plan for the day someone drives into it. That trade belongs in the design discussion, not the post-incident review. What attestation makes possible\u00b6 A fleet of machines in places you cannot visit: a substation, a wind turbine, a clinic in a town with one road into it. Each boots, measures itself and proves to your control plane what it is running before it is handed a key or allowed to join the network. A tampered device fails to join, and it fails remotely. What becomes cheap is trustworthy infrastructure in places with no staff. That reaches a lot of people who currently get worse service because their infrastructure is unsupervisable. Build provenance is the one that changes a working engineer\u0027s day. \"Where did this binary come from\" has been unanswerable in practice for the whole history of the industry, which is why supply chain attacks work. An attested build chain turns it into a lookup: this artefact, from this commit, by this builder, with these inputs. The class of incident that gets smaller is the one where you spend four days establishing whether you are affected rather than fixing it. And for people on bad connections or none: attestation lets you trust a device without trusting the network it arrived over. If the machine can prove its own state, you do not need a protected path to reach it. Who signs the keys\u00b6 A key you do not hold is not your root of trust. Follow the chains on a modern machine and see where they terminate: the silicon vendor\u0027s provisioning key, fused at manufacture and held under their revocation policy and jurisdiction; the OS vendor\u0027s signing key, which decides what may boot; the cloud\u0027s attestation service, the only thing that will vouch for the instance you rented. Three or four companies sign the statements the industry\u0027s security posture rests on. These are revocation powers as well as signing powers. A vendor who can decide a firmware level is no longer acceptable can decide your fleet is no longer attestable, on their schedule, for commercial reasons as easily as security ones. We have seen the pattern in device ecosystems where an attestation check becomes the enforcement point for which software may run: framed as integrity, functioning as a licence, identical mechanism either way. Then privacy. Device identity rooted in silicon is an unforgeable, unchangeable identifier for a piece of hardware and, by extension, its owner. That is because the design is built so it cannot be spoofed, rotated or lied about. That is what you want for a substation and what you do not want for a personal device. The property that makes a fleet manageable makes a population trackable. Open hardware roots of trust exist, are inspectable and can be provisioned with keys you generate. What they cannot do is vouch for the memory encryption on a CPU you did not make. So the open position today is split: hold your own root for everything you can, and where the vendor\u0027s root is unavoidable, record it as a named dependency rather than pretending it is a property of the maths. Here at AxisOps that is the rule: our own signing keys for everything we attest about our own infrastructure, plus an explicit list of the vendor roots we cannot avoid. Proving your own machine trustworthy should not be a service you rent from the company that made the chip. Run your own CA, hold your own keys and verify attestations yourself. Where it earns its place\u00b6 Despite all of the above, the alternative is worse: without attestation you\u0027re trusting the same parties with no way to check anything at all. The discipline is to write down, for any system that uses it: What the root is. Who controls it. What the attestation claims. What happens if the root is compromised. A design that can\u0027t answer those four is using attestation as a feeling rather than as a control. It\u0027s the same discipline as reproducible builds: replace a trust relationship with a checkable property, and be clear about which relationship you\u0027re still carrying.", "d": "2025-01-28", "g": ["security", "attestation", "supply-chain", "hardware"], "t": "Attestation only moves the trust", "u": "insights/attestation-chains/"},{"b": "A normal process on a normal operating system has ambient authority. It runs as a user, and it can reach anything that user can reach. open() takes a path, and whether the call succeeds comes down to checking the process\u0027s identity against the filesystem\u0027s permissions. That\u0027s the model almost all software is written against, and it\u0027s why a compromised process is so dangerous: it inherits everything its user could ever do. Hand a contractor your front door key and they have the run of the house, including the rooms you never mentioned. WASI takes the other approach. Handles, not paths\u00b6 In a capability model, a module can\u0027t name a resource it hasn\u0027t been given. There\u0027s no global namespace to reach into. To open a file in a directory, the module has to be holding a handle to that directory already, and from a handle it holds it can only derive handles with equal or lesser authority. The practical consequence: a WASM module handed a handle to /var/data cannot construct a path to /etc/passwd. Not denied; unable to express the request in the first place. There\u0027s no ambient root to traverse from. The contractor gets a key cut for one room and no idea the corridor exists. Why this beats a sandbox\u00b6 Sandboxes usually work by interposing on a syscall boundary and deciding what to allow: seccomp filters, LSM hooks, AppArmor profiles. They\u0027re policy applied to a program that\u0027s still asking for the whole world. A capability system changes what the program can ask for, and the difference shows up in the failure modes. A policy fails at its gaps: a denylist fails open outright, and even a strict allowlist still permits a vocabulary of syscalls whose combinations nobody enumerated. Capability systems fail closed, because authority nobody mentioned was never granted. It also makes the authority auditable. What a module can do is the list of handles it was given, and a list is a thing you can sit and read. Where the ecosystem is\u00b6 Dates need care here. Researching this produced three mutually contradictory dates for the same WASI release across different write-ups. One source corrected another about whether a February date referred to previews landing in Wasmtime or to the ratified specification. The secondary coverage of WASM is unusually unreliable, apparently because it\u0027s a popular SEO topic. So rather than repeat a date we can\u0027t stand behind: the component model and WASI\u0027s versioning are both moving, the async work is the significant recent addition, and anyone who needs a current answer should go to the specification repositories and the Bytecode Alliance\u0027s own records directly. What this makes possible\u00b6 The capability model plus a portable binary format gets you something that has not existed before: running somebody else\u0027s code without reviewing it for what it can reach. What it does with the authority you hand it still wants a look; the list of handles is the review\u0027s scope, not its replacement. Consider a plugin. Today, extending a product with third-party code means either trusting the author, reviewing every release or running it in a container with an operating-system userland and runtime footprint attached. With WASI the plugin gets a handle to the one directory it needs and nothing else, can start much faster, and the authority it holds is a list you can print on a receipt. That makes opening a plugin ecosystem to strangers a bounded risk rather than an act of faith, bounded by the soundness of the host runtime. That is the difference between a product with twelve integrations written by the vendor and one with four hundred written by its users. The same shape applies to multi-tenancy. If isolation costs microseconds rather than milliseconds, you can afford one instance per request rather than one per tenant. A whole category of bug stops being possible: the cross-tenant data leak caused by state left in a long-running process, which is the bug that ends companies. That class goes because no process outlives the request; bugs in the host runtime and side channels are not in the bargain. For engineering more broadly, the toil that goes away is the sandbox audit. Right now, working out what a seccomp profile permits is an afternoon with a strace log and a hopeful expression. Working out what a WASI module permits is reading the list of handles the host passed in. We would rather review a list than enumerate badness, and so would you. And it travels. A module gets the same declared authority in a server, an edge node and a browser tab. That matters most for the places with the worst infrastructure: compute that can run on a cheap box in a clinic with intermittent power, and run as the same module unchanged in a datacentre, holding the same declared authority in both. The code enforcing that authority is each runtime\u0027s own, and that is where the scrutiny moves. Where it falls down\u00b6 Capability discipline has to go all the way down. A host that hands a module a handle to / has a capability system in form only, and has given the contractor the master key after all. Performance isn\u0027t free. JIT or AOT compilation of WASM is good; it is not native. The ecosystem is young. Library support for capability-style APIs lags behind the languages compiling to WASM. It won\u0027t replace containers, whatever the marketing suggests. It\u0027s a different granularity, and it earns its place where you want many small isolated units that start fast. WASM can put the centre back in through the component registry, which is being decided right now while everybody argues about the specification. The component model\u0027s promise is that you assemble software out of parts that need not share a language. That promise requires a place to find the parts. We have watched this film: the default public registry becomes the namespace, the namespace becomes the ecosystem, and whoever runs it ends up holding the decision about what exists. npm, Docker Hub, the app stores. In every case the technical artefact was open and the discovery layer was not, and the discovery layer is where the power turned out to be. A rate limit on pulls is a tax on every build in the industry, imposed by a company nobody elected. The specific risk for WASI is sharper than for containers, because components are meant to be small and numerous. An application might pull two hundred of them. Two hundred dependencies resolved through one index, each one running with whatever authority the host grants it, is a supply chain with a single curator and a very large blast radius. Note also that a capability system makes the authority auditable without making the code trustworthy: a module confined to one directory can still have been replaced with a version that leaks what is in that directory. Where is the open version? Ahead, for once, and that is worth defending. The component model is specified in the open, the runtimes are open source, and registry work is happening in standards bodies rather than in one company\u0027s product. Running your own registry is currently a normal thing to do rather than an act of defiance. That window closes the moment one index becomes the default in the tooling, because defaults are destiny. The rule is simple: if you cannot rebuild your artefacts from source you control, and mirror them somewhere you own, then the registry is part of your infrastructure and somebody else is running it. Pin your components, mirror them and treat a dependency you can only fetch from one host as an outage you have scheduled for a date of their choosing.", "d": "2025-01-14", "g": ["wasm", "wasi", "security", "capabilities"], "t": "WASI starts with no ambient authority", "u": "insights/wasi-capabilities/"},{"b": "Nearly every isolation mechanism we build on assumes the host is trustworthy and the guest is suspect. Confidential computing turns that around and protects the workload from the infrastructure it runs on. That is a new capability, not a better version of an old one. It is also the most over-sold idea in the whole isolation business at the moment, so it pays to be exact about which part is real. The mechanism\u00b6 Memory encrypted with a key the hypervisor doesn\u0027t hold, generated and held by the CPU. AMD SEV-SNP, Intel TDX, Arm CCA and the older enclave model of Intel SGX are the implementations. A hypervisor can still stop the guest, and it can still deny it service. What it\u0027s supposed to be unable to do is read the guest\u0027s memory, or modify it without that being detected. What attestation proves\u00b6 This is the part that gets routinely overstated, so here it is plainly. Attestation produces a signed statement from the hardware saying: this is the measurement of the code that was loaded into this protected environment, and here is a signature chaining to a key the manufacturer vouches for. What that establishes: The launched image matched a known measurement. The environment is a real instance of the technology, per the manufacturer\u0027s signing key. What it does not establish: That the code is correct or safe. It attests to identity, not virtue. A measured, attested, vulnerable application is still vulnerable. That the environment is bug-free. Plenty of side-channel and architectural attacks against these technologies have been published. SGX in particular had a long run of them. Anything about runtime state after launch. The measurement covers what was loaded. Whatever has happened since isn\u0027t in it. The TCB, counted properly\u00b6 The claimed benefit is a small trusted computing base. Here\u0027s the full inventory: The CPU and its microcode The firmware inside the confidential-computing implementation The manufacturer\u0027s key management and provisioning infrastructure The attestation verification service, and your own code that checks it Your guest image, kernel and userspace included That last one is the big one for the VM-based implementations. A confidential VM carries a whole guest OS inside the boundary. The TCB is \"smaller than one that includes the hypervisor\"; it isn\u0027t \"small\". What this costs you\u00b6 Encrypted memory has overhead. Published figures vary by implementation and by workload, and most of them are vendor-reported, so the open question is a specific one: on a named part, running a named workload, what is the delta against an identical unprotected run? That is a measurement anyone with two identical hosts can take, and the number that would settle the sizing argument for everybody. It is on our bench list and we have not taken it yet. The second cost is operational and gets mentioned less. You now have a verification step in your deployment path that can fail for reasons that have nothing to do with your software: a firmware level that changes the measurement, a certificate that expires, an attestation service that is down. Budget for it as infrastructure, because that is what it is. Pay that price and the scenarios become concrete. A hospital can hand a cloud provider an encrypted dataset and a measured analysis container, get the results back, and the operator never had the records in readable form. The compliance conversation changes from \"we trust this supplier\u0027s staff vetting\" to \"the operator was cryptographically unable to look\". That is the difference between a two-year information governance negotiation and a procurement decision. Several banks can compute a fraud signal across their combined transaction sets without any of them seeing another\u0027s customers. Competitors can pool data on a shared risk without pooling their commercial position. That class of arrangement has been technically possible with multi-party computation for years and practically impossible because the performance was absurd. Encrypted memory on ordinary hardware makes it a normal engineering project. And for small operators the effect is levelling. If the host cannot read the workload, then renting a machine from somebody with a datacentre stops requiring that you trust their entire organisation. A three-person company can run something sensitive on rented metal and defend that choice to a regulator. Previously that required owning the building. Every one of those scenarios depends on who holds the root. \"The cloud cannot read your data\" is true and incomplete. The attestation chain terminates at a key held by the silicon vendor. You have not eliminated trust; you have moved it from Amazon or Microsoft to Intel or AMD. That may well be the better trade, because your cloud provider\u0027s commercial interest in your data is a good deal more direct than your CPU vendor\u0027s. It is a trade and not an escape. Two or three companies make the silicon that can do this. They operate the provisioning infrastructure, they sign the keys, they decide which firmware versions are acceptable and they can revoke. If confidential computing becomes the way regulated workloads run, then those companies hold a veto over which workloads are allowed to be trustworthy, and there is no appeal and no second supplier. A government that wants to compel something has a much shorter list of doors to knock on than it did when the answer was \"every datacentre operator\". The open version is thinner than it should be. You can build your own attestation verification service, and people do; the part you cannot build is the root, because it is fused into a die in a fab you do not own. Open hardware roots of trust exist and they do not run your SEV-SNP guest. Here at AxisOps our position is that this technology is worth having and worth being suspicious of in exactly the same breath. Run it, verify the attestation yourself rather than letting the cloud\u0027s console do it for you, and write down who you are now trusting instead of your hosting provider. A control whose root you cannot name is a feeling. Where it earns its keep\u00b6 The pattern: it is valuable where operator access is the threat. Against application vulnerabilities, which are what most compromises come down to, it is close to useless. Encrypting the memory doesn\u0027t help if the lock on the front door is the thing being picked.", "d": "2024-12-03", "g": ["security", "hardware", "attestation", "cloud"], "t": "Confidential computing, counted", "u": "insights/confidential-computing-counted/"},{"b": "Clock synchronisation is unglamorous, everything above it quietly depends on it, and the way it fails is rarely the way anyone expects. What the protocols give you\u00b6 NTP over a normal network gets you within a few milliseconds of reference, sometimes rather worse. That\u0027s fine for log correlation and for certificate validity, and no use at all for ordering events between machines. PTP (IEEE 1588) gets you to microseconds or better, by timestamping in hardware at the network interface and accounting for path delay. It needs network equipment that joins in (transparent or boundary clocks). Across gear that doesn\u0027t, the error grows with topology, load and path asymmetry, and can land anywhere from better-than-NTP on a quiet LAN to no better at all. Whether that still clears your accuracy target is the question to ask before ordering the switches. The protocol without the participating network often buys much less than the invoice suggests. GPS or atomic references anchor the whole thing to something that doesn\u0027t drift, which is what you want when the accuracy has to be absolute rather than relative. The mistake that causes the bugs\u00b6 Using a timestamp as an ordering. Two events on two machines, timestamped 5ms apart, clocks synchronised to \u00b13ms. The timestamps say A then B. The truth might be either order. Better synchronisation never removes this; it only shrinks the window in which it\u0027s true. Worse, clocks don\u0027t only drift, they step. An NTP correction can move a clock backwards. Code that assumes monotonically increasing time will then see a negative duration, and the usual symptom isn\u0027t a crash. It\u0027s a nonsensical metric, or an infinite retry, at three in the morning. The right habits here are boring and absolute: Measure durations with a monotonic clock (CLOCK_MONOTONIC, time.monotonic()). It can\u0027t step and it can\u0027t run backwards. Keep wall-clock time for showing times to humans and for things that really are about calendar time. Don\u0027t use timestamps to order events across machines. Use a logical clock, Lamport or vector, if you need ordering; use a consensus protocol if you need agreement. Treat a timestamp as an interval rather than a point, if you must use one. That\u0027s what Spanner\u0027s TrueTime formalises: it reports its uncertainty out loud and waits it out instead of pretending it isn\u0027t there. Where sub-microsecond really matters\u00b6 Financial trading, where regulation requires timestamp accuracy and sequence is money. Telecoms, where radio synchronisation is a physical requirement. Industrial control and audio, where jitter is the thing you perceive. Distributed tracing, where the span tree is rebuilt from timestamps and clock skew gives you spans that appear to start before their parent. That last one is the case most engineers meet, usually without realising the negative-duration span they\u0027re staring at is a clock problem rather than a tracing bug. Hardware timestamping used to mean a specialist card and a specialist budget. It is becoming a line item on ordinary server NICs, and that changes what an ordinary engineering team can build. Databases that can tell you the truth about order. TrueTime made externally-consistent distributed transactions possible, and it needed a private fleet of GPS receivers and atomic clocks in every datacentre. Once a bounded-uncertainty clock is a property of commodity hardware, that capability stops being something one company can offer as a service and becomes something you can build on a rack you own. A distributed database with real serialisable semantics across sites, without renting it: that is a different set of architectures available to a mid-sized business. Observability that stops lying. The negative-duration span is a daily annoyance now. With microsecond-bounded clocks across a fleet, a trace becomes a measurement rather than a reconstruction, and the whole category of \"is this a real latency spike or clock skew\" disappears from on-call. That is a class of bug gone, not reduced. Networks that schedule. Time-sensitive networking only works if the endpoints agree on the microsecond. Get that cheaply and you can run industrial control, professional audio and factory robotics over ordinary switched Ethernet instead of a proprietary fieldbus. A small manufacturer gets to buy commodity network gear for a job that currently requires a vendor\u0027s closed ecosystem, and the engineers maintaining it get to use tools they already know. The prediction we will make is that bounded uncertainty becomes a thing clocks report by default rather than a Google-scale luxury, and that the interval replaces the point as the normal representation of a timestamp in new protocols. The arithmetic is too useful to stay rare. Precision does not remove the need to trust whoever supplies the time. Your time comes from somewhere, and almost every estate\u0027s answer is a GNSS constellation operated by a military, relayed through a public NTP pool maintained by volunteers or handed to you by whichever cloud you are renting. Each of those is a dependency you did not negotiate. GNSS is spoofable and jammable, demonstrably so and at low cost. A clock you can move is a protocol you can break: certificate validity, token expiry, replay windows and the ordering guarantees above all rest on it. An adversary who can shift your clock by an hour does not need to break your cryptography. The concentration risk is sharper than people assume. If the regionally authoritative time for a large slice of an industry is one provider\u0027s service, then that provider is in a position to define when things happened, and to be compelled to. Time is a surveillance surface too: a high-resolution, authenticated time source correlated across services makes activity attribution much easier, which is useful for forensics and equally useful for tracking people. The open side is healthy and underused. NTS gives you authenticated time, and chrony and the public NTP pool are good work. A disciplined local reference, a GPS module and a modest oscillator is a few hundred pounds and an afternoon. It means your estate\u0027s notion of time survives a dropped upstream outright, and a jammed sky for exactly as long as the oscillator\u0027s holdover: a number you have to specify and test rather than assume. We think every serious estate should hold its own time, for the same reason it should hold its own DNS. A dependency you cannot inspect is one you cannot reason about, and you should not have to ask a hyperscaler what o\u0027clock it is. Report the interval, not a point\u00b6 The discipline is to make a property explicit and checkable rather than assumed. TrueTime\u0027s uncertainty interval reports what is actually known. A bare timestamp claims a precision it hasn\u0027t got, like a recipe that says \"bake until done\".", "d": "2024-11-12", "g": ["determinism", "time", "distributed-systems", "ptp"], "t": "Time is harder than it looks and duller than it sounds", "u": "insights/determinism-and-time/"},{"b": "Governments have been telling the software industry to change programming languages. The NSA said it first, in a short advisory in November 2022. CISA and six partner agencies across five countries made the case for memory safe roadmaps in December 2023, and the White House Office of the National Cyber Director put the full argument in Back to the Building Blocks in February this year. The documents agree on the diagnosis: for large codebases in C and C++, memory-safety defects have sat around two-thirds of serious vulnerabilities for as long as anyone has counted. The fix that works is changing the language new code is written in, because forty years of training, tooling and standards have not made the old ones safe at scale. None of these documents is law. They compel nothing, fine nobody and name no deadline. So it is fair to ask what a mandate that mandates nothing is for, and the answer is procurement. The documents exist to give buying organisations a sentence to put in a contract: show us your memory-safety roadmap. We made the longer version of that argument in procurement is an architecture: a requirement repeated across enough purchase orders behaves like regulation without ever being written as one. What a mandate cannot do\u00b6 It cannot rewrite the installed base, and the installed base is where the problem lives. The Linux kernel is tens of millions of lines of C. The TLS libraries, the image parsers, the PDF renderers and the baseband firmware that make up the actual attack surface of a working system are C and C++ almost without exception, maintained by teams sized for maintenance, not for rewrites. A document that said \"rewrite it\" would be waving at the tide. The documents mostly do not say that, to their credit. The ask is a roadmap: new code in memory-safe languages, old code triaged by exposure, the riskiest parsing and privilege boundaries rewritten first. Systems programming was already moving that way before any government noticed, which is one reason the guidance has been received as pressure in a direction the field was already moving rather than as interference. The evidence arrived from Android\u00b6 Android has been the production experiment since 2019, and Google has published results along the way: by late 2022 the platform carried around 1.5 million lines of Rust with no memory-safety vulnerabilities found in them, and last month\u0027s five-year figures complete the picture. Memory-safety issues were 76% of the platform\u0027s vulnerabilities in 2019; they are 24% in 2024, while the old C and C++ sat largely unrewritten underneath. The mechanism matters more than the headline. Vulnerability density decays with code age: most vulnerabilities are found in recent code, because recent code is where the churn is. Switch the language of new code and the payoff arrives years ahead of any rewrite schedule, because recent and actively modified code produces a disproportionate share of the new vulnerabilities. Everything the government documents ask rests on that observation, and Android is where it has been measured in production. The honest residuals\u00b6 Three things the mandate conversation keeps underweighting: The boundary moves, and concentrates. A Rust component calling a C library through FFI, or using unsafe blocks internally, has not eliminated the problem. At best it has gathered it into small, auditable regions, and that concentration happens only where the architecture enforces it. A large improvement when it holds, and a different claim from \"memory safe\". Language share is an input, not an outcome. A roadmap can report the percentage of code in safe languages while the exploitable surface lives entirely in the remainder. The ONCD report is candid that measuring software quality is an open research problem, and until it is solved, the roadmaps will be measured by proxies. The two-thirds figure has a denominator. It counts memory-safety CVEs in C and C++ codebases. Logic errors, injection and authentication failures do not go away in Rust, as every language community rediscovers on its first big incident. A forecast rather than a finding: the documents will age well, and not because anyone obeys them. The Android numbers make the roadmap argument on merit, procurement language will quietly make it binding, and in ten years the question \"what is your memory-safety roadmap\" will sound as unremarkable in a tender as \"do you have backups\" does today.", "d": "2024-10-14", "g": ["policy", "security", "memory-safety", "rust"], "t": "Memory safety by mandate", "u": "insights/memory-safety-by-mandate/"},{"b": "The xz backdoor had one property that most of the discussion sailed past: the malicious code was never in the git repository. The piece that activates it travelled only in the release tarballs. The obfuscated payload itself sat in the git tree as binary test files, where nobody thought to look. Anybody auditing the build scripts on GitHub would have found nothing wrong, because the part that wired it together was never in the tree. The gap\u00b6 Almost nobody builds from source. Distributions build from release tarballs, and users install the packages the distributions built. There\u0027s a chain of transformations between \"the source a human reviewed\" and \"the binary a machine runs\", and every link in it is an opportunity for somebody. Reproducible builds close the gap by making that transformation verifiable: the same source, built with the same declared toolchain, has to produce bit-identical output (Reproducible Builds). If it does, anyone can rebuild independently and compare against what was published, and a discrepancy becomes evidence rather than a shrug. The input matters as much as the determinism: rebuilding the release tarball reproducibly would have reproduced the backdoor faithfully. The gap only closes when the thing rebuilt is the tree a human reviewed, with the tarball checked against it or abandoned. That converts \"trust the build infrastructure\" into \"verify the build infrastructure\", which is the move the whole supply-chain argument keeps making in one costume or another. Why it is hard\u00b6 The obstacles are mundane and there are a great many of them, which is why the work took years: Timestamps baked into archives and binaries. Build paths embedded in debug info and __FILE__. Filesystem ordering, which differs between machines and changes link order. Locale and timezone affecting sorting and formatting. Parallelism making output order non-deterministic. Compiler non-determinism, including ASLR-influenced hash ordering. Embedded build IDs and signing timestamps. Not one of those is conceptually hard. All of them are spread across thousands of build systems, which makes the job a social one rather than a technical one. It is the same shape as the architecture portability cleanup. What it does and does not give you\u00b6 Does: establishes that a published binary corresponds to published source, so reviewing the source means something, and a compromised build host becomes detectable by anybody who rebuilds. Does not: tell you the source is any good. A reproducible build of backdoored source gives you a reliably backdoored binary. Reproducibility makes source review worthwhile; it doesn\u0027t perform the review for you. That distinction matters, because reproducibility gets sold now and again as a supply-chain answer all by itself. It\u0027s one necessary link in a chain. Once the property holds across a distribution, several things that are currently impossible become routine, and they are worth more than the security headline. Trust stops being a relationship and becomes a calculation. A bank, a hospital or a defence supplier can run its own rebuild farm and verify every package it installs against the published artefact, without asking the vendor for anything and without trusting the vendor\u0027s build estate. No contract, no audit visit, no questionnaire. You rebuild and you compare. For an organisation in a country that does not trust the country the software was built in, this is the difference between adopting open source and being told not to. A compromised build server stops being an invisible catastrophe. Right now, if someone owns a distribution\u0027s build host, the first sign is a researcher noticing half a second of SSH latency. With independent rebuilds the first sign is a mismatch, reported automatically, by anyone. The attack does not get quieter; the detection gets cheaper, and it moves from one lucky engineer to a cron job anybody can run. The toil that disappears is incident archaeology. When something does go wrong, \"which source produced this binary\" becomes a question with an answer rather than three days of forensics across a build farm\u0027s logs. Every deployed artefact has a verifiable provenance to a reviewable tree. That also turns the SBOM from a best-effort inventory into a derivation you can check, which is the input the licence-compliance question needed rather than the answer to it. And it is cheap. Unlike most security controls, this one has no per-seat price, no appliance and no vendor. It is engineering effort spent once per build system, after which the benefit accrues to everybody who ever installs the package, forever. There is not much else in security with that shape. The compiler is not the chokepoint. The independent build infrastructure is. Reproducibility means anyone can verify. Whether anyone does is a question about who runs the rebuild infrastructure, and rebuilding a distribution is not free: thousands of packages, repeatedly, on multiple architectures. That is a compute bill. The obvious places for it to land are the three clouds and the two large platform vendors, and if verification consolidates there then we have rebuilt the original problem one level up. \"Trust the vendor\u0027s build\" becomes \"trust the vendor\u0027s verifier\", and a signature from a rebuild service you cannot audit is just a longer way of taking somebody\u0027s word. The practical version of that risk is already visible in how build attestation is being packaged: as a feature of a particular hosted CI product, with the provenance signed by the platform and verifiable chiefly by tools that the platform supplies. Useful, and also a lock-in surface. If your proof that your software is what you think it is only validates inside one company\u0027s ecosystem, you have bought a dependency rather than a property. The open side here is in good shape and that deserves saying. The Reproducible Builds project\u0027s work is public, the diffoscope tooling is excellent, SOURCE_DATE_EPOCH is a de facto standard, and Debian\u0027s percentage is high enough to be an existence proof rather than an aspiration. The remaining gap has nothing technical in it: almost nobody runs a rebuild and compares, because it is nobody\u0027s job. Which is the whole opportunity. A rebuild-and-compare is within reach of a single competent engineer and a modest machine, and it is one of the few security controls where a small organisation can independently reach the same assurance as a large one. Here at AxisOps we rebuild what we deploy and we publish what we built it from, because the alternative is trusting a build estate we have never seen. Verification you run yourself is the only kind that cannot be withdrawn. From build-time evidence to runtime evidence\u00b6 Reproducibility swaps a trust relationship for a property you can check. It does it for the path from source to binary, and we expect the same move to arrive next for the running environment and for loaded code, by mechanisms nobody has settled on yet. The pattern is more durable than any of the implementations will be. Of the controls to come out of the xz incident this is the least technically interesting and the best value per unit of effort. The condition bears restating, because it decides everything: the rebuild must start from the tree a human reviewed, with the release source verified against it. Rebuild the tarball and you reproduce the backdoor faithfully. Rebuild the reviewed tree, and the attack that has demonstrably been used in anger has nowhere left to live.", "d": "2024-09-17", "g": ["supply-chain", "security", "open-source", "build"], "t": "The backdoor a second build can see", "u": "insights/reproducible-builds/"},{"b": "Right-to-repair legislation has made real progress on the things it set out to address: parts availability, access to service documentation, tools. The binding constraint moved while all that was going on. Parts pairing\u00b6 A replacement component (battery, display, sensor, drive) is cryptographically tied to the device it was first installed in. Fit a genuine part taken from another unit of the very same model and the device rejects it, degrades what it does or nags you about it forever. The stated justification is security and authenticity. Making sure a battery is genuine is a real safety concern, and a fingerprint sensor\u0027s integrity is a real security concern; nobody sensible disputes either. Both of those are legitimate. They\u0027re also indistinguishable, from outside, from controlling who\u0027s allowed to do the repair. The mechanism serves both purposes identically, and the vendor is the only party in a position to tell you which one is operating. Signed firmware\u00b6 The deeper version of the same problem. If a device will only run firmware signed by its manufacturer, then the day the manufacturer stops signing is the day the device\u0027s remaining life becomes fixed. The hardware hasn\u0027t failed; nobody else is permitted to update it. This is where repair and e-waste run into security policy, and the tension here is real rather than manufactured: Signed firmware is a real security control. It\u0027s what keeps persistent malicious firmware off the device, and tearing it out would be a serious regression. It also makes the vendor\u0027s support horizon the device\u0027s horizon. There\u0027s a resolution available in principle, and it\u0027s owner-controlled keys: the device still enforces signatures, and the owner can enrol a key of their own after a documented, physically-present procedure. Secure Boot on general-purpose PCs works roughly like this, so we know the approach is implementable. Elsewhere it\u0027s rare, and the reason isn\u0027t a technical one. Why an engineering site cares\u00b6 Because this is a design decision that anybody building hardware makes, and usually makes by default. Ship signed firmware with no owner key enrolment and you\u0027ve decided your product dies when you stop supporting it. That\u0027s a choice with environmental, commercial and ethical dimensions, and it\u0027s routinely made without anyone recognising it as a choice. For anything with a server component the adjacent question is sharper: what becomes of deployed devices if the company stops existing? A device that bricks when a service shuts down was sold to you as a product and delivered as a subscription. What owner-controlled keys would make possible\u00b6 Spell out the world where this is normal, because it is a better one and it is within reach. A hospital buys infusion pumps with a ten-year clinical life from a vendor whose support window is five years. Today year six is a procurement crisis. With owner key enrolment it is a maintenance contract: the hospital\u0027s own biomedical engineering team, or any firm it hires, signs and ships patched firmware for the back half of the device\u0027s life, and the pump keeps its verified boot chain the whole time. The security property survives the vendor. That is the entire argument in one sentence, and it is deliberately narrow. The key solves who may sign a boot, while the validation, certification and safety evidence a regulated device needs still have to exist for whatever is signed, whoever signs it. A council runs ten thousand streetlight controllers. A farm runs tractors with twenty-year hulls and five-year electronics. A school has a room of eight-year-old laptops that are perfectly adequate for the work and are refused the next operating system because nobody will sign a bootloader for them. In every case the hardware is fine and the signature is the thing that expired. The volume of e-waste produced by working devices that were locked rather than broken is large and entirely avoidable. For engineering, the class of problem that disappears is the end-of-life scramble: no more rushed hardware refresh driven by a signing calendar rather than by capacity or failure rates, no more fleet frozen on a known-vulnerable firmware because the only party who could fix it has moved on. You get to plan replacement on the device\u0027s physical condition, which is the thing you can measure. And it unlocks a repair industry. Independent workshops can offer firmware maintenance, not just component swaps, which is skilled local work in the sort of towns that lost their manufacturing. Repairability is an employment policy as much as an environmental one. The power sits in one specific place. The chokepoint is one private key, held by the manufacturer, which decides whether the hardware you bought will execute. Not the part, the manual, the diagnostic tool or the legislation. Ownership of the device is nominal while somebody else holds that key. You have possession, maintenance obligations and the depreciation; they have the veto. This concentrates in a way that should bother you. A small number of silicon vendors supply the root-of-trust blocks that nearly all this firmware chains to, and the key hierarchy above the device is theirs, not the device maker\u0027s. Several platforms now ship with attestation that reports to the vendor\u0027s service which firmware a device is running. It is sold as fleet management, and it is useful for that, while also being a reliable per-device presence and inventory feed flowing to a third party by default. A government that wants to know which devices are in a building has an easier job than it used to, and does not need to ask the owner. Where is the open version? Better than it was. Coreboot and its descendants are real and run on real machines, Secure Boot on PCs does permit owner keys, and the fwupd and LVFS work has made vendor firmware updates on Linux ordinary rather than heroic. But it is losing badly on anything integrated: phones, consumer appliances, most medical and industrial kit, anything with a cellular modem. On those platforms there is no owner key path at all and no sign of one, because no regulator has required it and no customer has refused to buy without it. Keep verified boot; change who owns it\u00b6 The sustainable position is verified boot with owner-controlled keys: keep the security property, drop the vendor\u0027s monopoly on how long the device gets to live. It costs engineering effort and it closes off an after-sales revenue stream, which between them explain its rarity perfectly well without anyone having to assume bad faith. We specify owner key enrolment when we buy, and we would rather pay more for hardware whose working life we control than save money renting the right to run our own machines. The question to put to a vendor is short: can I enrol my own key, and if not, what is your plan for my hardware after your last signature?", "d": "2024-08-20", "g": ["policy", "hardware", "open-source", "sustainability"], "t": "Repair ends at the signing key", "u": "insights/right-to-repair-and-firmware/"},{"b": "Qualys disclosed regreSSHion on 1 July 2024: a signal-handler race in sshd giving unauthenticated remote code execution as root on glibc-based Linux (Qualys). OpenSSH 9.8 shipped the fix the same day (OpenSSH). It is a regression, in Qualys\u0027s framing: the reintroduction of CVE-2006-5051, an eighteen-year-old bug that came back in OpenSSH 8.5p1 in March 2021. So the affected range has an odd shape. Below 4.4p1 vulnerable, unless backport-patched for CVE-2006-5051 or CVE-2008-4109. From 4.4p1 up to but not including 8.5p1, safe. From 8.5p1 through 9.7p1, vulnerable again; 9.8p1 is the fix. That shape is why this one is harder to scan for than it looks. Why the version number does not answer the question\u00b6 A distribution backports a security fix without changing the upstream version string. So a host reporting OpenSSH 9.6p1 sits squarely in the affected range and may be perfectly safe, because the vendor patched it last Tuesday and left the version alone. Reading sshd -V therefore tells you almost nothing on its own. The package version against the distribution\u0027s fixed version is what decides, and that only works where a package exists and where you know what the fixed version is for that release. Which leaves a third category, and getting it right is the only interesting decision in the tool: a packaged OpenSSH in the affected range, on a distribution our configuration has no fixed version for. The tempting answer is to call it vulnerable and let the operator sort it out. We report it as unknown. A false \"vulnerable\" on a patched host sends someone to apply a workaround that costs availability, for no gain. A false \"fixed\" is worse. \"Unknown\" is the only verdict that is true, and it carries the one instruction that is also true: go and look at this host yourself. The workaround is not free\u00b6 The vendor-documented mitigation is LoginGraceTime 0. Qualys is blunt about what it costs: it \"makes sshd vulnerable to a denial of service (the exhaustion of all MaxStartups connections)\". Connections that have not yet authenticated stop being timed out, so anyone can hold every slot open and nobody can log in. You are choosing which outage you prefer. Unauthenticated root on an internet-facing box is worse than an SSH denial of service, so for most estates the trade is correct, and it is still a trade rather than a fix. Red Hat pairs it with fail2ban and a firewall, which is sensible, and turns one moving part into three. Our role at github.com/axisops/CVE-2024-6387, MIT, upgrades openssh-server first and only applies the workaround to hosts still vulnerable afterwards. Same ordering as every tool in this family, for the same reason: the workaround should reach the smallest possible number of machines. One line, in the right place\u00b6 The line goes at the top of sshd_config, and that is not a style preference. sshd keeps the first LoginGraceTime it reads. Ubuntu 22.04 onwards puts an Include sshd_config.d/*.conf at the start of the file, so a directive appended at the end loses to anything in that directory that sets the same keyword, because the first value read wins. The workaround appears in the config, reads correctly to anyone reviewing it and does nothing at all. The other half of that trap is removal. Installing the fixed package does not take the lines out. An estate that applied the workaround in July and upgraded in August is still running without login timeouts in September, carrying the denial-of-service exposure it accepted as temporary, because nothing connects the two operations. Take the lines out as a deliberate step after the upgrade, and check for them on hosts you believe are clean. What it does not do\u00b6 The workaround trades a remote root for an availability risk, and this role applies it without asking whether your box is behind a bastion or facing the internet. That judgement is yours and the tool cannot make it. Version detection depends on sshd -V, falling back to ssh -V, and on dpkg or rpm. A self-built OpenSSH in /usr/local has no package to compare against and lands in \"unknown\" by design. Containers running their own sshd are not covered by a host scan at all; their fix is a rebuilt image. And it contains no exploit code, so it cannot tell you whether a host is exploitable, only whether it is in the affected set. Qualys\u0027s own exploitation figures are hours to days of sustained connection attempts against a tuned target. That is a real attack and not a drive-by, which should inform how you rank this against everything else in the queue rather than whether you act. The five days\u00b6 The fix was available on day one. We published on day five, and all of it went on the classification problem rather than the change. Reading the advisory takes twenty minutes. Working out which of your hosts are genuinely affected means knowing every distribution and release you run, every vendor\u0027s fixed package version for each and which of those vendors had shipped by the time you looked. None of that is in the advisory, and none of it is knowable from the version string the host reports. An estate that can answer \"what OpenSSH, from what package, on what release\" from a declaration does this on day one. We went looking, host by host: the same answer we reached in December 2021, and one we have not yet finished building our way out of.", "d": "2024-07-06", "g": ["security", "ansible", "linux", "python", "mitigation"], "t": "The workaround that trades one outage for another", "u": "insights/regresshion-saturday/"},{"b": "Almost every intuition you have about performance was formed on symmetric multiprocessing: N identical cores, uniform memory and a thread that runs at the same speed wherever it happens to land. That machine is becoming the unusual one. What changed\u00b6 Asymmetric cores. Performance and efficiency cores in the same package, with different clocks, different microarchitectures and sometimes different cache hierarchies. Arm\u0027s big.LITTLE brought it to phones; Apple Silicon and Intel\u0027s hybrid parts brought it to laptops and desktops. Unified memory. CPU and GPU sharing one physical pool, which removes the copy that used to dominate a lot of GPU programming and moves where the bottleneck sits. On-package accelerators. Neural engines, media encoders, crypto units. Silicon that sits idle for most workloads and is decisive for a few. Why this breaks things quietly\u00b6 Nothing crashes. The failure mode is measurements that mean rather less than they look like they mean: A benchmark result is now a placement result. Run the same binary on a P-core and an E-core and you get two different numbers. A benchmark that doesn\u0027t state and pin placement is reporting the scheduler\u0027s mood. Thread count is not a capacity number. \"16 cores\" where 8 of them are efficiency cores at a fraction of the throughput makes nproc-based pool sizing wrong. Work-stealing assumes equivalent workers. A classic work-stealing pool rebalances towards whichever worker finishes fastest, which on asymmetric hardware means the P-cores pull most of the work. Usually that\u0027s fine; occasionally it\u0027s pathological, when tasks have dependencies. Spin-then-block heuristics calibrated on uniform cores mis-tune. The scheduling problem is hard\u00b6 This is one of the narrow cases where it really is the scheduler\u0027s problem, and it isn\u0027t a solved one. The kernel has to work out, from observed behaviour, whether a task is latency-sensitive (P-core, please) or throughput-oriented and tolerant (an E-core is fine, and cheaper). It\u0027s guessing at intent from history, rather like a waiter deciding from your last order what you\u0027d like next. Energy Aware Scheduling and the utilisation-clamp interface exist so userspace can give hints, and most software gives none. A declared latency requirement is better input than an inferred one, and we expect the next few years of kernel scheduling work to be mostly about getting that declaration out of userspace and into something the scheduler can act on. The fix for guessing is not a cleverer guess. What to do about it\u00b6 For anyone measuring or building: Pin, or report placement. A benchmark without taskset or an explicit affinity statement isn\u0027t reproducible on a hybrid machine. Size pools by measured throughput, not by core count. Say what the hardware was. \"16 cores\" has stopped being a hardware description. The part number is one. The pessimism above is about measurement discipline, not about the hardware. The hardware is the best news in computing right now. A laptop that can hold a serious workload on its efficiency cores runs for a working day on a battery, silently, with the fans off. That is not a comfort feature. It means a field engineer can carry a real development machine to a substation with no mains and finish the job. It also means a school in a place with four hours of grid power a day can run a classroom of machines off a modest solar array. Perf-per-watt is an access story before it is a performance story. In the rack the same arithmetic compounds. Efficiency cores let you put the background work (log shipping, metric aggregation, the compaction job, the TLS termination) on silicon that costs a fraction of a watt per unit of throughput, and leave the performance cores for the thing a customer is waiting on. Done properly that is the same service in meaningfully less rack space, in a building whose thermal envelope you had already filled. Half is the number the vendor decks claim; we have not measured it on our own estate. Capacity you didn\u0027t have to build, if the claim survives contact with your workload. For engineering itself, on-package accelerators kill a whole class of toil. Video transcoding, hashing, inference: work that used to mean a separate fleet of specialist machines, with its own capacity planning and its own failure modes, collapses into a feature of the general-purpose node. The ops burden of \"the GPU box\" goes away because there is no GPU box. The useful scheduling work still leaves one question: can you buy the good silicon, or only rent it? Asymmetric, accelerator-laden packages are much harder to second-source than a symmetric x86 part was. The scheduling interface, the accelerator\u0027s driver, the firmware that decides which cores are even visible to you: all of it is vendor-controlled, and on the more integrated parts it is not documented in any useful sense. You can buy the chip. Whether you can tune it is a different question, and on several current platforms the answer is no. The allocation problem is sharper than the documentation problem. Leading-edge packages are fabricated in very few places, and the first call on the output goes to whoever buys in the largest volume. That is three or four hyperscalers, and increasingly they are not buying merchant silicon at all; they are designing their own and keeping it. Nobody outside gets to rent it. The result is a machine you can only reach by renting it from the company that commissioned it. That is a different industry from the one where you bought a server and owned it. Watch for the version of this where the efficiency cores are reserved for the platform\u0027s own telemetry and management agents, and the capacity you paid for turns out to have a landlord living in it. An undocumented scheduling interface is a defect in the product rather than a fact of life. The useful pressure to apply is boring and it works: ask for the perf-per-watt numbers, ask which cores you control and buy the part that answers. The open side of this is in reasonable shape at the kernel level, where the scheduling work is public and anyone can read it. It is in poor shape at the accelerator level, where the interesting units are behind a binary blob. If that does not change, the most efficient computer you can own will stay well behind the most efficient computer you can rent.", "d": "2024-04-18", "g": ["hardware", "scheduling", "arm", "performance"], "t": "Heterogeneous is the default now", "u": "insights/heterogeneous-default/"},{"b": "On 29 March 2024 Andres Freund reported to the Openwall security list that xz-utils 5.6.0 and 5.6.1 contained a backdoor (mjtsai\u0027s roundup collects the original disclosures). Red Hat rated it Critical and scored it 10.0. Freund is a PostgreSQL developer at Microsoft. He wasn\u0027t auditing xz. He noticed that SSH logins on a Debian testing box were taking about half a second longer than they ought to, and he followed it, which is the engineering equivalent of pulling over because the engine note changed. Why this one is different\u00b6 Most supply-chain incidents are opportunistic: a typosquatted package, a compromised credential, a dependency confusion trick. Crimes of convenience against automated systems. This was a multi-year social engineering campaign against a person. The pattern, as reconstructed: a project that felt unmaintained with one exhausted maintainer, sustained pressure from apparently-helpful new contributors complaining about the pace of releases, a gradual handover of trust and eventually commit and release-signing authority. The backdoored tarballs were signed legitimately, by the account that had been given the right to sign them. No part of that is a software vulnerability. Every control that failed was a social one. What it says about the dependency graph\u00b6 xz is compression. It\u0027s in the boot path of most Linux distributions, in kernel module loading, in package management. It\u0027s also, functionally, one person\u0027s unpaid evening project. That combination (critical to everything, maintained by nearly nobody) is the ordinary condition of a large fraction of the dependency graph, not an exception. The RIPE open-source working group thread titled \"xz incident shows the need for structural change\" is a fair summary of the response from people who think about this seriously. A container-infrastructure meetup at BlackRock\u0027s London office, January 2016. The slide is Aladdin\u0027s own account of what it is built on: Java, Solr, Spring, React, Storm, Hadoop. The big-ticket platforms run on the same building blocks as the rest of us. What would have helped\u00b6 Which controls would and wouldn\u0027t have caught this: Would not have caught it. SBOM, because the component was legitimate and the version was current. Signature verification, because it was correctly signed. Dependency pinning, because pinning to the backdoored release leaves you pinned to a backdoor. Most CVE scanning, because there was no CVE until after the discovery. Might have caught it. Reproducible builds, because the backdoor was in the release tarball and not in the git tree. Closing that gap is what reproducibility is for, and it\u0027s the single highest-value control to come out of the whole incident. Did catch it. One engineer with the habit of chasing a performance anomaly instead of shrugging at it. Nobody can mandate that. What we shipped this morning\u00b6 The detection problem is the one we could write code for, so we did: an Ansible role at github.com/axisops/CVE-2024-3094, MIT, and you are welcome to it. It reads the installed liblzma5 and xz-utils versions, scans the library directories for liblzma.so.5*, resolves symlinks so each real file is counted once, reports any 5.6.0 or 5.6.1 and says whether sshd links liblzma at all. Dry run by default; nothing changes without --apply. Two details in it are worth more than the rest of the code. Debian\u0027s fixed package is versioned 5.6.1+really5.4.5-1, which a naive version comparison reads as 5.6.1 and flags as backdoored. So the parser has to understand +really or it cries wolf across an entire Debian estate. And after the upgrade the script names any affected file the package manager does not own, built from source into /usr/local or shipped inside an application bundle, rather than reporting the host clean. An upgrade fixes what the package manager controls, and the honest output is a list of what it does not. The backups it takes are copies of a backdoored library, kept as evidence, and the README says in bold not to restore them. Scanning a host also says nothing about container images, chroots or application bundles, whose fix is a rebuild rather than a package upgrade. What the role cannot do is the part this incident was about. It finds a known-bad version of a known-bad library, because by this morning somebody had told us which one to look for. Freund\u0027s half-second is not a thing a script can be taught. What this incident makes possible\u00b6 An odd thing to say about a 10.0, but a disaster that gets fully reconstructed in public is worth a great deal, and this one will be. The whole attack is visible. The mailing list pressure, the commits, the build system trick, the signed tarballs: all of it is on the record, which means every distribution, every vendor and every engineer gets to study a complete professional, patient supply-chain compromise from the inside, for free. To this day it is publicly unattributed. Nobody gets that from a breach at a commercial vendor, where the incident report is three paragraphs and a lawyer. Open source just paid a large bill and received the full post-mortem in exchange. Use it. The practical consequence we expect within a couple of years is that building from a release tarball stops being normal. The tooling to build distribution packages straight from a signed git tag, reproducibly, is nearly all in place; what was missing was a reason urgent enough to finish it. There is now one. The engineer\u0027s day changes concretely: a discrepancy between the published binary and an independent rebuild becomes an alert rather than an unanswerable question, and the class of attack that hides in the gap between the repository and the tarball closes for good. The wider benefit is that this is the rare security improvement that helps everybody equally. A small company, a hospital and a government all get the same verifiable artefact, because the verification happens in public and costs nothing to consume. The chokepoint for xz was a single unpaid maintainer with no colleague, no deputy and no capacity to say no to help. Not a registry, not a signing key, not a cloud. A person. And notice who caught it. Not a vendor\u0027s threat intelligence team, not a scanner, not any of the paid supply-chain security products whose marketing will be rewritten around this incident by Friday. One engineer, working on something else entirely, noticing 500 milliseconds. Every expensive control in the industry had the backdoored package and said it was fine. That matters for where this goes next. The response from the large platforms will be to offer you a curated, scanned, vendor-blessed package feed: their registry, their attestations, their judgement about what you may install. It will be better than nothing, and it is also the mechanism by which a handful of companies end up deciding which open source exists in practice. A curated feed is a gate, and a gate has an owner. If a maintainer cannot get through it, their work stops shipping, whatever its licence says. The open, self-hostable version of this control is in decent shape: you can run your own mirror, do your own rebuilds and verify independently. It is losing on convenience, which is how these things are always lost. Fund the dependency or inherit the exhaustion\u00b6 The economics here don\u0027t work, and no amount of tooling fixes economics. An attacker willing to spend two years building trust will beat a volunteer maintainer who has no time, and the industry consuming that work for free has been externalising the cost of maintaining it for thirty years. Paying maintainers is the only control on this list that addresses the failure that happened; it is not charity. We treat it as a line in the budget rather than a gesture, because the alternative is trusting somebody else\u0027s exhaustion.", "d": "2024-04-02", "g": ["supply-chain", "open-source", "security", "mitigation"], "t": "The xz backdoor found the human dependency", "u": "insights/xz-and-the-maintainer-problem/"},{"b": "Six weeks ago, at the World Governments Summit in Dubai, Nvidia\u0027s chief executive told an audience of ministers that every country needs to own the production of its own intelligence. The phrase he used was \"sovereign AI\", and it has moved through policy circles since at a speed that has very little to do with the underlying engineering. Where the phrase came from matters, because the provenance is the argument. The phrase has a vendor\u0027s fingerprints on it\u00b6 \"Digital sovereignty\" is an older idea with a real constituency. European data protection authorities have been making it for a decade, and it is mostly about jurisdiction: whose courts can compel disclosure, whose law governs a transfer, which regulator you can actually call. That argument is about legal exposure and it is usually made by lawyers. \"Sovereign AI\" is a newer construction, and it was not introduced by a regulator. It entered general circulation on an Nvidia earnings call in November 2023 and reached ministers in February. The content of the phrase, as it is being used, is that a nation must build domestic capacity to train and run models on hardware it controls. Notice what that does. The older sovereignty argument was about who can compel you. The new one is about what you have bought. The first has an answer that might be \"change the contract\" or \"move the data\" or \"accept the risk and write it down\". The second has an answer that is a purchase order. We have no quarrel with a country wanting domestic compute. The objection is narrower: the definition currently in circulation makes a vendor\u0027s product the measure of a national capability, and that definition was supplied by the vendor. What the pitch leaves out\u00b6 Four dependencies sit between a sovereign AI programme and anything a minister could call sovereign. None of them is addressed by buying accelerators. The accelerators themselves. A datacentre full of hardware from one company in one jurisdiction remains a dependency. If the chips, the firmware, the interconnect and the driver stack come from a single foreign supplier, the question \"what is our position if this is withdrawn\" has the same answer it had before the money was spent. It may well be a better answer than renting the same hardware from a hyperscaler, since the asset is on your soil and nobody can switch it off remotely. Owning a dependency still leaves you dependent. The software that makes them useful. CUDA is the dependency that the hardware conversation keeps eliding. A national programme that trains on this hardware produces artefacts, skills and tooling bound to one company\u0027s proprietary software environment. There are portable alternatives, and they are behind. That gap is the lock-in, and it does not appear on the invoice. Export control. An accelerator\u0027s availability is set by the exporting government, in units of processing power, revised when it chooses. A capability whose supply can be restricted by another state\u0027s licensing decision is a capability held at that state\u0027s discretion. This has already happened, and not only to adversaries: allied purchasers have found their orders reclassified. A sovereignty programme that depends on an export licence is a sovereignty programme with a foreign veto. The weights. Most of what is being called sovereign AI is a national deployment of a model trained elsewhere, on data assembled elsewhere, with alignment decisions made elsewhere. The compute is domestic; the artefact is not. A country that can run a model it cannot inspect, retrain or reproduce has bought a hosting arrangement. The test we would apply\u00b6 Our usual question works here with no modification. Which of these capabilities could be withdrawn or compelled by a party outside our control, and what is our position if that happens? Run it over a sovereign AI proposal and the answers are uncomfortable but useful: If the export licence lapses, can we still expand? Usually no. If the proprietary software stack changes its terms, what is the migration cost? Usually unmeasured. If the model\u0027s originator stops publishing weights, do we have a reproducible path to the current one? Usually no. If the vendor\u0027s roadmap moves, is our programme stranded? Ask the people who standardised on a single storage vendor. A programme that can answer those has done the work. A programme that has bought a lot of hardware has bought a lot of hardware. What would actually move the needle\u00b6 Three things, none of them as photogenic as a ribbon-cutting at a datacentre. Reproducible training. Published data provenance, published recipe and enough detail that an independent party can produce an equivalent model. That is what turns a delivered artefact into a capability you hold. It is also the part no vendor has an incentive to fund. Portable software. Work that runs on more than one supplier\u0027s silicon. The alternatives being behind is a reason to invest in them, not a reason to deepen the dependency. A national programme is one of the few buyers large enough to change that calculus. Dependency accounting, written down. Every sovereignty claim should be accompanied by the list of things that could be withdrawn and by whom. That list belongs in the programme\u0027s published specification rather than in a risk register nobody reads, because a dependency you have named is one you can plan around. Why this matters beyond AI\u00b6 The pattern has a history, which is why we are writing this down now rather than waiting to see how the programmes work out. A technology arrives. It has a real sovereignty dimension. A supplier supplies the vocabulary for discussing it, and the vocabulary is shaped so that the supplier\u0027s product is the solution. The cloud argument went this way. \"Sovereign cloud\" came to mean a local region operated by the same company under the same parent jurisdiction, which addressed the data-residency question and left the control question exactly where it was. The accelerator version of this will be harder to unwind, because the capital commitment is larger and the alternatives are further behind. The useful move, for anyone specifying one of these programmes, is to insist on a definition that was not written by a party with something to sell. Sovereignty describes your position when a supplier changes its mind. An inventory of hardware says nothing about that position on its own.", "d": "2024-03-19", "g": ["policy", "ai", "infrastructure", "supply-chain"], "t": "The sovereignty pitch arrived from a GPU vendor", "u": "insights/sovereign-ai-arrived-from-a-vendor/"},{"b": "For thirty years formal verification was something that happened in papers about systems nobody ran. Two things changed that. seL4 produced a verified kernel that is deployed in real safety- and security-critical systems. And the tooling reached the point where an ordinary engineer can apply it to a bounded problem without a doctorate in the subject. The spectrum, because \"formal methods\" covers very different things\u00b6 These are worth separating, because the cost and the payoff differ by orders of magnitude: Type systems. The formal method everyone already uses without calling it one. A compiler rejecting a type error is a proof of a property. Rust\u0027s borrow checker is the same trick with a stronger property attached. Property-based testing. Not proof; it generates cases against invariants you\u0027ve stated. Cheap, and it finds a surprising proportion of what proof would. Model checking. Exhaustively explore a finite state space. TLA+ is the practical entry point and has the best industrial track record, particularly for distributed protocols where the bugs live in interleavings nobody imagined. Deductive verification. Prove a program satisfies a specification. Expensive, and what seL4 did. Most of the value on offer sits in the first three. The fourth is for things where failure is unacceptable. What seL4 demonstrated\u00b6 Not that verification is cheap. The original effort was famously on the order of twenty person-years for about ten thousand lines. What it demonstrated is that it\u0027s possible at all for real systems code, and that the proof can be maintained. The proofs have been carried forward across new architectures: 32-bit Arm first, then x86-64 and 64-bit RISC-V, with AArch64 and the MCS configuration both in progress as of this writing (seL4 news). A proof that rots the first time you modify the code would be a curiosity. One that survives a port is an engineering artefact, in the way that a bridge design you can reuse is an engineering artefact. Once the proof is an engineering artefact, a whole class of argument gets settled once and then reused. Take a railway signalling interlock, a battery management controller, an infusion pump. Today each is certified by process: documents, review boards, test campaigns, a trail of paper establishing that competent people were careful. That is expensive, it has to be redone for every variant and it produces confidence rather than proof. A verified kernel underneath changes what has to be argued. Our bet is that this is where the money goes next: not proving applications correct, which will stay out of reach, but proving the separation between them, so a safety case can be made about one component at a time instead of about the whole box. That is the difference between recertifying a medical device because the Wi-Fi stack changed and not having to. For engineering itself, the class of bug that disappears is the one nobody can reproduce. A proof covers every execution, including the interleaving that happens once a fortnight on one customer\u0027s hardware and never on your bench. The limits\u00b6 A proof is relative to a specification. If the spec is wrong, the proof is worthless and you now have high confidence in the wrong thing. Writing the spec is where the thinking lives. Assumptions matter enormously, and seL4 names its own. The proofs assume the hardware behaves as modelled, and exclude the boot process, the low-level assembly the kernel starts from, direct memory access by peripherals and timing channels. Spectre was the hardware not behaving as modelled, which is the first of those assumptions failing in public. \"Formally verified\" means verified against that model and those exclusions, and anyone quoting it as a general separation guarantee has dropped the half that matters. Cost scales badly with size. This is for kernels, protocols and crypto primitives, not for application code. Reusing a proof is cheap. The small labour market able to make one is not. There are perhaps a few hundred people in the world who can carry a functional-correctness proof through a non-trivial port. They are expensive, they are concentrated in a handful of research groups and a handful of large companies, and the companies are the ones who can also afford to wait three years for the result. If verification becomes the thing that distinguishes a certifiable product from an uncertifiable one, then the ability to produce a verified component is a competitive moat held by whoever employs that cohort. The optimistic reading is that proofs are artefacts, and an artefact can be published. seL4\u0027s proofs are open, which means a small team can build on top of a verified kernel without employing a single proof engineer. That is a different distribution of power from the one where the verified component ships as a binary blob under a per-seat licence with the proof held as commercial confidence. Both futures are available right now and the licence terms decide which one you get. Here at AxisOps we will take the open one, and the reason is practical rather than ideological: a proof you are not allowed to read is a claim, and a claim is what verification was supposed to replace. The practical recommendation\u00b6 The highest-value thing most teams could pick up is TLA+ for distributed protocol design, used before implementation. Distributed systems bugs live in interleavings, interleavings are what model checking explores exhaustively, and a design bug found before implementation is the cheapest bug there is. That\u0027s a days-to-weeks investment, not twenty person-years.", "d": "2024-03-05", "g": ["formal-methods", "verification", "rust", "microkernels"], "t": "Formal methods stopped being academic", "u": "insights/formal-methods-practical/"},{"b": "Procurement writes architecture before an engineer opens the editor. Public-sector technology procurement reliably delivers systems that are expensive, late and locked in. The engineers involved are no worse than anybody else; here at AxisOps we\u0027ve worked alongside plenty of them. The structure selects for the outcome. The mechanism\u00b6 Procurement exists to transfer risk to a supplier. That\u0027s a reasonable goal, and it produces a predictable set of second-order effects: Large contracts favour large suppliers, because the compliance overhead of bidding is a fixed cost. A \u00a340m framework excludes the ten-person firm that could do the work, and competence never enters into it. Risk transfer requires specification up front, which requires knowing the answer before the work begins. Everything learned during delivery becomes a variation, and variations are where the money goes. Exit is almost never specified, because whoever writes the contract is buying a system, not planning to leave it. The cost of leaving gets discovered later, by somebody else, usually in a hurry. None of that is corruption. It\u0027s an incentive structure paying out on what it rewards. The architectural consequence\u00b6 A system procured this way has its architecture settled before an engineer is anywhere near it. If the contract names a vendor\u0027s platform, the architecture is that platform. If it specifies an outcome and transfers the risk, the supplier picks whatever minimises their risk, which is rarely what minimises your long-run cost. So the technical decisions that matter most (data formats, interface boundaries, where state lives) get made implicitly, in a document written by people who would be surprised to hear they were making technical decisions at all. It\u0027s like signing off the floor plan before anybody has asked where the drains go. What changes the outcome\u00b6 Three things, none of them new, all of them structural rather than technical: Specify the exit, not just the entry. A requirement to export all data in a documented format, exercised annually as an acceptance test rather than promised in a clause. An untested export clause is worth nothing at all. Buy capability in increments small enough to cancel. The sunk-cost dynamic on a large programme is the single biggest driver of good money going after bad. Treat interfaces as the procured thing. Specify the boundaries and test them, and whatever sits behind one is replaceable; that\u0027s the only real protection against lock-in you can buy. It is worth being concrete about the prize, because the critique above is familiar and the upside rarely gets described. A council that specified interfaces rather than systems could buy its revenues and benefits processing from one supplier, swap it in four years for a better one and keep the data where it was. That is not a fantasy; it is how the payments industry works. The reason you can change bank without losing your transaction history is that somebody specified the boundary rather than the vendor. Applied to public services, the thing that becomes cheap is the second decade. Right now every programme pays twice: once to build, and once more in eight years to escape what was built. The escape is usually the larger number. The smaller prize is more immediate and would change a lot of working lives. A specification that names formats and interfaces is one a ten-person firm can bid into. There are a great many such firms who could deliver a school admissions system or a licensing portal better and cheaper than a systems integrator, and they are excluded today by the cost of bidding rather than by capability. Lower that barrier and the public sector gets access to a supplier base it currently cannot reach, and those firms get to exist in towns where there is no other technology employer. That is industrial policy done by contract drafting, which is cheaper than industrial policy done by grant. There is a resilience argument too. A service assembled from replaceable parts degrades when one part fails. A monolithic platform from a single supplier stops. Anybody who has watched a region\u0027s social care system go dark because one company had a bad weekend understands which of those they would rather own. The framework is where the choice narrows, because it is presented as the solution to the problem it causes. A framework agreement exists to save time: pre-qualify a set of suppliers, then call off against it without running a full procurement each time. Reasonable. What it does in practice is set the list of companies who are allowed to sell to government for the next several years. Its qualification criteria (turnover thresholds, insurance levels, reference contracts of comparable scale, security certifications priced per audit) are a filter on size rather than on competence. For any substantial lot the answer ends up being the same handful of integrators and, increasingly, the same three cloud providers underneath them. The hyperscaler layer is the part that has changed and the part least discussed in the contract. A department can switch integrator and stay on the same cloud, which looks like competition and is not, because the dependency that would be expensive to unwind is the one nobody tendered for. Egress pricing, managed services with no equivalent elsewhere and identity systems that the whole estate now assumes: each is a decision made by an engineer under delivery pressure, and together they are a sovereign commitment made without a sovereign decision. The data is in the country, which is what the clause asked for. The dependency is not. Where is the open, ownable version? It exists and it loses on procurement rather than on merit. Open-source infrastructure has no vendor to pre-qualify, no sales team to respond to a framework notice and no commercial entity to carry the liability the contract requires. So the better technical answer cannot bid, which is a structural defect in the buying process and not a judgement about the software. The fix is unexciting and known: buy the integration and the support as the contracted thing, specify the software as open and require the exit test annually. Here at AxisOps this is the business we are in, so treat it as a declared interest and an argued position: we build infrastructure that the customer owns, can inspect and can rebuild without us. A system you can only operate while the supplier\u0027s invoices are being paid is not an asset on your balance sheet, whatever the accountants have written down. Why it\u0027s on an engineering site\u00b6 Because engineers keep trying to fix this downstream, with better architecture inside a contract whose shape already decided the outcome. The leverage is upstream. An engineer who can read a procurement document and say \"this clause means we won\u0027t be able to leave in eight years\" is doing more valuable work than one optimising the thing that\u0027s going to be abandoned anyway. Learn to read a contract; it is the highest-leverage unglamorous skill in public-sector technology.", "d": "2024-02-13", "g": ["policy", "infrastructure", "open-source"], "t": "Government procurement is an architectural decision", "u": "insights/procurement-is-an-architecture/"},{"b": "A scan of the entire IPv4 address space, set in metal type from a time when a machine could not be reached, let alone compromised. The command is the same one an attacker runs; the difference is the operator and the intent, which is not a property the tool has. A port scanner is reconnaissance. A packet capture tool is interception. A fuzzer finds bugs for whoever is running it. A model that writes code writes whatever code it\u0027s asked for. That\u0027s the ordinary condition of nearly everything in security, and of most systems engineering besides. A hammer builds houses and breaks windows, and nobody has worked out how to make one that only does the first. Why \"don\u0027t build it\" isn\u0027t available\u00b6 Withholding a capability assumes the capability is scarce. Usually it isn\u0027t: The technique is published. Security research is adversarial and public; withholding a tool doesn\u0027t withhold the method. The adversary is resourced. The actors you\u0027d most like to deny are the ones best able to build it themselves. Defenders need it more. You can\u0027t test what you can\u0027t attack. Withhold offensive tooling and you disarm the defender, because the attacker was never waiting on your release. That\u0027s the standard argument for publication and it\u0027s largely right. It also gets used to wave away a real question: the marginal effect of a particular release. Where the argument weakens\u00b6 Weaponisation gap. A proof of concept that needs expertise to use is a different object from a packaged tool that needs none. Lowering the skill floor changes who can attack, and \"it was already public\" doesn\u0027t address that. Scale. A technique that works once differs from one that works against everything at once. The automation is itself the capability. Patch availability. Publishing before a fix exists and publishing after are different acts carrying the same information. That third one is why coordinated disclosure exists. The hard cases are the ones where a working proof of concept is public before the official patch ships, which is the scenario the norm was built to prevent and the one it handles worst. What open tooling makes possible\u00b6 An open security tool is how a competent person without a budget gets to be competent. Whoever runs IT for a housing association, a school trust or a regional hospital can scan their own estate, capture their own traffic and fuzz their own application. None of them is buying a commercial assessment; all of them can run nmap and Wireshark and a fuzzer tonight, because somebody published. Remove that and defensive capability becomes a purchase that tracks budget rather than need. The least-resourced organisations holding the most sensitive records then get the least protection. That is the situation in healthcare and it is not improving. The engineering benefit is more specific. Tooling that attacks your own system turns a class of bug from unknown to routine: the memory-safety fault on malformed input, the parser that accepts what it should reject, the authentication path that behaves differently on the fourth attempt. Before fuzzing was normal those shipped and were found by incident. Now a build server finds them overnight, because the tools were free. The same applies to automated vulnerability discovery. A tool that finds bugs faster than people can lands on both sides at once, and which side it helps more is decided by access. Cheap and open, the defender runs it over their own code continuously and the bugs are gone first; expensive and licensed, the defender runs it quarterly if at all while the attacker runs it constantly. Our bet is that this becomes the central security question of the next decade, settled by distribution rather than capability. The chokepoint is neither a registry nor a key. It is the ability to classify a technique as a controlled export. Intrusion software has been within scope of the Wassenaar Arrangement for years. The first attempts to implement those controls were drafted broadly enough that publishing a proof of concept, or emailing a colleague abroad a capture tool, was arguably a licensable export. The community pushed back and the language narrowed, but the statute remains, applying to a category defined by what software does rather than what it is. The burden is where it bites. A large company has an export compliance function and can file for a licence; an individual researcher, a university group or a four-person consultancy cannot, and will decline to publish rather than find out. A control written to stop surveillance vendors selling to abusive governments lands hardest on independent research and barely touches the vendors, who have lawyers and sell under government licence anyway. The surveillance industry gets regulated into legitimacy and the open community into silence. Watch for the same move applied to model weights. The argument that a capable code-writing model is a dual-use export is already being made, and it is more plausible than the intrusion-software version because the artefact is one file with a clear owner. If it lands, open weights become a customs question and the only parties who may lawfully hold a frontier capability are three companies and several governments. The capability does not get rarer; it concentrates, the opposite of what the policy says on the tin. Here at AxisOps our position is that defensive capability has to be ownable: tools you can read, run on your own hardware and keep running when a licence lapses or a jurisdiction changes its mind. A regime in which you may inspect your own systems only while somebody\u0027s subscription is current is the thing to refuse, and export control is the likeliest route there. The choice that remains\u00b6 A judgement rather than a measurement: Coordinate disclosure. Report first, publish once a fix exists or a reasonable deadline has passed. The deadline matters: unbounded delay serves the vendor alone. Publish the defence alongside the attack. A detection rule, a mitigation, a configuration change. The most practical item here and the one most often skipped. Be straight about what you built. A tool described as a \"security scanner\" that is in fact an exploitation framework does nobody any favours, least of all you when somebody reads it. That list is least sufficient against the state and military case. Technology built for defence gets used offensively; technology built for legitimate law enforcement gets used against journalists. The commercial spyware industry is the clearest example, and the mechanism isn\u0027t deception: capability is fungible and the customer list changes. For one engineer the leverage is small but real. It sits in who you build for and what you will work on, a career decision rather than a technical one. Stated plainly rather than resolved, because we don\u0027t think it resolves. So: build the tool. Publish the defence with it. Be specific about the threat model it serves. And treat \"it would exist anyway\" as an explanation rather than an excuse: true, and still not an answer to what you chose to spend your time on.", "d": "2023-11-21", "g": ["policy", "security", "ethics", "open-source"], "t": "Dual use is the default, not the exception", "u": "insights/dual-use-and-the-engineer/"},{"b": "A platform boundary becomes more valuable than the product inside it. A platform launches open. Documented APIs, generous terms, third-party clients encouraged. It grows, because being easy to build on is how you recruit the ecosystem that makes you valuable in the first place. Then it closes. Rate limits appear, terms change, the API that third-party clients depend on gets deprecated and the data export turns awkward. No moral failing of particular companies is required to explain this: it\u0027s what the incentives produce once the ecosystem has been recruited and the switching cost paid. Why engineers should treat this as a design input\u00b6 Because the closing is predictable, which means you can design for it. The practical question when you adopt a platform isn\u0027t whether its terms are good today. It\u0027s what the exit looks like, and specifically: Can the data be exported in a documented format, completely, without negotiation? Is the interface we depend on a published standard, or is it this vendor\u0027s API? If the terms changed tomorrow, what\u0027s the recovery path and how long is it? Those have real answers, and the answers are usually worse than people assume, because nobody has tested them. Specify the exit before you sign\u00b6 The only moment you hold leverage over a supplier is before the contract exists. After that, the party you are buying from has a predictable future incentive to make leaving hard, and no obligation beyond what you wrote down. So write it down. Put the export format, its completeness and its cadence into the specification alongside the functional requirements, and treat an unexercised export clause as a promise rather than a capability. Exercising it annually as an acceptance test is what makes the difference; a fire drill beats a fire certificate. What this makes possible\u00b6 Specified boundaries do more than protect you from a supplier. Competition on the merits of the product. When the interface is public, a small supplier can offer a better implementation and a customer can take it without rebuilding their business. That is how we got competing web browsers, competing mail servers and competing SQL databases. Every market where you can change supplier in a fortnight is one where suppliers have to be good. The small team gets to participate. If messaging, identity, payments or social graph access are specified rather than proprietary, a four-person company can build a client or a specialist tool for a niche the platform will never serve: a reader for someone with low vision, a client that works over a 2G link, a tool for a language with twelve thousand speakers. None of those are commercially interesting to a platform with a billion users, and all of them matter enormously to the people who need them. An open boundary is an accessibility policy by other means. Public institutions stop being hostages. A hospital, a council or a school that can move its records between systems can replace a supplier that has become expensive or negligent. Today it frequently cannot, which is why public bodies run software everyone involved knows is inadequate. Specified data portability is the cheapest improvement available to public procurement. And the engineering benefit: an interface with more than one implementation is an interface whose spec gets tested. Single-implementation standards drift into \"whatever the vendor does\", and every ambiguity becomes a production bug years later. The interoperability mandates turning up in digital-markets regulation are an attempt to make the boundary a public property rather than a private one: messaging interoperability, data portability, access to platform functions on non-discriminatory terms. Whether they work is an open question, and the engineering concern is a narrow one. A mandated interface that is technically compliant and practically unusable satisfies the regulation and achieves nothing. An API with a compliant schema, punitive rate limits and no stability guarantee meets the letter of the thing. Which makes the specification the battleground\u00b6 The quality of the specification is the whole game, so look at who is in the room. A standards working group is staffed by whoever can afford to send engineers to it for three years: the three or four largest implementers of the thing being standardised, plus a few academics and the occasional underfunded nonprofit. Those implementers have the strongest commercial interest in the specification being ineffective. They need not obstruct anything; they need only be patient, argue for optionality and win on detail. The tells are consistent, and you can read them off a draft: Everything important is optional. A MAY where there should be a MUST is a clause that will never be implemented by the one party whose implementation matters. No conformance suite. A standard with no executable test of compliance cannot be failed, and therefore cannot be enforced, whatever the statute says. Extension points with no registry. The documented escape hatch through which a vendor re-establishes its proprietary interface inside the compliant one. Performance unspecified. Correct, slow and rate-limited is a complete defence. Privacy is the stick used to beat every interoperability mandate. Opening a boundary does expose data to more parties, and \"we cannot do this safely\" is a real argument that is also very convenient for an incumbent. Encrypted messaging interoperability is hard and needs its protocol work done in public. A platform arguing for user privacy while running the largest behavioural advertising operation on earth is making a commercial argument in a safety costume. Both halves are true. Watch too for the mandate that entrenches rather than opens. A rule that only the largest platforms must comply, implemented through a certified-partner programme with an approval queue, becomes a licence scheme the incumbents administer. Non-discriminatory terms mean nothing if only a funded company can complete the onboarding. Protocols with multiple independent implementations and no corporate owner do exist, and they are short of people and money rather than short of ideas. An interface you cannot reimplement is a dependency rather than a standard. That is why we specify our own boundaries and publish them. The interoperability you offer is a commitment about your own future behaviour, and it is the first thing a later owner of the company will withdraw quietly. Writing it into the product as a documented format and a complete export, tested in CI, makes it harder to remove than a promise in the documentation.", "d": "2023-10-05", "g": ["interoperability", "standards", "policy", "open-source"], "t": "Interoperability is the feature", "u": "insights/interoperability-beats-features/"},{"b": "The arithmetic that used to govern this is gone. A guided anti-tank missile is an expensive, export-controlled object built by a handful of companies. An FPV quadcopter is a hobby aircraft, a commodity flight controller, an analogue video link and an explosive charge fastened on with tape. Reporting from the current war puts those in the low hundreds of pounds each, against targets costing a thousand times more. Both Karabakh in 2020 and Ukraine since have been read as drone wars, and the larger systems got the headlines, but the cheap end is where the cost inversion happened. The interesting question for this notebook is which component was ever the hard one. The parts, and which of them was difficult\u00b6 Airframe: commodity. Motors and batteries: commodity, driven by a consumer market that has nothing to do with weapons. Radio: commodity. Explosive: the oldest technology in the stack and not the constraint. The hard part was always the terminal guidance. Getting an object onto a moving target in the last few seconds is the thing that justified the price of a guided weapon, and it required either a trained operator holding a lock or a seeker head with real engineering in it. That is the part that has moved. Visual tracking of an object across frames at useful frame rates now runs on an inference accelerator costing tens of pounds, with models and training pipelines that are public because they were built for inspecting crops and watching car parks. A terminal guidance problem has become a computer vision exercise with an off-the-shelf answer. This is the pattern this notebook keeps finding in other domains: the clever part migrates out of bespoke hardware and into software, and the hardware underneath becomes a commodity. It happened to networking. It happened to sequencing. It has happened here, and nobody involved intended the dual-use consequence. What it costs\u00b6 The attrition logic that follows is brutal and, from a purely engineering standpoint, correct. If the weapon is cheap enough you do not need reliability, you need volume, and a thirty per cent success rate at three hundred pounds a unit beats a ninety per cent rate at three hundred thousand. Every quality argument our trade makes about building things properly runs backwards in this one domain, which is worth sitting with. The deeper cost is that the barrier to entry has fallen through the floor. Precision strike used to be an attribute of states with a defence industrial base. It is now within reach of any organised group with a budget, a soldering iron and an internet connection. There is no export control that addresses this, because the controlled item is a quadcopter you can buy in a shop and a model you can download. What this makes possible, which is not all bad\u00b6 The honest version of this argument has to include what the same technology does when it is not carrying a warhead, because it is the identical stack. The autonomy that flies a strike drone flies a search and rescue aircraft over a mountainside at night, finds a casualty by thermal signature and does it without risking a crew. The same airframes map flood damage, inspect infrastructure nobody can safely climb and deliver blood products across a country with no roads. Those are running deployments, not projections, and they exist because the components got cheap for military reasons. There is also a defensive argument worth making plainly. A cheap precision weapon is an asymmetric one, and asymmetry favours whoever is defending territory against a larger force. A country that cannot afford an air force can now impose real cost on an armoured column. Whether that is good depends entirely on who is invading whom, which is the problem with saying anything general about dual-use technology. Where the loop closes\u00b6 The part that matters, and the reason this is an engineering question rather than a policy one. Today most of these systems have a person in the loop, holding a view through goggles and steering until impact. The human is there partly for ethics and largely because the link usually works. Both reasons are eroding. Electronic warfare is making the link unreliable, and the engineering response to an unreliable link is to push more decision-making onto the aircraft. Terminal autonomy, where the operator designates and the machine finishes the job after the link drops, is the obvious next step and is a software change to systems already flying. Nobody can currently check the claim. \"Human in the loop\" is an assurance about a system\u0027s behaviour, and there is no mechanism by which an outside party can verify it. You cannot tell from the outside whether a given airframe had a person authorising the final second or a model doing it. You cannot tell afterwards from the wreckage. The distinction on which the legality of these systems is being argued is, at present, an unverifiable claim made by the operator. This lab has spent a lot of words on attestation and reproducible builds: on proving what code is running on a machine. That work has an obvious and uncomfortable application here. An attested autonomy stack, where a system can prove which model and which policy it was running at a given moment, is technically feasible with mechanisms that exist. Nobody is building it, because no operator wants their weapons to produce evidence. Who ends up owning it\u00b6 The models are open, the airframes are commodity, and the supply chain for small drones runs substantially through one country, which has already demonstrated that it will restrict exports when it suits. A military capability resting on consumer parts from a strategic competitor is a strange position, and it explains the scramble to build domestic drone manufacturing everywhere at once. The second chokepoint is the inference silicon, which is a much shorter list of suppliers than the airframes, and already subject to export control for unrelated reasons. Autonomy capability tracks access to accelerators, and access to accelerators is decided by export policy written for a different purpose. The ugly part is the civilian side. The counter-drone systems now being installed around airports, stadiums and borders are a detection and identification infrastructure pointed at the sky, and they do not distinguish between a weapon and a camera. The capability to find, classify and down a small aircraft in a city is being built for sound reasons and will be available for other ones. Here at AxisOps our position is that this is the clearest case we know of where a defensive capability and an offensive one are the same artefact, and that the right engineering response is verification rather than refusal. If autonomy in weapons cannot be prevented, and we do not think it can, then the thing worth building is the apparatus that makes claims about it checkable. That is infrastructure work of the kind this lab does, and nobody is funding it. Where this breaks\u00b6 Whether attested autonomy would ever be adopted without a treaty requiring it, and whether any such treaty is reachable while the capability is this cheap and this widely distributed. The technical problem looks tractable. The political one does not.", "d": "2023-09-12", "g": ["ai", "inference", "hardware", "policy", "ethics"], "t": "The expensive part stopped being the munition", "u": "insights/cheap-end-of-the-targeting-loop/"},{"b": "A software bill of materials is an inventory. Every component, its version, its origin and how it got there. It is now a compliance requirement in several jurisdictions, which means a great many organisations produce them and a rather smaller number use them. What it does do\u00b6 It answers \"where is this running?\" quickly. Which sounds modest until you remember Log4Shell. The exploit and the mitigation were never the hard problem that December. The hard problem was that nobody could enumerate where the vulnerable library had got to, because it arrives transitively, shaded into fat JARs, bundled inside appliances nobody has the password for. Our own answer at the time was a directory walker: a tool that goes out looking, because the estate couldn\u0027t be asked. An SBOM is the difference between searching the house and reading the inventory. When the next disclosure lands, nothing else will get you an answer in minutes. What it does not do\u00b6 The overselling, plainly: It won\u0027t tell you whether you\u0027re exploitable. The vulnerable component can be present with the vulnerable code path unreachable. That\u0027s what VEX exists to express, and VEX adoption lags SBOM adoption considerably. It won\u0027t tell you the component is what it claims to be. An SBOM records identity and version as asserted by the thing it describes. Give a hostile party commit access and the malicious release is a current, correctly-signed version, because that is what it is. event-stream in 2018 was a handover of maintainership to a stranger (npm), though its payload arrived through a newly added dependency, flatmap-stream: the kind of new edge a regenerated inventory would at least have surfaced, which makes it the weaker example here. SolarWinds in 2020 compromised the build, so the vendor\u0027s own signature attested a backdoor (CrowdStrike). Every downstream inventory was accurate. Nor does an SBOM stop a typosquat landing in the tree, the dominant way hostile code enters PyPI and npm today: once it is in your lockfile it is in your inventory, correctly listed under its own name. It\u0027s only as good as its generation. From a lockfile it is accurate about declared dependencies and silent about vendored code, statically linked libraries and anything somebody copied into the tree. From a scan of a built artefact it catches more and is noisier. It goes stale immediately. Produced at release and never regenerated, it describes a system that may have stopped existing. The gap nobody likes discussing\u00b6 The useful SBOM is the one for what\u0027s deployed right now, which is frequently not what was built: base image updates, sidecars, runtime-installed agents and the things people added at three in the morning during an incident. Producing an SBOM at build time is largely solved. Maintaining an accurate inventory of running systems is the harder, less automated problem that no auditor asks about, and the one that decides whether you can answer the question when it matters. What a real inventory makes possible\u00b6 Take the inventory seriously and expensive things become routine. Disclosure response collapses from days to minutes. When a CVE lands you run a query and start fixing. The uncertainty beforehand, during which nobody can tell a customer whether they are affected, is where almost all the cost of Log4Shell sat. Remove the uncertainty and you remove the weekend. The smallest supplier gets to compete. An inventory is the cheapest credible answer to \"prove your software is safe to buy\", cheap in a way an audit never is. A four-person software house shipping a regenerated SBOM with every release can satisfy a procurement question that previously needed a compliance department. That widens who gets to sell into regulated sectors, and it is the part of this mandate we support. Maintainer effort goes where it is needed. Aggregate inventories tell you which libraries hold the world up, as opposed to which have the most GitHub stars. That is the input a funding programme needs and nobody has had it. If this mandate directs money at under-resourced projects on the critical path it will have paid for itself. And a class of toil disappears. Every scanner, every reconciliation spreadsheet, every \"which version of OpenSSL is on that appliance\" email thread exists because the estate cannot be asked. Declare the deployment and the inventory is a by-product. The technical work leaves a political choice. There are two main formats and several competing attestation schemes above them. Whoever\u0027s format regulators name in statute sets the terms for everybody, and the organisations with people in those working groups are the ones who can afford to send people to working groups. That is not a conspiracy; it is who turns up. The risk, already visible in how this is being sold, is compliance as a moat. The mandate says \"produce a bill of materials\", which is cheap. What gets marketed is a platform: continuous generation, a signed attestation chain, a managed repository of other people\u0027s inventories, priced per artefact per month. A large vendor absorbs that and puts the badge on the brochure; a small software house cannot, so regulated supply chains consolidate around whoever can clear the paperwork bar. That makes the next incident worse, because too much of the world resting on too few components is the thing we are supposedly protecting against. So the specification is the whole game. The tests to apply: can a one-person project generate a compliant document with free tooling in under an hour, and can a buyer verify it without a subscription? A no to either means the format has been captured, whatever its licence says. There is a privacy edge that gets no attention. An SBOM maps what a target runs and at which versions, so shared widely it is a reconnaissance document. Treat it as sensitive operational data, share it under agreement and be wary of any central repository proposing to collect everybody\u0027s. Here at AxisOps we generate inventories from the declaration that deploys the system, keep them in a plain open format and regenerate them on every build. It should be readable with grep twenty years from now, because an inventory you need a vendor\u0027s platform to interpret is not one you own. Inventory first, control second\u00b6 An SBOM is a prerequisite rather than a control. It makes a class of question answerable; answering it is still your job. So treat the compliance artefact and the operational capability as two different things. An organisation can comfortably have the first without the second, and the audit won\u0027t tell them apart.", "d": "2023-06-21", "g": ["supply-chain", "security", "sbom"], "t": "An SBOM tells you what you have, not whether it is safe", "u": "insights/sbom-is-a-starting-point/"},{"b": "\"The three pillars of observability\" is a marketing frame that got repeated until it became a design principle. It describes three data types somebody can sell you storage for. It describes no property of a system whatsoever. The real property is narrower and a lot harder: can you answer a question you did not think to ask in advance? Why the distinction bites\u00b6 A metric is a pre-aggregated answer to a question somebody already had. It is cheap, it is fast, and it will tell you nothing about a dimension nobody thought to label. Here at AxisOps we see the same failure repeatedly. Your p99 latency is up. The metrics tell you that, and stop. They cannot tell you it is customers on one shard, in one region, on one API version, unless somebody anticipated that combination and added those labels. Had they anticipated it, they would probably have fixed it before you noticed. Cardinality is why you cannot label everything and have done. High-cardinality labels (user IDs, request IDs) blow metric storage out, so the dimensions most useful for debugging are the ones metrics structurally cannot hold. Rather like keeping a tally chart of how many cars left the car park and then being asked which one had the dent in it. What answers unanticipated questions\u00b6 Wide structured events. One event per unit of work, carrying every dimension you have as structured fields: user, region, shard, version, feature flags, timings of each phase. Then you query it however you like, after the fact, without deploying anything. That isn\u0027t a third pillar to stand alongside the others; it swallows most of them. Gauges, hardware counters and other continuous state are not per-request events and keep their own path. Metrics are aggregates over events, and traces are events with causal links between them. What it costs you is storage and ingestion, and sampling is how you manage that. Tail-based for preference, so you keep every slow and failed request and throw away the boring ones. Where kernel tracing changed the economics\u00b6 The newer in-kernel programmable tracing (eBPF programs attached to tracepoints, kprobes and the networking hooks) made it possible to collect a good deal of this without instrumenting the application at all. Syscall latency, TCP retransmits, block I/O queue depth, page faults, scheduler run-queue delay: all of it observable from outside the process, with no code change, at an overhead low enough to leave running in production. That matters most for the things you can\u0027t instrument; a vendor binary, a runtime you don\u0027t control or a problem in the layer below your code. It\u0027s also the answer when somebody says \"we can\u0027t add instrumentation to that\". What it won\u0027t give you is application semantics. The kernel can tell you a syscall was slow. It can\u0027t tell you whose invoice run it was part of. The two are complementary, and the sales pitch for each tends to leave that bit out. What this makes possible\u00b6 Treat telemetry as queryable events rather than three pre-bought answers, and the shape of an incident changes. The investigation finishes in the same session. Today\u0027s debugging loop for anything unanticipated is: form a hypothesis, add a log line, deploy, wait for the problem to recur, read the log, repeat. Each turn is hours at best and days if the fault is intermittent. Query a wide event store and each turn is twenty seconds. An engineer who can test forty hypotheses in an afternoon solves problems that are out of reach when they can test three in a week. Debugging stops requiring the person who wrote it. When the data is in fields rather than prose log lines, anybody can ask the question. That breaks the pattern where every hard incident waits for one named individual who knows which grep to run. Performance work becomes evidence-led. With per-request timings across every phase, the slow thing is a query result rather than a committee opinion. We expect most of the commercial value to land here: saying which customers, on which path, are paying for which bit of latency, and therefore which engineering week is worth spending. Energy and hosting cost are on the same curve; you cannot cut what you cannot attribute. The frontier bit: a system that can be interrogated after the fact without redeployment is one step away from one that can explain itself. Given wide enough events and a way to correlate them, \"what is different about the failing requests\" is a statistical question over structured data rather than a human reading dashboards. We expect that to be automated within a few years, and the teams who get it will be the ones already storing events rather than pre-aggregated counters. The protocol does not trap you here. The pricing model does. Wide events mean volume, and the dominant commercial model charges on ingest and retention: per gigabyte, per host, per million events, per custom metric. Your bill scales with your traffic, so the day you most need the data is the day it costs the most. Teams respond by sampling harder, dropping fields and shortening retention to thirty days, then discover the intermittent fault they are chasing has a six-week period. The pricing has quietly specified the engineering. Worse, telemetry you paid to ingest frequently cannot be got back out. Query languages are proprietary, dashboards are not portable, export is per-query or capped or billed again, and historical data sits in a store you have no bulk read access to. You have paid, repeatedly, for a dataset you do not possess, and switching vendor means starting your history from zero. Which is why nobody switches. Then consider what that dataset contains: user identifiers, IP addresses, request paths, feature flags, query parameters. Your observability store is the most detailed behavioural record of your users anywhere in your estate, shipped continuously to a third party, usually with no reviewed retention policy and often to a jurisdiction nobody checked. It is a data-protection surface and almost nobody treats it as one. Scrub at the point of emission, because after that it has already left. The open side is in better shape than in most of these stories. Columnar storage on object storage, OpenTelemetry for the wire format, Prometheus for metrics and a handful of credible open query engines make a self-hosted wide-event store a real option for a mid-sized estate; the gap is operational polish, not capability. Whatever you buy, insist on an open wire format at emission so the agent is not the lock-in. Insist too on raw bulk export as a contract term you have exercised and a copy of your own events landing in storage you own on the way past. Here at AxisOps we keep the originals. Telemetry is a record of how your systems and your customers behave, and renting the only copy of it is a strange thing to do with the most useful dataset you own. The practical test\u00b6 Instead of auditing whether you\u0027ve collected all three pillars, try this. Take a real incident from the last quarter and ask whether the data you already hold would have answered it without anybody deploying anything. If the answer is \"we added some logging and waited for it to happen again\", you haven\u0027t got observability. You\u0027ve got monitoring, which is a different thing and also a useful one.", "d": "2023-03-14", "g": ["observability", "ebpf", "operations"], "t": "Observability is not three pillars", "u": "insights/observability-not-three-pillars/"},{"b": "A modern pacemaker runs firmware. So does a cochlear implant, a deep brain stimulator, an insulin pump and a continuous glucose monitor. They have update mechanisms, radio interfaces, signing keys and a vendor with a roadmap. This trade has thirty years of accumulated practice for running computers: patch promptly, test the rollback, keep a maintenance window, replace the hardware every few years. Work through that list for a device sutured to somebody\u0027s heart and all four assumptions fail at once. Where the model breaks\u00b6 There is no maintenance window. The device is doing its job continuously, and its job is keeping somebody alive. A firmware update to a pacemaker is performed with a clinician present, with a plan for what happens if the device does not come back. In 2017 a manufacturer recalled roughly half a million implanted pacemakers for a firmware fix, which was administered at a clinic, one patient at a time. There is no fleet-wide push. Rollback is theoretical. The entire discipline assumes you can return to a known-good state. If a cardiac device misbehaves after an update, the recovery procedure may be a surgical one. Ship an update you cannot withdraw and the blast radius is measured in people. End-of-life is not a date you choose. Implants last ten to twenty years. Vendors do not. A company that is acquired, pivots or folds leaves devices running in bodies long after the last engineer who understood them has gone. The retina implants whose manufacturer discontinued support are the case everyone cites, and they will not be the last, because the economics are structural rather than accidental. The threat model includes the patient. People have asked for the data their own implant produces and been refused. Researchers have been threatened for analysing device firmware. The security model is written around a patient who is a potential adversary, which is a strange thing to discover about the computer in your chest. What it costs\u00b6 Precision, which is the expensive part. The failure mode of a sloppy embedded system is a reboot; here it is a person. That justifies the regulatory burden, the long certification cycles and the conservative engineering. Anyone arguing for faster updates in this domain should have to explain what they would do about the first fatality. The real cost is the one that falls out of that caution: devices stay vulnerable for years because the process to fix them safely is slow. That is the correct trade, made in the only sane direction. It is also the reason the ownership question below matters more here than anywhere else in this notebook. What it makes possible\u00b6 Set the problems aside for a moment, because the capability is extraordinary and increasing. A cochlear implant turns profound deafness into usable hearing. A deep brain stimulator stops a tremor that no drug touches. A closed-loop insulin system does continuously what a person with type 1 diabetes otherwise does by hand a dozen times a day, every day. These are not augmentations of healthy people; they are repairs, and the repairs work. The direction of travel is towards more of them, with more compute inside them, doing more inference locally. A closed-loop system has to decide without a network, because the network is not there when you are asleep in a field, and the only sensible place for that model is on the device. Which means implanted devices become edge inference nodes, with every property of local inference, good and bad, applying to hardware that is inside a person. Further out, the interesting line is bidirectional interfaces: devices that read neural signals rather than only stimulating. Restoring movement after spinal injury, restoring speech after a stroke. The engineering is early and the experimental results are real. Within a decade or two this is a field with working products rather than demonstrations. And then the obvious thing happens, which is that people who are not ill want them. The boundary between repairing a deficit and exceeding a baseline is a social one, not a technical one, and it will be argued about by people with no engineering stake in it. What this lab can say is narrow: the engineering does not distinguish. A device that restores hearing and a device that extends it are the same device with different parameters. Who holds the key to the thing inside you\u00b6 This is where it gets uncomfortable, and it is the sharpest version of an argument made repeatedly here about signing keys. An implanted device will only accept firmware signed by its manufacturer. That is correct engineering: the alternative is a device that accepts firmware from anybody, which is indefensible. But it means the manufacturer holds the sole authority over a computer inside a person\u0027s body, for the life of the device, which may be longer than the life of the company. The patient owns the object and cannot read its code, cannot audit its behaviour, cannot fix it and cannot authorise anybody else to. The data position is similar. Telemetry flows to the vendor. Patients have had to fight for access to their own cardiac records. Meanwhile the data has obvious commercial value, obvious insurance value and obvious value to a court. An implant is a witness that never stops recording, and the legal process for compelling that record is being established case by case, mostly without the people wearing the devices in the room. There is no clean answer and we are not going to pretend otherwise. Patients cannot audit firmware. Mandated escrow has its own failure modes. What is achievable and not yet normal: a legal right to your own device\u0027s data in a documented format; an obligation on manufacturers to deposit documentation and keys so that a discontinued device can be maintained by somebody else; independent security research protected rather than prosecuted; and a published support horizon, so a patient knows at implantation when the computer in their chest stops being maintained. Here at AxisOps we build infrastructure people can own, inspect and rebuild, and we think that argument does not weaken as the hardware gets closer to the body. It gets more urgent. The most important computer a person will ever depend on may be one they cannot switch off, and the current answer to who controls it is a company. The question we cannot answer\u00b6 Whether any of this is fixable through procurement and regulation, or whether the safety case requires a single controlling vendor. We suspect the first and cannot rule out the second, and the people best placed to settle it are the ones with the commercial interest in the answer.", "d": "2022-09-08", "g": ["hardware", "security", "operations", "policy", "identity"], "t": "An implant has no maintenance window", "u": "insights/implant-no-maintenance-window/"},{"b": "Immutable infrastructure: don\u0027t modify a running system. Build a new image, deploy it, throw the old one away. The technical mechanism is simple enough. The benefit comes from somewhere else entirely. The problem it solves\u00b6 Configuration drift. A server that\u0027s been patched, tuned and fixed by several people over three years is in a state nobody has written down and nobody can reproduce. That matters most at the worst possible moment. The machine dies, and you discover that the thing you can rebuild isn\u0027t the thing that was running. The difference is all the accumulated undocumented fixes that made it work in the first place. Configuration management (Puppet, Chef, Ansible) attacked this by continuously converging a machine towards a declared state. That helps a great deal, but it leaves a residual problem: convergence only covers what the manifests mention. Anything outside them persists invisibly, and so does anything a person did by hand between runs. Immutability removes the residue by removing the machine\u0027s history. If the machine never changes after boot, there\u0027s no drift to accumulate. Why the benefit is social\u00b6 The technical half (build an image, deploy it) is the easy half, and plenty of organisations have it and still have drift. The property that delivers the value is a commitment: nobody logs in and changes anything, ever, including at three in the morning during an incident. That\u0027s a hard commitment to keep, because at three in the morning the fastest route back to service is frequently a manual change. Keeping the discipline means accepting a slower fix in the moment to preserve a property that pays later. It\u0027s the same bargain as not taking the shortcut across the lawn; one person doing it once is nothing, and that\u0027s how the path gets worn. Where organisations fail at immutable infrastructure, it\u0027s nearly always here and not in the tooling. One emergency sudo and the guarantee is gone, with nothing to announce it. It is the same failure mode as the declarative model generally. What it costs you\u00b6 Deployment has to be fast and boring, or the discipline won\u0027t survive. If shipping a one-line fix takes forty minutes, people will log in. Debugging changes shape. You can\u0027t leave a debugger attached to something you\u0027re about to replace, so the data has to come out of the system while it runs. Plan for that before you need it, not during. State has to live somewhere. Immutability applies to compute. Databases and storage are as mutable as they ever were, and pretending otherwise is how people end up with an immutable web tier and an undocumented pet database in the corner. Image sprawl is real and wants its own lifecycle management. What it makes possible\u00b6 Once a running system is a build artefact rather than a history, things that were previously projects become routine. Recovery stops being a rehearsal nobody has run. The disaster-recovery plan and the ordinary deploy become the same procedure, executed daily. That is the only version of DR that works, because the only tested path is the one you use all the time. A business that can rebuild its entire serving tier from source in twenty minutes does not need a site-failover runbook with eleven manual steps. It needs the pipeline it already has, pointed somewhere else. Patching becomes cheap enough to do constantly. The reason estates run months behind on security updates is that patching a long-lived machine carries real risk of breaking it in a way nobody can undo. Rebuild instead and the risk moves to a place you can test, with a rollback that is \"deploy the previous image\". Teams that do this patch weekly rather than quarterly, which is worth more than most security products you could buy with the same money. The audit question gets a real answer. \"What is running in production, and what source produced it?\" is answerable by looking at an image digest and a commit. On a drifted estate that question takes a week and the answer is a guess. For anyone in a regulated sector this converts a recurring compliance ordeal into a query. And the one engineers feel immediately: the class of bug that only exists on one machine disappears. No more \"it fails on web-03 and nobody knows why\", no more three-year-old kernel module somebody installed by hand, no more afternoons diffing two servers that are supposed to be identical. That toil goes to zero, and nobody misses it. All of those gains now depend on the image pipeline. Immutability means your production estate is defined by an artefact built somewhere. Ask where, and the answer for most organisations adopting this pattern is: a hosted build service, from a base image pulled from a public registry, stored in a cloud provider\u0027s own image format, deployed by that provider\u0027s own machine-image API, through a managed pipeline product. Now ask the only question that matters about any dependency: if that account were closed tomorrow, could you still produce a bootable machine? For a great many estates the answer is no, and the reason is not the application code. It is that nobody holds the base image, nobody has the build running anywhere else and the image format is the provider\u0027s. This is a worse lock-in than the mutable estate it replaces, in one specific way. A hand-tuned server was undocumented, but it was yours, and you could at least take the disk. An estate whose every machine is derived from a pipeline you rent is a tenancy. The provider has done nothing wrong; you simply no longer possess the means of production. Treat the pipeline the way you would treat a single-sourced component. That means a base image you build yourself from a distribution\u0027s published packages, mirrored locally. A build that runs on a machine you own as well as in the hosted service, and that is proven to run there, not assumed to. An output format that boots under plain KVM and not only in one cloud. And the whole thing exercised on a schedule, because an untested rebuild path is a promise, not a capability. It is the same reason you test the fire alarm rather than admiring the box on the wall. Here at AxisOps we build images from distribution packages, in a container we control, and we boot the result on our own hardware before it goes anywhere else. Rebuilding without the vendor is a practical requirement rather than a preference. It marks the difference between owning infrastructure and renting the right to run it, and immutability is the technique that makes the distinction concrete: your estate is now exactly as portable as your build is. Keep the promise or keep the complexity\u00b6 It\u0027s a good idea whose benefit is roughly proportional to how strictly you keep it, and it degrades to ordinary infrastructure plus extra build steps the moment you don\u0027t. Adopt it if you\u0027ll hold the line, and build the pipeline somewhere you can still reach it when the invoice changes.", "d": "2022-07-19", "g": ["infrastructure", "operations", "deployment"], "t": "Immutable infrastructure, and the discipline it requires", "u": "insights/immutable-infrastructure/"},{"b": "Qualys disclosed PwnKit on 25 January 2022 (Qualys). Any local user can become root through pkexec, and every version since the first in May 2009 is affected. On a multi-user box that is as bad as it reads. The advisory carries a workaround for hosts with no patch available: remove the setuid bit, chmod 0755 /usr/bin/pkexec. One command, no reboot, effective immediately. Red Hat\u0027s bulletin answers the same question differently. Asked whether it is safe to remove the setuid bit, it says Red Hat does not recommend it, \"as it will cause breakage on the system\", and names Qualys\u0027s advisory while declining its advice (Red Hat). Red Hat offers a SystemTap probe instead, which has to be reloaded at every boot and does not work under Secure Boot. Both are right, which is the problem\u00b6 The disagreement is not about the facts. Removing the setuid bit does stop PwnKit, and it does break pkexec for everything legitimate. The two vendors are weighting those differently because they are picturing different machines. On a headless server nobody has logged into interactively since it was provisioned, pkexec is a liability with no users. Take the bit off and the only thing that notices is the vulnerability. On an engineer\u0027s desktop, or anything running a graphical administration tool, pkexec is how software asks for privilege, and removing it breaks the machine in ways that generate tickets. So the correct action is per host rather than per fleet, and that is a decision no advisory can make for you because neither vendor knows what your boxes are for. What the role does with that\u00b6 Ours is at github.com/axisops/CVE-2021-4034, MIT, and you are welcome to it. The ordering is the whole design: Upgrade polkit first. policykit-1 on Debian and Ubuntu, polkit on the Red Hat family. A host that can be patched gets patched and keeps a working pkexec, which means the workaround question never arises for it. Then compare the installed version against the distribution\u0027s fixed version, read from dpkg-query or rpm. Only then, and only where polkit is still unpatched, remove the setuid bit. Record the original mode before changing it, so --restore can put it back exactly. Dry run by default. Nothing changes without --apply. The workaround is the fallback, not the fix, and the role is built so that the set of hosts receiving it is as small as the estate allows. On a fleet where everything can be patched today, the destructive path never executes. The failure mode nobody warns you about\u00b6 A later polkit package update restores the setuid bit. That is the detail worth carrying away from this whole incident. If the update that restores the bit is not itself the fixed version, the host is quietly exposed again, and nothing announces it. The mitigation was correct when it was applied and is undone by routine maintenance. That is the most dangerous shape a control can have: it looks done, it is recorded as done and it is not. The role handles it by reporting a returned setuid bit on every subsequent run and keeping the original mode it first recorded, so re-running after package updates is both safe and informative. But this workaround needs a calendar entry, not just a playbook run, until every host is patched. What it does not do\u00b6 Published with the code rather than found later by somebody else. The package verdict is per host, so a pkexec outside the package, built into /usr/local/bin or shipped inside an application, is judged against the host\u0027s polkit package, whose version may say nothing about it. Red Hat\u0027s EUS and AUS streams carry their own fixed versions, but the version table is keyed on the major release only. A correctly patched EUS host is therefore reported as vulnerable and needs checking by hand against its own advisory. The role does not deploy Red Hat\u0027s SystemTap mitigation. If you are on RHEL and you accept Red Hat\u0027s reasoning over Qualys\u0027s, this tool is not the one you want, and we would rather say that plainly than imply a coverage we do not have. It also contains no exploit code and makes no attempt to test exploitability; it reads versions and file modes and reasons about them. The part that generalises\u00b6 A vulnerability with a one-line workaround is not a one-line job, because the line is only correct on some of your machines. Everything expensive about this week was classification: which hosts can be patched now, which cannot, which of those have interactive users and which will have the bit restored under them by an update three weeks from now. The chmod is instant. Knowing where to send it is the work, and an estate that can answer that from a declaration rather than a survey does this in an afternoon instead of a fortnight.", "d": "2022-01-31", "g": ["security", "ansible", "linux", "python", "mitigation"], "t": "Two vendors, one workaround, opposite advice", "u": "insights/pwnkit-setuid-bit/"},{"b": "Log4Shell went public on 9 December 2021. Six days later we published an Ansible role that walks application directories, finds every archive containing the vulnerable class, backs it up and rewrites the archive without it. It is at github.com/axisops/CVE-2021-44228, MIT, and you are welcome to it. The mitigation was right and the timing was ahead of the official guidance. The six days are the interesting number, because almost none of them went on the fix. The fix is a single deleted entry in a zip file. They went on answering \"where is Log4j running?\", and that is a question about your own estate, not about Apache\u0027s. Why strip a class instead of upgrading\u00b6 The decision that looks obvious now was nothing like obvious that week. Apache\u0027s first fix, 2.15.0, shipped on 10 December. It was incomplete: in non-default configurations an attacker controlling Thread Context Map input could still reach the lookup, which became CVE-2021-45046. 2.16.0 removed message lookups entirely. Two releases in a week, each the definitive answer until it wasn\u0027t, and nobody tonight would bet against a third. Meanwhile an upgrade means a dependency bump, a rebuild, a test cycle and a release per application, across every application you own, several of which are maintained by people who left. Removing the class is a change to a file on disk. It needs no rebuild, no rerelease and no cooperation from anyone, and it works identically on an application whose build system has rotted. Deleting the class is taking the handle off a door you cannot lock. It is not the repair. It stops anyone walking through while you arrange one. We had it on the 15th, which is worth exactly as much as it sounds: we read the same mailing lists as everybody else and the right answer was visible to anyone looking. One side effect our README did not name: removing JndiLookup breaks anything that legitimately performs JNDI lookups, JndiContextSelector and JMSAppender among them. If an application used either, this role broke it. \"Confirm your applications still start\" is not the same as telling an operator what specifically might not. The shape of the tool\u00b6 The code is a hundred lines and the interesting thing about it is its structure, not its cleverness. def strip_jndi(jar_path: Path, backup_dir: Path) -\u003e bool: backup_path = backup_path_for(jar_path, backup_dir) # mirrors the full path shutil.copy2(jar_path, backup_path) rewritten = jar_path.with_suffix(jar_path.suffix + \".mitigating\") with zipfile.ZipFile(jar_path) as source: with zipfile.ZipFile(rewritten, \"w\", zipfile.ZIP_DEFLATED) as target: for item in source.infolist(): if item.filename == JNDI_CLASS: continue target.writestr(item, source.read(item.filename)) rewritten.replace(jar_path) Back up before mutating, and mirror the source path into the backup directory: an estate has log4j-core-2.14.1.jar in a dozen places, one per service, and keying the backup on the basename sends every one of them to the same destination. Write to a new file and rename over the original, so a process killed mid-run leaves a working JAR rather than a truncated one. Log every action to a file, so the operator ends up with a record of which archives on which host were modified: what you need the following week when somebody asks. Catch the per-archive failure and carry on, because one unreadable file must not stop the other two hundred. Signed archives are the awkward case. This rewrite leaves the signature metadata in place, and a verifier may still accept the signatures on the entries that remain; verification is not an inventory check (Oracle\u0027s JAR specification). Then the part that matters most, and the one easiest to get wrong under pressure: say what you did not check. A WAR or an EAR with JARs inside it, or a Spring Boot fat JAR with BOOT-INF/lib/log4j-core.jar, has no matching entry at the top level. Walk the filesystem for archives and those are skipped, on exactly the packaging formats a Java estate is most likely to use. A log full of \"could not inspect\" invites investigation. A log that quietly omits a format reads as a clean bill of health. The role names every nested archive it cannot open and prints a count, because a tool\u0027s confidence has to be calibrated to its coverage, and at three in the morning nobody reads the source to find out what it skipped. The question underneath\u00b6 The reason this tool is a directory walker is the thing really worth taking away. If you can answer \"where is Log4j running?\" from a declaration, you do not go looking for JARs at all. You patch the declaration. The entire design here, walk the filesystem, inspect what you find, hope the directory list is complete, is an admission that in December 2021 that answer was discoverable but not declared. We went round the house trying every door because nobody had drawn the floor plan. The useful prediction from that December is how cheap the hard part becomes once the estate can answer questions about itself. A small team responds as fast as a large one. A team with a declared estate patches faster than a much larger one working from a wiki, because the work is a query and an apply rather than an archaeology project. That inverts the usual assumption that incident response is something only a big security team does properly. The out-of-hours call changes shape. The expense of a disclosure sits in the hours before the fix, during which nobody can tell a customer whether they are affected. The zip -d itself takes a second. Remove the uncertainty and you remove most of the cost, including a team\u0027s weekend spent producing an answer a declaration would have given in a minute. A whole category of tooling stops being necessary. This role exists because the estate could not be asked. Every directory walker, every scanner, every inventory reconciliation script is work that evaporates when the deployment description is the inventory. Not a faster search. No search. Our bet is that this is where the next few years of operations work goes, and that it arrives driven by incidents rather than by architecture reviews. The estate that can be queried will stop being a sophistication and start being a baseline, in the same way that version control did. Who was actually holding the ceiling up\u00b6 The uncomfortable part of Log4Shell was who had been carrying the bug, not the bug itself. Log4j is a small Apache project. The people who shipped 2.15.0 and then 2.16.0 within the week, under worldwide scrutiny and a great deal of abuse, were volunteers. The library sat underneath a very large share of enterprise Java: banks, insurers, governments and most of the Fortune 500. None of them were paying for it. Somebody\u0027s unpaid weekend was holding up the world\u0027s payment infrastructure, and nobody involved had agreed to that arrangement. Follow the money and the asymmetry is plain. The organisations that profited most from Log4j built products on it at zero marginal cost. The organisations profiting most from Log4Shell are the vendors selling scanning, SBOM generation, SCA licences and managed detection into the panic. Both are large companies. Neither is the maintainer. A disclosure like this transfers money from everyone to the security industry and nothing at all to the people who wrote and then fixed the code. Watch what happens next, because it has a predictable shape. The response to a supply-chain crisis is a compliance regime, and a compliance regime is a fixed cost. A large vendor absorbs it as a line item and markets the badge; a four-person software house and an unpaid Apache committer cannot. If the industry\u0027s answer to Log4Shell is paperwork, the effect is to push small and independent software out of regulated supply chains and consolidate them around whoever can afford the audit. That makes the next one worse, because it concentrates the dependency further. Here at AxisOps our position is that you do not get to depend on something for free and then be outraged when it breaks. If a library is in your critical path: fund it, mirror it, read it and know where it runs. The first three are cheap. The fourth is the one you can solve with code, and it is the one we spent six days discovering we could not answer. What we know is still wrong with it\u00b6 Two things, published alongside the code rather than discovered later by somebody else. The README says to confirm your applications still start. That is not the same as telling you what might not: removing JndiLookup breaks anything that legitimately performs JNDI lookups, JndiContextSelector and JMSAppender among them, so this role will break such an application. That belongs in the README and will be in it. The repository has one commit, because the working history was squashed before it went public. There is a mild irony in a supply-chain tool arriving with no reviewable history, and we would rather say so than let you notice. Defensible for something shipped in a week; a bad habit to carry into anything that outlives the incident. Not the exploit, not the mitigation, not even the six days: the hard problem here is not knowing where the vulnerable thing runs. That is the problem we are going to spend the next few years solving, and it will not be solved with a better scanner.", "d": "2021-12-15", "g": ["security", "ansible", "supply-chain", "python", "mitigation"], "t": "Six days, and the question nobody could quite answer", "u": "insights/log4shell-six-days/"},{"b": "The claim that most security vulnerabilities in large C and C++ codebases are memory-safety issues has been made repeatedly by people who have the data. Microsoft\u0027s security response centre put it at around 70% of the CVEs it assigns, measured across twelve years of patched vulnerabilities. Chromium reached the same figure from 912 high and critical bugs in its own codebase since 2015, with use-after-free making up half of the memory-unsafety share. Those are different populations counted different ways, and the fact that they land on the same number is a coincidence rather than corroboration. Take it as two large C++ codebases independently finding that memory safety dominates, not as a law. That number was never news. The news is that there is finally somewhere else to go. Why the previous answers failed\u00b6 Memory safety without garbage collection was well understood in theory long before Rust turned up. The reason it never displaced C was never the theory: Garbage-collected languages solve safety and hand you a pause. For a kernel, a driver or a hard-real-time path that\u0027s not a trade available to you. Static analysers on C catch a fraction and produce false positives at a rate that trains people to ignore the output. Safer C dialects needed everybody to adopt them, which is about the hardest deployment model there is. Rust\u0027s contribution is affine types and a borrow checker packaged with a toolchain, a package manager and error messages good enough that an intermediate programmer gets somewhere. The theory was the easy part; the theory is always the easy part. What it costs you\u00b6 The advocacy around this gets tiresome, so here is the bill. The learning curve is real, and the borrow checker is properly difficult for anyone whose instincts come from C. It improves, but \"it clicks after a few weeks\" understates things badly for a whole team. Compile times are not good. unsafe exists and has to exist. Rust doesn\u0027t eliminate memory bugs; it confines the code that can cause them to blocks that are marked as such, which makes them reviewable. That\u0027s a large improvement and a different claim from \"memory safe\". The interface to C is where the bugs go. An FFI boundary is unsafe by construction, and a mixed codebase has rather a lot of boundary. In the kernel\u00b6 Rust for Linux is the test case that counts, because the kernel is the least forgiving environment going: no standard library, custom allocators, pervasive C interfaces and a maintainer culture with strong opinions. The interesting engineering question is where the boundary sits, not \"is Rust good\": how much C API surface has to be wrapped in safe abstractions before writing a driver in Rust is safer rather than just differently shaped. Every wrapper is itself unsafe code that someone has to get right. That work is ongoing and it\u0027s the thing to watch, rather than the language advocacy going on around it. What this makes possible\u00b6 Our bet: within five years, writing a new network-facing parser in C will look like a choice that needs defending in review. That is a bigger deal than it sounds, and here is where it lands. A whole class of emergency disappears. The out-of-hours call that starts \"there\u0027s a remote heap overflow in the thing that parses untrusted input\" is the most expensive single category of security work in the industry. Image decoders, font shapers, TLS record layers, video demuxers, DNS resolvers: these are where the CVEs live, they are all small and they are all ideal candidates to rewrite. An engineer who no longer spends a fortnight a year on that gets a fortnight back, permanently. Safety stops costing performance. Previously the choice was unsafe and fast or safe and paused. A device driver, a hypervisor component or an audio path written this way carries no collector and no pause; the bounds checks that survive to runtime cost what they cost, and in practice that lands close enough to C that the argument stops being about speed. That means the safety argument finally wins on hardware that is cheap and small, not just on servers with headroom to spare: a router, a set-top box, a meter, a car. Embedded firmware is where memory-safety bugs are most common and least likely ever to be patched, and it is where a garbage collector was never an option. Review gets a usable map. grep -rn unsafe on a Rust codebase tells you where the memory-safety risk is concentrated. On a C codebase of the same size, the answer is \"everywhere\". For anyone who has to audit code they did not write, under a deadline, that difference is the whole job. Rust makes the same move as a load-time verifier or a bytecode validator, at a different point in the pipeline: establish a property by construction, check it once, then rely on it. Check at compile time and the runtime gets to be cheap. We expect that pattern to keep spreading, because it is the only one that scales to code you did not write and cannot read. The safety story ends at crates.io\u00b6 Now the part that should worry you, and it is not the language. Rust\u0027s adoption story depends on Cargo, and Cargo depends on a single default registry. One organisation runs the index, serves the tarballs and decides what can be published, yanked or kept. Flat dependency trees of three hundred crates are normal, which means an ordinary build pulls code from a hundred-odd maintainers through one distribution point. We have watched what a default public registry does to a container ecosystem. The same shape applies here, and the blast radius for a compromise of the registry or of a widely-depended-on crate is the entire language\u0027s output. Vendoring and lockfiles help with reproducibility. They do nothing about trust: pinning a hash guarantees you keep getting the same code, not that the code was ever any good. There is no meaningful namespacing, no mandatory signing and no practical way for a reviewer to audit a transitive tree of that size. Anyone who has shipped a Node application knows where this goes. The second chokepoint is the toolchain itself. One production-grade compiler, one package manager, one registry and the funding for all of it coming substantially from a handful of very large companies who now have Rust on their critical path. That alignment is useful right now; the interests of a hyperscaler and the interests of someone running ten machines are not permanently the same. The healthy signs to watch for are an independent governance foundation with real money in it, a second compiler implementation and a specification somebody other than the project could implement against. The absence of a written language specification is the gap that matters most, because without one, \"Rust\" means \"whatever rustc does this week\". Vendor the dependencies, build in a container you control and mirror the crates you depend on: an estate that cannot build without a third party\u0027s uptime is not an estate you own. The language is a gift. The distribution model is a single point of trust that the language\u0027s own safety guarantees say nothing whatsoever about. It is the half that needs the engineering attention now.", "d": "2021-06-15", "g": ["rust", "systems", "memory-safety"], "t": "Rust arrives where C had no competitor", "u": "insights/rust-reaches-systems/"},{"b": "We have built the platform layer from scratch for every client we have taken on. Ingress, TLS, database, queue, object store, the thing that watches the other things. Different names each time, the same shapes. It runs somewhere between three and six weeks per engagement before anybody writes a line of software the client asked for. Multiply that across a client list and the number stops being a rounding error and starts being most of a year. This year we stopped doing it. The internal name for the thing we are building instead is Andromeda, and the name carries the argument. Why a galaxy\u00b6 A galaxy applies the same handful of physical laws as a solar system, at a scale where different structure emerges: spiral arms, a bar, a halo. Nobody designs the arms. They fall out of gravity operating on a lot of mass over a long time. That is the bet. The structure that holds four machines together and the structure that holds four hundred should be the same structure, operated at a different scale, rather than two different systems with a migration project between them. If the small version is a toy and the large version is the real thing, the small version teaches nothing and the migration is where the year goes. Most platform tooling fails this test. It is either a laptop convenience that falls over at ten nodes, or a distributed system with a minimum viable size of five machines and a full-time operator. The gap between them is where small teams are forced to choose badly. What a component framework actually means\u00b6 Three commitments, and none of them is a new technology. One description, every environment. A service is declared once. Laptop, staging and production differ in values, not in structure. The moment the production description is its own artefact, maintained separately, it drifts and the drift is only discovered during an incident. Composition over configuration. A client stack is an assembly of components with declared relationships. Adding search is adding a component, not editing six files and hoping. Ansible made the single-machine version of this argument for us three years ago in Why We Never Repeat Ourselves; this is the same argument raised to a whole stack. The generated artefacts are disposable. The compose file, the ingress config, the environment files are output. They are never edited and never the source of truth. An operator who hand-edits generated output has broken the model, and the model should make that difficult rather than merely discouraged. None of this is novel. It is what every competent team ends up building privately, badly, under deadline, and then cannot reuse because it is tangled into one client\u0027s particulars. What it costs\u00b6 A framework is a bet that the shapes repeat. Where they do, the saving is large. Where they do not, you have added indirection between an engineer and the thing they are debugging, and that indirection is expensive at three in the morning. The specific risk is the client who needs something genuinely unlike the others. If the answer is \"the framework cannot express that\", the framework has started dictating architecture, which is the failure mode of every internal platform ever built. The escape hatch, dropping to plain compose for one service, has to stay open and has to stay unembarrassing to use. The second cost is that we now maintain a product nobody is paying for directly. It competes for attention with billable work, and the first time it gets starved during a busy quarter is the moment it starts rotting. What it makes possible\u00b6 The time saved is the dull half of this. The interesting half is what becomes affordable once the platform layer is nearly free. A four-person company gets the same platform layer as a forty-person one: the same ingress, the same TLS renewal, the same reproducible rebuild. That is the layer and not the posture. Review depth, on-call coverage, security testing and how much of the system lives in one person\u0027s head are all still functions of headcount, and nothing here touches them. But small teams run worse infrastructure mostly because the setup cost is roughly fixed and they amortise it over far less, not for want of engineering skill. Drive that one cost towards zero and the floor comes up. Rebuilding stops being frightening. If the whole stack is described, you can destroy it and bring it back. That changes the posture of everything else: you patch aggressively because rollback is cheap, you test restores because restoring is normal and ransomware becomes an afternoon rather than a company-ending event. We expect that to matter more each year. And the small operator can stay small. A clinic, a school, a council can run their own systems on their own hardware with a realistic maintenance burden, instead of concluding that the only sane option is somebody else\u0027s cloud. The technical argument for centralising was always partly an argument about operational cost. Lower the cost and the argument weakens. Who ends up owning the shapes\u00b6 The uncomfortable part. A framework that describes your entire estate is also the thing that decides what your estate can be, and whoever defines the components holds that. That is fine while it is us, for us. It stops being fine the moment it is a product and the components are defined by a vendor whose interest is that you keep paying. The large platform vendors have already worked this out: the free tier teaches your engineers a vocabulary, the vocabulary is proprietary and the migration cost away is measured in years. That is lock-in by habit rather than by contract, and it outlasts any contract. The defence is specific and we are committing to it now, before there is a product to be tempted about. The descriptions are plain YAML. The output is ordinary compose and ordinary config files that run without any of our tooling present. If Andromeda vanished tomorrow, every stack it has rendered keeps running, and an engineer who has never heard of us can read what it produced. A framework you cannot leave is a tenancy agreement. The part we cannot defend against is us: the pressure, later, to make the convenient path the proprietary one. Writing that down here is the cheapest guard available. Where this breaks\u00b6 Whether the shapes really do repeat enough. We have a strong sample of clients that look alike, and a weak sense of whether that is a fact about infrastructure or about our client list. The real test is the first engagement that does not fit, and we have not had it yet. Ask us again in two years. Either this is the foundation everything else sits on, or it is an elaborate way of having built the same platform layer one more time.", "d": "2020-10-14", "g": ["infrastructure", "deployment", "ansible", "architecture", "operations"], "t": "Galaxy-scale engineering, and why we named it Andromeda", "u": "insights/andromeda-galaxy-scale/"},{"b": "WireGuard merged into Linus\u0027s tree during the 5.6 merge window in late January 2020, having sat in Dave Miller\u0027s networking tree since December 2019, and shipped in the 5.6 release this month (kernel.org, The Register). The number everybody quotes is about 4,000 lines, against something north of 100,000 for OpenVPN. The project\u0027s own stated goal is that it be \"comprehensively reviewable by single individuals\". That\u0027s a design goal, not an audit result, and the distinction matters rather a lot. Why small is the argument\u00b6 Most security engineering tries to make a large attack surface safe. WireGuard goes the other way and makes the surface small enough that a competent person can hold all of it in their head at once. You can inspect a shed; you cannot inspect a cathedral. The mechanisms: No cryptographic agility. One cipher suite: ChaCha20, Poly1305, Curve25519, BLAKE2s, HKDF. No negotiation, so no downgrade attack, and no negotiation code to get wrong. Upgrading means a new protocol version rather than a new option. No key management in the data plane. Peers are public keys, configured out of band. There\u0027s no PKI, no certificate parsing, and therefore no X.509 parser, historically one of the richest sources of remote vulnerabilities in TLS implementations. Silence by default. An unauthenticated packet gets no response whatsoever, which removes a whole class of scanning and amplification behaviour. Stateless until authenticated. The handshake is built so the responder allocates nothing that an unauthenticated peer can make it allocate. Every one of those is a decision to not support something. The line count is the consequence, not the goal. What has and has not been verified\u00b6 This gets overstated, so be careful with it. In May 2019, researchers at INRIA published a machine-checked proof of the WireGuard protocol using CryptoVerif. That verifies the protocol design. It isn\u0027t a verification of the kernel implementation, and those are two different claims. A proven protocol with a buggy implementation is still a buggy system. The open question is the in-kernel code: a formal third-party audit of the merged implementation would settle it, and we have not found one published. The claim \"formally verified\" gets applied to WireGuard a good deal more loosely than the record supports, and the thing that would fix that is an audit of the C, not another proof of the handshake. The counter-example it provides\u00b6 The default direction of travel for a successful protocol is to accumulate options, because every option has an advocate with a real use case behind them. TLS, SSH and IPsec all went that way and all paid for it in implementation complexity and downgrade attacks. WireGuard is the demonstration that the other path is open, and that the cost of taking it, which is telling people no, gets paid in product management rather than in engineering. What this makes possible\u00b6 A VPN that is four thousand lines and configured with a public key per peer is a different kind of component from one that needs a PKI and a support contract. A small business gets a real private network without a consultant. A peer is three lines of config and a key pair. Two sites, a handful of laptops and a cloud box become one network in an afternoon, with no certificate authority to stand up, no CRL to publish and nothing to renew at two in the morning. That used to be a project; it is now a config file you can read out loud. Encryption becomes the default for links nobody bothered to encrypt. Database replication between two racks, a monitoring feed, a backup stream to a colocation box: all of those have historically gone in the clear across a private line because IPsec was too much trouble for the benefit. At a handshake this cheap and a kernel data path this fast, there is no longer an excuse, and one entire category of \"we assumed the link was trusted\" incident goes away. Roaming stops being special. The protocol identifies peers by key rather than by address, so a laptop moving from office Wi-Fi to a phone tether to a hotel network keeps the same tunnel. For someone working from a country with unreliable connectivity, or over a link that drops every few minutes, this is the difference between a usable connection and a support ticket. And for engineering itself: a boundary you can read in an afternoon is a boundary you can reason about. The whole class of bug that comes from negotiation, option matrices and protocol downgrade simply has nowhere to live here. The code can be readable while the endpoint still belongs to somebody else. The same simplicity that makes WireGuard easy to self-host makes it easy to build a consumer VPN business on, and that industry is already large, already advertising heavily and already selling privacy as a subscription. The pitch is that your ISP should not see your traffic. The mechanism is that a company you have never met sees it instead. Be precise about what the protocol does and does not protect. WireGuard protects the link. It does not protect you from the operator of the far end, who terminates your tunnel, sees every destination you connect to and is subject to whichever jurisdiction\u0027s legal process applies to them. Swapping your ISP for a VPN provider is a change of who holds the log, not a removal of the log. There is a specific technical wrinkle that makes the commercial version worse than it looks. WireGuard\u0027s interface is stateless and keeps no per-peer session concept, so a server assigns each peer a fixed tunnel address and keeps that mapping in its config. The provider therefore holds a durable identifier for you by construction. A \"no logs\" claim sits on top of a protocol whose server side needs a persistent key-to-address table to function at all. We are not accusing any provider. The architecture does not help them keep the promise. The code is published and small; ownership sits with whoever operates the endpoint you dial. That is the good news, because unlike a registry or a certificate authority, this chokepoint is one you can simply take back: a thirty-pound box in a cupboard at home, a key pair and a port forward. Run your own endpoint and the question of who to trust with your traffic stops being a procurement decision. Two things to keep an eye on as this spreads. First, whether the protocol stays small once large vendors want features in it. The pressure to add negotiation will arrive the moment somebody important needs an exception. Second, whether the self-hosted path stays as easy as the subscription path. Today the self-hosted version is simpler than the commercial one, which is a rare and valuable position to be in, and it is worth defending.", "d": "2020-03-30", "g": ["security", "kernel", "cryptography", "networking"], "t": "WireGuard: small enough to read", "u": "insights/wireguard-small-enough-to-read/"},{"b": "When we wrote about Docker 1.0 the distinction that mattered was between packaging and isolation: the industry had adopted a boundary weaker than the virtual machine it replaced, because the ergonomics were better, and the isolation question was left for later. For years the trade was treated as a law of nature. You could have fast, dense and weak, or slow, heavy and strong. 2018 produced three projects that refused the trade, and each one is a different bet about where the boundary should live. Three bets\u00b6 Kata Containers, which grew out of Intel\u0027s Clear Containers work, puts a conventional virtual machine underneath the standard container runtime interface. The orchestrator sees a container; the host sees a VM with its own guest kernel. Compatibility is what it buys, and the cost is carrying a full machine per pod. gVisor, open-sourced by Google in May 2018, makes the opposite bet: no second kernel. A user-space process re-implements the Linux system call interface and the workload talks to that re-implementation instead of the host. The boundary is strong in the sense that workload system calls never reach the host kernel directly. The re-implementation itself still talks to the host, though, through a narrowed, filtered interface, and the whole arrangement is as trustworthy as that re-implementation is correct. Syscall-heavy workloads pay the interception cost on every call. Firecracker, which AWS open-sourced in November 2018, keeps the hypervisor boundary and deletes the machine. KVM stays. What goes is everything a cloud guest never uses: no BIOS, no PCI enumeration, no video, no USB. The device model is virtio network, virtio block, a serial console and a one-button keyboard controller whose only job is to signal reset. The numbers are the argument\u00b6 The paper AWS published at NSDI last month puts figures on it, and the launch announcement carries the same claims: a microVM boots to init in around 125 milliseconds, costs under 5 MiB of overhead beyond the workload itself, and a single host can start them at a rate of 150 per second. The virtual machine monitor is roughly fifty thousand lines of Rust. QEMU, the device model most production VMs sit on, is well over a million lines of C, and it emulates decades of hardware that no cloud workload will ever touch. That last comparison is the security argument, and it is older than Firecracker. VENOM, in 2015, was an escape through QEMU\u0027s virtual floppy disc controller, a device present and reachable in guests that had never mounted a floppy in their lives. A device model with four devices has no floppy controller to be wrong in. Shrinking the device model removes whole classes of escape path before anyone audits a line. The language choice carries a related claim: the component that parses guest-controlled data sits on the host side of the trust boundary, and Firecracker\u0027s authors chose to write it in a memory-safe language. Whether that choice becomes the norm for new hypervisor work is one of the things the shared rust-vmm crates, which Firecracker and others now build from, will settle in practice. Why this matters now\u00b6 Two years ago the floor moved. Meltdown showed the user/kernel boundary failing outright, and the Spectre family made every boundary that shares a core suspect, virtual machines included. What changed was the arithmetic of defence in depth: fewer tenants per kernel and a smaller interface between tenant and host, at the exact moment multi-tenant density made the shared kernel most valuable. A hardware-backed boundary per tenant stopped looking like caution and started looking like the conservative default. Firecracker is what that default looks like when someone re-derives it from requirements instead of inheriting it from 2007. The paper is explicit that Firecracker exists to run other people\u0027s code, at density, on shared hosts, and that the Lambda and Fargate fleets run on it in production. Our view, stated as a view: the interesting consequence is for the orchestration layer. Schedulers still think in containers because virtual machines were too slow and too heavy to be the per-workload unit. A microVM still carries a guest kernel, so it will never be as cheap as a namespace. But at 125 milliseconds and single-digit MiB of monitor overhead, the blanket objection no longer holds. If the microVM becomes the unit the control plane allocates, the container quietly becomes what it always was underneath: a packaging format.", "d": "2020-03-10", "g": ["containers", "security", "architecture", "linux"], "t": "Firecracker: the virtual machine came back smaller", "u": "insights/firecracker-the-vm-came-back/"},{"b": "There is now a screen on the wall. It has the gauges, the sparklines and the green panels, and when a client walks past it they ask what it costs to get one. The screen is the cheapest part. Everything that makes it worth having sits behind it, and almost none of that is a purchasing decision. What the stack actually is\u00b6 The components are unexciting, and chosen for it. Prometheus scrapes metrics and holds them short-term. Exporters sit next to the things being measured and speak Prometheus\u0027s language on their behalf. Loki takes logs, indexed by label rather than by content, so a query starts from \"which service, which host\" instead of a full-text search across everything. Grafana puts a face on all of it. One piece of that list is a bet: Loki only reached 1.0 this month, and we treat it accordingly. The rest has been boring for years. Every piece is open, every piece runs on our own hardware, and if we stopped paying anyone tomorrow it would keep running. That mattered more to us than any feature comparison, because a monitoring system you can lose access to is a monitoring system that fails at the worst moment. The architecture is a pipeline and it is worth stating plainly, because the diagrams people draw of this are usually more complicated than the thing: applications and infrastructure expose metrics, exporters scrape them, Prometheus stores them logs ship to Loki and are found by label Grafana queries both and draws rules evaluate in Prometheus; routing and silencing happen somewhere else That last line is where the difficulty lives. The part that took the year\u00b6 Deciding what is allowed to wake somebody up. An alert is a claim that a human needs to act, now, and almost every default alert that ships with anything fails that test. Disk at eighty per cent is not an emergency; disk filling at a rate that reaches full before office hours is. CPU at ninety per cent is a fact, not a problem. A failed backup job matters enormously and almost never pages anyone, because nobody wrote the rule. What we ended up with is a short list of questions each rule has to answer, and rules that cannot answer them do not get to page: What has stopped working for a user? Not what metric moved. What will the person who is woken actually do? If the answer is \"look at it and go back to bed\", it is a dashboard panel, not a page. How do we know this is still true in ten minutes? An alert that fires on one scrape of one instance is noise waiting to happen. Who silences it, and for how long, during planned work? A system with no answer to this teaches people to ignore it, which is worse than no alerts. None of that is a product feature. It is a conversation per service, and the conversation is slow because the honest answer to \"should this wake someone\" is often \"no\" and somebody has to say so. Running it rather than handing it over\u00b6 The thing that is new for us is that we operate this, instead of building it and leaving it with a client. That changes the design in ways we did not anticipate. Every alert has an owner who is not the author. If we are the ones woken, we are also the ones who reject a rule that pages without a cause. That turns out to be the only reliable filter anyone has found. Dashboards decay and nobody notices. A panel that silently stopped reporting looks identical to a panel reporting that everything is fine. We now treat a dashboard the same as any other deployed artefact: it is in version control, and the thing that renders it is rebuilt rather than edited in place. Retention is a decision, not a default. Metrics kept forever at full resolution get expensive quickly, and nobody looks at per-second data from eighteen months ago. Short-term fine-grained, long-term coarse and the boundary chosen on purpose. What we would tell anyone standing one up\u00b6 The stack is a fortnight. Budget for the rest. Instrument the things a user would notice first, not the things easiest to measure. Host metrics are easy and tell you least; a synthetic check that logs in the way a customer does is harder and tells you most. Write the runbook before the alert. If the rule fires at three in the morning and the person on call has to work out what it means from the alert text alone, the rule is unfinished. And accept that most of this is organisational. The technology settled years ago. What has not settled, anywhere we have seen, is the discipline of deciding what deserves a human.", "d": "2019-11-30", "g": ["observability", "operations", "infrastructure", "prometheus", "grafana"], "t": "The dashboard on the wall is the easy part", "u": "insights/monitoring-stack-2019/"},{"b": "For about fifteen years, \"server\" has meant x86-64 so totally that it stopped being a decision anyone made. That is ending, and the instruction set is the least interesting part of it. What matters is what a monoculture hides. What a monoculture costs you\u00b6 Software that has only ever been built on one architecture collects assumptions it doesn\u0027t know it\u0027s carrying: Memory ordering. x86-64 is strongly ordered; ARM is weakly ordered. Concurrent code that is subtly wrong (a missing barrier, a non-atomic read of something assumed atomic) works by accident on x86 and fails intermittently on ARM. The bugs were always there. The architecture change makes them visible. Unaligned access. Tolerated on x86, historically not on ARM; AArch64 tolerates most of it now, with device memory and atomics still trapping. char signedness. Plain char is implementation-defined, and the common toolchains settled on signed for x86 and unsigned for ARM. Not an architectural rule, and -fsigned-char will override it, but it is the default you meet. A twenty-year-old C idiom that happened to work quietly stops working. Build toolchains. The deepest one. Every vendored binary, every curl | sh installer, every container image with no arm64 tag. The first three are compiler and code problems and you can mostly fix them. The fourth governs the timeline, because it\u0027s a social problem spread across thousands of projects, and you can\u0027t patch a social problem. Why a Pi was a useful filter years early\u00b6 We ran a Raspberry Pi cluster in 2016, and the compute was neither here nor there. What it did was tell us straight away which parts of a stack were x86-only, years before that became anyone\u0027s commercial problem. Anything that wouldn\u0027t build was something we\u0027d have to fix eventually. It cost us almost nothing to find that out early, which is about the best return you get on a shelf of forty-pound computers. What the shift unlocks\u00b6 Not performance, to begin with. The first credible server ARM parts are competitive rather than dominant on raw throughput. What changes is the design space. An organisation that can specify its own silicon can make trades a general-purpose vendor can\u0027t: more cores at lower clocks, memory bandwidth over single-thread latency, accelerators for the workload it runs rather than the workload in the benchmark suite. The benchmark headlines matter less than what follows once the architecture is a choice. Performance per watt becomes the number people buy on. A rack is limited by the power and cooling you can get into the building, not by the number of U. Once two architectures compete on work-per-joule, the cost of a colocation cabinet and the cost of a small on-premises room both fall in a way they haven\u0027t since the Xeon took over. A business that could not previously afford to run its own compute gets a credible option that is not a hyperscaler invoice. That is the benefit we care about most, because it widens who can own infrastructure at all. Developer hardware and production hardware converge. An engineer on an ARM laptop building for an ARM server is testing the thing they ship. The whole category of \"works on my machine, breaks in production because the compiler made a different choice about a struct layout\" shrinks. That bug class is a tax on every team that has ever cross-compiled; take it away and you get the time back. Portability becomes a tested property again rather than a claim in a README. Every project that gets a second build target gains a permanent regression test for undefined behaviour. The cleanup is already happening in public, in CI configs, and everything that comes after this benefits whether it knows it or not. RISC-V is the obvious next candidate to collect that benefit; it arrives into a far friendlier world than ARM did, because somebody else already paid for the port. That choice only exists if you can buy the silicon. The companies doing the interesting ARM server work at this scale are mostly designing parts for themselves. The design space opens up for whoever can commission a run at a leading foundry, pay for the IP licences and absorb a multi-year programme before the first machine boots. That is a very short list, and nothing about ARM\u0027s instruction set changes the list. So the question to ask of any ARM server announcement is not how fast the chip is. It is: can I buy one, in a quantity I want, and put it in a rack I control? For a lot of the best silicon in this generation, the answer is no. It exists only as an instance type. You rent the performance-per-watt win from the operator who commissioned the part, and the efficiency gain lands on their margin rather than your bill. The allocation queue is the chokepoint, not the architecture. That is a worse outcome than the x86 monoculture it replaces, in one specific respect. A commodity x86 server was something anyone could buy from several vendors at a published price. \"Available only as a managed instance\" is not commodity. If the end of the monoculture produces three architectures that are each a single operator\u0027s private advantage, we have swapped one lock-in for a better-performing one. The open side of this transition needs general-purpose ARM boards that a mid-sized business can order ten of, and a toolchain nobody has to ask permission to use. The toolchain is in good shape. The procurable hardware is the half that is losing, and it is the half worth pushing on.", "d": "2019-09-10", "g": ["arm", "hardware", "architecture"], "t": "ARM reaches the server room", "u": "insights/arm-server-room/"},{"b": "Two things happened in biology recently that belong in a computing notebook rather than a life sciences one. At CASP13 in December, DeepMind\u0027s AlphaFold took first place in free modelling, the category for proteins with no useful template. By the assessors\u0027 summed z-scores it finished a clear first, and it predicted roughly 24 of 43 hard targets to high accuracy where the next best managed about 14. CASP has run since 1994 and improvement is normally incremental. This was not incremental. The method is the interesting part. It predicts a distribution over distances between residue pairs with a deep network, turns that into a potential and then optimises the structure by gradient descent. No new physics, no new chemistry. A representation problem, solved with the same machinery that was doing image classification four years ago. Meanwhile a nanopore sequencer weighs about 100 grams, draws power over USB and produces gigabytes of sequence from a device you can carry in a coat pocket. Read lengths run to megabases. Accuracy is worse than the big short-read platforms and improving steadily. Why this is a computing story\u00b6 Because the bottleneck moved. The hard part of structural biology used to be the instrument: a crystallography beamline, a cryo-EM microscope, the better part of a year per structure. Those were capital assets with queues attached. The hard part now is partly a model, and a model is a file. The same inversion is happening at the other end. Sequencing used to mean a room. It now means a dongle and a laptop, and the limiting factor is the basecaller, which is a neural network converting raw ionic current into bases. The instrument is commodity and the intelligence is software. That is a shape this notebook has seen before. It is the same move as software-defined networking and, before that, as the general-purpose CPU displacing fixed-function silicon. When the clever part migrates from the device into code, the device gets cheap and the code becomes the thing worth owning. What it costs\u00b6 Confidence, mostly, and in a field where the failure mode is expensive. A predicted structure is a hypothesis with a plausible shape, carrying none of the authority of an experimental observation. The distinction will blur the moment predictions are good enough to be convenient. CASP reports confidence estimates; the literature that cites a predicted structure three papers downstream will not. Expect a decade of results resting on predictions nobody re-derived, and expect a few of them to be wrong in ways that take years to surface. The second cost is that the training data is the product of the slow experimental work being displaced. The PDB is a public good built over decades by people doing crystallography. A model trained on it, released as a black box, consumes that commons and does not replenish it. If structure prediction reduces the incentive to deposit new experimental structures, the next generation of models has less to learn from. What it makes possible\u00b6 Start with the work that simply could not be afforded before. A neglected disease has no structural biology programme because no one will fund a decade of beamline time for a parasite that infects people who cannot pay. If predicting a target\u0027s structure costs compute rather than capital, that calculation changes, and the bottleneck moves to the much cheaper question of who is willing to do the chemistry. The same applies to antimicrobial resistance, where the economics have been broken for thirty years. Then the pocket sequencer, which is the part this lab finds most interesting. A device that runs on a laptop in a field station means identifying an outbreak where it happens rather than shipping samples to a reference lab in another country and waiting. During an epidemic the latency between \"someone is ill\" and \"we know what this is\" is the whole game. Portable sequencing collapses it from weeks to a day, and it does so without requiring that the affected country own a genomics institute. Further out, the combination is what matters. Cheap sequencing produces variants; cheap structure prediction says what a variant does to a protein. That loop, run continuously rather than per-project, is a surveillance system for biology itself. We would expect it to be how the next pandemic is characterised, assuming anybody has built the plumbing by then. Who ends up owning the fold\u00b6 The uncomfortable arithmetic. The experimental data is public and the models trained on it need not be. AlphaFold\u0027s results were published; the weights are another question, and the pattern across machine learning is that the paper arrives and the artefact does not. If the definitive structure-prediction model is a service, then structural biology acquires an API key, and a field that ran on shared data for thirty years starts running on a terms-of-service agreement. Query limits are a research agenda. So is pricing. The sequencer has a different chokepoint and a sharper one. The hardware is cheap but the basecaller is proprietary and the flow cells are consumables sold by one company. A device whose consumables have a single supplier is a subscription with a plastic case. Open basecallers exist and lag. That gap is the whole question. And the data is human. A sequence is the most identifying record a person has: it implicates relatives who never consented, and it cannot be rotated after a breach. Population-scale sequencing is being assembled right now, mostly by national programmes with mostly good intentions, and the governance is downstream of the engineering rather than ahead of it. A genomic database amounts to a permanent register of people and their descendants. Here at AxisOps our position on this is the same as everywhere else: prefer the instrument you can own, run the model yourself where a model exists to be run and treat a biological dataset as the most sensitive thing in the building, because it is. The compute is the easy part. What would change our mind\u00b6 If predicted structures turn out to be reliable enough that experimental validation becomes optional, the commons argument above gets much weaker and this reads as alarmism. We do not think that happens soon, but we would rather be wrong about it than right.", "d": "2019-05-21", "g": ["ai", "inference", "verification", "hardware", "policy"], "t": "The protein problem became a data problem", "u": "insights/protein-problem-became-data/"},{"b": "Originally posted on The Network Factory, 26 March 2018. The Network Factory was acquired by AxisOps; this piece is reproduced here as published, unedited, as part of the engineering archive. We are often asked for proof-of-concept and implementation of IoT and embedded solutions. One example of this was for a hospitality client, who asked us to help out with token-based access to their public Wi-Fi, via a printed till receipt given with each purchase. Unlike the majority of our clients, this business had no existing systems in place to host a print server. After some consideration, we decided to provision a Raspberry Pi Zero (Wi-Fi model) for the job. The Raspberry Pi is an ideal fit for this situation, thanks to its extremely low cost, exceptional energy-efficiency and its tiny profile and size. We decided to set up the Raspberry Pi using our automation and provisioning tool of choice, Ansible. This approach means that the steps involved are documented and the setup can be quickly and easily deployed to other Raspberry Pis (or, with minor changes, other devices and distributions). For development of the system, we created a VM that made use of the official x86 variant of Raspbian (the Raspberry Pi\u0027s official Debian-based Linux operating system). This allowed to closely approximate the environment of an actual Raspberry Pi running Raspbian, but with the performance advantage of deploying to a local VM (the Raspberry Pi Zero is a brilliant little device but it\u0027s not lightning fast) and the convenience advantage of VM snapshots, allowing us to instantly restore the VM to a fresh, unmodified state for testing fresh deployments of the setup. For similar work in the future, we could even more closely replicate the Raspberry Pi environment by making use of Vagrant and Qemu to actually run an emulated ARM environment (allowing us to use the standard ARM version of Raspbian intended for deployment to actual Raspberry Pis, as opposed to the x86 variant). To generate the voucher itself we used PHP with the TCPDF library. There are several excellent options for generating PDFs (such as FPDF and DOMPDF), but TCPDF\u0027s feature list was pretty hard to beat, not to mention no requirement of additional external libraries. One nice touch we wanted to add was a QR code on each voucher. This allows the client\u0027s customers to connect simply by scanning the QR code, saving them having to locate the correct Wi-Fi network in the list. To achieve this we installed the qrencode Linux package, which allows us to quickly and easily generate QR codes that we then embed in the PDF voucher file. One final thing we wanted to integrate was notifications for when a voucher roll was running low. In the event of vouchers running out entirely, we put in place code to make the printer output a \u0027Voucher roll empty\u0027 error. Ideally though we\u0027re not expecting to ever reach that point, because notifications will kick in first, so that the voucher roll can be replaced with a fresh one long before it runs out entirely. For this we decided to send notifications to our company Slack\u0027s client notifications channel via curl and Slack WebHook. Ideally the provision of new voucher rolls would be handled automatically without the need for human intervention. In our next iteration of this project, we aim to achieve that via the Raspberry Pi sending a request when it runs low on vouchers that fetches a freshly-generated voucher roll. Overall this was a hugely educational and rewarding project and we (and our client) are delighted with the result. For IoT and embedded solutions (and for many other things too), we can help.", "d": "2018-03-26", "g": [], "t": "IoT - Wi-Fi Voucher Printing", "u": "insights/tnf-iot-wifi-voucher-printing/"},{"b": "Originally posted on The Network Factory, 31 January 2018. The Network Factory was acquired by AxisOps; this piece is reproduced here as published, unedited, as part of the engineering archive. Technology tools within the DevOps world move at lightning speed. You often see a Cambrian explosion effect then rapid consolidation, and survival of the fittest (it does the job well, it\u0027s very easy to implement, fits with your existing tools and processes, and most of all has excellent documentation), but this does not necessarily mean it won\u0027t reach an evolutionary dead end as something new, or better, or more likely a shiny new fork, will appear. As with any company we evaluate our toolset offering. A lot of expense goes into training staff and maintaining support for products. Here at The Network Factory we regularly have technology reviews; what\u0027s working, what\u0027s not working, what\u0027s new on the horizon and where the industry is going. From our most recent review one thing is clear; the industry has finally settled on Kubernetes. It\u0027s become boring and that\u0027s a positive thing. Kubernetes should be considered the engine in a car; it\u0027s a necessity and a standard. However, like an engine it\u0027s quite complex (Kubernetes has quite a steep learning curve) but you don\u0027t need to be a mechanic to drive a car so we\u0027re now looking for the optional extras that help make that process less of a challenge, like the tools and UI\u0027s that sit on top of Kubernetes. It\u0027s very easy as developers and engineers to chase shiny new toys but for clients and production systems it needs to be a mature, well tested product. So out of our last review there a few solutions that we\u0027ve chosen not to support; this doesn\u0027t mean that they don\u0027t do a job, they are right for some businesses - one size does not fit all. Products we are no longer supporting from 2018, primarily because we have introduced better fit technologies for our existing clients, are Mesos and Mesosphere. This leads us to the shiny new products we\u0027re evaluating in Q1! We\u0027re really excited about StorageOS; it has the potential to solve some of the problems we have with persistent container storage and we are actively testing, evaluating and integrating it into our containerised solutions. Whilst we\u0027re talking about good software, despite being a terminal junkie I must say I really love the Portainer container management system. I use it locally, if only just to have an overview of my images, containers and storage. It\u0027s very easy to let Docker run wild on your system leading to abandoned containers, volumes, and using valuable disk space and resources. Our next review will be in June. Who can really say what the landscape will look like in six-months time? Well\u2026we have a fair idea!", "d": "2018-01-31", "g": [], "t": "Technology Review", "u": "insights/tnf-technology-review/"},{"b": "Everything we thought we knew about isolation is downstream of one week in January 2018. Before that week, \"the MMU will handle it\" was a perfectly reasonable engineering position. The hardware privilege boundary was the solid floor under every design decision anybody made about isolation. Now the floor is a thing with known leaks, across essentially every speculating processor shipped for two decades. Not uniformly, as the Meltdown and Spectre split below shows, but industry-wide. What broke\u00b6 Speculative execution is a feature, and a very good one: predict the branch, run ahead, throw the work away if you guessed wrong. It is the central technique that made processors fast. The assumption was that discarded work is architecturally invisible, and it was wrong in one specific way. Speculation leaves traces in microarchitectural state, mostly the cache. A load that \"never happened\" still evicted a cache line, and cache timing is measurable from userspace. The burglar did not get in, but you can still see his footprints in the flowerbed. Meltdown uses that to read kernel memory from an unprivileged process on affected Intel parts: speculate past the permission check, touch the data, then read it back out through the cache side channel. Spectre is the harder and more general problem. You train a branch predictor so that another context speculates down a path of your choosing. There is no single bug to patch; it is a property of the technique. The mitigation, and the cost\u00b6 KPTI (kernel page-table isolation) deals with Meltdown by unmapping kernel memory from the user page tables, which means a page-table switch on every syscall. It merged into the 4.15 cycle, just before rc6 (LWN). That cost is entirely workload-dependent: Google reported the impact as depending on the rate of system calls an application makes (Google), which is the correct way to describe it. The figures flying around this fortnight are wild. Press reports of 5-30%, vendor statements that the impact \"should not be significant\", field reports converging on roughly 20% worst case for syscall-heavy database and web workloads and near-zero for desktop use. FreeBSD\u0027s own write-up was refreshingly blunt that PTI incurs a regression, some of which PCID work regains. For Spectre variant 2, Google\u0027s retpoline mitigation (LWN) is reported as having negligible impact across its fleet. So a single percentage figure for this is meaningless, and almost every number quoted this month is either a syscall-bound worst case or a desktop best case, dressed up as a general one. Why it changes architecture, not just patch cycles\u00b6 Three consequences that will run for years. Isolation got more expensive, so people will want it finer-grained. If you are paying for a boundary, you want it where the risk is, not at whatever granularity the hardware chose for you. Shared tenancy has a new question. \"Is a container enough?\" was a pub argument. Co-tenancy with a cross-privilege side channel makes it an operational one. Our bet on the answer: something that puts a hardware boundary back under each tenant while keeping the start-up time and density that made containers win. A hypervisor stripped down until it costs almost nothing to boot, or a kernel interface narrow enough to reason about. We would not bet on which, but we will bet that most of the next five years of isolation engineering is this. Hardware has stopped being a trust anchor by default. Expect a generation of mitigation to move into silicon, because the software workarounds are expensive and partial. Expect the harder shift to follow: a demand that the layer underneath you be verifiable rather than assumed. Buying a CPU on the vendor\u0027s word about its isolation properties has just been shown to be a position, not a baseline. What the mitigations make possible\u00b6 A boundary with a measurable price is a better engineering object than a boundary everybody believed in for free. Once isolation has a number on it, you can spend it where it matters: this workload gets a hardware boundary and pays the measured price, that one gets a namespace and costs nothing, and here is the arithmetic. Most systems today apply one boundary everywhere because nobody knew the alternative\u0027s cost. The next few years should produce infrastructure where a regulated workload and a cache warmer sit on the same hardware under provably different isolation, which is cheaper and safer than the two-estates answer everyone reaches for now. The toil this kills is the security argument conducted on intuition. \"Is a container enough\" has been unanswerable for five years because nobody could price either side. It is about to become a measurement. Cheap verifiable isolation is also what makes shared infrastructure safe for people who currently cannot use it. A GP surgery, a law firm, a local authority: all are told to keep data on their own kit because the best answer anyone could give about co-tenancy was \"probably fine\". Replace that with a boundary you can test and the economics of small organisations running proper systems change completely. Who got the fix first, and who cannot read it\u00b6 Two things about this month should make every engineer outside a hyperscaler angry, and both are about access rather than the bug. The embargo ran for roughly six months. Google\u0027s Project Zero reported this to Intel, AMD and Arm on 1 June 2017 (Project Zero), and the large cloud operators had working mitigations deployed across their fleets before the rest of us learned there was anything to mitigate. Defensible on disclosure grounds, and it still means the companies that most needed the protection had half a year of it while everybody else had none. If embargo access tracks market share, security scales with size. The second is worse because it is permanent. The microcode patching your CPU\u0027s branch predictor is signed, encrypted and undocumented. You cannot read it, verify what it changed or build it yourself. You are applying an opaque binary to the most privileged component in your machine, from the vendor whose design error caused the problem, on their assurance. Two companies hold the only keys that can modify the hardware everyone\u0027s security rests on. Every other isolation argument sits on top of that. If hardware trust has to be verified rather than assumed, the attestation machinery arriving over the next few years will itself be rooted in keys held by the silicon vendors and the platform operators. Done well, that gives you evidence about the machine you are running on. Done the way the incentives point, it gives three companies the power to decide which software may run and a cryptographic record of every machine that ran it. The difference is who holds the keys. We run on hardware we own, patch kernels we can read and treat any isolation claim we cannot test as marketing. Trustworthy computing in which the only verifiable machine is a rented one is the wrong destination, and the open tooling for measuring your own boundaries needs to keep up. Keep the measurement, not the panic\u00b6 A boundary you cannot inspect is a boundary you are trusting on reputation. Spectre has not made hardware untrustworthy. It has made explicit that hardware was always trusted without evidence, and that twenty years of optimisation was quietly spending a security property nobody was measuring. Rather like discovering the surveyor never looked at the foundations because the house was already standing. Every strand of the isolation work now starting is a different answer to the same follow-up: so where do we put the boundary now, and how do we check it?", "d": "2018-01-15", "g": ["security", "hardware", "kernel"], "t": "Speculative execution crossed the security boundary", "u": "insights/spectre-meltdown-the-floor-moved/"},{"b": "Originally posted on The Network Factory, 31 December 2017. The Network Factory was acquired by AxisOps; this piece is reproduced here as published, unedited, as part of the engineering archive. 2017 at The Network Factory\u00b6 As we come to that funny little time between Christmas and New Year when many companies are on hiatus, all my obscure outstanding support tickets have been closed and I have reached inbox nirvana, it gives me time to reflect on the last twelve months. One of our most exciting projects for the year has been delivering for a client our first end-to-end containerisation platform, \u0027Bedrock\u0027. This time last year Bedrock was just a mind-map on a white board, but in September this year we signed off on the fully-featured containerisation platform with a one-click deployment and an exciting road map for 2018. It was also the year that we moved to our new, larger office, allowing us to house our entire operation under one roof, as we were previously running from a number of smaller separate locations. Not only is this beneficial for productivity but it has also developed the family feel of our team \u2013 Slack is great, but it doesn\u0027t beat a discussion over a cup of our finest Aeropress\u0027ed coffee! So what happened with technology in 2017?\u00b6 Containerisation. Of course Kubernetes (one of the building blocks of our Bedrock platform) was crowned the undisputed king of container orchestration, with over 30 certified platforms and distributions this year alone, and with other competing platforms such as DC/OS announcing they are supporting it (and rightly so). It is the core foundation of all new projects that we have undertaken over the last two years. We value it for its sheer flexibility in managing an application\u0027s life-cycle, from discovery and scaling to enabling easy rolling updates and ensuring that the application is running how it should every time all the time. Microsoft fully embraced Docker and containers as part of their core strategy as if it were a long-lost child, not only supporting the core Docker functionality, but building tools around managing and running Docker containers within a Windows environment, reinforcing the general move to the \u0027holy grail\u0027 of, wait for it, an \"immutable self-contained microservices-based architecture\" or, simply put, you really can now build your technology stack like Lego. Data breaches, security flaws and ransomware. 2017, in my opinion, was the year where it seemed like it was the trend for big businesses to be hacked \u2013 and some may not yet know they were hacked. First, let\u0027s discuss data breaches. The number of breaches is staggering, however there are three particular examples that stick in my mind: In September, Equifax provided an early Christmas present for would-be identity thieves, with the personal data of over 145 million people, including all of their financial history and personal details, being compromised. In October, Yahoo\u0027s parent company Verizon admitted that possibly every one of Yahoo\u0027s 3 billion accounts had been compromised back in 2013. And this November, we discovered that Uber had covered up a 2016 attack where hackers stole the data of 57 million customers and were paid $100,000 by Uber to keep this under wraps. Now onto security flaws. Things used to be simple; the chip inside your electrical device used to do one job, couldn\u0027t be updated and the firmware that shipped with it was the only one necessary. But now, a single, tiny chip can be an entire ecosystem of interconnected roles, functions and jobs, and in fact have its own operating system and worse still, it\u0027s now connected to the internet. A critical flaw in the Intel Management Engine was recently exposed, which affects the majority of desktops and servers sold in the last ten years. This flaw could in theory allow the system\u0027s low level administrative features to be compromised, and worryingly it could be done in a way that makes it hard to detect by the device\u0027s main operating system. Which then brings us to ransomware. May saw the infamous WannaCry, which managed to cripple the NHS, with some hospitals severely disrupted for several days. 88 of the 236 English trusts were affected and the NHS identified that 6,912 patient appointments were cancelled, with an estimated 19,000 appointments in total being affected, costing the NHS around at least \u00a3180,000. This situation could have been avoided entirely had they simply updated their systems when updates became available several weeks previously. So what does this tell us? It tells us that security measures, technology and people are fallible. There are multiple ways that your data can be compromised and, worryingly, technology is becoming more prevalent in our daily lives but seemingly easier to compromise. But putting the correct procedures and technology in place greatly reduces the risk of being compromised by internal and external threats. This is an integral part of what we do \u2013 we build mechanisms and procedures into your work flow that minimise your risk of exposure. Let us give you a demonstration of centralised key rotation, encryption and user isolation and of how this will help secure your business. What\u0027s likely to become more prevalent in 2018?\u00b6 The big battleground of 2018 is undoubtedly going to be the smart speaker and home automation space, as companies will battle it out to develop meaningful and useful applications. Certainly the first hardware vendor that brings out cross-platform agnostic support, for not only the voice assistant but also the music service, will solidify themselves as an essential building block in the ecosystem. Prepare to see battle lines drawn between existing audio tech companies as they try to differentiate themselves in the market. Also look forward to the \u0027Big Three\u0027 (Amazon, Apple and Google) experimenting with you through subtle tweaks and features, as they look to monetise the platform. Think \u0027targeted paid-for ads\u0027, which I can foresee alienating lots of people. I recently noticed my podcasts now have ads in them; it\u0027s one thing having ads included on your browser but it\u0027s quite another having them invading your own home. Our clients are now seeing, and better still are able to measure, the benefits of implementing DevOps and dealing with their legacy sunset applications, moving these to containers and/or developing new greenfield applications using DevOps tooling. We expect to be helping a lot more businesses to achieve this in 2018, especially now with their understanding of requirements such as GDPR and how this affects their business. Having had this drilled into them during 2017, it is now starting to be addressed, albeit due to the impending deadline! And as the marketing teams at the cloud vendors have got their heads around \u0027Serverless\u0027, regardless of its merit as another viable technology (think of it as your app that doesn\u0027t exist until its called and you only pay when its being used, then it silently disappears again so you\u0027re not paying for idle resources), you will see this term bannered about and pushed more as the latest magic bullet. But something to keep an eye on is a subset of this technology, also under the Severless banner, is \"Function as a Service\" (FaaS) and, more to the point, the OpenFaaS Project that allows you to package up your code/app as a serverless function on Kubernetes-orchestrated infrastructure. I could not end the year without mentioning the meteoric rise in the value of Bitcoin, however the real story here for me is its underlying blockchain technology and the potential to fundamentally change how we build digital services and apps to take advantage of this technology. Finally, expect more hardware/chip/IOT-based vulnerabilities that will not be so easily mitigated by a simple patch - the only safe fix may be to put it in the bin. It\u0027s time for me to log off\u00b6 I\u0027d like end by wishing you all a very happy, healthy and prosperous New Year from all the team. We haven\u0027t been very vocal on our blog during 2017 (however we promise to rectify that in the year to come). We will be recruiting throughout 2018, so anyone wanting to work with an awesome team, with unique benefits should look here for available roles.", "d": "2017-12-31", "g": [], "t": "2017 in Review and a Look Ahead", "u": "insights/tnf-2017-in-review-and-a-look-ahead/"},{"b": "Originally posted on The Network Factory, 6 December 2017. The Network Factory was acquired by AxisOps; this piece is reproduced here as published, unedited, as part of the engineering archive. Here at The Network Factory, we set up a lot of computer systems from scratch. From virtualisation servers and company laptops to IoT and Raspberry Pi-based embedded solutions, we do a lot of software installation and configuration, so it\u0027s really important that this is done as efficiently as possible. Ansible is a crucial part of our approach to this process. For each possible role a computer can serve, a distinct selection of software will be required to enable the computer to serve that role. Take for example a laptop to be used for development of software or web services. That computer will require the ability to run the code of any number of programming languages, to store databases for use in the software or web services being developed, code editing software to use to actually write the code, version tracking software to document changes made to the code, and much more. It will also need all the usuals - web browsers, email software, an office suite, etc. It all adds up. The problem\u00b6 We could go through and manually install each of these, but it would be a bit inefficient and time-consuming. And what if we\u0027re asked to set up 10, or 20, or 50 of these laptops with the same software selection? Our bill would be far higher than it needs to be and we\u0027d get home late. This is why we use Ansible. The solution\u00b6 With Ansible, the deployment of these software selections can be automated in a way that is fast, consistent and efficient. Ansible also provides the flexibility to vary software selection based on different roles or combinations of roles that a particular computer needs to serve. For our software development example, we could define a software programmer role with a selection of commonly-used software development applications. That role could then be combined with the more general computer-user role (adding in our web browsers etc). Our programmer role could also be refined to become specific to a particular programming language. What if it fails?\u00b6 There\u0027s no need to worry about hardware failing. After all, you do have all your work backed up... right? When your hardware fails (and it\u0027s only ever a matter of time), there\u0027s no need for hours to be spent restoring a new laptop to the same state as the machine that failed. Ansible can restore in minutes what would otherwise take hours - simply replace the faulty hardware, tell Ansible what role(s) the computer needs to serve, then let it work its magic.", "d": "2017-12-06", "g": [], "t": "Why We Never Repeat Ourselves", "u": "insights/tnf-ansible/"},{"b": "Originally posted on The Network Factory, 10 April 2017. The Network Factory was acquired by AxisOps; this piece is reproduced here as published, unedited, as part of the engineering archive. Let\u2019s talk about legacy apps.\u00b6 First off we shouldn\u2019t really use the term \u2018legacy\u2019 any more, and rightly so. Legacy has a bad rap. It\u2019s always used as the go-to scapegoat word for anything in the organisation that is no longer shiny and no-one wants to take responsibility for. But let\u2019s be realistic; your legacy sunset app is probably one of your major revenue streams. However, over time your app has spaghettied out of control and you\u2019ve been applying band-aids to it for years. The original developers have long since been promoted, left or died, it\u2019s built on frameworks or technologies which have long since passed end-of-life, or have more security holes than a piece of swiss cheese. So how are we going to address this?\u00b6 With legal frameworks such as GDPR coming into force in May 2018, you, as an individual within an organisation, can no longer bury your head in the sand. This is where modern, agile DevOps practices come into their own. Here at The Network Factory we have expertise in taking your existing app, unmodified, packaging it up and moving it into a modern, compliant platform. Now this may not necessarily address all of the security flaws within your app or, let\u2019s face it, make it look any prettier, however it does go a long way towards fixing the vast majority of the underlying components your stack sits upon, which can be replaced and brought into the modern world, with all the benefits of auditable, compliant infrastructure. An additional benefit is the ability to deploy your app within minutes, as opposed to how long it currently takes. That is, if you can deploy it at all, or recover it from a failed disaster recovery situation without relying on the \u2018backup and keep your fingers crossed\u2019 method. I mean, you have tested this right? Why are we doing this?\u00b6 It is not a magic bullet, however it does go some way to reducing your attack footprint, whilst benefitting from modern practices such as Containerization, Infrastructure as Code, Continuous Integration, Intelligent Logging and Compliance as Code. So before you hail microservices as the solution to all your woes and chuck your sunset app out with the bath water, you may want to take a less risky, more cost-effective approach first, whilst planning for your next iteration. Building on a long-established foundation on Linux, Docker Enterprise Edition can now run containers on IBM z Systems and with the introduction of Microsoft Windows Server 2016, Docker Engine can now run containers natively on Windows. Now with a single cross-platform technology to leverage, you have never been in a better position to modernise your operations with minimal risk. Here at The Network Factory we can help you extend the life and value of your existing apps and automate your platform whilst helping you plan for the future.", "d": "2017-04-10", "g": [], "t": "We Have to Talk About Legacy", "u": "insights/tnf-we-have-to-talk-about-legacy/"},{"b": "Drive caddies pulled out of a chassis used to be a weekly sight round here. While we were moving them, the numbers underneath the storage hierarchy changed. The gap that shaped everything\u00b6 For about thirty years the memory hierarchy had a cliff in the middle of it. RAM access was tens of nanoseconds. A spinning disk seek was somewhere around ten milliseconds, roughly five orders of magnitude slower. The difference between reaching for something on your desk and driving to another city for it. Essentially every piece of systems design from that era is a response to that cliff: Buffer pools and page caches, to avoid going to disk at all. Log-structured writes, because sequential disk access was tolerable and random access was catastrophic. B-trees tuned to a block size, to minimise seeks per lookup. Query planners that count I/Os, because I/Os were the only thing that mattered. None of that was wrong. It was correct engineering against a real constraint. What NVMe did\u00b6 An NVMe SSD hasn\u0027t got the cliff. Latency drops to tens of microseconds, a few hundred times slower than RAM rather than a hundred thousand times. And crucially, random access stops being special; the seek penalty that justified half the designs above simply isn\u0027t there any more. That moves where the cost sits. When the device answers in 20\u00b5s, the kernel block layer, the syscall boundary and the filesystem stop being rounding errors and become a meaningful fraction of the total. Software written to hide a 10ms device is now the slow part. You\u0027ve optimised the drive to another city and left the car on the drive with a flat tyre. Which is why SPDK and userspace drivers exist, and why \"just add another cache layer\" has stopped being automatically correct. Our expectation is that the kernel grows a submission path built for this rather than ceding the fast case to userspace entirely. The current interfaces were designed when a syscall was free relative to the device, and that premise is gone. Existing systems get faster; more importantly, designs that were never worth building become practical. Databases are the clearest case. Much of what makes a serious database hard to operate is machinery for keeping the working set in RAM: sizing the buffer pool, the hours tuning it, the cliff when your data outgrows it, the capacity planning that exists because exceeding memory meant falling off a hundred-thousand-fold performance edge. With storage a few hundred times slower than RAM instead, that edge becomes a slope, and \"does it fit in memory\" stops being the question your whole architecture is organised around. What that changes: a mid-sized organisation gets to hold its full history online. The analysis that today requires a separate warehouse, a nightly export and a specialist team can run against the operational store, because scanning cold data is no longer something you must avoid. A hospital trust can query ten years of records on one box. That is a different set of questions being answerable at all by people without a data engineering budget. For engineering, the toil that goes is cache tuning, and with it an entire family of staleness and invalidation bugs that only existed to work around the disk. Less code, fewer failure modes, a system you can reason about. The energy argument is substantial too. A rack of spinning disks spends real power keeping platters moving so that a seek is merely slow rather than impossible. Flash idles at nearly nothing and answers faster. Collapsing the hierarchy means fewer machines holding fewer redundant copies of hot data to compensate for a slow tier: less hardware, less cooling, a smaller bill. Four drives, all reading FAILED, on one array, 2015. This is the failure that copies exist for. Faster storage removed the copies that were only there to hide a slow disk; it does not remove the ones that are there so the data survives this. Betting on the next tier down\u00b6 Persistent memory is the one to watch and the one to be careful about. Byte-addressable, non-volatile, sitting between DRAM and flash: it would collapse the hierarchy further still, and whole research programmes are assuming it. The first products are announced and due this year, and the research is real. Our view is that the engineering will be fine and the economics are the risk. This tier currently comes from essentially one vendor on one roadmap, needs specific memory controllers to use properly and asks you to rewrite your durability logic around a programming model nobody else implements. Any one of those is survivable. Together they mean a product whose core design assumes persistent memory is a product betting its architecture on one company\u0027s continued enthusiasm for a line it can discontinue without consulting you. So: build the research, use the hardware and keep your durability model able to fall back to flash. The lesson we would write down now rather than learn later is that a technically excellent thing can lose on economics and ecosystem, and that would be no verdict on the engineering. The ideas transfer; a lot of this work is already pointing at coherent interconnects over PCIe, which is the version with more than one vendor in it. Who controls the roadmap\u00b6 Storage looks like a commodity and is not. NAND and DRAM come from a handful of manufacturers, and the fast-interconnect and persistent-memory specifications are steered by a smaller handful still. Expect the capability to arrive first as a cloud instance type: the latency you want, available to rent, months or years before you can buy the part for your own chassis, with the interesting features exposed only through one provider\u0027s API. That is how a hardware advantage becomes a hosting advantage. The hyperscalers already buy flash at prices and volumes nobody else is offered, so the cost-per-IOPS gap between renting and owning widens even when the technology is open. The privacy consequence of cheap, fast, dense storage is that nothing gets deleted. Retention policies have always been enforced in practice by storage costing money. When a decade of logs costs less than the meeting to discuss deleting them, the default becomes keep everything, and every one of those stores is a subpoena target and a breach waiting for a configuration error. The decision to make scans cheap is also a decision to make total recall affordable, and somebody should be choosing that on purpose. Here at AxisOps we buy the drives and own the box. The collapse of the hierarchy is the best argument for self-hosting in a decade: the hardware that used to require a storage array and a consultant now fits in a 1U server. The practical check\u00b6 If you maintain anything with a cache in it, the question is whether that cache is still earning its keep or whether it\u0027s a monument to a device you no longer own. The answer is measurable, and it\u0027s frequently no. Caching costs you something: invalidation, memory and a whole family of staleness bugs. Against a 10ms seek it was obviously worth paying. Against 20\u00b5s it\u0027s sometimes a straight loss, and you\u0027re maintaining the complexity for nothing.", "d": "2017-01-06", "g": ["storage", "hardware", "performance"], "t": "The storage hierarchy collapsed and nobody rewrote the assumptions", "u": "insights/storage-hierarchy-collapsed/"},{"b": "For most of networking\u0027s history the dataplane was something you configured. Vendor hardware, vendor firmware, a CLI and a feature set you either had or waited a release cycle for; rather like a car where the heater comes with the trim level and that\u0027s the end of the discussion. Over the last four years that has stopped being true, from four directions at once. The four strands\u00b6 SDN and OpenFlow split the control plane from the forwarding plane. The idea was right and the first implementation was far too rigid, with a fixed set of match fields and a spec revision needed for every new protocol. OpenFlow is fading; the separation it argued for isn\u0027t. DPDK went the other way: leave the hardware alone and haul the dataplane up into userspace, bypassing the kernel network stack altogether. Poll-mode drivers, hugepages, cores pinned to packet processing. Enormous throughput gains, and the price is owning a parallel universe with none of your kernel tooling in it. XDP splits the difference. Run an eBPF program at the earliest point in the driver, before sk_buff allocation, and drop or redirect right there. You get most of the bypass benefit and you\u0027re still in the kernel, with kernel tooling and kernel safety guarantees. The core support is queued for 4.8, which is in release candidates as we write (LWN): the program type, context and verifier support land, with the driver hooks following one NIC at a time. P4 goes to the far end and makes the hardware programmable: describe the parser and the match-action pipeline, compile it for the target. The ambition is the biggest of the four, a switch whose protocol support is software. What they have in common\u00b6 Each one replaces \"the vendor decided what this device can parse\" with \"we decide\". The pattern appears in half a dozen places at once: take something hardware fixed for you and make it specifiable instead. The dataplane version happened in networking first, which is why it tends to get filed on its own. Which of these we would bet on\u00b6 This field has produced rather more conference talks than deployments, so a prediction rather than a tally: XDP wins the general case. Verified program, kernel tooling, no second universe to operate. We expect DDoS mitigation and load balancing first, because those are the workloads where a packet you can drop in the driver is worth real money, and then everything else. DPDK keeps telco and appliances, where the throughput earns back the operational isolation, and does not escape them. P4 stays narrow, tied to whichever programmable ASICs ship in volume. OpenFlow mostly goes, and its central argument wins outright. The specific protocol dies and the architectural argument wins. That pattern is one to notice, because it keeps happening. It is roughly what is becoming of OpenStack\u0027s API ambitions too. The practical gain is a deploy cycle measured in minutes. A new attack appears on a Tuesday. Today you open a vendor case, you are told the match field you need is in next quarter\u0027s firmware, and in the meantime you filter in userspace at a tenth of line rate or you buy a box. With a programmable dataplane you write the filter, load it and drop the traffic in the driver on hardware you already own. The university or regional ISP that cannot afford a scrubbing-centre contract gets to defend itself with a commodity server, which is a real redistribution of capability away from the people who can afford appliances. The second scenario is the one we care about more. Protocol work becomes possible for people who are not a vendor. If you can describe a parser and compile it, you can deploy a new transport, a new encapsulation or a new congestion signal on your own network and measure it on real traffic, without persuading a chip company that your use case is worth a tape-out. That is the difference between a standards process driven by whoever ships silicon and one driven by whoever can demonstrate something working. For engineering the toil that disappears is the vendor-feature wait, and with it a whole category of architecture written around what the box happened to support. Designs stop being shaped by a datasheet. The compiler and the silicon behind it decide who may program the pipeline; the language is open in principle and closed in practice. P4 is a published language with an open specification (SIGCOMM CCR, 2014 for the design; the Consortium\u0027s spec is at version 1.0.2), and the only way to get a P4 program onto a fast switch is a vendor\u0027s backend compiler for a vendor\u0027s ASIC, under NDA, with the pipeline\u0027s real constraints documented only to licensees. A language anybody can write and one company can execute is not an open platform; it is a very good funnel. The way this goes wrong is P4 becoming the marketing name for one vendor\u0027s SDK, with each generation\u0027s quirks forcing rewrites and the portability story never quite arriving. We would like to be wrong about that and would not bet on it. The merchant-silicon market is already down to a handful of suppliers for the parts that matter, and a programmable pipeline makes that concentration worse rather than better, because the software investment is per-target. Every line of pipeline code you write against one ASIC family is a switching cost you pay later. Then there is the surveillance consequence. A dataplane you can program at line rate is a dataplane that can classify, mirror and log at line rate, on boxes already sitting at every exchange and every carrier edge. The same capability that lets a regional ISP defend itself lets a national operator implement anything a lawful-intercept order asks for, as a software update, with no procurement and nothing to notice. Interception stops being a box someone has to install. XDP is where the open version lives, and it is keeping up: the program runs on the NIC you bought from anyone, the verifier is in the kernel you can read and nobody has to approve your filter. Here at AxisOps that is the one we invest in. We would rather have a commodity server doing 10Gb of programmable forwarding we can inspect than a clever switch whose pipeline we are licensed to describe but not to understand. Why it matters beyond networking\u00b6 Because XDP is evidence that the eBPF model generalises. A verified program, loaded at runtime, running in a hot kernel path, fast enough for line rate. If that works for packets, the obvious question is what else it works for. We expect the answer to be observability, security enforcement and eventually scheduling, which is why this counts as one of the five strands rather than a networking footnote.", "d": "2016-09-14", "g": ["networking", "ebpf", "sdn"], "t": "The network stopped being fixed-function", "u": "insights/programmable-dataplane/"},{"b": "Five Pis, a desktop switch and an unreasonable quantity of green patch cable, photographed at some unflattering hour. The compute is negligible by any 2016 standard; about as much total RAM as one modest server. That was never the point. What a real cluster teaches\u00b6 Running a distributed system on one machine with processes on different ports teaches you the happy path and nothing else. It won\u0027t teach you: Partition. Pull a cable out and the system has to have an opinion. Pulling a cable out is very easy; simulating it on one host is fiddly enough that people quietly skip it. Asymmetric failure. A node that can receive but not send is a different animal from a node that\u0027s down, and almost every naive implementation treats them as the same thing. Clock behaviour. Five machines with no PTP and indifferent NTP drift relative to one another in a way a single host simply doesn\u0027t. Boot order. Nothing ever comes up in the order the diagram implies. Those are the properties that make distributed systems hard, and they\u0027re the ones a single-host test quietly elides. Before all the cabling. The dev boards alone are not a distributed system; it is the cables, the software and then the testing that teach you something. The ARM part, which is going to matter\u00b6 The Pi is a toy for this purpose and the ARM cores are slow. What they are not is x86, and anything that only builds on x86 announces itself on the first attempt. Endianness assumptions, inline assembly, unaligned access that x86 forgives and some ARM instruction sequences still fault on, a binary dependency with no port: all of it surfaces in the first hour on a \u00a330 board rather than in the first week of a migration. We will take the bet that this is the useful part. ARM in the server room is coming, and the argument is performance per watt rather than performance. The Pi 2, the size of a credit card, is doing real work on a phone charger, and a rack full of cores built to that constraint changes the electricity bill, which is the bill that limits a datacentre in the end. Somebody will ship a serious ARM server part and sell it on energy rather than single-thread speed, and the people who can take it will be the ones whose software already builds clean on a second architecture. That is a cheap option to hold. Keeping your build working across two architectures costs a CI runner and some discipline; acquiring the capability after the hardware arrives costs a project. The wider benefit is access, and it is larger than the hobbyist framing suggests. A cluster used to be a thing you needed an employer to see. Partition behaviour, quorum loss, split brain: the engineers who understood those were the ones who had worked somewhere with a datacentre. \u00a3150 of boards puts the same education in a bedroom, which means the next generation of distributed systems engineers will not be drawn only from people who got the right graduate job. We think that shows up in the quality of the software within a decade. The same arithmetic travels. A school, a clinic or a field station in a place with expensive power and bad connectivity can run real local services on hardware that costs less than the annual bill for a hosted equivalent, on a solar panel if it has to, and keep working when the link drops. That is not a compromise version of computing; for a lot of the world it is the only version that is dependable. That accessibility depends on supply rather than software. The Pi is affordable because one foundation negotiated one SoC deal with one vendor and sells at near cost. There is no second source. The boards are already hard to get in volume at points in the year, and a board you cannot buy is not a platform. If ARM in the server room arrives the way we expect, the parts that matter will be designed by a handful of companies and, more to the point, fabbed by two or three. The first customers in the queue for any constrained process node are the companies buying a hundred thousand units. Everyone else waits. Cheap, inspectable, generally available ARM hardware is a market condition rather than a law of nature, and market conditions revert. The architecture licence is the other chokepoint. ARM is one company\u0027s instruction set, licensed on its terms, and a world where the energy-efficient future runs on it is a world where that company sets a price for participating. The open instruction sets being worked on now are years behind and worth following for exactly that reason. The privacy angle is the small nodes themselves. The same \u00a330 computer that lets a clinic run its own records is the one that makes a camera with a network stack cost nothing. The cheap-sensor build-out arriving over the next few years will put far more surveillance capability in far more hands than any policy process is prepared for. The technology is neutral and the deployment will not be. Here at AxisOps we buy hardware we can own, keep two architectures in the build and prefer the version of any system that runs on a board we can replace from a different supplier. A platform with one source is a dependency wearing a friendly face. Would we do it again\u00b6 Yes, and we have, with better hardware. A physical cluster costs so little that the alternative (reasoning about partition behaviour off a diagram) is a false economy. You can hire a van and move house, or you can draw the van on a whiteboard and hope the sofa fits. The photograph is here because the cabling is the real part. The whiteboard version of this has five neat boxes and five neat lines.", "d": "2016-02-03", "g": ["hardware", "distributed-systems", "arm"], "t": "A Pi cluster teaches what a diagram cannot", "u": "insights/raspberry-pi-cluster/"},{"b": "A box of unboxed kit, 2015. The physical layer kept right on existing throughout a decade of people explaining that it had been abstracted away. Each wave was a real improvement that created the next wave\u0027s problem. Wave one: virtualise the machine\u00b6 VMware, Xen, KVM. One physical server becomes many logical ones. The win was utilisation; physical servers were running at single-digit percentages because every application wanted one of its own. What it didn\u0027t change was the unit. A VM is still a machine. It has a kernel, an init system, a package manager, a hostname and somebody whose job it is to patch it. You now had more machines than you started with, which is why configuration management (CFEngine, Puppet, Chef, later Ansible) rose in this period: the machine count had outgrown hand-tending. The new problem: VM sprawl. Provisioning got easy and de-provisioning never did. Rather like a garage: getting things in is effortless, and nobody ever takes anything out. Wave two: abstract the machine away\u00b6 OpenStack\u0027s ambition was an open API over pooled compute, storage and network, so that a machine became an API call. The API argument has won completely. OpenStack itself largely hasn\u0027t, and the reason is instructive: the operational burden of running it crept up on the burden it was supposed to remove. A control plane with that many services, each with its own database and message queue, is a distributed system that needs its own team. The new problem: the unit was still a machine, so you still had machines to patch. You just patched them via an API. Wave three: schedule against a pool\u00b6 Containers plus an orchestrator. Now the unit is a process with its dependencies, and the machine really does recede: you declare what should run and a scheduler places it. This one is sticking, and it\u0027s a real advance. What the declarative model changed: the system has a desired state written down, and a controller continuously reconciling towards it. That\u0027s the durable idea, far more than containers ever were. Reading a VMFS datastore from Ubuntu with vmfs-fuse. What used to fill a server room now runs in one box, abstracted until it looks like a single machine. When the hypervisor that owned this filesystem will not boot, the state on the disk is still there and still yours, which is the whole argument for knowing where your state actually lives. The new problem, and it\u0027s a subtle one: the abstraction is excellent right up until it leaks, and when it leaks you need to understand every layer below it at once. Kernel, container runtime, the network plumbing, the scheduler\u0027s own control loops. The operational floor is higher than before, not lower. Teams adopting it in search of simplicity are generally not finding any. What reconciliation makes possible\u00b6 Take the declarative loop seriously and some long-standing expensive things get cheap. Recovery stops being a procedure. Today a failed host means a runbook, a person and an hour; under a reconciling controller it means the controller notices the declared state is unmet and places the work elsewhere. The 3am call becomes a graph you look at on Monday. That is the single largest quality-of-life change available to anyone carrying a pager, and it is available now rather than in five years. It also puts real infrastructure within reach of organisations that cannot staff a team for it. A small council, a charity, a two-person engineering shop: the thing that used to stop them running their own systems was the number of humans needed to keep twelve machines patched and consistent, not the hardware cost. Declared state plus a reconciler is a real substitute for headcount, and that is the first time it has been true. Our bet is that this is what lets self-hosting come back after fifteen years of losing to convenience. For engineering the class of bug that dies is configuration drift. The machine that works because somebody fixed it by hand in 2013 and nobody knows what they did has been the standard shape of operational risk for a generation. A system whose state is written down and continuously enforced does not accumulate those. Who owns the pool\u00b6 The reconciliation model is open and the schedulers are open source, so the chokepoint sits elsewhere: the control plane offered as a service. Watch what is being built: the orchestrator offered as a managed endpoint, with the cloud\u0027s own load balancer, its own identity system, its own storage classes and its own network plugin wired in beneath. Each of those integrations is reasonable on its own and the sum is an orchestrator that only reconciles against one company\u0027s infrastructure. The declarative file you wrote looks portable and is not, because every resource it names resolves to a proprietary service. That is a far better lock-in than VMware ever managed, because it arrives as convenience rather than as a licence. The surveillance consequence needs stating plainly. A managed control plane knows every workload you run, every image you deploy, every secret you mount and every scaling event, which together describe your business in more detail than your accounts do. Three companies holding that for most of the economy is a concentration of commercial intelligence with no precedent, and it is available to any government that can compel disclosure from them. Here at AxisOps we build the version you can own: your hardware or someone else\u0027s, the same declared state, the controller running where you can read its logs. We are against the shape of this technology that only works if you rent it. The open implementations are keeping up today, and keeping up is not automatic; it depends on people continuing to run them on their own kit. And the declaration can still lie\u00b6 Legibility. A 2005 server was knowable. You could log in and read the whole thing: which processes, which config files, which cron jobs. Tedious, certainly, but bounded. A modern cluster isn\u0027t knowable that way by any one person. The compensation is supposed to be that it\u0027s declared rather than inspected, and that holds only while the declaration really is the source of truth. The moment anybody makes a change outside it, the system is in a state nobody has written down, and nothing whatever announces the fact. That\u0027s the cost of the trade, and it\u0027s why the discipline that matters most in this model has nothing to do with tooling: it is the refusal to make manual changes.", "d": "2015-11-24", "g": ["virtualisation", "containers", "infrastructure"], "t": "The machine disappeared; the state did not", "u": "insights/virtualisation-to-orchestration/"},{"b": "Control groups merged into Linux 2.6.24, released January 2008 (kernel.org). For most of the six years that followed, they were something your init system set up and you never thought about again. Then containers staked real isolation promises on them, and a design that had grown one controller at a time started showing its seams. The v1 problem\u00b6 v1 lets you have a separate hierarchy per controller. You can mount memory with one tree and cpu with a different one, and a process can sit at inconsistent positions in each. That sounds like flexibility. In practice it makes two things very hard: Reasoning about a process. Answering \"what limits apply here\" means walking several unrelated trees and hoping. Delegating a subtree safely. Without a single hierarchy there\u0027s no single thing to hand over. It also produces real ambiguity in accounting. There\u0027s no unarguable answer to which cgroup a page gets charged to when it\u0027s shared between processes in different cgroups, and v1 made several choices inconsistently across controllers. What v2 changes\u00b6 A single unified hierarchy. Every controller sees the same tree. Delegation becomes a coherent operation, because a subtree is a subtree for all controllers at once. Tejun Heo\u0027s rework has been landing piecewise since 3.16 behind the __DEVEL__sane_behavior mount flag (kernel source), and the direction is settled even if the merge is not. The cost is a compatibility break, and that is the part to plan for. Every container runtime, every init system and every monitoring agent that reads /sys/fs/cgroup is written against v1 paths. Our bet is that distributions will keep defaulting to v1 for years after v2 lands, that runtimes will carry both code paths for most of a decade, and that you will be reading cgroup.controllers and memory.stat with two parsers for the whole of that period. Budget for the second parser now. Delegation is the part worth being excited about, and it is the part nobody is talking about yet. A coherent subtree is a thing you can hand to someone who is not root. Here at AxisOps that is the capability we want. Give a developer a slice of a real machine, with a hard memory ceiling and a CPU weight, and let them own it. No hypervisor, no second kernel to patch, no ticket to the infrastructure team to resize anything. The university department that today buys four underused servers because three research groups will not share one can run all three on one box and have the accounting to prove nobody is being robbed. The hosting company that today sells you a VM because that is the only unit it can bill safely can sell you a process instead, at a fraction of the overhead. For engineering the win is a whole class of outage disappearing. The runaway job that eats all the RAM and takes the host with it is the oldest shared-box failure there is, and memory.max on a delegated subtree contains it: the job dies inside its own budget instead of taking the box with it. The box still shares a kernel, CPU and I/O, so the neighbour is contained rather than abolished. Every on-call engineer has been paged for the runaway job at least once. And the mechanism is in your kernel, readable, with no licence and no account. That matters more than it sounds. Who ends up owning this\u00b6 The kernel side stays open; nobody can take /sys/fs/cgroup away from you. Ownership changes one layer up, at the registry. docker pull goes to one default public index run by one company. The image format is open and so is the daemon, but the place everybody fetches from is neither distributed nor accountable. That is where the leverage sits: whoever runs the default registry decides what your build depends on, sees every pull your estate makes and can price or revoke access once enough of the industry has wired it into CI. A pull log is a remarkably good map of what a company is building and when it ships. Our view, and we will keep saying it: run your own registry, mirror what you depend on and treat the public index as a convenience you can survive losing. The technology does not require a landlord. The default configuration supplies one anyway, and defaults are how estates get captured. Why this one matters\u00b6 It\u0027s the clearest example of a pattern that keeps returning: a mechanism designed as plumbing becomes an interface the moment something popular depends on it, and decisions that were fine as plumbing become constraints everyone downstream inherits, overnight. It\u0027s the extension lead running under the carpet because it was only meant to be there for a fortnight; now the whole office is plugged into it. The lesson isn\u0027t \"design for the future\". Nobody was going to predict containers from 2008. It\u0027s that the second system has to carry the first one for years, and that cost belongs in the estimate.", "d": "2015-03-11", "g": ["linux", "kernel", "containers"], "t": "cgroups, and the decade before anyone noticed", "u": "insights/cgroups-v2-pre-history/"},{"b": "Docker 1.0 was announced at DockerCon on 9 June 2014 (LWN). The reaction was immediate. The useful question for a lab is narrower: what is actually new? Not the isolation. Control groups were merged in Linux 2.6.24, January 2008 (kernel.org). Namespaces arrived incrementally over roughly a decade, with the user namespace the last to be completed, in 3.8 in February 2013 (LWN). LXC has been using both since 2008 (LKML). Everything Docker 1.0 isolates with, the kernel already did. What is new\u00b6 Three things, and none of them a kernel feature: A layered image format. Copy-on-write filesystem layers, so a base layer is fetched once and then shared. A registry. docker pull against a default public index. This is the one that will matter most and gets discussed least. A one-line build description. A Dockerfile is a worse build system than almost any build system you could name. It is also one anybody can write in five minutes, which is why it is winning. That is a distribution story, not an isolation story. Why the distinction matters\u00b6 Because what is getting adopted at scale is weaker than what it replaces, and the reason is ergonomics. A VM gives you a hypervisor boundary. A container gives you a kernel-shared boundary: one kernel, with the guest separated by namespaces, cgroups and whatever the seccomp and capabilities posture happens to be that week. Those are real mechanisms. But a kernel vulnerability is a shared-fate event across every container on the host in a way it simply isn\u0027t across VMs. Flats with a party wall against houses on separate plots; both are homes, and only one of them lets you hear next door\u0027s plumbing. The industry is making that trade knowingly at the top and largely unknowingly in the middle. Today the default docker run is root in the container, with no seccomp filtering and no user namespace remapping at all. Both get talked about, and neither has shipped. Here at AxisOps we will take the bet that the packaging problem is the one that was worth solving, and that it is now largely solved. Nobody is going to argue about tarballs again. Which leaves the isolation bill outstanding, and it will be paid. The shape we expect: something that puts a real boundary back underneath the container without giving up docker run. A thinner hypervisor, a syscall interposition layer, a kernel that can be convinced to lie to a process more thoroughly than namespaces currently manage. We do not know which of those wins and we would not bet on one. We will bet that the next five years of container engineering is mostly this, because the ergonomics are already settled and the security posture plainly isn\u0027t. If you are putting containers into production now, that is the thing to plan for: the boundary you are running today is the weakest one you will run. Build as though it is going to be replaced, because it is. Who ends up owning the registry\u00b6 One thing in the list above is not like the others, and it is the one nobody is arguing about. A layered image format is a specification; anybody can implement it. A Dockerfile is a text file. But docker pull redis resolves against a single default public index operated by one company, and that default is doing an enormous amount of work. It decides what \"the\" Redis image is. It sees every pull, which over an estate is a rather good map of what you are building and when you are deploying it. And it is the point at which a free service can later become a metered one, because by then every Dockerfile, CI pipeline and runbook in the industry has the unqualified name baked into it. Here at AxisOps we think the registry, not the runtime, is the part to watch. Run your own, mirror what you depend on and write image references that name the registry explicitly so that moving is an afternoon\u0027s work rather than a project. The format being open is what makes that possible; the default being convenient is what will stop most people doing it. The same question will arrive for every layer of this. Whoever runs the index everybody defaults to is the one collecting the rent, and conveniences are how defaults get set. The plain version\u00b6 Containers are winning because docker run works on a laptop and in production and the difference stops mattering. That is a real engineering achievement and we should say so plainly rather than retrofitting a security story onto it afterwards.", "d": "2014-06-18", "g": ["containers", "linux", "kernel"], "t": "Docker 1.0, and what we actually adopted", "u": "insights/docker-1-0/"}]