Running a capable language model on your own hardware stopped being exotic. The engineering underneath that shift is more interesting than the fact of it.
Quantisation, and what it costs¶
A model's weights are trained in 16- or 32-bit floating point. Quantisation stores them in fewer bits (8, 5, 4, sometimes fewer) with a scale factor per block to keep the range. Dropping from 16-bit to 4-bit cuts memory by roughly four and degrades quality far less than that ratio implies, because the information is spread redundantly across an enormous number of parameters.
It isn't free. The degradation is uneven. It shows up first in long-context coherence and in tasks needing precise recall rather than in fluency, which means casual testing systematically understates the loss. A quantised model sounds just as good while being measurably worse at the things you'd want to use it for. Rather like a cover band; fine from the back of the room, less convincing up close.
Memory bandwidth is the constraint¶
Generating one token means reading essentially every weight. A 7-billion parameter model at 4-bit is roughly 4GB, so each token needs around 4GB of memory traffic. At 50GB/s you can't exceed about 12 tokens per second however fast the arithmetic unit is. The ceiling for single-stream inference is memory bandwidth divided by model size, and the compute sits idle, waiting. Which is why:
- Unified memory architectures do unexpectedly well; high bandwidth to a large pool beats a fast GPU whose VRAM can't hold the model at all.
- A GPU that has to swap layers across PCIe performs dramatically worse than its specification implies.
- Batching helps throughput enormously (the weights get read once and many sequences advance) and does nothing for single-user latency.
So for local single-user inference the amount of fast memory matters more than the arithmetic throughput, which is the opposite of the intuition most people bring over from gaming hardware.
The open question, and the measurement that settles it¶
The number everyone wants is tokens per second on a named part, and it is the one nobody should quote loosely: it moves with the model, the quantisation, the context length, the runtime and the build flags, so a figure without all five is noise.
The bandwidth analysis above is arithmetic from published specifications and holds regardless. What it does not tell you is how far real runtimes fall short of that ceiling. The experiment is cheap: fix the part, the model, the quantisation, the context length and the runtime version, publish the command and report the measured rate against the bandwidth-derived bound. That ratio is the interesting figure, almost nobody publishes it and it is on our bench next.
No prompt leaves the building. A GP surgery can run clinical summarisation over patient notes on a machine in the back office, and no record, no identifier and no query crosses the practice's firewall. A solicitor can run discovery across privileged material without the question of whether a third-party API waives privilege, because the question does not arise. The blocker in both cases is not capability; the models are good enough for a draft a professional then checks. It is that the only capable option means posting the most sensitive text the organisation holds to somebody else's server, under terms that can change, in a jurisdiction whose legal process does not answer to the data subject. Local inference removes that transfer, and with it the risks that ride on the transfer.
The data protection story is substantially different. No processor agreement, no transfer mechanism, no third party in the chain, because nothing was transmitted. The duties that attach to holding the data stay exactly where they were: retention, access control and breach notification are the controller's whether the model ran locally or not, and the machine in the back office is now part of the attack surface. Anybody who has filled in a DPIA for an API-based AI feature knows how much of that document disappears, and which pages remain. For a regulated organisation it is a year of legal review against an afternoon.
It works where the network does not, so a clinic with four hours of grid power and no uplink can run the same model a teaching hospital runs. And it does not change underneath you: no deprecation notice, no silent model swap, no rate limit, no price rise, no provider deciding your use case is off-policy. The weights on your disk will behave the same way in five years, which for anything in a clinical process beats the last few percent of benchmark performance. The toil that disappears with all of it is the apparatus of guarding an outbound connection: prompt sanitisation, token budgeting, API key management, reconciling a bill nobody can predict.
The version of AI worth building toward is the one that still works when you stop paying somebody else to look at your most sensitive data.
The chokepoints, and there are three¶
The weights. You cannot run locally what you cannot obtain. Open-weight models are released at the discretion of a small number of large companies that also sell the hosted alternative. Licences are increasingly "open" with usage restrictions that would not pass as open source anywhere else, and the strongest models are not released at all. Open weights trail the frontier by something like a year and the gap is not obviously closing. If it widens into a permanent two-tier arrangement, local inference becomes the cheap option rather than the good one. This one is the hinge.
The accelerator supply chain. High-bandwidth memory and leading-edge packaging are made in very few places, and the output is on multi-year allocation to the largest buyers. When the hardware that makes local inference viable is allocated to the companies selling hosted inference, the market is not functioning as a market, and export controls add political allocation on top of the commercial kind. The bandwidth analysis above is the optimistic note: local inference wants large fast memory, not exotic compute, and that is a much broader supply base.
The distribution of what you run. If the practical way to get a model becomes a vendor's hub with its own terms, telemetry and acceptable-use enforcement, then "running it locally" means running a file whose availability somebody else controls. Keep your own copies, hash them and treat a model you cannot re-download as a dependency you have already lost.
Prompt logs are a record of what an organisation is worried about and what an individual is thinking, retained by default, discoverable by legal process and valuable as training data. Hosted inference is the most intimate telemetry stream ever built; local inference makes that record not exist rather than promising it will be handled well.
The open ecosystem is in better shape than the gloom suggests: excellent runtimes, first-rate community quantisation work and a private assistant you can stand up in an afternoon on hardware costing less than a year of API fees. It is losing on one axis only, access to the best weights, and that is a licensing fight rather than a technical one. Which means it can be won.