If the world is short of data centres, how much of its AI could run on machines people have already bought?
A colleague summed up one response to the compute crunch in a single sentence: individuals and companies should buy their own GPUs. He is right about the direction and, I think, wrong about the purchase. Most people and most companies already own a great deal of compute. It sits in their laptops, desktops and phones. It is already paid for, already powered and already connected, and for most of the day it does very little.
In The Compute Strait I argued that scarce AI will reward those who waste less of it: code before models, memory before inference, small models before frontier ones. That essay was about how much intelligence a job needs. This one is about where the intelligence that remains should run. The question is no longer only which model should do a task. It is also where the task should execute.
The argument is this. Language understanding is becoming a standard capability of the device, like graphics or encryption: something an application calls rather than buys. Some inference can therefore move from the data centre to the browser, the operating system and the enterprise’s own machines. It is not free, because cost moves from the provider to the user and the integrator. It wins where use is repeated, data is sensitive, latency matters or connectivity is patchy, and only on devices that qualify. The durable assets are the workflow, memory, permissions, evaluation and routing, not the local model. And a new strait forms around the companies that own the browsers and operating systems.
1 The question after the strait
AI has two workloads. Training creates or improves a model, and it needs large clusters of chips joined by very fast connections, because the chips exchange enormous amounts of data as they work. Inference is everything that happens afterwards: every summary, translation, classification, recommendation and answer. A trained model can be copied, and each copy can run on its own without talking to the others. Training has to happen where the clusters are. Inference can happen wherever the model can be copied.
Inference is also where demand is heading. McKinsey’s analysis of scaling AI inference expects it to account for about 60% of AI demand by 2030, against about 40% for training, growing at around 36% a year. That is a forecast, not a measurement. But the direction is consistent with what any business deploying agents can see in its own bills: training is paid occasionally, inference is paid on every request, for ever.
Almost all of that inference is assumed to travel to a data centre. AI strategy, for companies and countries alike, is organised around building and renting central capacity. Meanwhile, billions of existing devices are gaining the ability to run useful models locally. Before asking how many more GPUs the world needs, it is worth asking a smaller, more practical question: how much inference can we stop sending to the cloud?
2 Intelligence becomes a device capability
An application does not own a data centre to draw graphics, play video or encrypt a message. It asks the device, which already has the hardware and the software to do it. Language understanding is joining that list. A website or an app can now ask the device to classify a request, extract the fields from a document or summarise a thread, and the device answers without sending anything to a server.
Three routes lead there. The first is the browser. Chrome made its Prompt API available to web pages as a stable feature from Chrome 148, running Google’s Gemini Nano on the user’s machine, and the model is downloaded once and shared across every site that uses it. The second is the operating system. On Android, apps reach a shared on-device Gemini Nano through ML Kit’s GenAI APIs, built on a system service called AICore, and Google has reported that Gemini Nano now runs on more than 140 million devices. Apple’s June 2026 model family includes a dense 3-billion-parameter on-device model and a 20-billion-parameter model that is stored in flash memory and activates only 1 to 4 billion parameters per request, a design that lets device models grow beyond the limits of working memory. On Windows, Microsoft offers three layers: built-in AI APIs such as the Phi Silica language model, which supports fine-tuning; Foundry Local for running open models; and Windows ML for running a developer’s own models. The third route is developer-supplied. Runtimes such as WebLLM run open-weight models inside the browser on the device’s graphics chip, at up to about 80% of native speed in benchmarks, and ONNX Runtime Web does the same for a wider class of compact models.
Figure 1. Three routes to the device
| Browser-native | Operating-system native | Developer-supplied | |
| Who supplies the model | The browser maker | The phone or computer maker | You |
| Cost to distribute | None to the developer; shared across sites | None to the developer; shared across apps | You ship and update the weights |
| Can you tune it to your domain? | No | Limited: some platforms allow fine-tuning or adapters | Yes |
| Reach | Eligible desktops and Chromebook Plus devices | Eligible phones and computers | Any device with the runtime and the memory |
The trade-off is the one that matters for anyone building on top. A platform model is free to distribute and impossible to make your own. Your own model knows your domain, and you pay to deliver it. Most real systems will use both: the platform model for general language work, and small models of their own for the narrow tasks where domain knowledge decides the result.
It also helps to be precise about what “local AI” means, because it is not one technology. It covers three different things: deterministic logic, which has always run on the device; specialised machine-learning models, often a few megabytes, that classify, rank or recognise; and small generative language models of a few billion parameters. Android shows the difference. Its on-device speech recognition has a basic mode built on a conventional model that runs on most Android devices, and an advanced mode built on Gemini Nano that currently runs only on Pixel 10 phones. The narrower the intelligence, the more devices can run it.
3 Where should inference run?
The Compute Strait put code and memory ahead of any model. That still holds. What changes is that the remaining model work now has four possible locations, not two.
Figure 2. Four places inference can run
The two edges have different buyers. On the user’s device, the user supplies the compute. At the enterprise edge, the company controls it and manages it across employees.
Routing decides which location gets each request, on five criteria: whether the location can do the task, whether the data may leave the device, how fast the answer must arrive, what it costs, and whether this particular device qualifies. The router must itself be cheap. It should start with rules and capability checks: is a local model present, does it support this language, does this task class complete reliably on it? Only then should evaluation decide whether to escalate. A system that asks an expensive model whether it needs an expensive model has already lost. The same applies to councils of small models: useful when a specialist is called for a specific case, wasteful as a committee convened for every request.
One version of this idea does not work, at least not yet. It is the vision of idle devices pooled into a public cloud that computes for strangers. Browsers restrict background work. On Android, on-device generative inference is permitted only when the app is in the foreground, and each app has an inference quota. Devices go offline, users reasonably object to strangers spending their battery, and large models cannot be split efficiently across distant consumer connections. The device computes for its owner. Asking it to compute for strangers is a different and much harder business.
Inside a company, the picture changes. An office pool does not split a model across machines. Each idle machine runs a complete model, and an office server routes each question to whichever machine is free, so nothing has to cross a slow connection mid-calculation. The machines sit on the office network, IT manages them, and the employer owns them, so consent is a matter of employment policy rather than a marketplace. A machine can serve while its owner is away and it is plugged in; if it leaves in the middle of an answer, another machine continues from the last word. In one working prototype I have seen, the gap at that hand-off measured a few seconds. Well-specified desktops and laptops, particularly those whose memory is shared between processor and graphics chip, can hold capable open models, so the pool can answer colleagues’ questions interactively, not just run overnight jobs. The line is not between batch and interactive work. It is between colleagues and strangers.
Three questions decide whether an office pool pays. The first is peak-hour coverage. Supply and demand run against each other: laptops are free when their owners are away, which is also when fewer people are asking. An always-on machine is therefore the backbone, and idle machines are burst capacity; the number to measure is the share of answers at the busiest hour that the idle machines serve. The second is who can read the question. Traffic between machines can be encrypted, but the text must be decrypted on the machine that answers, so a sensitive question served from a colleague’s laptop sits, briefly, in that colleague’s memory. Sensitive classes of question, such as legal or HR matters, should run only on machines IT controls, by default rather than by request. The third is the owner’s experience: when the owner returns, the machine must yield at once, with no noise, slowdown or drained battery. The comparison that settles the economics is the pool against a single dedicated office server, fully loaded.
One boundary is worth stating plainly. Content sent inside an email cannot invoke arbitrary local models on the reader’s device, even if the email application’s own software uses local AI. For a brand, local AI belongs on surfaces it owns, such as its apps and web pages.
4 Not free, just paid by someone else
Local inference removes the provider’s per-request charge. It does not remove cost. The provider still pays for integration, distribution, testing across many device types, security and a fallback for every device that cannot run the model. The user pays in electricity, battery and disk: Chrome’s built-in models need 22 GB of free disk space. Whether that adds up to a saving depends on what the device is being compared with.
An illustration makes the point. Take a service with a million users, each asking for ten small language tasks a day: summarising and classifying a work thread, say. That is 300 million tasks a month. Assume a batched central small model costs $0.0002 a task, a frontier model $0.01, and the device route carries $40,000 a month in integration, testing and fallback costs. Assume 30% of devices are eligible and 70% of their tasks complete locally, so 21% of all tasks never reach the cloud.
Figure 3. The device against two alternatives (all inputs assumed; equal completion assumed)
| Design | Monthly cost to the provider |
| All tasks on a central small model | $60,000 |
| Device first, central small model as fallback | $87,400 |
| All tasks on a frontier model | $3,000,000 |
| Device first, frontier model as fallback | $2,410,000 |
Device-first costs = 237 million fallback tasks at the central price + $40,000 fixed. Break-even against the central small model: the device route must add less than $12,600 a month in operating cost. The frontier rows are a reference point, not evidence for the device: a task a central small model completes at equal quality should never be benchmarked against the frontier, and a task that needs the frontier cannot be assumed to complete on a device model. The user’s electricity and battery are not included.
Against the frontier, the device saves about $590,000 a month. Against a well-batched central small model, it loses money: the $12,600 of inference it avoids does not cover its fixed costs. The device competes with the cheapest central option, not the most expensive one. Once a system has followed the frugal ladder and moved routine work to small central models, the remaining cost case for the device is thin unless its fixed costs are small. In this example the break-even is explicit: moving 21% of tasks saves $12,600 a month, so the device route must cost less than that to run. The commercial opportunity depends as much on lowering the fixed cost of deploying local AI as on avoiding cloud calls. That is why platform-managed models matter: they spread the cost of distributing, updating and testing a model across every application on the device.
So the device does not win on cost alone. It wins where something else is at stake: data that should not leave the machine, interactions that must feel instant, places where connectivity is unreliable, and very high-volume, repeated tasks where the fixed costs spread thin. The test, as in the previous essay, is cost per completed task at equal quality, not cloud tokens avoided.
5 The gate: eligibility and language
The device route is open only to devices that qualify. Chrome’s documentation sets the terms: a graphics chip with more than 4 GB of memory, or a processor route with at least 16 GB of RAM and four cores, plus the 22 GB of free disk. Chrome’s foundation-model APIs are not yet supported on Android or iOS. And from Chrome 149, the built-in models support five languages for input and output: English, Spanish, Japanese, German and French. Hindi and India’s other languages are not among them.
This turns the opportunity into arithmetic. The share of all requests that moves to the device is the product of three numbers: the share of devices that are eligible, the share of tasks suited to local execution, and the share of those tasks that complete correctly on the device.
Figure 4. How many requests move to the device (all inputs assumed)
| Scenario | Eligible devices | Tasks suited to local | Local completion | Share of requests moved |
| Narrow | 20% | 70% | 60% | 8% |
| Middle | 40% | 70% | 70% | 20% |
| Broad | 60% | 80% | 80% | 38% |
A local model that works brilliantly on a fifth of devices moves a twelfth of the work. Eligibility is the first number to measure, and it should be measured on the machines people use, not on a developer’s high-end laptop.
A second distinction matters even more for the compute crunch. Requests moved are not compute freed. Devices will first absorb the easiest tasks, which are also the cheapest to serve centrally. A thousand short classifications may consume less data-centre capacity than one long reasoning task. Moving a fifth of requests to devices may relieve far less than a fifth of central inference spending, energy or capacity. Any test of the device route should therefore measure the inference spending and compute it displaces, not the number of calls it avoids. Read this way, the device does less to relieve the strait than the request counts suggest; its larger contributions are privacy, latency and resilience.
In India the edge is mostly the phone, and mostly the mid-range and lower-priced Android phone. Language support varies by platform: Chrome’s built-in models do not cover Hindi, while Apple’s third-generation models list Hindi among their supported languages, but on devices at the premium end of the market. Generative platform models are largely absent from the phones most Indians use. That does not mean India must wait. The three kinds of local intelligence separate here. A tiny model that recognises intent in a Hindi search query, ranks products or extracts a field from a form can run on a mid-range phone that could never run a multi-billion-parameter generative model, and it can work offline today. Those models are tens of megabytes rather than gigabytes, shipped inside apps for narrow jobs. Generative work goes to a platform model where one exists and to a central small model where it does not. India should not wait for frontier-quality local chatbots on affordable phones. It can start by moving narrow, repetitive intelligence onto the devices that already exist. The gap to close is small, dependable Indian-language models that fit in a phone’s memory, with shared evaluation sets to prove they work. That is a public-infrastructure problem as much as a commercial one.
6 The new strait
In Every Business Has a Strait I argued that power accumulates at narrow passages in valuable flows. On-device AI creates one. If the browser and operating-system owners decide which local models exist, which languages they support, which devices may run them and how often an app may call them, then dependency has not disappeared. It has moved from the cloud inference provider to the device platform. The controls are already visible: Android’s per-app inference quotas and foreground-only rule, Chrome’s declared language list, hardware requirements that differ by model and interface, and terms that govern how platform providers process usage data.
The choice for a developer is between platform-managed and developer-controlled intelligence. Platform-managed models are easier to distribute and maintain, and give the platform owner more control over what the application can do. Developer-controlled models offer customisation and portability, and make the developer responsible for installation, updates and compatibility. Most serious products will need both, which raises the question of what must stay constant as the models underneath change.
That changes what a business should own. The local model is the platform’s. The durable assets are the layers above it: the workflow, the memory, the permissions, the evaluations that prove a task completed correctly, and the routing logic that can send a task to whichever model a device offers. In From Rented to Owned I argued for owning the assets that compound and renting the rest. The same rule holds on the device as in the cloud: own the workflow and the memory, rent the model. A business does not need to own every model. It needs to own the ability to change models without losing its accumulated intelligence.
Two cautions follow. On-device inference does not make a system private by itself; an application can still send what the model produced, or what it read, somewhere else. And a result computed on the user’s device is not authoritative. A client can be tampered with, so anything a bank, retailer or employer must rely on, such as an eligibility decision, a price or an approval, needs verification on the server. Local is fast and private. It is not, on its own, trusted.
7 Three experiments
The claims above can be tested in sixty to ninety days, on real devices, with clear criteria.
Figure 5. Three experiments
| Experiment | What it compares | Success criterion | What would invalidate it |
| A repeatable language task on typical Indian office devices, such as extracting commitments from work email | Rules and code; local model; central small model; local-first hybrid | At least 50% of eligible tasks completed locally with no material loss of quality (a proposed threshold, not a prediction) | Eligibility too low; missed or invented commitments; review costs rise |
| A narrow on-device model inside an engagement SDK in brand apps, on mid-range phones | The on-device model against the server-side equivalent and the existing rules or machine-learning baseline | Beats the baseline on completion, at lower cost or latency | No gain over the baseline; battery or app-size cost too high |
| An office pool: idle, managed machines answering colleagues’ questions, with an always-on machine as backbone | The pool against one dedicated office server and a central small model | Lower fully loaded cost at equal answer quality, with a material share of peak-hour answers served by idle machines | Idle machines serve little at peak; owners notice the load; sensitive questions cannot be kept on IT-run machines |
Every experiment should report the same measures: the share of devices eligible, completion, errors, latency, energy, human review and cost per completed task. The first experiment should publish two results, not one: quality and cost among eligible devices, and the share of all tasks completed locally once the eligibility denominator is applied. Hiding that denominator is how pilots become misleading. Every experiment should also report the central compute displaced, not just the requests avoided, and each should set a stop rule before work begins: if the threshold is missed, use the central route and move on.
The second experiment carries a discipline of its own. Many personalisation decisions are already made cheaply by rules or conventional machine learning. An on-device model earns its place only by beating the baseline it replaces, not by being on the device.
8 How the thesis could fail
The essay makes three bets of different strength, and they should be judged separately. A task can run locally; that does not make it cheaper to run locally; and even when it is cheaper, it may not free much data-centre capacity. Each is a separate proposition needing its own evidence.
High conviction: more useful inference will run on devices. Browser and operating-system models are shipping, and device models are growing beyond the limits of working memory. This does not imply that most inference will leave the cloud, only that the set of worthwhile local tasks will keep widening.
Promising but unproven: routing across device, enterprise and cloud cuts cost per completed task. The wins on privacy and latency are clear. The cost case depends on fixed costs, on the central alternative and on how much compute the moved tasks would have consumed.
Speculative: strangers’ idle devices become a shared public inference network. Background limits, offline devices, consent, compensation and the physics of splitting models across consumer connections all stand in the way. An office pool of colleagues’ managed machines is a different and nearer proposition, and the third experiment tests it.
Figure 6. Five ways the thesis fails
| Failure | What it would mean | What would show it |
| Eligible share stays low | Local models remain a feature for premium devices | Eligibility measured on typical devices stays below a third |
| Central small models win | Batched cloud inference beats the device on a fully loaded basis | Fully loaded comparisons favour the central route at equal completion |
| Platforms restrict access | Quotas, terms and device lists narrow what developers can do | Access rules tighten rather than widen |
| Indian languages lag | The mass market cannot use platform models | No platform model supports major Indian languages by 2028 |
| Compute glut | Cloud inference becomes cheap enough that nobody bothers | Small-model inference prices fall faster than device integration costs |
Five tests, each answerable by 2028, will settle most of it:
- What share of active devices in India can run a platform language model in at least one Indian language?
- Fully loaded cost. Do published comparisons show the device beating a batched central small model on cost per completed task at equal quality, for any high-volume workflow, and how much central compute does it displace?
- Platform access. Have quotas, background rules, device lists and language support widened or narrowed for developers?
- Small models. Do small models in Indian languages that fit in a phone’s memory exist and complete narrow tasks reliably?
- Can an application move between local-model providers without losing its workflow, memory or evaluations?
If eligible devices stay a minority, platforms keep tight control and central small models keep getting cheaper, the device remains a privacy and latency feature rather than an economic tier. That is a smaller claim, and still worth building for.
**
The answer to a compute crunch is not only to buy more GPUs. It is to use the compute people have already bought, for the work it fits, and to rent the rest. The Compute Strait reduces the quantity of inference each job needs. This essay decides where what remains should run. Together they suggest that the next advance in AI affordability may come not only from better models and cheaper chips, but from three better decisions: whether inference is needed, which model should provide it and where it should run. Stop treating location as fixed. A task should go where it can be completed reliably, privately and economically. Often that will still be a server. Increasingly, it may be the computer in front of the person asking.
The cloud made computing available to everyone by moving it away from their devices. The next AI revolution could make intelligence affordable to everyone by moving some of it back.
