A phone that runs a language model is the most convincing demo in AI right now. It is also the most misleading, because the first thirty tokens arrive fast enough to feel like magic and the same conversation is noticeably slower two minutes later. Nothing has changed about the model. Something has changed about memory, and the difference between those two observations is the whole subject.
Table of Contents
If you are trying to get a model running locally, or wondering why the one you installed keeps getting killed, the useful mental model has four parts: weight size, cache size, memory bandwidth, and heat. They explain almost every symptom you will see.
Prefill and decode are two different jobs
A language model does two things per request, and they have opposite bottlenecks.
Prefill is reading your prompt. Every token in the prompt goes through the model at once, which makes it a large matrix multiplication and mostly a compute problem. Decode is generating the answer, one token at a time. For each token, the phone has to read essentially the entire model, and it does roughly two arithmetic operations per value it reads. The maths units finish long before the memory does.
That is the single most important fact about local inference: decode speed is set by how fast the chip can move weights, not by how many operations per second it can do. A phone with a very fast GPU and a modest memory bus will beat a slower chip with a wider bus on the same model, and no advertised TOPS figure tells you which one that is.
In practice, tokens per second on a single conversation is bounded roughly by memory bandwidth divided by the bytes read per token, which is close to the size of the quantised weights plus the cache being read. Everything else is a correction on top of that ceiling.
The memory budget is smaller than the spec sheet
The most common mistake is starting from the phone’s RAM. An 8GB phone does not offer 8GB to your model. The operating system, the app, the runtime’s buffers, the image assets and the cache all come out of the same pool, and Android enforces per-app limits that are lower than the headline figure. On iOS, the equivalent limit is enforced by the system’s memory-pressure kill, which shows up as an app that vanishes rather than one that reports an error.
The realistic calculation is a subtraction, and it usually leaves you with one to three gigabytes for a comfortable margin on a mid-range device. Anything that fits exactly will die the first time the user opens the camera.
So the budget has three lines: the weights on disk and in memory, the cache that grows with the conversation, and the app itself. Most guides only mention the first, which is why models that “fit” still crash.
Weights: what quantisation actually buys
Weights are the numbers learned during training. They are stored in a fixed number of bits each, and reducing that number is quantisation.
The arithmetic is simple enough to do in your head, and Hugging Face’s quantisation overview states it the same way: a model with 8 billion parameters needs about 16GB of memory in 16-bit precision, roughly 8GB at 8 bits, and about 4 to 5GB at 4 bits. That is why 4-bit quantisation turned local models from a desktop hobby into something a phone can attempt.
GGUF builds go further and mix precisions per tensor, which is why you see names like Q4_K_M rather than a clean “4-bit”. That particular mix averages somewhere around 4.8 to 4.9 bits per weight because it keeps a few sensitive tensors at higher precision. The naming is not decoration: a Q4_K_M file is meaningfully larger than a Q4_0 file of the same model, and usually worth it.
The published evidence also points at Q4_K_M specifically. A 2025 ACL study on small language models for edge deployment measured quantisation reducing decode latency by 17 to 75 percent depending on method, with Q4_K_M consistently the strongest and averaging around a 50 percent latency reduction compared with the full-precision baseline. The same study found Q6_K and Q3_K becoming ineffective on long prompts, where their irregular bit widths cost enough in alignment and padding to erase the advantage.
Two rules fall out of that. Drop to 4-bit before you do anything else. Then stop dropping: 3-bit and 2-bit builds save memory you probably did not need and cost quality you will notice, especially on reasoning and code.
The KV cache is the number that grows
Every generated token needs to attend to everything before it. To avoid recomputing that, the runtime stores the keys and values for every token in context. That is the KV cache, and its size is the thing that turns a working demo into a phone that dies on turn four.
The formula is short: two (keys and values) times the number of layers, times the number of key/value heads, times the head dimension, times the bytes per element, times the number of cached tokens.
Take a typical 8B model with 36 layers, 8 key/value heads and a head dimension of 128, running in 16-bit cache. That is about 147 kilobytes per token. At 4,096 tokens of context, you are holding roughly 600MB, and at 32,768 tokens you are holding close to 4.8GB. The weights did not change at all.
This is why long context on a phone is a memory feature, not a quality feature. It is also why grouped-query attention matters so much on mobile: reducing the number of key/value heads cuts the cache by the same factor, which is often the difference between a 4k context and a 32k one.
Practical consequences follow immediately. Set the context size to what the task needs, not to the maximum the model supports. Clear the conversation when it is done. And if you are building an app, do not keep three sessions alive “just in case”.
If you plan to run this alongside other on-device work, the thermal and power picture in our guide to smartphone battery life is a decent companion, because sustained inference and sustained screen-on use fail in similar ways.
Where the speed actually goes
Once the weights and the cache fit, speed is a bandwidth story with three interruptions.
The first is memory traffic per token. Because decode reads the whole model for every token, a 4GB model at 60GB/s of effective bandwidth cannot exceed roughly 15 tokens per second, before any overhead. Real numbers land below that ceiling, sometimes well below, because the runtime, the tokeniser and the sampling all take time.
The second is prefill on a long prompt. A 4,000-token prompt is a single large computation; on a phone CPU it can take many seconds before the first token appears. That is why a model can feel instant on a one-line question and unusable when you paste a document. Time to first token and tokens per second are different measurements and should be reported separately.
The third is heat. A phone has no fan, so sustained inference will eventually be limited by the thermal budget, and performance drops. This is not a bug in the runtime. It is the reason short benchmarks flatter phones and long sessions do not, and it is why any tokens-per-second figure worth trusting states how long it ran and on which device.
A starting configuration that usually works
If you want a baseline rather than a research project, this is the shape that tends to hold on current mid-to-high-end phones.
Pick a model between one and four billion parameters, in a Q4_K_M GGUF build. Set the context to 2,048 or 4,096 tokens. Use the platform’s normal acceleration path rather than fighting it: llama.cpp’s Android binding handles CPU feature detection itself, and on iOS the Metal backend is the default worth trying. Keep the cache in 16-bit unless memory forces you down. Measure time to first token, decode tokens per second, and peak memory, then repeat after five minutes of continuous generation to see what throttling does.
Choose the runtime from the constraint, not the other way round. llama.cpp with GGUF is the most portable option and covers Android and iOS. TFLite/LiteRT is the least painful route when your model is already a small vision or classification network. ONNX Runtime Mobile is reasonable when you need one runtime on both platforms. Core ML only makes sense if you are willing to own a conversion pipeline.
And be honest about the shape of the problem. A phone can run a small model quickly, keep it private and work offline. It cannot hold a frontier model’s weights, and no amount of prompt engineering changes that. The interesting split is the one described in why AI is moving from the cloud to the edge: small models handling the frequent, private, latency-sensitive work locally, with the big model called only when the task genuinely needs it. The same hardware logic applies to other constrained devices, which is why IoT design constraints run into the same wall of memory and power, and the same battery-versus-capability trade-offs shape what ships in smart wearables.
The four numbers to write down
Before you tune anything, record four measurements on the actual device: weight file size, cache size at your chosen context length, decode tokens per second after one minute, and peak memory during a long generation. Almost every “it works on my phone” claim falls apart at the fourth one.
Everything else in this space, from quantisation choices to runtime benchmarks, is a way of moving those four numbers. When someone quotes a tokens-per-second figure with no device, no quantisation level, no context length and no duration, the number is not comparable to anything, including itself an hour later once the phone is warm.





