What are small language models? SLMs for phones and agents
Small language models are LLMs of about 1B to 14B parameters that run on a phone or laptop. How they are made, how much memory they need, and when to use one.

A small language model (SLM) is a language model small enough to run on hardware you already own, a phone or a laptop, with low enough latency to be useful for one person. In practice that means roughly 1 to 14 billion parameters, compressed to 4 bits or fewer, made with the same transformer recipe as frontier models but trained on more carefully chosen data and often distilled from a larger teacher.
There is no official cutoff. NVIDIA’s 2025 position paper Small Language Models are the Future of Agentic AI defines an SLM by what it does instead of a parameter count: “a LM that can fit onto a common consumer electronic device and perform inference with latency sufficiently low to be practical when serving the agentic requests of one user.” I like that definition because it moves with the hardware. A 14B model was a datacenter model in 2022 and is a laptop model now.
What counts as a small language model?
The models people mean when they say SLM, with the numbers from their own papers and posts:
- Phi-3-mini (Microsoft, April 2024): 3.8 billion parameters trained on 3.3 trillion tokens, 69% on MMLU. The paper quantizes it to 4 bits, where it takes about 1.8 GB, and runs it on an iPhone 14 with an A16 chip, offline, at more than 12 tokens per second.
- Llama 3.2 1B and 3B (Meta, September 25, 2024): text-only models with a 128K context, described as fitting “onto edge and mobile devices”, enabled on day one for Qualcomm and MediaTek hardware.
- Gemma 3 (Google, March 2025): 1B to 27B parameters with at least a 128K context. Google reports the 4B instruction-tuned model as competitive with the previous generation’s 27B.
- Phi-4 (Microsoft, December 2024): 14B parameters, trained with synthetic data throughout, and reported to beat its own teacher, GPT-4, on STEM question answering.
- Apple’s on-device model (June 2025): about 3 billion parameters, compressed to 2 bits per weight with quantization-aware training, and exposed to app developers through the Foundation Models framework.
The pattern across all five: the architectures stay close to a standard transformer, with tweaks aimed at memory. Most of the work went into data and compression.
How are small language models made?
Three techniques do most of the work, and the best SLMs use all three.
Better data per parameter. The Phi line started with phi-1 (2023), a 1.3B code model trained on about 6B tokens of “textbook quality” web data plus about 1B tokens of synthetic textbooks written by GPT-3.5. It reached 50.6% pass@1 on HumanEval. A small model has little capacity to waste on noise, so filtering and synthetic data matter more than they do at 400B.
Overtraining. Scaling laws such as Chinchilla tell you the cheapest way to reach a given loss at training time, about 20 tokens per parameter. A model that will serve billions of requests should be trained much longer than that, because inference cost scales with parameters and training cost is paid once. Phi-3-mini saw 3.3 trillion tokens with 3.8 billion parameters, close to 870 tokens per parameter.
Pruning and distillation. Meta built Llama 3.2 1B and 3B by structurally pruning Llama 3.1 8B, then training them with the 8B and 70B models’ logits as token-level targets. That is knowledge distillation: the small model learns from the teacher’s full probability distribution, not only from the one correct next token.
Then comes quantization, which is what makes the model fit on the device.
How much memory does a small language model need?
The first-order answer is parameters times bits per weight. This script prints the weight memory for common sizes at four precisions and checks what fits on three devices.
# slm_memory.py: how much memory do a model's weights need? Python 3, stdlib only.
# Weights only. The KV cache, activations, the runtime and the OS need room on top.
MODELS = {"1B": 1e9, "3B": 3e9, "3.8B (Phi-3-mini)": 3.8e9,
"8B": 8e9, "14B (Phi-4)": 14e9, "70B": 70e9}
BITS = [16, 8, 4, 2]
DEVICES = {"8 GB phone": 8, "16 GB laptop": 16, "24 GB laptop": 24}
def gib(params, bits):
return params * bits / 8 / 2**30
print(f"{'params':<18}" + "".join(f"{str(b) + '-bit':>9}" for b in BITS))
for name, n in MODELS.items():
print(f"{name:<18}" + "".join(f"{gib(n, b):>8.1f}G" for b in BITS))
# My rule of thumb: weights should take at most half of the device's memory.
print("\nlargest model whose 4-bit weights fit in half the memory:")
for dev, gb in DEVICES.items():
fits = [m for m, n in MODELS.items() if gib(n, 4) <= gb / 2]
print(f" {dev:<13} {fits[-1]}")
Output (Python 3.13.11):
params 16-bit 8-bit 4-bit 2-bit
1B 1.9G 0.9G 0.5G 0.2G
3B 5.6G 2.8G 1.4G 0.7G
3.8B (Phi-3-mini) 7.1G 3.5G 1.8G 0.9G
8B 14.9G 7.5G 3.7G 1.9G
14B (Phi-4) 26.1G 13.0G 6.5G 3.3G
70B 130.4G 65.2G 32.6G 16.3G
largest model whose 4-bit weights fit in half the memory:
8 GB phone 8B
16 GB laptop 14B (Phi-4)
24 GB laptop 14B (Phi-4)
The Phi-3-mini row lands on 1.8 GB at 4 bits, the same figure the paper reports, which is a good sign the arithmetic is the right first approximation. Apple’s 3B model at 2 bits is the 0.7 GB cell. The “half the memory” rule is mine, not a standard: the operating system, other apps and the KV cache all need the rest, and the KV cache grows with the context window you actually use. An 8B model technically fits a phone at 4 bits; a 3B model is what ships.
Memory is half the story. Each generated token reads every weight once, so decode speed on a phone or laptop is mostly memory bandwidth divided by model size in bytes. Smaller and more compressed models are faster for the same reason they fit.
Why do small language models matter to a builder?
Four reasons, in the order I would weigh them.
Privacy and data residency. From a data-platform and privacy-compliance seat, the line I would circle is that an on-device model never sends the prompt anywhere. For text that is regulated, contractual or simply personal, that removes a whole review.
Cost and latency at volume. NVIDIA’s paper estimates that serving a 7B SLM is 10 to 30 times cheaper in latency, energy and FLOPs than serving a 70B to 175B LLM. Classification, extraction, routing and formatting run millions of times a day and rarely need a frontier model.
Agents are mostly narrow calls. The same paper’s argument is that most agent subtasks are repetitive, narrowly scoped and non-conversational. An agent that calls a big model to plan and a small one to fill a JSON schema or pick a tool is the heterogeneous system the paper recommends, and it is what model routing formalizes.
Offline. A model on the device works on a plane, in a basement and during a provider outage.
Where do small language models break?
Knowledge. Parameters store facts. A 3B model knows far less than a 400B one, and it fills gaps with confident text, which is how hallucinations start. Give it the facts through retrieval, and do not use it as an encyclopedia.
Long, open-ended reasoning. Multi-step planning across a long context is where the gap to frontier models is widest. The Phi papers show strong benchmark numbers per parameter, but benchmarks are narrower than real tasks.
Quantization has a floor. Going from 16 to 4 bits usually costs a little quality; going to 2 bits needs the model trained for it, as Apple did with quantization-aware training. Do not assume a 2-bit file of an arbitrary model behaves like the original.
Context costs memory. A 128K context window on a phone is a specification, not a promise. The KV cache for a long prompt can be larger than the weights.
What I would do
I would start every narrow task, such as classification, extraction, redaction, or tool selection inside an agent, on a 3B to 8B model, and move up only when an eval on my own data shows it failing. I would run the candidates locally with a 4-bit build first (the setup is in how to run LLMs locally), because an afternoon of real prompts tells you more than a leaderboard. For anything that needs broad knowledge or long planning, I would keep a frontier model in the loop and let the small one handle the high-volume steps around it. And for regulated text, I would make on-device the default and justify every exception.
FAQ
What is the difference between an SLM and an LLM?
Size and where it runs. Both are transformer language models trained the same way. An SLM has roughly 1 to 14 billion parameters and runs on a phone or laptop; an LLM in the usual sense has tens to hundreds of billions and runs in a datacenter. The line is informal and moves as hardware improves.
Can a small language model run on a phone?
Yes. Microsoft ran a 4-bit Phi-3-mini on an iPhone 14 at more than 12 tokens per second, Meta’s Llama 3.2 1B and 3B target mobile chips, and Apple ships a roughly 3B model on its devices.
Are small language models good enough for agents?
For the narrow steps, often yes: tool selection, argument filling, extraction and formatting. For planning across many steps, a larger model still does better. A mixed setup, with a big model planning and small models executing, is the common answer.
How are small models so good for their size?
Curated and synthetic training data, many more training tokens per parameter than the compute-optimal recipe, and distillation from a larger teacher model.
Sources
- Belcak et al. (NVIDIA), Small Language Models are the Future of Agentic AI, June 2, 2025
- Abdin et al. (Microsoft), Phi-3 Technical Report, April 22, 2024
- Abdin et al. (Microsoft), Phi-4 Technical Report, December 12, 2024
- Gunasekar et al. (Microsoft), Textbooks Are All You Need (phi-1), June 20, 2023
- Meta, Llama 3.2: Revolutionizing edge AI and vision with open, customizable models, September 25, 2024
- Gemma Team (Google), Gemma 3 Technical Report, March 25, 2025
- Apple Machine Learning Research, Updates to Apple’s On-Device and Server Foundation Language Models, June 9, 2025