How to run LLMs locally on a Mac: Ollama, llama.cpp, MLX
How to run an LLM locally on a Mac: pick a model that fits unified memory, choose Ollama, llama.cpp, LM Studio or MLX, and call it from code. With a fit calculator.

To run an LLM locally on a Mac, install a local runtime (Ollama is the shortest path; llama.cpp, LM Studio and Apple’s MLX are the others), download a quantized open-weight model small enough to fit in your unified memory, and talk to it in the terminal or over a local HTTP API. On Apple Silicon the GPU and CPU share one pool of memory, so the question that decides everything is how many gigabytes the model and its context need.
My machine is an M4 MacBook Air with 24GB of unified memory, so that’s the budget I’ll use below. The arithmetic transfers to any Mac. I ran the calculator on 2026-10-31 with Python 3.13.11 and no packages.
Why does a Mac work for local LLMs?
Two reasons, both about memory.
Unified memory. On a PC, a model has to fit in the graphics card’s VRAM, which is usually much smaller than system RAM. On Apple Silicon, the GPU reads the same memory as the CPU. A 24GB Mac can give a model most of 24GB without a discrete GPU. That’s why how much VRAM an LLM needs translates to “how much unified memory” on a Mac.
Bandwidth sets speed. Generating each token means reading every weight once. So decode speed is bounded by memory bandwidth divided by model size in bytes. Apple’s tech specs list the 13-inch M4 MacBook Air at 120GB/s of memory bandwidth, alongside a 10-core CPU, an 8-core or 10-core GPU and a 16-core Neural Engine. That bandwidth figure is the ceiling I’ll use, and it’s why smaller, quantized models feel faster: fewer bytes to read per token.
Which model fits in 24GB?
Weights take parameters times bytes per weight. At 16-bit that’s 2 bytes per parameter, at 8-bit 1 byte, at 4-bit half a byte. Then the KV cache grows with context length. And macOS, a browser and an editor need room too.
Here is that as a script. The model shapes (layers, KV heads, head size) come from the Mistral 7B and Llama 3 papers; parameter counts are the nominal sizes in their names. The 8GB reserve for the OS and apps is my assumption, so it’s a variable.
# macfit.py: will this model fit in a 24GB Mac, and what's the decode-speed ceiling?
# Python 3 stdlib only. Arithmetic, not a benchmark.
GB = 1e9
TOTAL_GB = 24 # unified memory on an M4 MacBook Air (24GB config)
RESERVE_GB = 8 # my assumption: macOS, browser, editor. Tune this knob.
BANDWIDTH_GBS = 120 # Apple's spec for the M4 MacBook Air
models = { # name: (params in billions, layers, kv_heads, head_dim), configs from each paper
"Mistral 7B": (7, 32, 8, 128),
"Llama 3 8B": (8, 32, 8, 128),
"Llama 3 70B": (70, 80, 8, 128),
}
def kv_gb(layers, kv_heads, head_dim, ctx, bytes_per=2):
return 2 * layers * kv_heads * head_dim * bytes_per * ctx / GB
budget = TOTAL_GB - RESERVE_GB
print(f"budget for model + KV cache: {budget} GB of {TOTAL_GB} GB\n")
print(f"{'model':<12}{'bits':>5}{'weights GB':>12}{'KV 8k':>8}{'KV 32k':>8}{'fits @8k':>10}{'ceiling tok/s':>15}")
for name, (b, layers, kvh, hd) in models.items():
for bits in (16, 8, 4):
w = b * 1e9 * bits / 8 / GB
k8, k32 = kv_gb(layers, kvh, hd, 8192), kv_gb(layers, kvh, hd, 32768)
fits = "yes" if w + k8 <= budget else "no"
# each decoded token reads every weight once, so bandwidth / weight bytes bounds speed
print(f"{name:<12}{bits:>5}{w:>12.1f}{k8:>8.1f}{k32:>8.1f}{fits:>10}{BANDWIDTH_GBS / w:>15.0f}")
Output (Python 3.13.11):
budget for model + KV cache: 16 GB of 24 GB
model bits weights GB KV 8k KV 32k fits @8k ceiling tok/s
Mistral 7B 16 14.0 1.1 4.3 yes 9
Mistral 7B 8 7.0 1.1 4.3 yes 17
Mistral 7B 4 3.5 1.1 4.3 yes 34
Llama 3 8B 16 16.0 1.1 4.3 no 8
Llama 3 8B 8 8.0 1.1 4.3 yes 15
Llama 3 8B 4 4.0 1.1 4.3 yes 30
Llama 3 70B 16 140.0 2.7 10.7 no 1
Llama 3 70B 8 70.0 2.7 10.7 no 2
Llama 3 70B 4 35.0 2.7 10.7 no 3
Read the last column carefully. It’s a theoretical ceiling from bandwidth alone, not a speed I measured. Real decode is slower: kernels don’t hit peak bandwidth, and the KV cache is read on every token too. Real 4-bit files are also a little larger than half a byte per weight, because quantization formats store scales alongside the weights.
The shape of the answer still holds. On a 24GB Mac, 7B to 8B models at 4-bit or 8-bit fit with room for a long context. A 70B model doesn’t fit at any of these precisions. Somewhere in between, models in the low teens of billions at 4-bit fit with a modest context. And small language models in the 1B to 4B range leave plenty of headroom and are the fastest option.
Which tool should I use: Ollama, llama.cpp, LM Studio or MLX?
They overlap a lot. The differences are in how much they hide.
Ollama is the easiest. Its macOS docs ask for macOS Sonoma (14) or newer, install by dragging the app from ollama.dmg into Applications, and store models in ~/.ollama. The README quickstart is one line:
ollama run gemma4
It downloads the model, loads it and opens a chat. Behind that, it runs a local server on port 11434.
llama.cpp is the C/C++ engine underneath much of local AI. Its README says “Apple silicon is a first-class citizen”, optimized via ARM NEON, Accelerate and Metal, and supports 1.5-bit to 8-bit integer quantization. Models use the GGUF format, “a file format for storing models for inference with GGML and executors based on GGML”. The install docs list Homebrew. The README’s examples pull a GGUF model straight from Hugging Face:
brew install llama.cpp
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF # chat in the terminal
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF # OpenAI-compatible API server
Pick llama.cpp when you want control over quantization, context size and server flags.
LM Studio is a desktop app with a model browser and a chat window. Its system requirements are Apple Silicon, macOS 14.0 or newer and “16GB+ RAM recommended”. The mlx-engine README says LM Studio 0.3.4 and newer for Mac ships with an engine built on Apple’s MLX.
MLX is Apple’s array framework for Apple Silicon, and mlx-lm is its package for “generating text and fine-tuning large language models on Apple silicon”. Pick it if you live in Python, want to fine-tune locally, or want MLX-format models from Hugging Face:
pip install mlx-lm
mlx_lm.generate --prompt "How tall is Mt Everest?"
The README’s default model for that command is mlx-community/Llama-3.2-3B-Instruct-4bit, and mlx_lm.convert --model <repo> -q quantizes a model yourself. One detail worth knowing on a 24GB machine: if a model fits in RAM but runs slowly, the README suggests raising the GPU’s wired memory limit with sudo sysctl iogpu.wired_limit_mb=N, with N above the model size and below total memory. Wiring requires macOS 15.0 or later.
What about vLLM? It’s built for serving many concurrent users on datacenter GPUs, and its quickstart lists Linux. For one person on a laptop, the tools above fit better. I explain why in what is vLLM.
How do I call a local model from code?
Every tool above exposes a local HTTP API. Here’s Ollama’s /api/generate from the Python standard library. The response includes eval_count (output tokens) and eval_duration (nanoseconds), so you can measure your own tokens per second instead of trusting someone else’s chart.
# ask_ollama.py: call a local Ollama model with the stdlib. Requires `ollama run <model>` first.
import json, urllib.request
body = {"model": "gemma4", "prompt": "Explain unified memory in one sentence.", "stream": False}
req = urllib.request.Request("http://localhost:11434/api/generate",
data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
r = json.load(urllib.request.urlopen(req))
print(r["response"])
print(f"{r['eval_count']} tokens at {r['eval_count'] / (r['eval_duration'] / 1e9):.1f} tok/s, "
f"prompt {r['prompt_eval_count']} tokens in {r['prompt_eval_duration'] / 1e6:.0f} ms")
I haven’t pasted output for this one because the numbers depend on the model, quantization, prompt and what else the machine is doing. Run it on yours. The two figures it prints map onto time to first token and decode speed, which LLM latency explained covers.
Where does local AI break?
- Memory pressure. If the model, its context and your apps exceed unified memory, macOS starts swapping and speed collapses. Watch Memory Pressure in Activity Monitor and drop to a smaller model or quantization before blaming the tool.
- Long contexts cost memory. The KV columns in the table grow linearly. A model that fits at 8k context can stop fitting at 32k.
- Quality drops with size. A 4-bit 8B model is a different product from a frontier API model. Test it on your actual task.
- Batteries and heat. Decoding keeps the GPU busy. On a thin laptop, sustained runs can be slower than short ones. I haven’t measured this, so treat it as a reason to test longer jobs, not a number.
- Licenses. Open weights come with licenses that differ by model. Read them before shipping anything.
What I would do
On a 24GB Mac, I’d install Ollama first and pull a 7B to 8B class instruct model at 4-bit, because it fits with a long context and room for a browser. I’d keep a 1B to 4B model alongside it for quick tasks like summarizing a file or classifying text in scripts.
When I need control, I’d move to llama.cpp’s llama serve with an explicit context size, or to mlx-lm if the work is in Python or involves local fine-tuning. llama serve exposes an OpenAI-compatible endpoint, so the same client code can switch between local and hosted models with a base URL change.
For anything with private data, local inference has one property I value from my data-platform and privacy-compliance years at Walmart: the prompt never leaves the machine, so there’s no third-party data processing to review. That’s often the real reason to run locally, more than cost.
Before trusting any tokens-per-second claim, including the ceiling in my table, I’d run the Ollama script above on my own prompts and record the numbers.
FAQ
Can a MacBook Air run an LLM?
Yes. A 7B to 8B model at 4-bit needs roughly 4 to 5GB for weights plus the KV cache, which fits on a 16GB or 24GB MacBook Air. Smaller models fit on 8GB machines with short contexts.
How much RAM do I need to run an LLM on a Mac?
Roughly parameters times bytes per weight, plus KV cache, plus room for macOS. LM Studio recommends 16GB or more. On 24GB, 7B to 8B models are comfortable; 70B models don’t fit even at 4-bit.
Is Ollama or LM Studio better on a Mac?
Ollama suits terminal and API use; LM Studio suits browsing models and chatting in a GUI. Both run models locally and both expose a local server. Pick the interface you prefer.
What is GGUF?
The model format used by llama.cpp and other GGML-based executors. It stores weights, often quantized, together with the metadata needed to load them.
Does running an LLM locally need internet?
Only to download the model. After that, inference runs entirely on your machine.
Sources
- Apple, MacBook Air (13-inch, M4, 2025) Tech Specs (read 2026-10-31)
- Ollama, GitHub README, macOS docs and Generate API (read 2026-10-31)
- ggml-org, llama.cpp README and install docs (read 2026-10-31)
- ggml-org, GGUF specification (read 2026-10-31)
- LM Studio, System requirements and mlx-engine README (read 2026-10-31)
- Apple ML Explore, mlx-lm README (read 2026-10-31)
- Jiang et al., Mistral 7B, October 10, 2023
- Llama Team, Meta, The Llama 3 Herd of Models, July 31, 2024