What is LoRA? LoRA vs QLoRA for fine-tuning LLMs
LoRA fine-tunes an LLM by training two small low-rank matrices per layer while the model stays frozen. QLoRA does it on a 4-bit base. How both work, in Python.

LoRA (low-rank adaptation) is a way to fine-tune a large language model by freezing all of its original weights and training two small matrices per adapted layer whose product is added to the frozen weight. QLoRA is the same idea with the frozen base model stored in 4-bit precision, so a fine-tune that needed a cluster fits on one GPU. LoRA saves trainable parameters and optimizer memory; QLoRA also shrinks the weights themselves.
I ran the snippet below on 2026-10-31 with Python 3.13.11 and no packages.
How does LoRA work?
Take one weight matrix in a transformer, say the query projection W0 with shape d x k. Full fine-tuning learns an update ΔW of the same shape and adds it. LoRA, from Hu et al. at Microsoft (June 2021), constrains that update to a low-rank product:
W = W0 + BA, where B is d x r, A is r x k, and the rank r is much smaller than d or k.
Only A and B are trained. The paper’s argument is that the change a model needs during adaptation has a low “intrinsic rank”, so you lose little by forcing it to be low-rank. Three details make it work in practice:
- Initialization.
Astarts as random Gaussian andBas zero, soBAis exactly zero on step one. Training starts from the pretrained model’s behavior, not a perturbed one. - Scaling. The update is multiplied by
α/r. The authors setαto the firstrthey try and leave it, which means changingrlater doesn’t require retuning the learning rate. - Merging. After training you can compute
W0 + BAonce and store it. The result is a normal matrix of the original shape, so a merged LoRA model has no extra inference latency, unlike adapter layers that add sequential compute.
The paper’s GPT-3 175B numbers are the reason LoRA took over. With r = 4 and only the query and value matrices adapted, training VRAM dropped from 1.2 TB to 350 GB and the checkpoint from 350 GB to 35 MB, about 10,000 times smaller. Training ran 25% faster than full fine-tuning, because there are no gradients to compute for most parameters. And a deployment that keeps the base model in memory can serve many tasks: the authors note that storing 100 adapted models takes about 354 GB instead of 35 TB.
The most surprising result is how low the rank can go. On GPT-3 with the WikiSQL and MultiNLI datasets, a rank as small as one was enough when adapting both Wq and Wv, in the authors’ words “to our surprise”.
What does LoRA look like in code?
Here is LoRA on one toy layer in standard-library Python, followed by the memory math for a real 7B model. The first part freezes a 16 x 16 matrix, trains a rank-2 B and A to absorb a low-rank change, then merges them. The second part uses the Mistral 7B shapes from its paper (dim 4096, 32 layers, MLP hidden 14336, 8 KV heads of 128, vocabulary 32000).
# lora.py: LoRA on a toy layer, then the memory math for a 7B model. Python 3 stdlib only.
import random
# 1. Train a rank-2 update B @ A on a frozen 16x16 weight W0, then merge it.
random.seed(0)
d, r, n = 16, 2, 64
rand = lambda rows, cols, s: [[random.gauss(0, s) for _ in range(cols)] for _ in range(rows)]
mm = lambda X, Y: [[sum(a * b for a, b in zip(row, col)) for col in zip(*Y)] for row in X]
add = lambda X, Y: [[a + b for a, b in zip(p, q)] for p, q in zip(X, Y)]
T = lambda X: [list(c) for c in zip(*X)]
W0 = rand(d, d, 0.3) # "pretrained", frozen
target_shift = mm(rand(d, r, 0.5), rand(r, d, 0.5)) # the task needs a low-rank change
X = rand(d, n, 1.0) # 64 training inputs as columns
Y = mm(add(W0, target_shift), X) # what the adapted layer should output
A = rand(r, d, 0.1) # LoRA init from the paper: A random Gaussian ...
B = [[0.0] * r for _ in range(d)] # ... and B zero, so B @ A starts at exactly zero
def loss():
P = mm(add(W0, mm(B, A)), X)
return sum((p - y) ** 2 for pr, yr in zip(P, Y) for p, y in zip(pr, yr)) / (d * n)
print(f"trainable params: {2 * d * r} of {d * d} ({2 * d * r / (d * d):.0%})")
print(f"step 0 loss {loss():.2e}")
lr = 0.02
for step in range(1, 1501):
E = add(mm(add(W0, mm(B, A)), X), [[-y for y in row] for row in Y]) # residual
G = mm(E, T(X)) # dL/dW (unscaled)
gB, gA = mm(G, T(A)), mm(T(B), G) # chain rule into B, A
B = [[b - lr * g / n for b, g in zip(br, gr)] for br, gr in zip(B, gB)]
A = [[a - lr * g / n for a, g in zip(ar, gr)] for ar, gr in zip(A, gA)]
if step in (100, 300, 1500):
print(f"step {step:>4} loss {loss():.2e}")
W_merged = add(W0, mm(B, A)) # fold the adapter in: one matrix again, no extra latency
x = rand(d, 1, 1.0)
diff = max(abs(a[0] - b[0]) for a, b in zip(mm(W_merged, x), add(mm(W0, x), mm(B, mm(A, x)))))
print(f"merged vs unmerged max difference: {diff:.2e}\n")
# 2. Memory for fine-tuning Mistral 7B (shapes from the Mistral 7B paper, Table 1).
dim, layers, hidden, kv, vocab = 4096, 32, 14336, 8 * 128, 32000
linears = [(dim, dim), (dim, kv), (dim, kv), (dim, dim), # q, k, v, o
(dim, hidden), (dim, hidden), (hidden, dim)] # gate, up, down
lin = layers * sum(i * o for i, o in linears)
emb = 2 * vocab * dim # input embeddings + output head; norms ignored
base = lin + emb
lora = lambda rank: layers * sum(rank * (i + o) for i, o in linears)
GB = 1e9
print(f"base params: {base / 1e9:.2f}B")
print(f"{'method':<28}{'trainable':>14}{'GB weights+optimizer':>23}")
print(f"{'full fine-tune (16 B/param)':<28}{base:>14,}{base * 16 / GB:>23.1f}")
for rank in (8, 64):
p = lora(rank)
print(f"{f'LoRA r={rank}, bf16 base':<28}{p:>14,}{(base * 2 + p * 16) / GB:>23.1f}")
print(f"{f'QLoRA r={rank}, NF4 base':<28}{p:>14,}{(lin * 4.127 / 8 + emb * 2 + p * 16) / GB:>23.1f}")
trainable params: 64 of 256 (25%)
step 0 loss 1.38e+00
step 100 loss 5.97e-01
step 300 loss 3.86e-04
step 1500 loss 3.14e-26
merged vs unmerged max difference: 6.66e-16
base params: 7.24B
method trainable GB weights+optimizer
full fine-tune (16 B/param) 7,241,465,856 115.9
LoRA r=8, bf16 base 20,971,520 14.8
QLoRA r=8, NF4 base 20,971,520 4.5
LoRA r=64, bf16 base 167,772,160 17.2
QLoRA r=64, NF4 base 167,772,160 6.8
The toy layer starts at the frozen model’s output (loss 1.38, because B is zero) and learns the shift to floating-point precision with a quarter of the parameters. Merged and unmerged outputs agree to 6.66e-16, rounding error. In a 16 x 16 matrix, rank 2 is a quarter of the size; in a 4096-wide layer, rank 8 is under half a percent.
The second table is the real reason to care. I applied LoRA to all seven linear layers per block, which is what QLoRA recommends. Full fine-tuning at 16 bytes per parameter (fp16 weights and gradients plus fp32 master weights and two Adam moments, the accounting from the ZeRO paper) needs about 116 GB before activations. LoRA keeps the base in bf16 at 2 bytes per parameter and pays the 16 bytes only on 21M to 168M adapter parameters. QLoRA stores the linear layers at about 4.127 bits each. These numbers are a floor: activations, the KV of long sequences and framework overhead come on top, so treat them as lower bounds, not a hardware spec. The rest of that budget is in how much VRAM an LLM needs.
How does QLoRA work?
QLoRA (Dettmers, Pagnoni, Holtzman and Zettlemoyer, University of Washington, May 2023) keeps LoRA’s adapters and changes how the frozen base is stored. Its headline: fine-tuning a 65B model drops from more than 780 GB of GPU memory to under 48 GB, without degrading performance against 16-bit fine-tuning. Three pieces do it:
- 4-bit NormalFloat (NF4). Pretrained weights are roughly normally distributed, so the 16 levels of a 4-bit code are placed at quantiles of a normal distribution instead of evenly. The paper calls it information-theoretically optimal for normally distributed data, and it beat 4-bit integers and 4-bit floats in their tests. Weights are quantized in blocks of 64, each with its own scale.
- Double quantization. Those per-block scales are themselves 32-bit numbers, costing 0.5 bits per parameter. QLoRA quantizes them again to 8-bit in blocks of 256, bringing the overhead to 0.127 bits, a saving of about 0.37 bits per parameter or roughly 3 GB on a 65B model. That’s where the 4.127 in my script comes from.
- Paged optimizers. Optimizer state uses NVIDIA unified memory, so a memory spike during a long batch pages out to CPU RAM instead of crashing the run.
The 4-bit weights are storage only. Whenever a layer runs, QLoRA dequantizes the weight to BFloat16 and does the matrix multiplication in 16-bit. Gradients flow through the frozen 4-bit weights into the bf16 adapters, which are the only thing updated. For more on the storage formats themselves, see what is quantization.
The paper also corrected a LoRA default. The original paper adapted only the query and value matrices. The QLoRA authors could not match full fine-tuning that way, and found that the number of adapted layers was the hyperparameter that mattered: “LoRA on all linear transformer block layers are required to match full finetuning performance”, while r made little difference. Their Guanaco models, trained with QLoRA, reached 99.3% of ChatGPT’s performance on the Vicuna benchmark after 24 hours of fine-tuning on one GPU.
LoRA vs QLoRA: what’s the difference?
Both train the same low-rank adapters. The difference is the frozen base:
- LoRA keeps the base in 16-bit. Faster per step, no quantization error, and needs roughly 2 bytes per base parameter of GPU memory.
- QLoRA keeps the base in NF4. Roughly a quarter of the base memory, at the cost of extra dequantize work on every forward and backward pass.
Pick QLoRA when the base model won’t fit in 16-bit on the GPU you have. Pick LoRA when it does. At 7B, the table above says a single 24 GB card holds LoRA’s model state before activations; at 70B, only QLoRA gets close to one card. In Hugging Face PEFT, the switch is a BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4") when loading the model, with the same LoraConfig either way:
from peft import LoraConfig, get_peft_model
config = LoraConfig(r=16, lora_alpha=16, target_modules="all-linear", lora_dropout=0.1)
model = get_peft_model(model, config)
PEFT’s merge_and_unload() then folds the adapter into the base weights for serving, which is the W_merged line in my script.
Where does LoRA break?
It learns less. Biderman et al. (2024) found LoRA “substantially underperforms full finetuning” on code and math with typical ranks, and that full fine-tuning learns weight changes with 10 to 100 times higher rank than common LoRA configurations. The same paper found the upside: LoRA forgets less of what the base model could do. If your task needs a large change to the model, such as a new language or a new programming domain, a low-rank update may not hold it.
Adapters are tied to one checkpoint. A LoRA trained on one model is a delta on those exact weights. A new base version means training again.
Defaults lag the research. Query-and-value-only adapters were the original default; QLoRA’s results say adapt every linear layer. Check what your framework’s default target_modules actually covers.
Quantization error is real. QLoRA’s results match 16-bit fine-tuning on the paper’s benchmarks. Your task isn’t on those benchmarks, so compare a QLoRA run against a LoRA run on your eval once if you can afford it.
What I would do
For a fine-tune on an open-weight model, I’d start with QLoRA on every linear layer, rank 16, lora_alpha equal to the rank, and a few hundred clean examples. That’s the cheapest experiment that tells you whether fine-tuning helps at all. If it helps and memory allows, I’d rerun as plain LoRA to remove quantization from the comparison, and only consider full fine-tuning if the eval shows the adapter running out of capacity.
For serving, I’d keep adapters separate while iterating, since one base model plus many small adapters (about 42 MB at rank 8 for the 7B above, in bf16) is cheap to store and swap, then merge the winner for production. And I’d remember that LoRA is a training trick: a merged model is the same size and speed as the base, so a 7B LoRA fine-tune still needs 7B worth of inference hardware. If that’s too big, the answer is a smaller model, not a smaller adapter.
FAQ
What does LoRA stand for?
Low-Rank Adaptation, from Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021).
What rank should I use for LoRA?
Small ranks often suffice: the LoRA paper found rank 1 enough for GPT-3 on WikiSQL and MultiNLI, and QLoRA found r mattered less than adapting all linear layers. 8 to 64 is a common range; raise it if your eval shows the adapter underfitting.
Is QLoRA worse than LoRA?
In the QLoRA paper, NF4 with double quantization matched 16-bit LoRA and full fine-tuning on their benchmarks. It’s slower per step because of dequantization, and any quantization adds some error, so test on your own eval.
Does LoRA slow down inference?
Not once merged. W0 + BA is a single matrix of the original shape. Unmerged adapters add a small extra matrix multiply per adapted layer.
Can I fine-tune with LoRA on a Mac?
The memory math is the same on any hardware. QLoRA as released relies on the CUDA bitsandbytes library and NVIDIA unified memory, so on Apple Silicon you’d use a framework built for it rather than the paper’s exact stack.
Sources
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, June 17, 2021 (method in section 4.1, GPT-3 savings in section 4.2, rank study in section 7.2)
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, May 23, 2023
- Biderman et al., LoRA Learns Less and Forgets Less, May 15, 2024
- Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, October 4, 2019
- Jiang et al., Mistral 7B, October 10, 2023 (Table 1)
- Hugging Face PEFT docs, LoRA developer guide (read 2026-10-31)