What is a transformer? The architecture behind LLMs, explained
A transformer is the neural network architecture behind modern LLMs. How its layers, attention and feed-forward blocks work, with a Python parameter counter.

A transformer is a neural network architecture that processes a sequence of tokens by repeating one block many times: an attention layer, which lets every token pull information from other tokens, followed by a feed-forward layer, which transforms each token on its own. It was introduced in the 2017 paper Attention Is All You Need for machine translation, and it’s the architecture the GPT-3 and Llama 3 papers build their large language models on.
Why did the transformer replace RNNs?
Before 2017, language models were mostly recurrent networks (RNNs and LSTMs). They read a sentence one token at a time and carried a hidden state forward. That has two costs. Training can’t be parallelized across the sequence, because step 50 needs the result of step 49. And information from token 1 has to survive 49 updates to reach token 50.
Attention had already been bolted onto recurrent translation models. Bahdanau, Cho and Bengio’s 2014 paper argued that squeezing a whole source sentence into one fixed-length vector “is a bottleneck”, and let the decoder “(soft-)search for parts of a source sentence” instead. The transformer’s move was to drop the recurrence entirely and keep only attention.
The paper’s Table 1 states the trade in one line. A self-attention layer connects any two positions with O(1) sequential operations and a maximum path length of O(1); a recurrent layer needs O(n) for both. The price is O(n²d) work per layer, because every token is compared with every other token. That quadratic term is the reason long context is expensive, and the reason work like sparse attention exists.
The results were good enough to end the argument. The big model reached 28.4 BLEU on WMT 2014 English to German and 41.8 BLEU on English to French, trained in 3.5 days on eight NVIDIA P100 GPUs.
What is inside a transformer block?
The original model had an encoder and a decoder, each a stack of N = 6 identical layers with a model width (d_model) of 512. Each layer has two sub-layers:
- Multi-head self-attention. Each token is projected into a query, a key and a value. Queries are matched against keys to decide how much of each value to mix in. The base model splits this into 8 heads of 64 dimensions each, so different heads can track different relationships. The mechanism has its own post: what attention is.
- A position-wise feed-forward network. Two linear layers with a nonlinearity between them, applied to each token independently. In the base model the inner width (d_ff) is 2,048, four times d_model. Most of a transformer’s parameters live here.
Each sub-layer is wrapped in a residual connection and layer normalization, so the block computes x + Sublayer(x) and normalizes. Residuals let you stack dozens or hundreds of these blocks without the signal degrading.
Two more pieces finish the picture:
- Embeddings in and out. Tokens enter as embedding vectors, and the last layer projects back to a score for every vocabulary token. The paper shares one weight matrix between the two embedding layers and that final projection.
- Position information. Attention on its own has no sense of order; shuffle the tokens and the scores don’t change. The original model adds fixed sine and cosine signals to each embedding,
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))and the cosine for odd dimensions. Llama 3 and many other current LLMs use rotary position embeddings instead, which get their own post in what RoPE is.
Encoder, decoder or decoder-only?
The 2017 model was an encoder-decoder. The encoder reads the whole source sentence with attention in both directions. The decoder writes the translation one token at a time, and its self-attention is masked: the paper sets illegal connections to negative infinity before the softmax so that each position can only attend to itself and earlier positions. The decoder also has a third sub-layer, cross-attention, that looks at the encoder’s output.
GPT-style LLMs keep only the decoder stack, without cross-attention. There’s no separate source sentence: the prompt and the answer are one sequence, and the causal mask is what makes next-token prediction a fair training task, since the model can’t peek at the token it’s being asked to predict. That’s the architecture the GPT-3 and Llama 3 papers describe. Encoder-only models in the BERT family survive in embedding and reranking models, where seeing the whole input at once is the point.
How many parameters does a transformer have?
You can work it out from the config. Kaplan et al.’s scaling laws paper gives the approximation: per layer, 4 * d^2 for the attention projections (query, key, value, output) plus 2 * d * d_ff for the feed-forward block. With d_ff = 4 * d that’s 12 * d^2 per layer, so non-embedding parameters come to about 12 * n_layers * d_model^2. The same paper estimates training compute at about 6 FLOPs per parameter per token.
This script applies that to the GPT-3 configs from Table 2.1 of the GPT-3 paper and to the original Transformer.
# param_count.py: count a transformer's parameters from its config, the way
# Kaplan et al. (2020) do: each layer has 4*d^2 attention weights plus
# 2*d*d_ff feed-forward weights. Python 3, stdlib only.
def decoder_only(n_layers, d_model, d_ff=None):
d_ff = d_ff or 4 * d_model # GPT-3 uses d_ff = 4 * d_model
attn = 4 * d_model * d_model # W_q, W_k, W_v, W_o
ffn = 2 * d_model * d_ff # up and down projections
return n_layers * (attn + ffn) # = 12 * n_layers * d_model^2
# GPT-3 family, configs from Table 2.1 of "Language Models are Few-Shot Learners"
gpt3 = {"GPT-3 Small": (12, 768, "125M"), "GPT-3 XL": (24, 2048, "1.3B"),
"GPT-3 13B": (40, 5140, "13.0B"), "GPT-3 175B": (96, 12288, "175.0B")}
for name, (layers, d, reported) in gpt3.items():
n = decoder_only(layers, d)
print(f"{name:<12} layers={layers:>2} d_model={d:>5} non-embedding={n/1e6:>9,.0f}M paper says {reported}")
# The 2017 Transformer (base): 6 encoder + 6 decoder layers, d_model=512, d_ff=2048,
# ~37,000 shared BPE tokens. The decoder has a second attention block (cross-attention).
d, d_ff, vocab = 512, 2048, 37_000
enc = 6 * (4 * d * d + 2 * d * d_ff)
dec = 6 * (8 * d * d + 2 * d * d_ff)
emb = vocab * d # one matrix shared by both embeddings and the output layer
print(f"\nTransformer base: encoder {enc/1e6:.1f}M + decoder {dec/1e6:.1f}M + embeddings {emb/1e6:.1f}M"
f" = {(enc + dec + emb)/1e6:.1f}M (paper: 65M, before biases and layer norms)")
# Training compute rule of thumb: about 6 FLOPs per parameter per token.
n = decoder_only(96, 12288)
print(f"\nGPT-3 175B, 300B tokens: 6*N*D = {6 * n * 300e9:.2e} FLOPs")
# A 2024 block: Llama 3 405B (Table 3 of the Llama 3 paper). Grouped-query attention
# (8 key-value heads of 128 dims) and a SwiGLU feed-forward with three matrices.
L, d, ffn, kv, vocab = 126, 16_384, 53_248, 8 * 128, 128_000
per_layer = 2 * d * d + 2 * d * kv + 3 * d * ffn
total = L * per_layer + 2 * vocab * d # input embeddings + output layer, assumed untied
print(f"Llama 3 405B: {L} layers x {per_layer/1e6:,.0f}M + embeddings {2*vocab*d/1e9:.1f}B = {total/1e9:.1f}B")
Output (Python 3.13.11):
GPT-3 Small layers=12 d_model= 768 non-embedding= 85M paper says 125M
GPT-3 XL layers=24 d_model= 2048 non-embedding= 1,208M paper says 1.3B
GPT-3 13B layers=40 d_model= 5140 non-embedding= 12,681M paper says 13.0B
GPT-3 175B layers=96 d_model=12288 non-embedding= 173,946M paper says 175.0B
Transformer base: encoder 18.9M + decoder 25.2M + embeddings 18.9M = 63.0M (paper: 65M, before biases and layer norms)
GPT-3 175B, 300B tokens: 6*N*D = 3.13e+23 FLOPs
Llama 3 405B: 126 layers x 3,188M + embeddings 4.2B = 405.8B
Four things to read from this. First, the formula lands within 1% for GPT-3 175B, so a model’s size really is just depth times width squared. Second, the gap for GPT-3 Small (85M against 125M) is the embedding table: a vocabulary of tens of thousands of tokens times 768 dimensions is a big share of a small model and a rounding error in a large one. Third, the compute estimate matches the 3.14 × 10^23 FLOPs the GPT-3 paper reports for training the 175B model, which is the bridge to how LLMs are trained. Fourth, the same arithmetic works on a 2024 config once you account for its two changes, grouped-query attention and a three-matrix feed-forward block: Llama 3 405B comes out at 405.8B.
What changed in transformers since 2017?
The block is recognizably the same. The changes are refinements, mostly for efficiency at inference:
- Pre-normalization, normalizing before each sub-layer rather than after. The GPT-3 paper lists it among the GPT-2 changes it kept.
- Rotary position embeddings in place of the fixed sinusoids. Llama 3’s config table lists RoPE with a base frequency raised to 500,000 for longer contexts.
- A gated feed-forward block. Llama 3 uses SwiGLU, which has three weight matrices instead of two.
- Grouped-query attention. Several query heads share one key-value head, which shrinks the KV cache. The GQA paper reports quality close to full multi-head attention at speed comparable to a single shared head. Llama 3 405B has 128 query heads and 8 key-value heads.
- Mixture of experts. The feed-forward block is replaced by many expert networks and a router that sends each token to a few of them, so total parameters grow faster than compute per token. See mixture of experts.
- Faster attention kernels, which compute the same math with less memory traffic; FlashAttention is the best known.
The Llama 3 paper is blunt about how small these changes are: it calls the 405B model “a standard, dense Transformer architecture” that “does not deviate significantly” from Llama 2, and credits the gains to data and scale. None of these changed the core contract: a stack of blocks, each mixing information across tokens with attention and then transforming each token with a feed-forward network.
What I would do
You don’t need to implement a transformer to build with one, but you should be able to read a model config. When a model card lists layers, hidden size, heads, key-value heads and vocabulary, run the 12 * L * d^2 sum. It tells you roughly how much memory the weights need, how big the KV cache gets per token (two vectors, keys and values, times layers times key-value heads times head size), and whether the “8B” in the name includes the embedding table.
From a data-platform seat, the line I’d circle is the O(n²) in Table 1. Every architectural trick since 2017 is a negotiation with that term, and so is every pricing page that charges more for long prompts. When a vendor claims a much longer context, ask how they paid for it.
FAQ
Is ChatGPT a transformer?
ChatGPT runs on OpenAI’s GPT models, and the GPT-3 paper describes a decoder-style transformer. Labs that publish their architectures, like Meta for Llama 3, describe transformers too. Closed labs share fewer details about their current models.
What is the difference between a transformer and an LLM?
A transformer is an architecture, a way of wiring a neural network. An LLM is a trained model, typically a transformer with billions of parameters trained on trillions of tokens. Transformers also run inside models that aren’t LLMs: the Llama 3 paper’s own speech encoder is built from rotary transformer layers.
Do I need to understand transformers to use an LLM API?
No. You need tokens, context windows and sampling to use an API well. The architecture matters when you choose a self-hosted model, size its hardware, or reason about why long prompts cost more.
Are transformers being replaced?
Not in the models whose architectures are published. Alternatives that avoid the quadratic attention cost are active research, but Meta’s own 405B model chose “a standard, dense Transformer” in 2024 and won on data and scale. Bet on the transformer until a paper with production numbers says otherwise.
Sources
- Vaswani et al., Attention Is All You Need, June 12, 2017 (v7, August 2, 2023)
- Bahdanau, Cho and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, September 1, 2014
- Kaplan et al., Scaling Laws for Neural Language Models, January 23, 2020 (Table 1 and Section 2.1)
- Brown et al., Language Models are Few-Shot Learners, May 28, 2020 (Table 2.1 and Appendix D)
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, May 22, 2023
- Llama Team, Meta, The Llama 3 Herd of Models, July 23, 2024 (Section 3.2 and Table 3)