What is a large language model? How LLMs work, explained
A large language model is a neural network trained to predict the next token of text. How LLMs work, what 'large' means, and a 25-line Python demo.

A large language model (LLM) is a neural network, almost always a transformer, trained on a very large amount of text to predict the next token given the tokens before it. Generate one token, append it, predict again, and that single trick produces answers, code, translations and tool calls. “Large” refers to the parameter count (billions) and the training data (trillions of tokens), not to any formal threshold.
How does a large language model work?
Every LLM does the same three things at inference time:
- Tokenize. Your text becomes a list of integer IDs. I covered how that split happens, and why it decides your bill, in what tokens are.
- Predict. The network reads the whole list and outputs a probability for every token in its vocabulary as the next one.
- Sample and repeat. One token is picked from that distribution, appended to the input, and the loop runs again until the model emits a stop token or hits a length limit.
That loop is called autoregressive generation. The model never plans a paragraph and then writes it; it writes one token at a time, and everything it “knows” about the conversation so far is whatever sits in the input, which is why the context window matters so much.
The network in the middle is a transformer, the architecture from Google’s 2017 paper Attention Is All You Need. Its core operation, attention, lets each token look at every earlier token and decide which ones matter for the next prediction.
What does next-token prediction look like in code?
Here is the whole idea with the neural network removed. It counts which word follows which in a tiny corpus (that’s the “training”), then generates by sampling a next word, appending it and repeating (that’s the “inference”). No packages, no key.
# tiny_lm.py: a language model in 25 lines. Count which word follows which,
# then generate by predicting the next word over and over. Python 3, stdlib only.
import random
from collections import Counter, defaultdict
corpus = """the model reads the prompt . the model predicts the next token .
the next token is added to the prompt . the model reads the prompt again .
a large model predicts the next token better than a small model .""".split()
follows = defaultdict(Counter) # "training": count bigrams
for a, b in zip(corpus, corpus[1:]):
follows[a][b] += 1
def next_token_probs(word):
counts = follows[word]
total = sum(counts.values())
return {w: c / total for w, c in counts.most_common()}
print("P(next | 'the') =", {w: round(p, 2) for w, p in next_token_probs("the").items()})
print("P(next | 'model') =", {w: round(p, 2) for w, p in next_token_probs("model").items()})
random.seed(7)
for start in ("the", "a"): # "inference": sample, append, repeat
out = [start]
while out[-1] != "." and len(out) < 12:
probs = next_token_probs(out[-1])
out.append(random.choices(list(probs), weights=probs.values())[0])
print("generated:", " ".join(out))
Output (Python 3.13.11):
P(next | 'the') = {'model': 0.33, 'prompt': 0.33, 'next': 0.33}
P(next | 'model') = {'reads': 0.4, 'predicts': 0.4, '.': 0.2}
generated: the model reads the model predicts the model predicts the prompt .
generated: a large model .
The second line of output is the shape of every LLM’s job: a probability over possible next tokens. The generated sentences show the weakness of this toy. It only sees one word back, so “the model reads the model predicts” is locally plausible and globally nonsense.
A real LLM replaces the count table with a transformer that conditions on thousands of previous tokens, and replaces counting with gradient descent over billions of weights. The loop at the bottom, sample then append then repeat, stays exactly the same. How it picks from that distribution is the subject of temperature, top-p and top-k.
What makes a language model “large”?
Two numbers: parameters and training tokens.
Parameters are the learned weights. GPT-3, described in Language Models are Few-Shot Learners (May 2020), had 175 billion, which the authors called “10x more than any previous non-sparse language model”. It was trained on 300 billion tokens. Four years later Meta’s Llama 3 paper describes a 405 billion parameter model trained on 15.6 trillion tokens with 3.8 × 10^25 FLOPs, almost 50 times the compute of the largest Llama 2.
Scale matters because loss falls predictably as you add parameters, data and compute together, which is what scaling laws describe. The GPT-3 paper’s headline finding was a side effect of that: a large enough model could do a new task from a few examples in the prompt, “without any gradient updates or fine-tuning”. That’s in-context learning, and it’s why prompting works at all.
There’s no line where a model becomes “large”. In 2026 people call a 3B model a small language model and still mean an LLM-style transformer; what changed is the expectation, not the definition.
How is an LLM different from ChatGPT?
An LLM is the model. ChatGPT, Claude and Gemini are products built on top of one.
A freshly pretrained model (a base model) only continues text. Ask it a question and it may answer, or it may write three more questions in the same style, because that’s what often follows a question on the web. Turning a base model into an assistant takes post-training. OpenAI’s InstructGPT paper (March 2022) showed how much that step matters: human raters preferred outputs from a 1.3 billion parameter InstructGPT model to the 175 billion parameter GPT-3, “despite having 100x fewer parameters”. Their recipe was supervised fine-tuning on human-written demonstrations, then reinforcement learning from human feedback. I walk through the stages in how LLMs are trained, and the moment that recipe reached the public in ChatGPT’s research preview.
The product then adds a system prompt, a chat template, safety filters, tools, memory and a user interface. When people compare “ChatGPT vs Claude” they’re comparing all of that, not only the weights.
What can’t a large language model do?
It can’t look things up. Everything the model knows comes from its training data up to a cutoff date, compressed into weights. Ask about something rarer or newer than that and it still produces a fluent answer, because producing fluent text is the one thing it was trained to do. That’s the root of hallucinations, and the reason retrieval and tools exist.
It has no memory between calls. Each API request is independent; the “memory” in a chat app is the app resending the history, which costs tokens every turn.
It doesn’t run your code or check its own math unless the system around it gives it a tool to do so. A modern agent is an LLM in a loop with tools, which is a different thing from the LLM alone.
Its output is a sample, not a lookup. Same prompt, different answer, because the next token is drawn from a distribution. Lowering randomness narrows the spread; it doesn’t turn the model into a database.
What I would do
Treat the model as a component with a clear contract: text in, a probability distribution over text out. From a data-platform seat, that framing is what keeps a system sane. You wouldn’t put an unvalidated upstream feed straight into a report, and you shouldn’t put unvalidated model output straight into a database or a customer email either.
Concretely:
- Pick the model last. Write down the task, the inputs, what a correct output looks like and how you’ll check it. Then try two or three models against that check.
- Ground facts. If the answer depends on facts that change or live in your systems, retrieve them and put them in the prompt rather than trusting the weights. That’s the case for RAG.
- Count tokens early. Price, latency and context limits are all measured in tokens, and they shape the design more than model choice does.
- Validate structure. If you need JSON, parse it and reject what doesn’t fit, every time.
On a search product like rSearch, the model is one line in a config; the retrieval, the prompt and the checks around it are the system. I’d design for the model being swapped, because it will be.
FAQ
Is an LLM the same as AI?
No. An LLM is one kind of AI model, a neural network trained on text to predict the next token. Image generators, speech recognizers, recommendation systems and chess engines are AI too, and most of them aren’t language models. Many current LLMs are multimodal, reading images and audio as well as text.
What is a base model?
A base model is an LLM straight out of pretraining: it continues text and nothing more. An instruct or chat model is a base model after post-training (supervised fine-tuning and preference training), so it answers questions, follows instructions and stops when it’s done. Open-weight releases often ship both.
How many parameters does an LLM have?
Anywhere from about a billion to several hundred billion for dense models, and more for mixture-of-experts models that only use a fraction of their weights per token. GPT-3 had 175 billion; Llama 3’s largest had 405 billion. Parameter count alone doesn’t predict quality; training data and post-training matter as much.
Do LLMs understand language?
They model it statistically well enough to translate, summarize, write code and follow instructions. Whether that counts as understanding is a philosophical argument. For a builder, the practical answer is to judge a model by measured task performance, not by how fluent it sounds.
Sources
- Vaswani et al., Attention Is All You Need, June 12, 2017
- Brown et al., Language Models are Few-Shot Learners (GPT-3), May 28, 2020
- Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT), March 4, 2022
- Llama Team, Meta, The Llama 3 Herd of Models, July 23, 2024