What are tokens in AI? A builder's explainer
Tokens are the chunks of text an AI model reads and bills you for. See how tokenizers split text, why Gujarati costs more, and how to count them.

A token is the unit of text a language model reads and writes: a whole word, a piece of a word, a space-plus-word, a punctuation mark or a single byte. Every model turns your text into a list of integer token IDs before it does anything else, and API providers bill you per token, so tokens decide your cost, your latency and how much fits in a prompt.
I ran every snippet below on 2026-10-23 with Python 3.13.5, tiktoken 0.14.0 and Hugging Face tokenizers 0.23.3. The outputs are pasted exactly as they printed.
Why should a builder care about tokens?
Because the bill, the context limit and the speed of a response are all counted in tokens. OpenAI’s help article on counting tokens puts it plainly: input, output, cached and reasoning tokens each count toward usage, and the context window limits how many tokens a request can hold.
The catch is that a token count isn’t a word count, and it changes with the model and the language. If you estimate a budget in words, you’ll be wrong by a margin you can’t predict until you measure.
When I ran the same question in Gujarati through three tokenizers for this post, the counts came out 58, 37 and 8. Same sentence, three tokenizers, and a 7x spread in the number that both the bill and the context limit are measured in. That test is further down.
What does a token look like in code?
Here is the smallest thing that works: encode one sentence with the tokenizer that GPT-4o uses and print what comes back.
# pip install tiktoken==0.14.0
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
text = "Hello world! Tokens are how models read."
ids = enc.encode(text)
print(len(text), "characters ->", len(ids), "tokens")
print(ids)
print([enc.decode([i]) for i in ids])
print(enc.decode(ids) == text)
40 characters -> 9 tokens
[13225, 2375, 0, 124325, 553, 1495, 7015, 1729, 13]
['Hello', ' world', '!', ' Tokens', ' are', ' how', ' models', ' read', '.']
True
Most words here are one token each, and the space belongs to the word after it (' world' starts with a space). The model receives only the list of integers: "Hello" is ID 13225 and nothing more.
o200k_base is the encoding name; tiktoken’s model table maps GPT-4o, GPT-4.1, GPT-5 and the o-series to it, and GPT-4 and GPT-3.5 Turbo to the older cl100k_base.
How does a tokenizer split text into tokens?
Most LLM tokenizers today use byte-pair encoding (BPE). It started as a data compression trick: Philip Gage’s 1994 algorithm (cited in the Sennrich paper below) repeatedly replaces the most frequent pair of bytes with a new symbol. Sennrich, Haddow and Birch (2016) adapted it to split words into subword units for machine translation, and the GPT-2 paper (Radford et al., 2019) ran it on raw UTF-8 bytes, so the base vocabulary is just 256 symbols and any string can be encoded. Compression is a topic I’ve spent time on before; one of my papers reviews parallel LZW lossless compression with OpenMP, and BPE is the same instinct pointed at text.
I only trust an algorithm once I’ve written it, so here is BPE from scratch in about 30 lines. It starts from bytes and performs five merges.
# Toy byte-pair encoding, pure Python 3.13, no dependencies.
text = "low lower lowest slow slower"
ids = list(text.encode("utf-8")) # start from raw bytes: 256 possible symbols
print("start:", len(ids), "tokens")
def pair_counts(ids):
counts = {}
for a, b in zip(ids, ids[1:]):
counts[(a, b)] = counts.get((a, b), 0) + 1
return counts
def merge(ids, pair, new_id):
out, i = [], 0
while i < len(ids):
if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
out.append(new_id)
i += 2
else:
out.append(ids[i])
i += 1
return out
vocab = {i: bytes([i]) for i in range(256)}
for step in range(5):
counts = pair_counts(ids)
pair = max(counts, key=counts.get)
new_id = 256 + step
vocab[new_id] = vocab[pair[0]] + vocab[pair[1]]
ids = merge(ids, pair, new_id)
print(f"merge {step + 1}: {vocab[pair[0]]!r} + {vocab[pair[1]]!r} -> {new_id} {vocab[new_id]!r} ({counts[pair]}x), now {len(ids)} tokens")
print([vocab[i].decode() for i in ids])
start: 28 tokens
merge 1: b'l' + b'o' -> 256 b'lo' (5x), now 23 tokens
merge 2: b'lo' + b'w' -> 257 b'low' (5x), now 18 tokens
merge 3: b'low' + b'e' -> 258 b'lowe' (3x), now 15 tokens
merge 4: b'low' + b' ' -> 259 b'low ' (2x), now 13 tokens
merge 5: b'lowe' + b'r' -> 260 b'lower' (2x), now 11 tokens
['low ', 'lower', ' ', 'lowe', 's', 't', ' ', 's', 'low ', 's', 'lower']
Twenty-eight bytes became 11 tokens after five merges. A real tokenizer does the same thing tens of thousands of times over a huge training corpus: GPT-2 ended with 50,257 tokens, and o200k_base has 200,019 IDs (enc.n_vocab in tiktoken 0.14.0).
Merge 4 also shows a bug. My toy glued a trailing space onto 'low ', which wastes vocabulary on every word-plus-space combination. The GPT-2 paper fixes this by never merging across character categories (letters, digits, punctuation), with an exception that lets a space attach to the start of the next word. That rule is why ' world' carries its space at the front. Karpathy’s minbpe implements that regex split and reproduces the GPT-4 tokenizer exactly, and it’s the code I’d read next.
Once the text is IDs, the model takes over. It reads the IDs and produces a score for every ID in the vocabulary as the next token; one gets picked, decoded back to text and appended, and the loop runs again. Each trip around that loop is one output token.
Why does the same text cost different amounts on different models?
Each model family trains its own tokenizer, and the vocabulary decides how many pieces your text breaks into. I asked one question, “Where is the railway station?”, in English, Hindi and Gujarati, and counted it with three tokenizers: GPT-2’s (from the Hugging Face Hub), cl100k_base and o200k_base.
# pip install tiktoken==0.14.0 tokenizers==0.23.3
import tiktoken
from tokenizers import Tokenizer
samples = {
"English": "Where is the railway station?",
"Hindi": "रेलवे स्टेशन कहाँ है?",
"Gujarati": "રેલવે સ્ટેશન ક્યાં છે?",
"Python": "def add(a, b):\n return a + b",
"Number": "3141592653589793",
"Emoji": "🚀🔥",
}
gpt2 = Tokenizer.from_pretrained("gpt2") # Hugging Face Hub
encoders = {
"gpt2": lambda s: gpt2.encode(s).ids,
"cl100k_base": tiktoken.get_encoding("cl100k_base").encode,
"o200k_base": tiktoken.get_encoding("o200k_base").encode,
}
print(f"{'sample':<9} {'chars':>5} {'bytes':>5}", *(f"{n:>11}" for n in encoders))
for name, s in samples.items():
counts = [len(f(s)) for f in encoders.values()]
print(f"{name:<9} {len(s):>5} {len(s.encode('utf-8')):>5}", *(f"{c:>11}" for c in counts))
sample chars bytes gpt2 cl100k_base o200k_base
English 29 29 6 6 6
Hindi 21 55 34 20 6
Gujarati 22 58 58 37 8
Python 31 31 15 11 11
Number 16 16 6 6 6
Emoji 2 8 6 6 3
(The script also prints a warning to stderr about unauthenticated Hugging Face Hub requests. It’s harmless.)
English costs 6 tokens everywhere. Gujarati costs 58 tokens on GPT-2, one per UTF-8 byte, because that vocabulary has almost no Gujarati merges. On cl100k_base it’s 37, and on o200k_base it drops to 8. Same meaning, a 7x spread in what you’d pay and how much of the context window it eats.
This matters to me because I build from Surat, where Gujarati and Hindi are everyday languages. If your product serves Indian-language text, check the tokenizer on your own samples before you pick a model or set a budget.
It isn’t only an OpenAI thing. Anthropic’s token counting docs say Claude Opus 4.7 and later models use a newer tokenizer that produces about 30 percent more tokens for the same input text than earlier Claude models (checked 2026-10-23), and they tell you to recount prompts against the model you plan to use.
How do tokens turn into a bill?
Providers price input and output tokens separately, per million tokens. OpenAI’s pricing page lists gpt-4o at $2.50 per 1M input tokens and $10.00 per 1M output tokens on standard processing (checked 2026-10-23). Here is the arithmetic for one small call.
# pip install tiktoken==0.14.0
import tiktoken
# gpt-4o standard prices, USD per 1M tokens (OpenAI pricing page, checked 2026-10-23)
PRICE_IN, PRICE_OUT = 2.50, 10.00
enc = tiktoken.encoding_for_model("gpt-4o") # -> o200k_base
prompt = "Summarise this support ticket in two lines: my order #4521 arrived damaged."
input_tokens = len(enc.encode(prompt))
output_tokens = 300 # an assumption: a chatty answer; set max_output_tokens to cap it
cost = input_tokens * PRICE_IN / 1e6 + output_tokens * PRICE_OUT / 1e6
print(enc.name, input_tokens, "input tokens,", output_tokens, "output tokens")
print(f"input ${input_tokens * PRICE_IN / 1e6:.6f}")
print(f"output ${output_tokens * PRICE_OUT / 1e6:.6f}")
print(f"total ${cost:.6f} per call, ${cost * 10_000:.2f} per 10,000 calls")
o200k_base 18 input tokens, 300 output tokens
input $0.000045
output $0.003000
total $0.003045 per call, $30.45 per 10,000 calls
The 300 output tokens are my assumption for a long-winded answer, so treat the dollar figure as a worked example. The ratio is the real lesson: for a short prompt, output dominates, at four times the input rate per token on gpt-4o. Asking for “two lines” and capping max_output_tokens (Responses API) or max_completion_tokens (Chat Completions) does more for this bill than trimming the prompt.
Reasoning models add a hidden line item. OpenAI’s help article says reasoning tokens don’t appear in the answer text but count toward output usage and get billed as output tokens. I’ll cover reasoning models, and the full per-task cost math, in separate posts.
Where do tokens trip you up?
A few behaviours catch people out, and they all follow from the model seeing IDs instead of letters.
# pip install tiktoken==0.14.0
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
def show(s):
ids = enc.encode(s)
print(f"{s!r:<18} {ids} {[enc.decode([i]) for i in ids]}")
# 1. Same word, different tokens
for s in ["hello", " hello", "Hello", "HELLO"]:
show(s)
# 2. The model never sees letters
show("strawberry")
# 3. Numbers get chunked
show("3141592653589793")
show("1,000,000")
'hello' [24912] ['hello']
' hello' [40617] [' hello']
'Hello' [13225] ['Hello']
'HELLO' [111642, 2699] ['HEL', 'LO']
'strawberry' [302, 1618, 19772] ['st', 'raw', 'berry']
'3141592653589793' [28968, 19218, 25799, 31887, 48822, 18] ['314', '159', '265', '358', '979', '3']
'1,000,000' [16, 11, 1302, 11, 1302] ['1', ',', '000', ',', '000']
Spacing and case change the ID. hello, hello and Hello are three unrelated integers, and HELLO splits into two. OpenAI’s help article makes the same point with “red” and “ red“. A stray leading space or an all-caps prompt changes what the model receives.
Letters are invisible. “strawberry” arrives as three IDs (st, raw, berry). Counting the r’s means the model has to know the spelling of each piece from training. My view is that this explains a lot of the classic letter-counting mistakes, though I wouldn’t claim it’s the only cause.
Numbers come in chunks of three. o200k_base cuts long digit strings into three-digit groups, and cl100k_base gave me the same split for the 16-digit number. If you need exact arithmetic, have the model call code instead of doing digits in its head.
The 4-characters rule is English-only. OpenAI’s rule of thumb is about 4 characters or three quarters of a word per token, for English. My Gujarati sentence was 22 characters and 37 tokens on cl100k_base, so a character-based estimate would have been off by a factor of about 7.
Text isn’t the whole request. Message roles, tool definitions, JSON schemas, images and files all add input tokens. Both OpenAI and Anthropic say this in the docs linked above, and both offer an endpoint that counts the full request.
How do I count tokens accurately?
Use the model’s own tokenizer, and for a full API request use the provider’s counting endpoint. This is what I’d do:
- For OpenAI models,
tiktoken.encoding_for_model("<model>")for plain text, and OpenAI’s input-token counting API when the request has tools, images or files. - For Claude, Anthropic’s free
count_tokensendpoint with the exact model ID you’ll call, since their tokenizer changed with Opus 4.7. - For open-weight models, load the model’s tokenizer from the Hugging Face Hub with
tokenizersortransformers. I run local models on the Mac Mini listed on my setup page, and there the model’s own token count decides what fits in its context window. - After a call, log the
usagefields the API returns (input_tokensandoutput_tokensin Responses,prompt_tokensandcompletion_tokensin Chat Completions). That’s the number you get billed on.
I’d skip character or word heuristics for anything that touches money, and I’d re-measure whenever I switch models.
FAQ
How many words is one token?
For English text, about three quarters of a word, or roughly 4 characters, per OpenAI’s estimate. For other languages and for code it varies a lot; measure with the tokenizer.
Do spaces and punctuation count as tokens?
Yes. Punctuation is often its own token ('!' was ID 0 above), and a space usually sticks to the start of the next word, so ' world' is one token.
Do all AI models use the same tokenizer?
No. GPT-4 uses cl100k_base, GPT-4o and later OpenAI models use o200k_base, Claude models have their own tokenizer (which changed with Opus 4.7), and open-weight models ship their own. The same text gives different counts on each.
Are images and files counted in tokens too?
Yes, on the APIs I’ve checked. OpenAI says images and files affect the input count, and Anthropic’s count_tokens endpoint returns token counts for image and PDF inputs. The exact cost depends on the model and the input size, so count it rather than guessing.
Sources
- OpenAI Help Center, Understanding and counting tokens (read 2026-10-23)
- OpenAI, API pricing (gpt-4o prices checked 2026-10-23)
- Anthropic, Token counting (read 2026-10-23)
- openai/tiktoken, README and model.py (model to encoding table)
- Hugging Face, Tokenizers documentation and the GPT-2 model card
- Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (ACL 2016)
- Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, 2019), section 2.2
- Andrej Karpathy, minbpe
What I’m writing next
This is the first post in a series on LLM basics. Next up: what an LLM is, what a context window is, and how to work out LLM cost per task. Tokens come up in all of them.