Malhar Ujawane

Blog

Post·

What are tokens in AI? A builder's explainer

Tokens are the chunks of text an AI model reads and bills you for. See how tokenizers split text, why Gujarati costs more, and how to count them.

What are tokens in AI? A builder's explainer

A token is the unit of text a language model reads and writes: a whole word, a piece of a word, a space-plus-word, a punctuation mark or a single byte. Every model turns your text into a list of integer token IDs before it does anything else, and API providers bill you per token, so tokens decide your cost, your latency and how much fits in a prompt.

I ran every snippet below on 2026-10-23 with Python 3.13.5, tiktoken 0.14.0 and Hugging Face tokenizers 0.23.3. The outputs are pasted exactly as they printed.

Why should a builder care about tokens?

Because the bill, the context limit and the speed of a response are all counted in tokens. OpenAI’s help article on counting tokens puts it plainly: input, output, cached and reasoning tokens each count toward usage, and the context window limits how many tokens a request can hold.

The catch is that a token count isn’t a word count, and it changes with the model and the language. If you estimate a budget in words, you’ll be wrong by a margin you can’t predict until you measure.

When I ran the same question in Gujarati through three tokenizers for this post, the counts came out 58, 37 and 8. Same sentence, three tokenizers, and a 7x spread in the number that both the bill and the context limit are measured in. That test is further down.

What does a token look like in code?

Here is the smallest thing that works: encode one sentence with the tokenizer that GPT-4o uses and print what comes back.

# pip install tiktoken==0.14.0
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
text = "Hello world! Tokens are how models read."
ids = enc.encode(text)
print(len(text), "characters ->", len(ids), "tokens")
print(ids)
print([enc.decode([i]) for i in ids])
print(enc.decode(ids) == text)
40 characters -> 9 tokens
[13225, 2375, 0, 124325, 553, 1495, 7015, 1729, 13]
['Hello', ' world', '!', ' Tokens', ' are', ' how', ' models', ' read', '.']
True

Most words here are one token each, and the space belongs to the word after it (' world' starts with a space). The model receives only the list of integers: "Hello" is ID 13225 and nothing more.

o200k_base is the encoding name; tiktoken’s model table maps GPT-4o, GPT-4.1, GPT-5 and the o-series to it, and GPT-4 and GPT-3.5 Turbo to the older cl100k_base.

How does a tokenizer split text into tokens?

Most LLM tokenizers today use byte-pair encoding (BPE). It started as a data compression trick: Philip Gage’s 1994 algorithm (cited in the Sennrich paper below) repeatedly replaces the most frequent pair of bytes with a new symbol. Sennrich, Haddow and Birch (2016) adapted it to split words into subword units for machine translation, and the GPT-2 paper (Radford et al., 2019) ran it on raw UTF-8 bytes, so the base vocabulary is just 256 symbols and any string can be encoded. Compression is a topic I’ve spent time on before; one of my papers reviews parallel LZW lossless compression with OpenMP, and BPE is the same instinct pointed at text.

I only trust an algorithm once I’ve written it, so here is BPE from scratch in about 30 lines. It starts from bytes and performs five merges.

# Toy byte-pair encoding, pure Python 3.13, no dependencies.
text = "low lower lowest slow slower"
ids = list(text.encode("utf-8"))  # start from raw bytes: 256 possible symbols
print("start:", len(ids), "tokens")

def pair_counts(ids):
    counts = {}
    for a, b in zip(ids, ids[1:]):
        counts[(a, b)] = counts.get((a, b), 0) + 1
    return counts

def merge(ids, pair, new_id):
    out, i = [], 0
    while i < len(ids):
        if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
            out.append(new_id)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out

vocab = {i: bytes([i]) for i in range(256)}
for step in range(5):
    counts = pair_counts(ids)
    pair = max(counts, key=counts.get)
    new_id = 256 + step
    vocab[new_id] = vocab[pair[0]] + vocab[pair[1]]
    ids = merge(ids, pair, new_id)
    print(f"merge {step + 1}: {vocab[pair[0]]!r} + {vocab[pair[1]]!r} -> {new_id} {vocab[new_id]!r}  ({counts[pair]}x), now {len(ids)} tokens")

print([vocab[i].decode() for i in ids])
start: 28 tokens
merge 1: b'l' + b'o' -> 256 b'lo'  (5x), now 23 tokens
merge 2: b'lo' + b'w' -> 257 b'low'  (5x), now 18 tokens
merge 3: b'low' + b'e' -> 258 b'lowe'  (3x), now 15 tokens
merge 4: b'low' + b' ' -> 259 b'low '  (2x), now 13 tokens
merge 5: b'lowe' + b'r' -> 260 b'lower'  (2x), now 11 tokens
['low ', 'lower', ' ', 'lowe', 's', 't', ' ', 's', 'low ', 's', 'lower']

Twenty-eight bytes became 11 tokens after five merges. A real tokenizer does the same thing tens of thousands of times over a huge training corpus: GPT-2 ended with 50,257 tokens, and o200k_base has 200,019 IDs (enc.n_vocab in tiktoken 0.14.0).

Merge 4 also shows a bug. My toy glued a trailing space onto 'low ', which wastes vocabulary on every word-plus-space combination. The GPT-2 paper fixes this by never merging across character categories (letters, digits, punctuation), with an exception that lets a space attach to the start of the next word. That rule is why ' world' carries its space at the front. Karpathy’s minbpe implements that regex split and reproduces the GPT-4 tokenizer exactly, and it’s the code I’d read next.

Once the text is IDs, the model takes over. It reads the IDs and produces a score for every ID in the vocabulary as the next token; one gets picked, decoded back to text and appended, and the loop runs again. Each trip around that loop is one output token.

How text becomes tokens and backFive rows. 01 text: Hello world!. 02 tokens: Hello, world with its leading space shown as a dot, and an exclamation mark. 03 token IDs in o200k_base: 13225, 2375 and 0. 04 the model reads the IDs and scores all 200,019 possible next IDs. 05 one ID is picked, decoded to text and appended; an orange line loops back to row 03 because this repeats once per output token. Input tokens are rows 01 to 03, billed once per request; output tokens are billed once per loop.From text to tokens and backO200K_BASE · TIKTOKEN 0.14.001TEXTHello world!02TOKENSHello·world!03TOKEN IDS132252375004MODELLLMscores 200,019 IDs05OUTPUTpick one ID → decode → appendrepeat onceper output tokenBILLINGInput tokens: rows 01 to 03, counted once per request.Output tokens: one per trip around the orange loop.

Why does the same text cost different amounts on different models?

Each model family trains its own tokenizer, and the vocabulary decides how many pieces your text breaks into. I asked one question, “Where is the railway station?”, in English, Hindi and Gujarati, and counted it with three tokenizers: GPT-2’s (from the Hugging Face Hub), cl100k_base and o200k_base.

# pip install tiktoken==0.14.0 tokenizers==0.23.3
import tiktoken
from tokenizers import Tokenizer

samples = {
    "English": "Where is the railway station?",
    "Hindi": "रेलवे स्टेशन कहाँ है?",
    "Gujarati": "રેલવે સ્ટેશન ક્યાં છે?",
    "Python": "def add(a, b):\n    return a + b",
    "Number": "3141592653589793",
    "Emoji": "🚀🔥",
}

gpt2 = Tokenizer.from_pretrained("gpt2")  # Hugging Face Hub
encoders = {
    "gpt2": lambda s: gpt2.encode(s).ids,
    "cl100k_base": tiktoken.get_encoding("cl100k_base").encode,
    "o200k_base": tiktoken.get_encoding("o200k_base").encode,
}

print(f"{'sample':<9} {'chars':>5} {'bytes':>5}", *(f"{n:>11}" for n in encoders))
for name, s in samples.items():
    counts = [len(f(s)) for f in encoders.values()]
    print(f"{name:<9} {len(s):>5} {len(s.encode('utf-8')):>5}", *(f"{c:>11}" for c in counts))
sample    chars bytes        gpt2 cl100k_base  o200k_base
English      29    29           6           6           6
Hindi        21    55          34          20           6
Gujarati     22    58          58          37           8
Python       31    31          15          11          11
Number       16    16           6           6           6
Emoji         2     8           6           6           3

(The script also prints a warning to stderr about unauthenticated Hugging Face Hub requests. It’s harmless.)

English costs 6 tokens everywhere. Gujarati costs 58 tokens on GPT-2, one per UTF-8 byte, because that vocabulary has almost no Gujarati merges. On cl100k_base it’s 37, and on o200k_base it drops to 8. Same meaning, a 7x spread in what you’d pay and how much of the context window it eats.

Token count for the same question in three languages“Where is the railway station?” counted with three tokenizers. English: 6 tokens on gpt2, cl100k_base and o200k_base. Hindi: 34, 20 and 6. Gujarati: 58, 37 and 8.One question, three languages, three tokenizers“Where is the railway station?” Token count, lower is cheaper.gpt2cl100k_baseo200k_base0102030405060English29 UTF-8 BYTESgpt26cl100k_base6o200k_base6Hindi55 UTF-8 BYTESgpt234cl100k_base20o200k_base6Gujarati58 UTF-8 BYTESgpt258cl100k_base37o200k_base8MEASURED 2026-10-23 · TIKTOKEN 0.14.0 · TOKENIZERS 0.23.3 · GPT2 ON GUJARATI = 1 TOKEN PER BYTE

This matters to me because I build from Surat, where Gujarati and Hindi are everyday languages. If your product serves Indian-language text, check the tokenizer on your own samples before you pick a model or set a budget.

It isn’t only an OpenAI thing. Anthropic’s token counting docs say Claude Opus 4.7 and later models use a newer tokenizer that produces about 30 percent more tokens for the same input text than earlier Claude models (checked 2026-10-23), and they tell you to recount prompts against the model you plan to use.

How do tokens turn into a bill?

Providers price input and output tokens separately, per million tokens. OpenAI’s pricing page lists gpt-4o at $2.50 per 1M input tokens and $10.00 per 1M output tokens on standard processing (checked 2026-10-23). Here is the arithmetic for one small call.

# pip install tiktoken==0.14.0
import tiktoken

# gpt-4o standard prices, USD per 1M tokens (OpenAI pricing page, checked 2026-10-23)
PRICE_IN, PRICE_OUT = 2.50, 10.00

enc = tiktoken.encoding_for_model("gpt-4o")  # -> o200k_base
prompt = "Summarise this support ticket in two lines: my order #4521 arrived damaged."
input_tokens = len(enc.encode(prompt))
output_tokens = 300  # an assumption: a chatty answer; set max_output_tokens to cap it

cost = input_tokens * PRICE_IN / 1e6 + output_tokens * PRICE_OUT / 1e6
print(enc.name, input_tokens, "input tokens,", output_tokens, "output tokens")
print(f"input  ${input_tokens * PRICE_IN / 1e6:.6f}")
print(f"output ${output_tokens * PRICE_OUT / 1e6:.6f}")
print(f"total  ${cost:.6f} per call, ${cost * 10_000:.2f} per 10,000 calls")
o200k_base 18 input tokens, 300 output tokens
input  $0.000045
output $0.003000
total  $0.003045 per call, $30.45 per 10,000 calls

The 300 output tokens are my assumption for a long-winded answer, so treat the dollar figure as a worked example. The ratio is the real lesson: for a short prompt, output dominates, at four times the input rate per token on gpt-4o. Asking for “two lines” and capping max_output_tokens (Responses API) or max_completion_tokens (Chat Completions) does more for this bill than trimming the prompt.

Reasoning models add a hidden line item. OpenAI’s help article says reasoning tokens don’t appear in the answer text but count toward output usage and get billed as output tokens. I’ll cover reasoning models, and the full per-task cost math, in separate posts.

Where do tokens trip you up?

A few behaviours catch people out, and they all follow from the model seeing IDs instead of letters.

# pip install tiktoken==0.14.0
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def show(s):
    ids = enc.encode(s)
    print(f"{s!r:<18} {ids}  {[enc.decode([i]) for i in ids]}")

# 1. Same word, different tokens
for s in ["hello", " hello", "Hello", "HELLO"]:
    show(s)
# 2. The model never sees letters
show("strawberry")
# 3. Numbers get chunked
show("3141592653589793")
show("1,000,000")
'hello'            [24912]  ['hello']
' hello'           [40617]  [' hello']
'Hello'            [13225]  ['Hello']
'HELLO'            [111642, 2699]  ['HEL', 'LO']
'strawberry'       [302, 1618, 19772]  ['st', 'raw', 'berry']
'3141592653589793' [28968, 19218, 25799, 31887, 48822, 18]  ['314', '159', '265', '358', '979', '3']
'1,000,000'        [16, 11, 1302, 11, 1302]  ['1', ',', '000', ',', '000']

Spacing and case change the ID. hello, hello and Hello are three unrelated integers, and HELLO splits into two. OpenAI’s help article makes the same point with “red” and “ red“. A stray leading space or an all-caps prompt changes what the model receives.

Letters are invisible. “strawberry” arrives as three IDs (st, raw, berry). Counting the r’s means the model has to know the spelling of each piece from training. My view is that this explains a lot of the classic letter-counting mistakes, though I wouldn’t claim it’s the only cause.

Numbers come in chunks of three. o200k_base cuts long digit strings into three-digit groups, and cl100k_base gave me the same split for the 16-digit number. If you need exact arithmetic, have the model call code instead of doing digits in its head.

The 4-characters rule is English-only. OpenAI’s rule of thumb is about 4 characters or three quarters of a word per token, for English. My Gujarati sentence was 22 characters and 37 tokens on cl100k_base, so a character-based estimate would have been off by a factor of about 7.

Text isn’t the whole request. Message roles, tool definitions, JSON schemas, images and files all add input tokens. Both OpenAI and Anthropic say this in the docs linked above, and both offer an endpoint that counts the full request.

How do I count tokens accurately?

Use the model’s own tokenizer, and for a full API request use the provider’s counting endpoint. This is what I’d do:

  1. For OpenAI models, tiktoken.encoding_for_model("<model>") for plain text, and OpenAI’s input-token counting API when the request has tools, images or files.
  2. For Claude, Anthropic’s free count_tokens endpoint with the exact model ID you’ll call, since their tokenizer changed with Opus 4.7.
  3. For open-weight models, load the model’s tokenizer from the Hugging Face Hub with tokenizers or transformers. I run local models on the Mac Mini listed on my setup page, and there the model’s own token count decides what fits in its context window.
  4. After a call, log the usage fields the API returns (input_tokens and output_tokens in Responses, prompt_tokens and completion_tokens in Chat Completions). That’s the number you get billed on.

I’d skip character or word heuristics for anything that touches money, and I’d re-measure whenever I switch models.

FAQ

How many words is one token?

For English text, about three quarters of a word, or roughly 4 characters, per OpenAI’s estimate. For other languages and for code it varies a lot; measure with the tokenizer.

Do spaces and punctuation count as tokens?

Yes. Punctuation is often its own token ('!' was ID 0 above), and a space usually sticks to the start of the next word, so ' world' is one token.

Do all AI models use the same tokenizer?

No. GPT-4 uses cl100k_base, GPT-4o and later OpenAI models use o200k_base, Claude models have their own tokenizer (which changed with Opus 4.7), and open-weight models ship their own. The same text gives different counts on each.

Are images and files counted in tokens too?

Yes, on the APIs I’ve checked. OpenAI says images and files affect the input count, and Anthropic’s count_tokens endpoint returns token counts for image and PDF inputs. The exact cost depends on the model and the input size, so count it rather than guessing.

Sources

What I’m writing next

This is the first post in a series on LLM basics. Next up: what an LLM is, what a context window is, and how to work out LLM cost per task. Tokens come up in all of them.

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.