TIL: the OpenAI CLI counts tokens offline
OpenAI's official CLI has a tokenizer command that counts and inspects tokens locally, with no API key or network. Commands and real output.
The OpenAI CLI (openai, the official command-line tool for the OpenAI REST API) has a tokenizer command that counts tokens locally, with no API key and no network. Its help text says it “Tokenizes exact UTF-8 text without credentials, network access, or saved input.”
Run on Linux x86_64 on October 24, 2026, with v1.40.6 (released October 23). Outputs are pasted as printed.
Install from the release tarball and check the hash first:
V=1.40.6
curl -sLO https://github.com/openai/openai-cli/releases/download/v$V/openai_${V}_linux_amd64.tar.gz
curl -sLO https://github.com/openai/openai-cli/releases/download/v$V/openai_${V}_checksums.txt
grep linux_amd64.tar.gz openai_${V}_checksums.txt | sha256sum -c -
tar xzf openai_${V}_linux_amd64.tar.gz
./openai --version
openai_1.40.6_linux_amd64.tar.gz: OK
openai version 1.40.6
It ships four encodings:
./openai tokenizer encodings
o200k_base (default)
cl100k_base
r50k_base (legacy)
p50k_base (legacy)
Take the Gujarati sentence from What are tokens in AI?, where tiktoken and the GPT-2 tokenizer gave 8, 37 and 58 tokens. The CLI gives the same three numbers:
for e in o200k_base cl100k_base r50k_base; do
./openai tokenizer count --text "રેલવે સ્ટેશન ક્યાં છે?" --encoding $e
done
Encoding: o200k_base
Input bytes: 58
Tokens: 8
Encoding: cl100k_base
Input bytes: 58
Tokens: 37
Encoding: r50k_base
Input bytes: 58
Tokens: 58
To check the “no network” claim, run it inside a network namespace with nothing but a downed loopback interface. curl can’t resolve a host in there; the tokenizer still works:
unshare -rn curl -sS -m 5 -o /dev/null https://api.github.com
unshare -rn ./openai tokenizer count --text "રેલવે સ્ટેશન ક્યાં છે?"
curl: (6) Could not resolve host: api.github.com
Encoding: o200k_base
Input bytes: 58
Tokens: 8
inspect shows each token ID with its byte range, and --format json gives a script something to parse:
./openai tokenizer inspect --text "Hello world!"
./openai tokenizer count --text "Hello world! Tokens are how models read." --format json
Encoding: o200k_base
Input bytes: 12
Tokens: 3
0 ID 13225 [0, 5)
"Hello"
hex: 48656c6c6f
1 ID 2375 [5, 11)
" world"
hex: 20776f726c64
2 ID 0 [11, 12)
"!"
hex: 21
{"encoding":"o200k_base","input_bytes":40,"token_count":9}
Same IDs (13225, 2375, 0) and the same 9-token count as the tiktoken snippet in the tokens post.
One limit, in the CLI’s own words: “Plain-text counts are not complete request counts, billed usage, prices, or context-limit guarantees.” Tool definitions and message structure add tokens a plain-text count can’t see. For the number you’ll be billed, the help points to openai responses input-tokens count, which calls the API and needs a key. That one wasn’t run for this note.
Also handy if you run it in scripts: set OPENAI_CLI_NO_UPDATE_CHECK=1 to turn off update checks.