Gemini 1.5 Pro: the million-token context and the needle test
Google announced Gemini 1.5 on 15 February 2024 with a 128k default and a 1 million token preview. What the needle test measures, and what it leaves out.

On February 15, 2024, Sundar Pichai and Demis Hassabis announced Gemini 1.5. The model in early testing is 1.5 Pro, mid-size and multimodal. The standard context is 128,000 tokens. Limited developers and enterprise customers can try up to 1 million tokens in AI Studio and Vertex AI, as a private preview, at no cost during testing, with longer latency expected. In research they had tested up to 10 million tokens.
The statistic next to that window is a needle in a haystack. The model found the embedded text 99% of the time in blocks as long as 1 million tokens. A needle test measures retrieval of a sentence someone planted. It does not measure whether the model can reconcile two contradictory sections of a codebase. Google’s demos are the capability claim. The needle definition in their post is the limit of the 99%. Pricing tiers are “coming soon”, from 128k up to 1 million, so there is no rate to multiply. The script below prints their figures. It does not estimate tokens.
What is Gemini 1.5 Pro?
1.5 Pro achieves comparable quality to 1.0 Ultra using less compute, through mixture-of-experts: selective expert pathways. “Less compute” is their quality claim. The price tiers are still unpublished.
On the benchmarks they use for developing their LLMs, 1.5 Pro outperforms 1.0 Pro on 87% of that panel and is broadly similar to 1.0 Ultra on the same panel. The 87% is a share of those development benchmarks.
They say 1.0 started at 32,000 tokens. The standard 1.5 Pro window is 128,000. The private preview is 1 million. The research test is 10 million. The length to plan a product on, from this post, is the standard 128,000. The 1 million figure has an access list, a private-preview label, no cost while testing, and longer latency. The 10 million figure is a research length.
Where the tokens go is a separate question from how long the window is. The 1 million token preview is the one Google puts in AI Studio and Vertex AI. No cost during testing is the preview’s bill during testing. Whether a repository, a recording, or a transcript may leave the network is a separate review.
How long is the context in Google’s own units?
For the 1 million token window they give equivalences: 1 hour of video, 11 hours of audio, codebases with over 30,000 lines, and over 700,000 words. The script copies those strings, the context lengths, and the needle line. It prints their figures. It does not estimate tokens.
# context_equivalents.py — figures from Google's 15 February 2024 post. Not a tokenizer.
rows = [
("standard context", "128,000 tokens"),
("private preview", "1,000,000 tokens"),
("research test", "10,000,000 tokens"),
("1M as words, their figure", "over 700,000 words"),
("1M as code, their figure", "over 30,000 lines"),
("1M as audio, their figure", "11 hours"),
("1M as video, their figure", "1 hour"),
("needle test at 1M", "found 99% of the time"),
]
for name, value in rows:
print(f"{name:<28} {value}")
Output (Python 3):
standard context 128,000 tokens
private preview 1,000,000 tokens
research test 10,000,000 tokens
1M as words, their figure over 700,000 words
1M as code, their figure over 30,000 lines
1M as audio, their figure 11 hours
1M as video, their figure 1 hour
needle test at 1M found 99% of the time
“Over 700,000 words”, “over 30,000 lines”, “11 hours”, and “1 hour” belong to the 1 million token window. The standard context is 128,000 tokens. The research length is 10 million tokens, with those equivalences left on the 1 million row.
The note on the chart is the limit on the statistic: the 99% is retrieval of a planted sentence. “Over 700,000 words” is their figure for the 1 million window. The script prints it and does not turn words, lines, or hours into a token estimate.
What does the needle test measure?
The result, in their words, is that the model found the embedded text 99% of the time in blocks as long as 1 million tokens. Embedded text is the passage placed in the block for the test to find. My reading of that result is narrow, and it is mine. A needle test measures retrieval of a sentence someone planted. It does not measure whether the model can reconcile two contradictory sections of a codebase.
A long repository often holds two sections that disagree: a comment and the code under it, or two passages that give different rules. Finding a planted sentence shows the sentence was still available at that length. Deciding which passage governs is the test I would add. The announcement’s 99% remains the retrieval result.
What did the demos and the Kalamang test show?
The capability claim I would cite is Google’s demo list. They describe 402-page Apollo 11 transcripts, a 44-minute silent Buster Keaton film, and a prompt with more than 100,000 lines of code. The film is silent, so the input they describe is the picture across 44 minutes, next to their equivalence of 1 hour of video for the 1 million window. The code prompt is more than 100,000 lines, a different figure from “codebases with over 30,000 lines”, the general equivalence for that same window. I would quote the demo and the equivalence as two separate lines. The Ultra comparison and the 87% belong to their development panel.
The other long-context result is in-context learning: Machine Translation from One Book. The language is Kalamang, with fewer than 200 speakers. Given a grammar manual, translation came out “at a similar level to a person learning from the same content.” The manual sits inside the request, which is why the window is what makes the setup possible. I would keep this beside the needle result as its own measurement: a translation from the manual, at the level of a person learning from the same content. The 11 hours of audio stay an equivalence they state for the 1 million window. The film they describe is the silent picture.
Where do Llama 2 and GPT-4 sit next to these windows?
The earlier windows are what make 128,000 and 1 million readable. Llama 2, from July 2023, ships the 7B, 13B, and 70B models at 4k, under the Llama 2 license. GPT-4, from the March 14, 2023 launch, is an 8,192-token model with limited access to 32,768 tokens, described then as about 50 pages, and with long-context quality on that longer model still being improved. At that launch the 8,192-token model was $30 per million input tokens and $60 per million output tokens, and the 32,768-token model was $60 and $120 per million. Those dollars are the March rates in that note. Gemini’s tiers, from 128k up to 1 million, are the ones marked coming soon.
Google says 1.0 started at 32,000 tokens. GPT-4’s long model at launch was 32,768. 1.5 Pro’s standard window is 128,000, and the private preview is 1 million. Llama 2 remains the local-weights option at 4k when the prompt cannot be sent to AI Studio, Vertex AI, or a paid API. A longer hosted window is what you can send when sending is allowed. The 10 million research length stays off a delivery plan. The 99% is retrieval of embedded text. Two contradictory sections of a codebase are the test I would add before trusting a repository to the preview.
What I would do
I would plan shipping features against the standard 128,000-token context. I would treat 1 million tokens as a private preview for limited developers and enterprise customers, on AI Studio and Vertex AI, at no cost during testing, with longer latency expected. I would treat 10 million as a research length they reported.
I would use the 99% for the measurement they published: embedded text, found 99% of the time, in blocks as long as 1 million tokens. On a codebase I would add the test the needle skips, two sections that disagree and a question about which one governs, and I would score that run myself.
I would cite the demos as theirs: the 402-page Apollo 11 transcripts, the 44-minute silent Buster Keaton film, and the prompt with more than 100,000 lines of code. I would cite Kalamang as in-context translation from a grammar manual, fewer than 200 speakers, “at a similar level to a person learning from the same content.” I would keep the 1 million token equivalences labeled as theirs: over 700,000 words, over 30,000 lines, 11 hours of audio, 1 hour of video. The script prints them and does not estimate tokens.
I would leave the unit price blank until the tiers they said are coming soon, from 128k up to 1 million, are published. No cost during testing is the preview’s bill during testing. If the corpus has to stay on hardware you control, I would use Llama 2’s 4k weights, under that license. GPT-4’s windows are a send as well.
Update, 29 October 2026
Two later notes pick up the measurements this announcement states as headlines. What a context window is is where I treat a long window as a limit you can put a price on. What are tokens is where a word count and a token count come apart, which is why the script above only reprints Google’s February 2024 figures.