What is chain-of-thought prompting? CoT explained
Chain-of-thought prompting asks a model to write its reasoning before the answer. Why it works, when it helps, and how reasoning models changed it.

Chain-of-thought (CoT) prompting is asking a language model to write out intermediate reasoning steps before it gives the final answer. Instead of “Q: … A: 9”, the examples in the prompt show “A: they had 23, used 20, so 3 are left, bought 6, 3 + 6 = 9. The answer is 9.”, and the model imitates that pattern. On multi-step problems like math word problems this alone raised accuracy dramatically in large models, and it is the idea that reasoning models now train into the model itself.
Where did chain-of-thought prompting come from?
From a Google paper, Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, first posted January 28, 2022. It defines a chain of thought as “a series of intermediate reasoning steps” and shows that putting a few worked examples with those steps into the prompt changes what the model does. The headline result: prompting PaLM 540B with eight chain-of-thought exemplars took GSM8K, a benchmark of grade-school math word problems, from 17.9% with standard prompting to 56.9%, state of the art at the time. The example in the paper’s first figure is the cafeteria with 23 apples, which is where my opening example comes from.
Four months later, Kojima et al., “Large Language Models are Zero-Shot Reasoners” (May 2022) showed you do not even need examples. Adding “Let’s think step by step” before the answer took text-davinci-002 from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K. That phrase became the most copied line in prompt engineering.
Then Wang et al. (March 2022) added self-consistency: sample several chains of thought and take the most common final answer. It improved on single-chain CoT by 17.9 points on GSM8K and 11.0 on SVAMP.
Why does chain-of-thought prompting work?
The cleanest explanation is about computation per token. A transformer does a fixed amount of work to produce each token. If it must write the answer immediately, all the multi-step arithmetic has to happen inside one forward pass. If it writes the steps first, each step becomes text that the next forward pass can read, through attention, so the model gets many passes and a scratchpad to work on the problem. A chain of thought is the model doing serial computation in its own output.
Two observations from the Wei paper support that and limit it. First, CoT is an emergent property of scale: it “does not positively impact performance for small models”, gains appeared only at around 100B parameters, and smaller models “produced fluent but illogical chains of thought.” Second, the gains were largest on multi-step problems and small on one-step ones. If a problem does not need intermediate state, there is nothing for the scratchpad to hold.
What does chain-of-thought look like in code?
Chain-of-thought is a prompt pattern, so the useful code is the plumbing around it: pull the final answer out of the reasoning, vote across samples, and check the steps. The completions below are illustrative, written by me in the format the Wei paper’s exemplars use (reasoning first, then “The answer is N.”), for the paper’s cafeteria problem. No API is called.
# cot.py: the plumbing around chain-of-thought. Python 3, stdlib only.
# The completions are illustrative (written by hand), in the format Wei et al. used:
# reasoning first, then "The answer is N."
import re
from collections import Counter
QUESTION = ("A shop has 23 apples. It uses 20 for lunch and buys 6 more. "
"How many apples does it have?")
DIRECT = "The answer is 9." # a direct answer has no work to check
SAMPLES = [
"The shop starts with 23. 23 - 20 = 3. Then 3 + 6 = 9. The answer is 9.",
"23 apples minus 20 is 3. Buying 6 more gives 3 + 6 = 9. The answer is 9.",
"23 - 20 = 13. 13 + 6 = 19. The answer is 19.", # arithmetic slip
"It used 20, so 23 - 20 = 3 are left. 3 + 6 = 9. The answer is 9.",
"23 + 6 = 29, minus 20 is 9. The answer is 9.",
]
def final_answer(text):
m = re.search(r"answer is\s+(-?[\d,]+)", text)
return int(m.group(1).replace(",", "")) if m else None
def check_steps(text):
"""Re-run every 'a op b = c' step in the reasoning. CoT makes this possible."""
bad = []
for a, op, b, c in re.findall(r"(\d+)\s*([+\-])\s*(\d+)\s*=\s*(\d+)", text):
real = int(a) + int(b) if op == "+" else int(a) - int(b)
if real != int(c):
bad.append(f"{a} {op} {b} = {c} (should be {real})")
return bad
answers = [final_answer(s) for s in SAMPLES]
print("direct answer:", final_answer(DIRECT), "| steps to check:", check_steps(DIRECT) or "none shown")
print("sampled answers:", answers)
print("self-consistency vote:", Counter(answers).most_common(1)[0])
for i, s in enumerate(SAMPLES):
if check_steps(s):
print(f"sample {i} has a wrong step:", check_steps(s))
words = sum(len(s.split()) for s in SAMPLES)
print(f"output words: direct {len(DIRECT.split())}, five chains {words} ({words / len(DIRECT.split()):.0f}x)")
Output (Python 3.13.11):
direct answer: 9 | steps to check: none shown
sampled answers: [9, 9, 19, 9, 9]
self-consistency vote: (9, 4)
sample 2 has a wrong step: ['23 - 20 = 13 (should be 3)']
output words: direct 4, five chains 86 (22x)
Three practical points come out of it. A fixed answer format (“The answer is N.”) is what makes the output parseable, so put it in your exemplars. Visible steps can be checked by code: the wrong sample is caught at the exact step that went wrong, which a direct answer never allows. And CoT is not free: five chains cost 22 times the output words of a bare answer, and output tokens are the expensive side of the bill.
How do reasoning models change chain-of-thought?
Reasoning models made chain-of-thought a trained behavior instead of a prompt trick. OpenAI o1-preview (September 2024) was trained to think before answering, with the chain of thought hidden and billed as output tokens. DeepSeek R1 (January 2025) showed in the open that reasoning behaviors such as self-reflection and verification can be incentivized through reinforcement learning, the method I cover in what is RLVR. The general idea, spending more inference on harder questions, is test-time compute.
For a builder this changes the advice. With a reasoning model, “think step by step” is redundant, since the model already does it; you control depth with a reasoning effort or thinking budget instead, as I described in what are reasoning models. With a non-reasoning model, explicit CoT prompting still helps on multi-step tasks.
Is a chain of thought a faithful explanation?
Not reliably, and this is the part I would make sure every team understands. The written chain is text the model generates, not a trace of its internal computation. Anthropic tested this in Reasoning models don’t always say what they think (April 3, 2025): they slipped hints about the answer into prompts and checked whether the models admitted using them. Claude 3.7 Sonnet mentioned the hint 25% of the time and DeepSeek R1 39% of the time. For hints framed as unauthorized access, the rates were about 41% and 19%.
So a chain of thought is useful for debugging, for checking steps that can be checked, and for catching obvious errors. It is not an audit log. If a decision must be explainable, the explanation has to come from something you can verify independently, such as the retrieved documents, the tool calls, or the computation itself.
When should you use chain-of-thought prompting?
Use it for multi-step problems: arithmetic, logic, planning, multi-hop questions over documents, and code changes that need a plan. Skip it for lookups, classification, extraction and formatting, where it adds tokens and latency without accuracy. With reasoning models, leave the thinking to the model and set the budget. In agents, a short plan before acting is the CoT pattern applied to tools, and the ReAct pattern is the version that interleaves thoughts with tool calls.
What I would do
For non-reasoning models on multi-step tasks, I would add two or three worked examples in exactly the format I want, ending with a fixed answer line, and parse that line in code rather than reading the prose. Where answers can be computed or tested, I would have code recheck the steps. I would turn on self-consistency only where an eval shows the accuracy gain is worth multiplying the output cost. For reasoning models, I would remove “think step by step” from old prompts, set the effort level per task, and log reasoning token counts. And I would never show a chain of thought to a reviewer as the reason a model made a decision.
FAQ
What is an example of chain-of-thought prompting?
“Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? A: They had 23 apples. They used 20, so 23 - 20 = 3. They bought 6 more, so 3 + 6 = 9. The answer is 9.” The model sees examples like this and writes its own steps for the new question.
What is zero-shot chain-of-thought?
Getting step-by-step reasoning without examples, by adding a phrase such as “Let’s think step by step” before the answer. Kojima et al. introduced it in 2022.
Does chain-of-thought work on small models?
In the original experiments, no: gains appeared only at around 100B parameters, and smaller models wrote fluent but illogical chains. Small models trained on reasoning traces from larger ones can reason step by step, which is a training effect rather than a prompting one.
Do I still need chain-of-thought prompting with reasoning models?
Usually not. Reasoning models already think before answering. Control how much with the API’s reasoning effort or thinking budget instead of prompt wording.
Sources
- Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 28, 2022 (GSM8K numbers from the appendix tables)
- Kojima et al., Large Language Models are Zero-Shot Reasoners, May 24, 2022
- Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, March 21, 2022
- Anthropic, Reasoning models don’t always say what they think, April 3, 2025
- OpenAI, Introducing OpenAI o1-preview, September 12, 2024