Malhar Ujawane

Blog

Post·

Why are AI coding agents slow? Where the time goes

An agent task's time is generation, prefill, tool runs and per-call overhead. A runnable model shows what 1000 tokens per second fixes and what it doesn't.

Why are AI coding agents slow? Where the time goes

A coding agent is slow because one task is many model calls in a row, and each call pays for four things: generating the output tokens, processing the input (prefill), running the tool the model asked for, and the network round trip. A faster model shrinks only the first of those, so past a certain speed the rest of the loop decides how long you wait.

What shipped this week?

On February 12, 2026, OpenAI released GPT-5.3-Codex-Spark as a research preview for ChatGPT Pro users in the Codex app, CLI and VS Code extension. It’s “a smaller version of GPT-5.3-Codex”, text-only with a 128k context window, served on Cerebras’ Wafer Scale Engine 3 at “more than 1000 tokens per second”. It’s the first result of the OpenAI and Cerebras partnership announced on January 14, 2026, which adds 750MW of what OpenAI calls “ultra low-latency AI compute”.

Five days earlier, on February 7, Anthropic made a “2.5x-faster version of Claude Opus 4.6” available in Claude Code as fast mode (/fast), at a higher cost per token. Two labs shipped speed as its own product tier within five days.

The most useful line in OpenAI’s post is a footnote under its benchmark chart. Task duration is estimated as the sum of “(1) output generation time (output tokens ÷ sampling speed), (2) prefill time (prefill tokens ÷ prefill speed), (3) total tool execution time, and (4) total network overhead.” That’s the whole model of agent latency in one sentence, so here it is in code.

Why does agent speed matter to a builder?

It decides how you work with the agent. If a task takes ten minutes, you write a careful spec, hand it off and review the result later. If it takes ten seconds, you steer it like an editor: try, look, redirect. OpenAI pitches Codex-Spark for exactly that loop, “making targeted edits, reshaping logic, or refining interfaces and seeing results immediately.”

It also has a price. Anthropic’s fast mode costs more per token than standard Opus 4.6, and Codex-Spark is a smaller model than GPT-5.3-Codex. Speed gets bought with money or with capability, so it’s worth knowing which part of your wait it actually removes. I wrote about pricing a whole task, rather than a single call, in LLM cost per task.

How do you estimate how long an agent task takes?

Add up OpenAI’s four terms over every step. This script makes a tiny repo, measures real tool calls on it (search, read, edit, a 40-test pytest run, git diff), and then times one made-up nine-step task at different model speeds. Everything marked “assumed” is an input you should replace with your own numbers.

# agent_time.py: where the wall-clock time of one coding-agent task goes.
# Python 3.13, pytest==8.4.2, ripgrep 14.1.1, git 2.47. Linux.
# Tool times are MEASURED on this machine. Model speeds, token counts and per-call overhead are
# INPUTS you choose (marked "assumed"); they are not measurements of any specific model.
import os, subprocess, sys, tempfile, time, statistics, textwrap

# 1. A tiny repo for the tools to work on: one module, 40 tests, one commit.
repo = tempfile.mkdtemp()
def write(name, text):
    with open(os.path.join(repo, name), "w") as f:
        f.write(textwrap.dedent(text))
write("calc.py", "def add(a, b):\n    return a + b\n\ndef mul(a, b):\n    return a * b\n")
write("test_calc.py", "from calc import add, mul\n" + "".join(
    f"def test_add_{i}():\n    assert add({i}, {i}) == {2 * i}\n\ndef test_mul_{i}():\n    assert mul({i}, 2) == {2 * i}\n\n"
    for i in range(20)))
git = lambda *a: subprocess.run(["git", "-C", repo, *a], capture_output=True, check=True)
git("init", "-q"); git("add", "."); git("-c", "user.name=x", "-c", "user.email=x@x", "commit", "-qm", "init")

def edit():                                       # what an "apply patch" tool does: rewrite a file
    with open(os.path.join(repo, "calc.py"), "a") as f:
        f.write("\ndef sub(a, b):\n    return a - b\n")

TOOLS = {
    "search":    lambda: subprocess.run(["rg", "-n", "def ", repo], capture_output=True),
    "read":      lambda: open(os.path.join(repo, "calc.py")).read(),
    "edit":      edit,
    "run tests": lambda: subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider"],
                                        cwd=repo, capture_output=True),
    "git diff":  lambda: git("diff", "--stat"),
}

def measure(fn, n=7):
    runs = []
    for _ in range(n):
        t = time.perf_counter(); fn(); runs.append(time.perf_counter() - t)
    return statistics.median(runs)

tool_s = {name: measure(fn) for name, fn in TOOLS.items()}
print("measured tool time (median of 7):")
for name, s in tool_s.items():
    print(f"  {name:10} {s * 1000:8.2f} ms")

# 2. One made-up task: find the code, read it, edit, test, fix, test, show the diff, answer.
#    (tool, output tokens the model writes, tokens the tool result adds to the context)  [assumed]
STEPS = [("search", 120, 300), ("read", 80, 800), ("read", 80, 800), ("edit", 600, 100),
         ("run tests", 60, 300), ("edit", 300, 100), ("run tests", 60, 300), ("git diff", 60, 200),
         (None, 250, 0)]
FIRST_PROMPT = 12_000      # system prompt + tool definitions + task, tokens   [assumed]
PREFILL_TPS = 5_000        # input tokens per second the server processes      [assumed]

def task_time(out_tps, overhead_s, run_tests=True):
    gen = prefill = tools = over = 0.0
    new_in = FIRST_PROMPT                       # with a cached prefix, only new tokens are prefilled
    for tool, out_tok, result_tok in STEPS:
        if tool == "run tests" and not run_tests:
            continue
        prefill += new_in / PREFILL_TPS
        gen += out_tok / out_tps
        over += overhead_s
        if tool:
            tools += tool_s[tool]
        new_in = out_tok + result_tok
    return gen, prefill, tools, over

print("\nseconds per task        generate  prefill    tools  overhead    total  generate share")
for label, tps, ovh, tests in [("50 tok/s", 50, 0.5, True), ("100 tok/s", 100, 0.5, True),
                               ("1000 tok/s", 1000, 0.5, True),
                               ("1000 tok/s, -80% ovh", 1000, 0.1, True),
                               ("  ... and no tests", 1000, 0.1, False)]:
    g, p, t, o = task_time(tps, ovh, tests)
    total = g + p + t + o
    print(f"{label:22} {g:8.2f} {p:8.2f} {t:8.2f} {o:9.2f} {total:8.2f}  {g / total:6.0%}")
print("\noverhead per model call [assumed]: 0.5 s, or 0.1 s after an 80% cut")

The output:

measured tool time (median of 7):
  search         3.68 ms
  read           0.01 ms
  edit           0.01 ms
  run tests    162.15 ms
  git diff       1.46 ms

seconds per task        generate  prefill    tools  overhead    total  generate share
50 tok/s                  32.20     3.25     0.33      4.50    40.28     80%
100 tok/s                 16.10     3.25     0.33      4.50    24.18     67%
1000 tok/s                 1.61     3.25     0.33      4.50     9.69     17%
1000 tok/s, -80% ovh       1.61     3.25     0.33      0.90     6.09     26%
  ... and no tests         1.49     3.11     0.01      0.70     5.30     28%

overhead per model call [assumed]: 0.5 s, or 0.1 s after an 80% cut

At 50 tokens per second, generation is 80% of the task. Make generation 20 times faster and the task gets about 4 times faster, from 40.3 to 9.7 seconds, because prefill and the per-call overhead didn’t move. At 1000 tokens per second, generating output is only 17% of the wait.

Stacked horizontal bars of seconds per task, with generation in orange and prefill, tools and overhead in shades of black and grey. 50 tokens per second: 40.3 s, mostly orange. 100 tokens per second: 24.2 s. 1000 tokens per second: 9.7 s, with orange now a thin sliver. With per-call overhead cut 80%: 6.1 s. With test runs skipped as well: 5.3 s. A note says generation got 20x faster while the task got about 4x faster.

Why does each part take the time it does?

Generation is sequential. A model writes one token per forward pass, and each pass has to read the weights it uses from memory, so output speed is mostly limited by memory bandwidth. I go through the arithmetic in LLM latency explained. OpenAI’s partnership post describes Cerebras’ approach as “putting massive compute, memory, and bandwidth together on a single giant chip”, which goes straight at that limit.

Prefill is parallel but grows with context. The model processes all input tokens of a call at once, which is much faster per token than generating, but an agent’s context keeps growing with every tool result. Prompt caching lets the server skip the prefix it has already seen, which is why the script only prefills the new tokens after step one. Without a cache, every step would pay for the whole history again.

Tool time is your machine. In this toy repo the tools are nearly free, and the test run is the only one that registers, at 162 ms. A real test suite, a TypeScript build or an install can take minutes, and no model speed touches that.

Overhead is per call. Each step is a full request: connect, send the context, wait for the first token, stream the rest back. An agent makes dozens of these, so a fixed cost per call adds up fast.

What did OpenAI change besides the model?

The harness. OpenAI writes that as it trained Codex-Spark, “it became apparent that model speed was just part of the equation for real-time collaboration.” It added a persistent WebSocket connection and changes inside the Responses API that “reduced overhead per client/server roundtrip by 80%, per-token overhead by 30%, and time-to-first-token by 50%”, and says the WebSocket path will become the default for all models. In my toy numbers, the 80% cut alone takes the task from 9.7 to 6.1 seconds, a bigger saving than anything left on the generation side.

There’s a second, quieter change. Codex-Spark “makes minimal, targeted edits and doesn’t automatically run tests unless you ask it to.” That cuts tool time by skipping verification. In my toy that saves very little, because 40 tests run in a sixth of a second; in a real project with a slow suite, it could be most of the saving. It’s also the step that catches broken edits.

Where does this model break?

My numbers are assumptions. The 0.5 seconds of overhead, the 5,000 tokens per second of prefill and the token counts per step are inputs I picked to make the arithmetic visible. Only the tool times are measured, on a toy repo. Measure your own loop before you trust any split.

Smaller and faster can mean more steps. Codex-Spark is a smaller model than GPT-5.3-Codex. If a faster model needs more attempts to land a fix, the extra steps eat into the speedup, and the only way to know is to run your own tasks several times and compare. My notes on that are in how to evaluate AI agents.

Agents don’t always run one step at a time. Parallel tool calls and subagents overlap work, so the sum becomes the length of the longest chain rather than the total.

What would I do?

Log a timestamp at the start and end of each part of every step for a week of real tasks: request sent, first token, last token, tool start, tool end. The biggest term tells you where to spend. If it’s generation, a fast tier like Spark or fast mode is the right purchase. If it’s tests or builds, speed those up first, for example by running only the tests near the change; a faster model won’t help.

My opinion on the industry: this week shows the labs splitting coding agents into two modes. OpenAI says so directly, describing “two complementary modes” for Codex: long-running autonomous work and real-time collaboration. For interactive work I’d expect the harness, the round trips and the test loop to get as much engineering attention as the model, because past about a thousand tokens per second, that’s where the seconds are. The coding agent that feels fastest will be the one with the shortest whole loop.

FAQ

Is 1000 tokens per second fast?

For generating code, yes: in the script above, 1,610 output tokens take 1.6 seconds at that speed. The catch is that the rest of the loop doesn’t speed up with it.

What is prefill?

Prefill is the model processing your input (system prompt, history, tool results) before it writes the first output token. It runs in parallel across the input, so it’s faster per token than generation, but it grows with context and shows up as time to first token.

Does a faster model make an agent cheaper?

Not by itself. You pay per token, and fast tiers can cost more per token, as Anthropic’s fast mode does. A faster model saves your time; whether it saves money depends on the price and on how many steps it needs.

Does prompt caching make agents faster?

Yes, for prefill. With a cached prefix, each step only processes the new tokens, which keeps time to first token from growing with the whole history. It does nothing for generation, tools or network overhead.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.