Anthropic computer use: screenshots, a mouse, and prompt injection
On 22 October 2024 Anthropic put computer use in public beta. Claude scored 14.9% on OSWorld screenshots. Build around the prompt-injection warning.

On 22 October 2024 Anthropic put computer use in public beta. An upgraded Claude 3.5 Sonnet is available to all users that day. A new Claude 3.5 Haiku is announced for later in the month, so this note stays with the upgraded Sonnet and the beta. Computer use is in public beta on the Anthropic API, on Amazon Bedrock, and on Google Cloud’s Vertex AI. Anthropic calls it the first frontier model to offer computer use in public beta, and calls the feature experimental, sometimes cumbersome and error-prone.
The capability number they print is 14.9% on OSWorld screenshots. The model looks at a screen, moves a cursor, clicks, and types, and the next screenshot comes back as input. That channel is wider than a JSON tool with a schema. The failure mode they name is prompt injection, and the rest of this note is that failure mode plus the defensive posture in the posts.
What did Anthropic release on 22 October 2024?
The upgraded Claude 3.5 Sonnet is the model available to all users that day. Haiku’s release is later. Computer use is the new capability, in public beta, on the API, Amazon Bedrock, and Vertex AI.
The June model and this October model share a family name and are different releases. In the June Sonnet post Anthropic’s internal agentic coding evaluation had Claude 3.5 Sonnet at 64% and Claude 3 Opus at 38%, in a harness where the model writes, edits, and executes code from a natural-language description. That 64% is their harness, from 21 June 2024. The 22 October figure on OSWorld is 14.9% for screenshot-only tasks. One is a coding eval. The other is a desktop of screenshots.
There is no new token price in the 22 October launch post or in the research post from the same day. This note has no bill. The diagram is doing the work a price script would have done, and the only script below prints the three OSWorld figures.
What does the computer-use loop actually do?
Computer use, in their description, is: look at a screen, move a cursor, click, and type. The API translates an instruction into commands. The example they give is specific enough to design around: use data from the computer and the web to fill a form.
The loop matches the diagram.
Claude receives a screenshot. It counts pixels in that image to decide where the cursor should go. The research post the same day says training the model to count pixels mattered. The model then returns a mouse action or a keyboard action. The desktop changes. The next turn begins with a new screenshot.
That next screenshot is the part I would underline. It can contain text the user did not write. A page loaded to fill the form, a document already open, a dialog the last click summoned: any of those can sit in the bitmap. The research post names the Trust and Safety consequence. Malicious instructions in content the model sees can override the directions it was given, or cause unintended actions. A computer on the internet can put that content into a screenshot. Public beta users should take precautions. Anthropic points at a reference implementation as the place those precautions are shown.
Scrolling, dragging, and zooming are hard, on their account. Together with “experimental, sometimes cumbersome and error-prone,” that is why they say to start with low-risk tasks.
o1-preview, from 12 September 2024, shipped an API without function calling, streaming, or system messages, and spent the extra compute on a hidden chain of thought. Computer use spends the turn on a mouse and a keyboard, then reads the next screenshot. The October observation channel is the whole screen.
What does 14.9% on OSWorld measure?
The launch post gives three numbers. This script prints those three, from constants. There is nothing to price, which is the point of keeping the script this small.
# osworld.py — scores from Anthropic's 22 October 2024 launch post.
print("screenshot-only, Claude 3.5 Sonnet 14.9%")
print("screenshot-only, next-best cited 7.8%")
print("Claude, more steps allowed 22.0%")
Output (Python 3):
screenshot-only, Claude 3.5 Sonnet 14.9%
screenshot-only, next-best cited 7.8%
Claude, more steps allowed 22.0%
Screenshot-only, Claude 3.5 Sonnet scores 14.9%. The next-best AI system they cite scores 7.8% on that setup. When more steps are allowed, Claude reaches 22.0%. I would use 14.9% as their capability claim for screenshot-only OSWorld, and I would judge a form-filling task by running that task. Their advice is still to start with low-risk tasks. The 22.0% line is the same model with a longer step budget.
The June 64% stays off this row. Different month, different release, different harness: a bug fix or a feature, described in natural language, with tools that write and run code.
Why is the screen a wider boundary than a JSON tool?
On 13 June 2023 OpenAI shipped function calling with an injection caveat. A proof-of-concept exploit showed that untrusted data, coming back from a tool’s output, can instruct the model to perform unintended actions. The guidance was to consume information only from trusted tools, and to confirm with the user before real-world actions such as sending email, posting, or making a purchase. The model never ran the function. Your process did. The untrusted text arrived as a string you chose to pass back into the conversation.
Computer use keeps the idea that untrusted text can redirect the model, and it uses a different channel. I would keep the mechanisms distinct. In the June shape, the observation is a tool result inside a schema you defined, and you can decide which fields to forward. In the October shape, the observation is the whole screen. The model counts pixels, returns a click or a keystroke, and the next bitmap is whatever the desktop now shows. A page on the internet is enough, on Anthropic’s account, to place instructions where the model will see them. A screenshot has no list of required fields.
A JSON tool is a function you named, with arguments you can validate, the way the June note checks hours and rate_inr before it multiplies. A desktop is every window the session can reach. The screenshot is the new boundary. The user’s prompt is no longer the only data in the call. The beta’s example, filling a form with data from the computer and the web, goes looking for whatever else is on that screen.
Anthropic tells public beta users to take precautions and points them at the reference implementation. They say to start with low-risk tasks. The June confirmation rule still belongs here, with the channels kept distinct: confirm with the user before email, posting, or a purchase, because those actions leave the machine and the model can click the control that sends them. The failure mode stays the one they named. Content the model sees can carry instructions. Those instructions can override the task or cause an action the user did not ask for. A desktop on the internet can deliver that content inside a screenshot.
Their classifiers, and the extra monitoring they describe around election-related activity, are controls they run. The activities they list are generating and posting to social media, registering domains, and government websites. Sending a request does not move that monitoring into the caller’s process. They assess the updated model, including computer use, as AI Safety Level 2. I would cite ASL-2 as their published assessment, and I would still watch the first tasks. The same posts call the beta error-prone.
What I would do
I would ship a schema-backed tool for any action I can name in advance, and I would keep computer use for the residue: a graphical interface, a low-risk task, a desktop whose contents I would be willing to show. I would read the reference implementation before I wrote my own loop, because that is where they pointed beta users. I would confirm with the person in the loop before any email, post, or purchase, carrying the June 2023 caveat across and keeping the channels labeled as different.
I would put 14.9%, 7.8%, and 22.0% in the design doc with “experimental, sometimes cumbersome and error-prone,” and I would treat scrolling, dragging, and zooming as known hard cases. I would measure the task on a screen that does not contain data I cannot explain sending. Pixel counting is how the cursor gets its coordinates, which is why a layout change can move the click. The October sources add no token price. The question I would ask instead is which channel the observation uses, and what a wrong click can reach. The classifiers and the election-related monitoring stay Anthropic’s. The caller owns the desktop and the confirmation step.
Update, 29 October 2026
The failure mode in the research post is the one I wrote up later for browser agents in AI browser prompt injection: a model that reads a page has a hard time keeping the page’s words distinct from the user’s. Computer use is the October 2024 version of that problem, with the page arriving as pixels. What is MCP? is the later note on the narrower channel, a host calling a tool through a schema instead of handing the model a screen. The June 2024 coding score and this beta stay separate. 14.9% on screenshot-only OSWorld remains the number from launch day, and the prompt-injection warning remains the line to build around.