Malhar Ujawane

Inspiration

Inspiration·

Anthropic: analysts sided with their agent's coin flip 86% of the time

Anthropic gave 50 analysts' Claude Code agents a hidden coin-flip lean. Analysts matched it 86% of the time and 15% credited the agent.

Anthropic: analysts sided with their agent's coin flip 86% of the time

How AI agents shape experts’ judgment calls is a new study from Anthropic’s Alignment team (October 27, 2026). They gave 50 professional analysts a realistic causal-inference task in Claude Code, running Claude Opus 4.8 at High effort. The task had four judgment calls with two reasonable answers each, like whether to measure farm productivity per hectare or per unit of operating cost. On every call, a coin flip decided which option each analyst’s agent would lean toward, through a system prompt the analysts never saw.

Across the four analytical choices, analysts went with their agent’s lean 86% of the time. If the agent’s input made no difference, we’d expect about 50%. When analysts had already expressed their own preference before the agent weighed in, they switched away from it 60% of the time if the agent disagreed, compared to 6% if it agreed.

Bar chart of the share of analysts’ choices that matched their agent’s hidden lean: outcome measure 92%, comparison method 82%, which farms 81%, weighting 90%, all four combined 86%. A dashed line marks 50%, the rate expected if the agent had no effect.

The number that stuck with me is a different one. Afterward, 77% of analysts explained their choices using the agent’s arguments, sometimes word for word, and only 15% credited the agent. Experience, education and a causal-inference quiz score didn’t predict who deferred.

The caveats are real and Anthropic lists them: the lean was deliberately strengthened, it’s one task, one field and one model, and the 60% vs 6% switching figure rests on 38 cases. Without an assigned lean, Opus 4.8 made the same five choices in 112 of 123 runs, so real agents do have defaults of their own. The study also found that the 11 people who skipped the agent made more serious errors (6 of 11 vs 3 of 50), though that group was small and less experienced.

Why this matters to me: when I hand a client an analysis or an architecture call, it goes out under my name. This study says the person signing may not be the one who decided. I wrote about agents quietly anchoring on their own earlier choices in Context inertia. This is the same drift, one level up, in the human.

Two habits I’m taking from Anthropic’s own suggestions. Write down my call on each open decision before the agent weighs in. And ask the agent for the strongest case for the option it didn’t pick, since the paper notes the agents “rarely made a strong case for why an expert might prefer the other option.”

This lands a few days after Apollo’s note on AI payoff outside tech. If sign-off stops meaning independent judgment, “a human reviewed it” stops being much of a selling point too.

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.