Sonnet 5.5 another game changer, Benchmarked with Gemini 3.8 Flash and Argon

Sep 2026

ai-tools claude-code benchmark

Benchmark results at a glance

So excited to see the big jump of Sonnet 5.5 in quality in Anthropic's announcement, but how much faster and cheaper is Sonnet 5.5 compared to other top-tier models? Here's my benchmark, modelled on my real project's review work: Sonnet 5.5 came close to Opus 5.5 on task fulfilment, faster and much cheaper, completing 183 of 210 tasks (87%) against Opus's 195 (93%), and a run took a median of 53 seconds at $0.11.

Updated on 2026-10-17 with Google's Gemini 3.8 Flash and Gemini 4 Argon. All five models ran the same new test, so the numbers below replace the first version's.

Notes:

• Task fulfilment is the share of check tasks completed over twenty-one scored runs each.

• Opus took 148 seconds and $0.37, higher task fulfilment, slower with higher cost than Sonnet.

• Surprisingly, the top tier Fable 5.1 did no better than Opus, at more than twice its cost

• Both Gemini models have a low price per token, but at high thinking they wrote far more tokens per run and often ran out of room, so a Gemini run took longer and completed fewer tasks than a Claude run.

I was running long build and review sessions on Opus 5.5 and thinking about moving them to Sonnet 5.5. The work is review-heavy: reading merged code, checking it against a design document, and finding where it breaks. So the question was simple. Does the cheaper, faster model give up quality on that kind of work, and does the top-tier model earn its price? When Google released Gemini 3.8 Flash and Gemini 4 Argon, I added both to the same test.

I measured it two ways: the sessions' own logs, and a blind test on code written for the test, where the right answers were known. Here is the short version, with the best model first.

• Speed. Sonnet 5.5 was the fastest by far, at a median of 53 seconds a run against about two and a half minutes for Opus 5.5 and Fable 5.1. Both Gemini models were the slowest, at five to six minutes, because at their high thinking setting they wrote several times more tokens.
• Performance. All three Claude models found nearly every bug in the answer key, and Opus 5.5 and Fable 5.1 found the most real bugs beyond it. Sonnet 5.5 never marked a bug as proven when it was not; the others did four to seven times. Both Gemini models often ran out of room or returned nothing, so they completed fewer tasks.
• Cost. Sonnet 5.5 was the cheapest per task completed, at about a third of Opus 5.5 and a seventh of Fable 5.1. The Gemini prices are promotional and double after November (Argon) and December (Flash).

Three runs per model per case is a small sample, so read cost and speed as strong evidence and quality as weak evidence. The real-session numbers from the first version cover the Claude models only, so they sit in the test procedure below.

The test procedure

The sessions' logs give speed and cost but not quality, and they mix jobs. So I built a controlled test.

The real sessions. These numbers are from the first version of this post and cover the three Claude models only. The logs gave 391 Sonnet 5.5 calls, 2,963 Opus 5.5 calls and 734 Fable 5.1 calls, with a median of 5.4 s, 6.4 s and 12.2 s, at $0.14, $0.18 and $0.66 a call. Fable ran at a higher effort setting there and the jobs differed, so this describes, not compares. The timestamps include the wait for the first token, so I cross-checked with a direct test: the same prompt three times per model on a wall clock. In the logs a Sonnet call took a median 16% less time than an Opus call and Sonnet wrote 28% more tokens a second. The direct test gave 21% less wall-clock time (14.5 s against 18.4 s) and 25% more tokens a second (96 against 77). The two methods agree on direction and roughly on size.

The team's reviewing session also checked Sonnet 5.5's early planning and review work and found every claim it checked held up. Later the same builder ran a full build on Sonnet 5.5: 116 tests passing on a live database, 51 of 51 deliberate faults caught, and a push. Its reports held up under checking, apart from one claim that had no log behind it, which was caught before it went into the merge request. One build is a small sample.

The blind test. The first version used two pieces of that project's code. I couldn't send that code to a second vendor, so for this version every model ran on new code written for the test, at two levels of difficulty. Four cases are Python services of about 600 lines each, with 370 to 460 lines of tests: a job queue with leases, webhook delivery from an outbox, a usage-metering ledger and a feature-flag evaluator. All five models found almost every bug in those, so I added three harder cases of about 1,200 to 1,450 lines in nine files each, with at least five bugs per case that need reasoning across files: a consent gate and a transactional publish loop, which follow the topics of the first version's test and were written from this post's own description of them, and an order-fulfilment workflow. Each case has a design note, a few short rulings, seeded defects each confirmed by a test that proves it, and decoy code that looks wrong but is correct. Every model got the same brief, files, design note and rulings, in the same prompt word for word, read-only: no running code, no search, no shell. Claude ran at effort high and Gemini at thinking level high, each through its own vendor's API, with the same 64,000-token output cap and a 30-minute limit. I ran each model three times per case, 21 runs in all.

The outputs got random labels and lost any model name, and two scorers from two vendors, Claude Opus 5.5 and Gemini 3.8 Flash, scored every reported defect against the source without knowing which model wrote it. Where they disagreed, a third reviewer ran the model's own failing input against the code and ruled. Only then did I open the sealed list mapping labels to models. A task is one bug to find: the 174 in the answer key over all 21 runs, plus 36 real bugs the key missed, counted once per run. Three of those bugs were found by a reviewer reading the code after the first round, and nine in the models' own answers in the second round, so that second set leans toward what these five models find. Each was reproduced by the third reviewer and then checked for in every answer to its case. I count a key bug and a bug beyond the key the same, and a run that returned nothing as finding nothing. The cases and keys were written by Claude Opus, so the key may lean toward bugs a Claude model finds. The third reviewer was Sonnet 5.5, one of the models under test, and Sonnet's own count of wrong CONFIRMED labels rests on its rulings: over both rounds the Opus scorer found none, the Flash scorer four.

What the test found

Each explanation below is my inference from the runs, not something I proved. The full results table, with every metric, is right after this section, and the per-round and per-case tables are in the appendix.

Opus and Fable found the most. Opus 5.5 completed 195 of 210 tasks, Fable 5.1 191 and Sonnet 5.5 183. Opus over Fable is within the run-to-run spread, so I would not rank those two. All three found nearly every bug in the answer key; the gap is in the real bugs beyond it, where Opus found 21 of 36, Fable 17 and Sonnet 10. My reading: with the rules in front of the model, the key bugs are within reach of all three, and extra capability shows up as extra bugs found beyond the key.

Sonnet had zero wrong CONFIRMED labels. In 21 runs, Sonnet 5.5 never marked a bug as proven when it was not. Opus 5.5 and Gemini 3.8 Flash did it 4 times, Fable 5.1 5 times and Gemini 4 Argon 7 times. A wrong CONFIRMED label is a bug the model says it proved, with an input that does not actually fail. Because a reader acts on a CONFIRMED label, this is the most costly mistake in a review.

Sonnet was clearly faster and cheaper. It finished in about a third of Opus's time, at about a third of the cost per task completed.

Both Gemini models wrote far more. At their high thinking setting, Gemini 3.8 Flash and Gemini 4 Argon wrote several times more tokens a run than the Claude models. So the cheapest model per token was not the cheapest per task, and both Gemini models were the slowest. When the promotional prices end, their cost per task roughly doubles.

Gemini runs often did not finish. On the harder cases Gemini 3.8 Flash spent most of its 64,000 output tokens thinking and was cut off mid-answer in 7 runs, and 1 more call returned nothing in 30 minutes, so 13 of its 21 runs ended normally. Gemini 4 Argon ended 17 normally: it was cut off once and 3 calls returned nothing. On the runs that ended normally, both found 99% of the bugs in the answer key. The cap was the same for every model and set before the run, so a cut answer is scored as it was. If you run a fleet on any model, set timeouts and keep the retry idempotent.

Fable did no better than Opus, and cost the most. At about twice Opus's price per run it completed 191 of 210 tasks against 195, with one more wrong CONFIRMED label. I do not know why. A model tuned for long, hard problems may not gain on a small, well-specified one. This is a result about this job, not about Fable in general.

The results in full

Gemini's median times count every run, including those that waited 30 minutes, and its cost per task counts those runs at $0, since they sent nothing back.

Side note: how I set the difficulty of the harder cases. I tuned them on four older models outside the test, then froze them before any of the five saw them. Each older model ran once per harder case, on the same cases, prompt and settings, scored by the same two blind scorers:

The new Claude models found 99% to 100% of the key bugs on the same cases. The older Gemini models found more than the new ones across all runs, in three runs each, because the new ones thought much longer and hit the output cap or stalled.

What Anthropic reports about Sonnet 5.5 (vendor-reported, not tested by me)

Everything here comes from Anthropic's announcement of Sonnet 5.5 on 2026-09-28, at https://www.anthropic.com/claude-sonnet-5-5. I did not test any of it against Sonnet 5, so my data can neither confirm nor contradict these figures.

• Speed. "Sonnet 5.5 generates outputs 30%+ faster than Sonnet 5, making it our fastest Sonnet model to date."
• Cost. "Sonnet 5.5 is priced the same as Sonnet 5 at $2 per million input tokens, $10 per million output tokens, and $0.20 per million tokens for cache reads, but it typically needs far fewer tokens to do the same work. In our testing, it costs up to 30% less per task than its predecessor." The price per token did not change; the saving comes from fewer tokens per task. Anthropic's "task" means a whole job, not my check task, which is one bug detection.
• Agentic coding. "Sonnet 5.5 scores 70.6% on Terminal-Bench 4.0, an agentic coding evaluation, compared to Sonnet 5's 10.3%." That is roughly a jump from 10% to 71% on that benchmark.
• Vision and long tasks. "And it's strong on long-horizon work and image understanding—it's the first Sonnet model to beat Pokémon Red working only from screenshots."

My own data, without testing Sonnet 5: Sonnet 5.5 was the fastest and cheapest of the five per task on the same review job and wrote about half the tokens of Opus 5.5. That fits the direction of the fewer-tokens claim, but it is a comparison with Opus, not with Sonnet 5.

What Google publishes about Gemini 3.8 Flash and Gemini 4 Argon (not tested by me)

I took these from Google's model and price pages and did not test them.

• Availability. Gemini 4 Argon is listed as a stable model on the Gemini API models page.
• Price. Output prices include thinking tokens, which is why the token counts above matter for cost. Per million tokens:

What to take from it

If you choose models for engineering work:

• Test on your own work, with answers you can trust, such as an independent review or a post-mortem you already have.
• Score blind, count what the key missed, and count confident wrong answers on their own line.
• Report runs, not only averages, and treat timeouts as results.
• Use cost per task completed: under two cents on Sonnet and about twelve cents on Fable here.
• Do not assume the top tier wins every job. On a bounded, well-specified job it did not lead.
• A model with a low price per token that thinks longer can cost more per task, and it needs an output cap with room for its answer.

Appendix: per round, per case, and what each metric means

First round: four smaller cases (medium difficulty)

Runs of one model were close: Sonnet's twelve runs took 31 to 56 seconds and Opus's 66 to 152. Three real defects outside the key, in two of the cases, were found by a reviewer reading the code; every answer to those two cases was then checked for them, and Opus 5.5 reported each of them in every run.

Second round: three harder cases (higher difficulty)

The Claude models still found almost everything. Sonnet 5.5 missed one confirmed task in 90, and Opus 5.5 and Fable 5.1 none.

Caps and stalls drove the gap. At its high thinking setting, Gemini 3.8 Flash spent about 60,000 of its 64,000 output tokens thinking and was cut off mid-answer in 7 of 9 runs. On the 2 runs that ended normally, it found 19 of 20. Gemini 4 Argon was cut off once, and 3 of its 9 calls returned nothing in 30 minutes. On the 5 runs that ended normally, it found 49 of 50. The cap was the same for every model and set before the run, so a cut answer is scored as it was. Most of the gap in the table comes from the cap and the stalls.

Opus and Fable found the most beyond the key. Nine real defects outside the answer key were found in the answers themselves and then checked in every answer. Opus 5.5 and Fable 5.1 reported 12 each of the 27 possible, Gemini 4 Argon 8, Sonnet 5.5 5 and Gemini 3.8 Flash 3. On finding bugs, this is where the Claude models pulled apart.

Argon's median time counts all 9 runs, including the 3 that waited 30 minutes, and its cost per task counts those 3 at $0, since they sent nothing back.

Per case

In the first round, confirmed key tasks completed per case, over three runs:

In the second round, confirmed key tasks completed per case, of 30:

Across all 105 runs, every Claude run but one found every confirmed task; the exception was one Sonnet 5.5 run on the second round's consent gate, which missed one. Gemini 3.8 Flash's seven missed tasks on the first test are its one run that returned nothing. On the harder cases, Flash finished one run of nine with every task found, and Argon four.

I applied one counting rule throughout: a defect named only in a run's "what I could not check" section does not count. It is a stated limit, not a claim. For the real defects outside the key, the rule was added after a first count; it lowered four counts and put no model ahead of another.

What each metric means

• Run, task and task fulfilment. A run is one code review of a piece of code. A check task, or task for short, is one bug detection: one verified defect the model is asked to catch in it. Task fulfilment is the share of check tasks completed. The 210 tasks are the 174 bugs in the answer key over all runs, plus the 36 real bugs beyond the key.
• Real bugs found beyond the key. Real bugs the key missed: three found by a reviewer reading the code after the first round, nine found in the models' own answers in the second round. Each was reproduced by the third reviewer, then checked in every answer to its case by both scorers. Counted once per run.
• Plausible key tasks completed. The key's weaker items, kept apart.
• Other new bugs. Reported bugs not in the key that the scorers verified from the source but that were not checked in every answer, so they are not in the 210.
• Unsupported claims. Claims the scorers could not support, or found wrong: the precision side.
• Wrong CONFIRMED labels. A task marked as completed (CONFIRMED) whose failing input does not fail. The costliest error in a run, so it is counted apart.
• Decoys wrongly flagged. Correct code, placed to look wrong, reported as a defect. Two decoys turned out to hide a real defect, one in each round; reports of those were counted as real findings, not as false flags.
• Runs cut at the output cap, and runs with no answer. Runs that used all 64,000 output tokens, thinking included, before finishing the answer; and calls that returned nothing within my 30-minute limit.
• Median time per run. Wall-clock time to a finished answer, as the median over a model's runs.
• Median cost per run. Tokens times list price, as the median over a model's runs; cloud contracts may differ. A call that sent nothing back is counted at $0.
• Cost per task completed. Total cost divided by tasks completed.

Sources

• Anthropic, Claude Sonnet 5.5 announcement, https://www.anthropic.com/claude-sonnet-5-5 (fetched 2026-09-29).
• Anthropic, Claude Sonnet 5.5 model page, https://platform.claude.com/docs/en/models/sonnet-5-5/overview (fetched 2026-09-29).
• Anthropic, Models overview, https://platform.claude.com/docs/en/about-claude/models/overview (fetched 2026-10-15).
• Google, Gemini API models page, https://ai.google.dev/gemini-api/docs/models (fetched 2026-10-15).
• Google, Gemini 4 Argon model page, https://ai.google.dev/gemini-api/docs/models/gemini-4-argon (fetched 2026-10-15).
• Google, Gemini API price list, in the Gemini API documentation at ai.google.dev (fetched 2026-10-16).

Test environment

• Claude Sonnet 5.5, Opus 5.5 and Fable 5.1 through Anthropic's Messages API; Gemini 3.8 Flash and Gemini 4 Argon through Vertex AI on Google Cloud. Each model returned the same id it was asked for.
• Claude at effort high, Gemini at thinking level high, a 64,000-token output cap, streaming, no system prompt, and the same prompt bytes for every model.
• Gemini ran the first round twice. The first pass, without streaming, had three runs that did not end normally, so the scored Gemini answers come from a second pass with streaming, with the same prompts and settings.
• The first round's model calls took 3 h 25 m, plus 2 h 31 m for Gemini's second pass, and 1 h 45 m to score. The second round took 5 h 6 m to run and 2 h 26 m to score.
• Costs are list-price estimates. The tools were Python 3.14 and Claude Code 2.1.284, which ran the sessions but made none of the model calls.