A curated LLM eval package with a benchmark

Model, effort, cache: which one moves your agent bill? My LLM benchmark tells

Oct 2026

ai-tools benchmark claude gemini

In production at scale, the biggest challenge is cutting LLM cost while keeping the quality. Anthropic's article Reducing cost and improving performance with Claude Platform shows how to optimize it: keep the prompt cache hitting, clear out old prompt habits that burn tokens, and set effort to fit the task. Each of those fixes needs an eval to prove it on your own work. This package is that eval.

What it solves

Every new model launch says faster, cheaper, smarter. Your agents run your tasks at your quality bar, so measure there. I had my AI agent team build one curated eval and run Claude and Gemini through it: same claims, same tools, same pass bar.

It answers three questions on your own work:

• Which model clears your bar at the lowest cost?
• Which effort level is worth paying for?
• What does the prompt cache save on each task?

One run answers all three.

What you get

⚡ One eval, any model. Claude or Gemini, on the same tasks.
⚡ Your bar, your answer. The lowest-cost setting that still clears your quality bar.
⚡ Every dial measured. Model, effort and cache, from the API's own usage record.
⚡ Prove a fix before you ship it. Anthropic's cost guide says to measure on your own workload. This package does it.
⚡ Open and rerunnable. Plain Python, with the plans written before the runs.

Model

Model is the first dial most teams reach for. Anthropic's docs say to compare models on cost per task, because you pay for completed tasks, and that's how this benchmark reports it.

• Sonnet 5.5: as fast or faster at every effort level, with every claim passed in both trials.
• Gemini 3.8 Flash: the lowest-cost model that passed, at about half of Sonnet's cost per task at medium, taking 2.5 times as long.
• Opus 5.5: every claim passed too, at about twice Sonnet's cost.

Effort

Anthropic's docs say to sweep effort on your own workload, because what effort buys depends on the task. Here is that sweep.

Effort is nearly free on Claude: high costs 5% to 7% more than low. On Gemini 3.8 Flash it's the quality dial. Flash passed every claim from medium thinking up, and high costs 50% more than low and takes 2.4 times as long.

Cache

Anthropic's docs put it plainly: turn on prompt caching before any other lever. The article's cache section lists the habits that keep it hitting.

The automatic prompt cache cut Claude's cost per task about in half. On a long shared prompt, a cached read cost about 8 to 9 times less.

• Every Claude model, about half, with the automatic cache. At high effort, Opus 5.5 went from $0.197 to $0.097 per task and Sonnet 5.5 from $0.092 to $0.049.
• A 148,000-token prompt on Sonnet 5.5, with an explicit cache breakpoint: a cached read cost 9.2 times less than no cache. The first request writes the cache at 25% more, in line with the 1.25x cache-write price in Anthropic's docs.
• Gemini 3.8 Flash caches follow-up turns on its own. On a 171,000-token prompt, a follow-up turn read about 98% of it from the cache and cost 8 times less on 8 of 10 tries.
• In practice: on Claude, a new question reuses the cache when an explicit breakpoint ends the shared prompt. On Gemini, follow-up turns reuse it.

💸 Cache alert: turn on the prompt cache and Claude's cost per task drops by about half.

How I tested

The eval set starts from Anthropic's cost-optimization cookbook (MIT licence): ten fictional insurance claims with known right answers and twelve tools. The agent calls tools until it approves, denies, routes to a supervisor or refers the claim to the special investigations unit. A claim passes with the right decision within six turns, and a trial passes at 10 of 10. Every setting ran twice. Cost per task is the cost of one claim.

• Claude: Opus 5.5 and Sonnet 5.5 on Anthropic's API at low, medium and high effort, with the cache on and off.
• Gemini: Gemini 3.8 Flash on Google's Vertex AI at low, medium and high thinking.
• Cost comes from each response's usage record.
• Each paid run had a written plan in the repo first, and an audit of the code led to fixes, all public.
• Sources: Anthropic's article Reducing cost and improving performance with Claude Platform and its docs page Optimizing for cost and intelligence, both read on 2026-10-24.

Rerun it

Everything is open: the eval set, the tool, the plans and the raw records, in ai-benchmark-v2. Add your model, your cases and your bar. My earlier post compared the same model families on a code-review task: Sonnet 5.5 benchmarked with Gemini 3.8 Flash and Argon.

Before you switch models, turn on the cache.

Disclaimers

• Results are from my benchmark on one task: ten fictional insurance claims from Anthropic's cookbook.
• Claude and Gemini ran on different APIs at different times, so compare the two families loosely.
• Gemini 3.8 Flash at low thinking scored 7 and 8 of 10, under the pass bar.
• Costs are at list price. For Gemini, "effort" in the tables means its thinking level.
• The Claude cache figures come from the automatic cache inside the agent loop.
• Gemini's cache is on by default; only a project administrator can switch it off.
• The figures are directional, as Anthropic says of its own. Models and prices change, so measure on your own workload.