Databricks tested every model and harness on its own multi-million-line codebase — here's what's actually worth the money
- Databricks ran the leading AI coding assistants through a test built on its own multi-million-line codebase — the questions were adapted from PRs its engineers had already merged, so nothing is searchable online or sitting in a model's training data
- The most counterintuitive finding: picking a model by its per-token price gets you the wrong answer. Sonnet 5 is 1.7x cheaper per token than Opus 4.8, yet costs more to actually finish a task — $2.09 versus $1.94 — while scoring 6 points lower
- The harness matters more than the model for cost. Same model, same thinking-effort setting — swap the harness and per-task cost can more than double. The difference comes down to how much context gets stuffed back into the model each turn: Opus on claude code burns 742k tokens per task versus just 236k on pi
- An open-source model made the top tier: GLM 5.2 matches Opus 4.8 on quality (87% each) at $1.28 per task versus $1.94
- More thinking effort isn't always better: pushed to the max setting, Opus 4.8 costs $3.60 per task for just 82% — worse and pricier than the high setting's $0.93 for 85%
This is an internal benchmark Databricks published itself, built to answer one question: in a multi-million-line codebase, which AI coding agent — meaning which model-plus-harness combination — actually delivers the best value and the smoothest experience. It went up on July 8; the five authors include CTO and Spark creator Matei Zaharia.
Who is Databricks
A data-and-AI platform company founded in 2013 by the original team behind Apache Spark — one of the most widely used open-source engines for big-data processing, running data pipelines and machine-learning workloads at companies worldwide. Databricks now serves more than 12,000 enterprise customers, closed a Series L round in February 2026 at a $134 billion valuation, and runs at $5.4 billion in annualized revenue.
The authors carry weight. Two of the five named authors are Databricks co-founders: Matei Zaharia, Spark's creator and the company's current CTO, and Patrick Wendell, also a co-founder.
The codebase is genuinely complex. Scala, Go, Rust, Java, Python, TypeScript, Bazel, and Protobuf are all in play, and engineers merge thousands of changes a day. That's a level of complexity no public benchmark can simulate.
The motive is saving its own money. Databricks built this to make AI more cost-effective for its own engineering org — an internal check-up that it happened to publish, methodology and all, not a product review aimed at outsiders.
The method itself is simple enough: take real code its engineers wrote, turn it into test questions, run it against the leading model-and-harness combinations, and see who gets it right and for how much. Several of the conclusions run directly against how most people currently pick their tools.
Six findings follow, with a checklist you can copy at the end.
Models split into three tiers — you don't need the priciest one for everything
Databricks didn't fixate on rankings, since a few points of difference wash out in real-world tasks. What's actually useful is the grouping: every model-and-harness combination falls cleanly into one of three capability tiers.
| Tier | Pass rate | Who's in it |
|---|---|---|
| Tier 1 | 82–90% | Opus 4.8, GLM 5.2, high-end GPT 5.5 configurations |
| Tier 2 | 71–82% | Sonnet 5, Sonnet 4.6, mid-tier GPT 5.5, GPT 5.4 |
| Tier 3 | 51–60% | GPT-5.4-mini, Haiku 4.5 |
Routine work doesn't need the flagship. Flipping a switch or updating a config is well within reach for mid- or lower-tier models, at a fraction of the cost — but Databricks' default had always been the most expensive option.
Division of labor beats blanket upgrades. After seeing the data, Databricks is routing more work to the Haiku and GPT 5.4 Mini tier, saving the smartest models for genuine architecture exploration and deep debugging.
Which tasks to downgrade? The usage logs did the math
Before making any changes, Databricks pulled every AI coding session log to see what engineers were actually asking the AI to do.
About a quarter of the work is low-complexity, six in ten sessions are medium difficulty, and only 12.1% genuinely call for deep design exploration. Yet the default had always been the most expensive model available — the waste was sitting right there in the data.
Open-source models have arrived — good enough to be your daily driver
One open-source model made it into Tier 1: Zhipu AI's GLM 5.2.
Quality ties Opus 4.8. Both land at an 87% pass rate — statistically indistinguishable.
About 30% cheaper per task. GLM 5.2 runs $1.28 a task versus $1.94 for Opus 4.8.
Already headed for daily-driver status. Internal developers had already been trial-running it for everyday work, and these scores line up with what they reported — enough for Databricks to start rolling it out more broadly.
Cheap per token doesn't mean cheap overall
Almost everyone picks a model by looking at the same number: dollars per million tokens. That number can't predict what you'll actually pay.
What matters is cost per completed task. Between that and per-token price sits one variable: how much the model actually has to read and how long it has to think to get the job done.
Sonnet 5 is the counterintuitive case. Its per-token price runs about 1.7x cheaper than Opus 4.8's, yet it costs $2.09 per task versus Opus 4.8's $1.94. The cheaper sticker price ends up with the bigger bill.
The extra cost goes toward taking the scenic route. Sonnet 5 reads more files, thinks longer, and goes back and forth more, burning through 1.9x the tokens — while scoring 6 points lower (81% versus 87%).
The harness swings costs more than you'd think
Same model, same thinking-effort setting — swap only the harness, and per-task cost can more than double while quality barely moves. Most people never even treat the harness as something worth choosing.
Same chef, same dish, different kitchens — the plating time and cost can differ by double. The kitchen is the harness.
Cost differs by double, quality barely moves. pi came out cheaper in every single comparison, by margins ranging from 1.20x to 2.08x; five of the six pairings differed by 3 points or less on pass rate.
The difference comes down entirely to context management. The cumulative content fed back into the model per completed task: 742k tokens for Opus 4.8 on claude code, versus just 236k on pi.
| Model · thinking effort | pi cheaper by | Pass rate (pi / native) |
|---|---|---|
| Opus 4.8 · high | 2.08× | 85% / 87%（−2） |
| Opus 4.8 · xhigh | 1.46× | 90% / 88%（+2） |
| Opus 4.8 · max | 1.20× | 82% / 89%（−7） |
| GPT-5.5 · medium | 1.54× | 83% / 80%（+3） |
| GPT-5.5 · high | 1.22× | 81% / 83%（−2） |
| GPT-5.5 · xhigh | 1.44× | 78% / 80%（−1） |
pi: a coding assistant with only four tools in its toolbox
pi belongs to the same category as Claude Code and Codex: install it on your own machine, run it in a terminal, tell it what you want in one sentence, and it goes off reading code, editing files, and running tests before reporting back.
The entire toolbox has four tools. Read a file, write a file, edit a file, run a command. Anything else loads on demand only when actually needed.
Its permanent system prompt runs under 1,000 tokens. Roughly a few hundred words.
Who built it. Armin Ronacher and Mario Zechner. Ronacher is well known in Python circles as the creator of Flask and Jinja2, two frameworks that have been in daily use for over a decade. The project is free and open-source, sits around 54,000 stars on GitHub, and plugs into any vendor's model.
Being thin is the whole reason it saves money. Every single turn, a harness has to re-stuff a bundle of material back into the model: the permanent system prompt, instructions for every tool, everything said so far, and whatever code needs reviewing this round. More tools, a longer system prompt, more retained history — all of it inflates that bundle, and you pay for the whole thing again every single turn.
The information that isn't force-fed doesn't disappear — the model just calls "read file" to fetch it when it actually needs it. The whole approach only works if the model is willing to go looking, and is good enough to find what it needs.
Maxing out the thinking-effort setting: pricier and worse
Cranking a model's thinking-effort setting to the max doesn't guarantee better accuracy. It does guarantee a bigger bill.
| Combination | Cost per task | Pass rate |
|---|---|---|
| Opus 4.8 · pi · high | ~$0.93 | 85% |
| Opus 4.8 · pi · xhigh | ~$2.20 | 90% |
| Opus 4.8 · pi · max | ~$3.60 | 82% |
| GPT 5.5 · codex · med | ~$1.33 | 79.5% |
| GPT 5.5 · codex · xhigh | ~$2.42 | 79.5% |
Effort level and score aren't a straight line. Opus 4.8 on pi scores 85% for under a dollar at the high setting, then drops to 82% at max for $3.60. The sweet spot for this pairing sits in between, at xhigh — 90%.
Sometimes the extra spend buys you nothing. GPT 5.5 on codex scores an identical 79.5% at both the medium and xhigh settings — and xhigh costs almost twice as much.
Effort-level names don't translate across models. High, xhigh, and max are each vendor's own labels — don't assume the same name means the same thing across providers.
Why not just use an established benchmark like SWE-Bench
Whether any of this holds up depends on how the test questions were built. A public benchmark couldn't answer the question Databricks was actually asking.
The answers leak. Both the questions and solutions are public, and given enough time they end up in training data — at which point the test stops measuring ability.
It doesn't match the real environment. Databricks' codebase spans more than a dozen languages — Scala, Go, Rust, Java, Python, Bazel, Protobuf — plus a sprawl of microservices. A generic benchmark can't reflect that kind of complexity.
So Databricks turned the code its own engineers merge every day into test questions. A PR that's already cleared review is good raw material on its own: it comes with a developer's step-by-step commit history, a human review, and tests that verify whether the change actually works.
Filter first, then adapt
| Filter | Why |
|---|---|
| Recent | Using recent history keeps the questions grounded in current frameworks, conventions, and style |
| Human-written | Bot commits, service accounts, and AI- or auto-generated changes are all excluded |
| Backed by solid tests | No tests means no way to grade the answer |
| Focused scope | Touches only a handful of modules, not a sprawling rewrite across half the codebase |
| Full-stack coverage | Scala backends, Rust systems code, React/TypeScript frontends, protobuf/gRPC interfaces, and Bazel configs all need representation |
The solution gets stripped out when the prompt is rewritten. A PR description usually explains why a bug should be fixed a certain way — leaving that in is basically writing the answer into the question. What's left is only the desired outcome and the constraints.
The tests get pulled out and hidden separately. Databricks' build system can already work out which tests depend on the changed files, so that whole set becomes the grading standard.
Every single question gets a human pass. Scripts and AI are only used to generate candidates. Some original tests need rewriting — ones that only allow a single implementation get loosened, ones that are too lax get tightened — and all of that rewriting is done by hand, not by AI.
No AI graders, because that rewards sounding right
Grading works like this: the moment the model declares itself done, the code gets snapshotted, the hidden tests get restored, and the suite runs — passing all of them means the task passes. No LLM ever acts as judge anywhere in this process. Databricks' reasoning:
We did not use an LLM judge to evaluate correctness, since we've found that this rewards sounding right over being right. Databricks
Whether tests pass is binary — there's no room for interpretation. That's exactly why the "must have solid tests" filter can't be skipped: the credibility of the grading standard rests entirely on that test suite.
How Databricks guarded against cheating
In early runs, a few models scored suspiciously well. Pulling the full traces and reading them by hand traced the problem to the environment: every question comes from an already-merged commit, and the complete git history was sitting right there in the working directory. Any agent that can run shell commands can simply scroll back through that history and dig up the original "reference answer."
The fix is blunt: during every run, the working copy gets completely severed from the repository so the git history simply isn't reachable.
Every coding agent's harness and model ran with its standard out-of-the-box configuration, paired with the everyday tools Databricks engineers normally have access to — nothing was specially tuned for the test.
Takeaways
None of this requires Databricks' scale to pull off. Any team with a stack of merged PRs already has, sitting right there, a benchmark no model has ever trained on — and the grading standard is just the tests it already wrote.
Mix and match — don't bet on one model. The frontier right now spans OpenAI, Anthropic, and open-source models; no single vendor owns every tier.
Route tasks by difficulty. Six in ten jobs are medium difficulty — send the simple ones to cheap models and save the flagship for complex design work. Databricks' next step is automating that routing.
Measure the harness layer first. Picking the right harness can sometimes save more than switching models — and it's a layer most teams have never even benchmarked.
Cost by completed task, not by token. Per-token price can't predict the total bill, and more thinking effort isn't automatically better.
Databricks plans to keep adding questions, especially harder ones, and will run every new agent or harness through this test going forward.
Databricks tested it on its own multi-million-line codebase: picking a model by token price gets it wrong
Databricks built its test questions from engineers' already-merged code, then ran 7 models, 3 harnesses, and 4 thinking-effort settings through it — the whole thing, with charts, in one page.
↓ One page, one animated chart
Public benchmarks like SWE-Bench have their questions and answers sitting online, and given enough time they leak into training data — at which point the test stops measuring ability. Databricks took a different approach: turning code its own engineers merged just last week into test questions. All figures were collected and graded by Databricks itself; no third party has reproduced them yet.
Same questions, same grading standard: run 7 models across 3 harnesses and 4 thinking-effort settings, and the scores settle into three tiers. One open-source model made Tier 1 — Zhipu's GLM 5.2 scores 87%, tying Opus 4.8, at $1.28 per task versus $1.94.
Almost everyone shopping for a model looks at dollars per million tokens. Between that number and what you actually pay sits one variable: how much the model has to read and how long it has to think to finish the job.
A harness is the software layer that turns a model's output into actual actions: it decides how much code the model sees each turn, which tools it can call, and how it retries on failure. Claude Code, Codex, and pi are all harnesses, and any of them runs with any vendor's model underneath. Same model, same thinking-effort setting — swap only the harness, and per-task cost can more than double.
Same Opus 4.8, same test set — the cumulative content fed back into the model per completed task differs by 3.2x. The information that isn't force-fed doesn't vanish; the model just calls "read file" when it needs it. pi is an open-source project from Armin Ronacher (creator of Flask and Jinja2) and Mario Zechner, sitting around 54,000 stars on GitHub — background the source post doesn't cover. The winner here is a third-party open-source project; Databricks' own product, Omnigent, is built to let you swap models and harnesses on the fly.
cheaper
per-token price
supposed to
be cheap!
why's the cost
double?
↓
236k
fed per task, claude
code vs. pi
a dial for how long
the model thinks first
- × Every number is self-collected and self-graded by Databricks — no third party has reproduced it
- × "Swapping harnesses doesn't change quality" holds for five of six pairings — the max setting differs by 7 points
- × Effort labels like high / xhigh / max are defined per vendor — not comparable across models
per-token price column.
Members can copy or download the full Markdown, and let their AI subscribe and ask.
Found this useful? Leave your email — every new explainer, the moment it's out. Unsubscribe anytime.