Databricks tested every model and harness on its own multi-million-line codebase — here's what's actually worth the money
- Databricks ran the leading AI coding assistants through a test built on its own multi-million-line codebase — the questions were adapted from PRs its engineers had already merged, so nothing is searchable online or sitting in a model's training data
- The most counterintuitive finding: picking a model by its per-token price gets you the wrong answer. Sonnet 5 is 1.7x cheaper per token than Opus 4.8, yet costs more to actually finish a task — $2.09 versus $1.94 — while scoring 6 points lower
- The harness matters more than the model for cost. Same model, same thinking-effort setting — swap the harness and per-task cost can more than double. The difference comes down to how much context gets stuffed back into the model each turn: Opus on claude code burns 742k tokens per task versus just 236k on pi
- An open-source model made the top tier: GLM 5.2 matches Opus 4.8 on quality (87% each) at $1.28 per task versus $1.94
- More thinking effort isn't always better: pushed to the max setting, Opus 4.8 costs $3.60 per task for just 82% — worse and pricier than the high setting's $0.93 for 85%
This is an internal benchmark Databricks published itself, built to answer one question: in a multi-million-line codebase, which AI coding agent — meaning which model-plus-harness combination — actually delivers the best value and the smoothest experience. It went up on July 8; the five authors include CTO and Spark creator Matei Zaharia.
Who is Databricks
A data-and-AI platform company founded in 2013 by the original team behind Apache Spark — one of the most widely used open-source engines for big-data processing, running data pipelines and machine-learning workloads at companies worldwide. Databricks now serves more than 12,000 enterprise customers, closed a Series L round in February 2026 at a $134 billion valuation, and runs at $5.4 billion in annualized revenue.
The authors carry weight. Two of the five named authors are Databricks co-founders: Matei Zaharia, Spark's creator and the company's current CTO, and Patrick Wendell, also a co-founder.
The codebase is genuinely complex. Scala, Go, Rust, Java, Python, TypeScript, Bazel, and Protobuf are all in play, and engineers merge thousands of changes a day. That's a level of complexity no public benchmark can simulate.
The motive is saving its own money. Databricks built this to make AI more cost-effective for its own engineering org — an internal check-up that it happened to publish, methodology and all, not a product review aimed at outsiders.
The method itself is simple enough: take real code its engineers wrote, turn it into test questions, run it against the leading model-and-harness combinations, and see who gets it right and for how much. Several of the conclusions run directly against how most people currently pick their tools.
Six findings follow, with a checklist you can copy at the end.
Models split into three tiers — you don't need the priciest one for everything
Databricks didn't fixate on rankings, since a few points of difference wash out in real-world tasks. What's actually useful is the grouping: every model-and-harness combination falls cleanly into one of three capability tiers.
| Tier | Pass rate | Who's in it |
|---|---|---|
| Tier 1 | 82–90% | Opus 4.8, GLM 5.2, high-end GPT 5.5 configurations |
| Tier 2 | 71–82% | Sonnet 5, Sonnet 4.6, mid-tier GPT 5.5, GPT 5.4 |
| Tier 3 | 51–60% | GPT-5.4-mini, Haiku 4.5 |
Routine work doesn't need the flagship. Flipping a switch or updating a config is well within reach for mid- or lower-tier models, at a fraction of the cost — but Databricks' default had always been the most expensive option.
Division of labor beats blanket upgrades. After seeing the data, Databricks is routing more work to the Haiku and GPT 5.4 Mini tier, saving the smartest models for genuine architecture exploration and deep debugging.
