LangChain: Tune the harness, not the model — Nemotron 3 Ultra scores near Opus 4.8 at ~1/10 the cost, so enterprises need not buy closed APIs at premium prices
- As a Nemotron Coalition member, LangChain kept Nemotron 3 Ultra weights fixed inside Deep Agents and only changed the harness (system prompt, tool descriptions, middleware around model and tool calls).
- Method: an eval-driven loop — run evals → read failing traces → cluster behavior patterns → change one harness piece → re-eval; candidates first pass a cheap representative screen, then the full suite.
- On prompts: short, single-purpose instruction blocks, not broad rewrites. On middleware: hard enforcement (loop caps, retries) plus putting the right signal where it will actually be read.
- Key finding: the same “keep reading after a full-page file read” rule failed in the tool description and worked when injected into the tool result; Nemotron also obeys mid-conversation injected messages more reliably than standing system-prompt rules.
- Typical score ~0.80→0.84, best 0.86 near Opus 4.8’s best 0.87; full suite ~$4.48 vs Opus ~$43.48; median latency ~10s per test for both.
What they actually did
LangChain published a Nemotron 3 Ultra harness-tuning playbook: leave the model alone, change only the scaffolding around it, and see how high an open model can go on agent work.
In one line: put an open model that already handles tools and multi-step work into a harness tuned for it, match Claude Opus 4.8’s best run quality, and cut full-suite cost by about an order of magnitude.
Why it matters: Same weights, different scaffolding, different scores. They previously took gpt-5.2-codex on Terminal-Bench 2.0 from 52.8 to 66.5 (about Top 30 to Top 5) without touching the model. This time they apply the same idea to open Nemotron 3 Ultra and report a near-Opus cost/quality comparison.
Here, harness means the system prompt, tool descriptions, and middleware around model and tool calls. Generation settings stayed at vendor defaults — temperature, top-p, thinking budget unchanged — so the gains are not from sampling knobs.
