LangChain: Tuning the Harness, Not the Model — Nemotron 3 Ultra Closes In on Opus 4.8 at About 1/10 the Cost, Proving Enterprises Don't Need to Pay Top Dollar for Closed APIs
- LangChain froze the Nemotron 3 Ultra weights and only changed the harness (system prompt, tool descriptions, middleware) — no model changes, no touching temperature or other generation parameters.
- The method is an eval-driven loop: evaluate → read failure traces → cluster behaviors → change one part of the harness → re-evaluate; screen cheap on a small sample first, then run the full suite.
- Prompts use short, single-purpose instruction blocks; middleware handles enforcement plus injecting signals right at the failure point.
- Key mechanism: the same rule written in a tool description does nothing, but stuffed into the tool's return result it works; injecting mid-conversation is more reliable than a standing system-prompt rule.
- Typical run ~0.80→0.84, best run 0.86 versus Opus 4.8's best of 0.87; the full suite costs about $4.48 vs $43.48, with median latency around 10 seconds per question on both sides.
What They Did
LangChain published a harness-tuning playbook for Nemotron 3 Ultra: leave the model untouched, change only the scaffolding around it, and see how high an open model can be lifted on agent tasks.
In one line: same open model, a better-fitting shell — the best run closes in on Claude Opus 4.8, at roughly an order of magnitude lower cost for the full eval suite.
Why this matters: same weights, different scaffolding, and the score gap can be sizable. LangChain previously took gpt-5.2-codex from 52.8 to 66.5 on Terminal-Bench 2.0 without touching the model. This time they apply the same idea to an open model, with a cost comparison against Opus.
Generation parameters stayed at the vendor's recommended defaults. The gains in this piece didn't come from cranking temperature and hoping for the best.
When the Shell Doesn't Fit, Where Does the Horsepower Go
Open models can already carry real agent workloads at a fraction of the cost of frontier APIs. But even a capable model, dropped into a harness that wasn't built for it, will misfire.
The model is the engine; the harness is the chassis and gearing. However good the engine, if the gearing and transmission don't match, horsepower gets wasted spinning and slipping. Tuning the harness means adjusting the gearing and transmission — not swapping the engine.
They picked Nemotron 3 Ultra because NVIDIA's post-training already made it stable across a variety of agent harnesses, not just tuned for single-turn chat. The no-profile baseline sits around 0.80 — the floor is already decent, which is what makes the shell worth building on.
Tuning Runs on a Loop, Not Guesswork
Without eval signal, you're just guessing. An agent's input space is large and there are many acceptable outputs — fixing A can break B. So the workflow is a closed loop, with LangSmith traces as the source of truth.
What gets clustered is a failure pattern (e.g., "answers halfway through reading a file"), not the wording of one specific question. Changes that boost a score by gaming one question's phrasing get thrown out.
Candidates first pass a representative small sample; only then do they run on the full suite. The screening sample has to cover different behavior types, or tasks the screen never touched can regress across the board.
Prompts: Don't Rewrite the Whole Page — Write One Small Block Against a Failure
The system prompt is the cheapest lever, and the easiest to overrate. Sweeping rewrites and "be a better agent" instructions often get diluted, because a capable model's real failure points are usually not about wording.
When a task finishes but doesn't report the concrete result, add only this block — nothing else:
<final_answer_completeness> After tool calls succeed, the final answer must report the concrete result, not just that the task is done. Include the key entity, action, identifier, title, recipient, service, status, or value that answers the user's request. If the user asked multiple questions, answer each one from its matching tool output. </final_answer_completeness>
Same Sentence, Different Placement, Opposite Outcome
This is the single most important mechanism in the whole piece. They wanted one rule: when a file read returns exactly a full page, assume there's more and keep reading.
Write that sentence into the read_file tool description, and the model stops after page one. Move it, word for word, next to the tool's return result, and the model starts reading on.
What the model is actually processing is "the chunk of content right in front of it." The farther a rule sits from that chunk, the more it's as if it were never written; only sitting right next to that chunk does it actually get read.
What Middleware Actually Does: Two Kinds of Hooks
Some behaviors have to happen every single time — prompting alone isn't enough. Middleware wraps around model calls and tool calls:
Mid-Conversation Injection: More Reliable Than the System Prompt
Nemotron listens better to "a message that shows up exactly when needed" than to a standing rule baked into the system prompt. So middleware injects at two points, timed to that rhythm:
What Survives, What Gets Cut
Most of what gets tried doesn't survive. A tuning loop that only reports its wins is usually already overfitting.
How the Numbers Move After Tuning
A single overall score only tells you better or worse. They scored by category, matching what agents actually do: files, tools, retrieval, multi-turn conversation, long-context summarization.
With no profile, there was no compression guidance: content that should be compressed wasn't, or the model answered off the summary alone without reading the offloaded history file. Adding explicit guidance pushed it nearly to the max.
Many tool failures come down to how tools are described, how retries work, how to close things out — not just the model "not knowing how to use tools."
Already strong out of the box (a post-training artifact); a small shell tweak still squeezes out a bit more.
Problems around maintaining long-horizon backend state barely budge. They didn't twist the profile to game the score here.
Where Tuning the Shell Stops Working
Harness tuning has a ceiling. Knowing the boundary matters as much as knowing the payoff.
A second line gets drawn inside the harness itself: what belongs in the "core shared by every agent," and what's just "one model's profile."
The question worth asking again and again on any project: can this change honestly be pushed a little closer to "core"?
When a failure doesn't respond to anything you change around the model, that's the signal it lives in the weights, and the fix is post-training, not another hook. LangChain · Tuning the harness, not the model
Getting an Open AI Model Close to the Top Tier: Skip the Stronger Model — Just Get the "Shell" Around It to Fit
LangChain touched only the shell, never the model, and tuned the open Nemotron 3 Ultra until its best run closed in on Claude Opus 4.8 — this one illustrated page explains how, and how much cheaper it gets.
↓ One page, done · includes one animated figure
An AI agent — one that can call tools on its own and work through a task step by step, not just chat — and when people want it more capable, the first instinct is always a stronger model. This playbook takes a different road: wrapped around the model is a "shell" (the harness) — its opening instructions, each tool's manual, the processing logic that runs at every step in between. That shell is where this round of tuning actually happens.
✘ But drop it into a shell that wasn't designed for it, and it misfires: asking the same thing over again, quitting before the job's done, spinning in circles
Why: the instructions and tool descriptions in the shell don't match this model's temperament, so its effort goes into fighting the scaffolding instead of the task
The idea is simple: don't change a single byte of the model's weights — only turn three screws on the shell: the opening instructions, the tool manuals, the processing logic in the middle. Tune it piece by piece through a closed loop: run the eval (scoring against a fixed set of questions) → pull up the AI's full record for each step (the trace) → find where it keeps tripping → change just one small part of the shell → re-run the eval to see if the score moved. Screen cheap on a small sample first, and only run the full suite after it passes.
The most basic move: don't rewrite the whole system prompt (the opening block that sets the rules) — the longer it gets, the more it gets drowned out. Against a specific failure, add just one small, single-purpose block.
plan carefully, mind the edge cases…"
→ Too long, too vague, gets diluted
After a tool succeeds, the answer
must report the concrete result,
not just "done"
</final_answer_completeness>
The full, copy-ready block is in the article. But the single most counterintuitive lesson in the whole playbook isn't what to write — it's where to write it.
They wanted the AI to follow one rule: when reading a file, if the returned content exactly fills a page, assume there's more and keep reading. Same sentence — written into the tool's manual, the AI stops after page one; moved, word for word, next to the tool's return result, and the AI dutifully keeps paging.
Along the same lines, the middleware also drops a live "plan first, then execute" message into the conversation right before the AI is about to act — this kind of "shows up exactly on cue" reminder works better than a standing rule baked in at the start.
Running the same Deep Agents eval (LangChain's own fixed question set for scoring agents), with quality nearly matched, the two bills differ by an order of magnitude.
- × Re-asking the same thing
- × Quitting before done
- × Spinning in circles
of the weights — just tune the "shell" around it.
rule-setting text
each tool
at every step
AI stops after page 1
AI keeps paging on
Members can copy or download the full Markdown, and let their AI subscribe and ask.
Found this useful? Leave your email — every new explainer, the moment it's out. Unsubscribe anytime.