Why does the same AI model perform so differently in a different Agent framework?

The Hugging Face and Liquid AI teams have published a technical guide: The ultimate guide to multi-harness RL.

It addresses a practical problem in building AI Agents: the same large language model can behave very differently inside different Agent harnesses. Training it for one harness does not guarantee that its skills will transfer to another.

The guide explains where these differences come from and demonstrates how existing tools such as Claude Code, Codex, and OpenCode can be connected to reinforcement learning, allowing a model to practice completing tasks and avoiding unnecessary operations across several real tools. We will first explain the problem, then examine the training method and results.

Why does changing the harness change the same model so much?

Researcher Joel Niklaus tested the same GLM-5.2 model across harnesses on the SWE-bench Pro coding benchmark. Changing the surrounding program noticeably changed its task-solving rate. More surprisingly, harness rankings did not transfer directly between models: among ten scaffolds, Codex was GLM-5.2's second-best partner, but it ranked second from last, ninth, with Gemma 4 26B-A4B.

Read the blue and red points separately. Mini-SWE-Agent is GLM's strongest partner, whereas Crush is Gemma's. Moving upward means solving more tasks; moving left means a lower cost per task. Higher cost does not guarantee a higher success rate. A claim that a tool is “best” therefore needs to specify the model it is paired with. The horizontal axis also does not represent the training cost of this RL experiment.

The guide also cites a controlled evaluation: the same 100 SWE-bench Verified tasks, containers, 50-step action budget, and timeout, with only the harness configuration changing. The minimal configuration has no compression or retries. The improved configuration streamlines tools and adds compression and retries. The full configuration adds self-checking, drift checks, and rollback.

The winners under the three configurations are GPT, Kimi, and GLM, respectively. This makes the effect of the harness on model rankings concrete: a model that ranks higher on a leaderboard may not stay ahead inside your execution loop. The figures come from Table 2 in Zhang et al., cited by the guide.

In research and industry, the program wrapped around the underlying model is called an Agent harness.

Distinguishing the roles

To understand the mechanism, first separate several commonly confused concepts:

• Policy / base model: the model weights themselves. A prompt goes in, generated text or tokens come out. The model has no memory across calls. It selects tools and arguments; the surrounding program executes them.
• Harness: the program surrounding the model. It constructs context, exposes tools, intercepts and executes tool calls, parses arguments, handles errors, and decides when to retry or stop.
• Agent: the complete system formed by a policy inside a harness. Users interact with this entire program running in a loop.
• Sandbox: the isolated environment where tools execute code, such as a Docker container, a lightweight virtual machine, or a remote Linux session.
• Benchmark: a fixed set of test tasks together with grading rules.

The model proposes a tool call or final answer. The harness performs the operation, organizes the next round of context, and retries or stops according to its own rules. Changing the scaffold changes both what the model sees and what tools it can call.

An RL environment is also distinct from a harness. The environment stores task state, receives actions, changes that state, returns observations, and supplies rewards when the relevant conditions are met. The harness keeps the model interacting with it. Files and calculation results, for example, are environment state; the harness loop decides when to read a file, ask the model again, or stop.

Why can training on one tool lead to narrow skills?

When deploying an open model yourself, weights alone do not ensure compatibility with the interface of your chosen tool.

The KwaiKAT team at Kuaishou explains in its KAT-Coder research that Agent RL conducted within one fixed scaffold can teach a model to follow that scaffold's particular conventions rather than solve problems more generally. They identify three kinds of overfitting:

1. Format: tool-call API dialects vary widely. Some tools use the standard OpenAI format, some use Anthropic's structure, and others, such as Aider, require specific code-edit blocks in plain text. Tool names matter too: A calls a tool bash, B calls it terminal, and C calls it execute_command. A model that memorizes A's argument names can have its calls rejected by B for invalid arguments.
2. Context: harnesses compress history and assemble context differently. Some use Markdown, while others maintain structured message lists. A model accustomed to one layout may struggle with another.
3. Control flow: some harnesses automatically repair syntax and protect retries; others are strict and terminate after an error. A model that relies on those recovery mechanisms may struggle outside its familiar environment.

The Orchard paper recorded a substantial regression. OpenSWE-32B, fine-tuned in OpenHands, lost 7.5 percentage points when run in the minimal Mini-SWE-Agent. In the unfamiliar Kimi-CLI, its SWE-bench Verified score fell from roughly 60% to 3.6%, and it scored zero on Terminal-Bench 2.0. The same paper found that another system, Scale-SWE, could not produce valid tool calls outside its native tool. Lower success rates and invalid formats are two different failure modes.

The guide surveys work including Poolside Laguna, Kimi K3, Qwen, and Xiaomi MiMo, all increasing the variety of harnesses encountered in training. Their methods differ. Poolside mixes external tool trajectories into supervised training but uses only its production tool, pool, for subsequent RL. Kimi combines modules into different execution configurations. Qwen3-Coder-Next generates coding data in six tools. “Multi-tool training” therefore needs to specify whether diversity enters data collection, supervised fine-tuning, or RL. Open-source developers still face an engineering problem: mature tools are standalone programs written by others, not designed for training. How can RL be connected to them?

Training a black-box harness: why final text is not enough

In conventional RL, the environment is usually a standard function library, such as the familiar Gymnasium interface. Microsoft's Agent Lightning team describes this distinction in terms of white-box and black-box environments:

• White-box environment, with the trainer controlling the loop: the training engine controls every generated token and every env.step() call. Training frameworks such as TRL were originally built around this assumption.
• Black-box environment, with the harness controlling the loop: once a real tool such as Claude Code or Codex starts, it follows its own logic in the sandbox. It compresses context, retries silently, and repairs malformed JSON. Outside that loop, the trainer sees a series of HTTP requests to the model endpoint.

Here, “white-box” and “black-box” refer to who controls the interaction loop. Other reports sometimes use these terms to distinguish compressed from uncompressed histories; that is a different meaning. Even a harness with a simple internal loop can control its own task execution.

An intuitive approach is to wait until the tool finishes, save the full conversation printed on screen, and use it with a reward to train the model.

For this policy-gradient training route, which requires exact sampling records, text and a final score cannot reliably reconstruct what happened during generation.

Token drift and missing log probabilities

Policy-gradient algorithms such as PPO and GRPO need two precise records to determine the correct direction of a parameter update:

1. Which exact token IDs were sampled during generation?
2. What probabilities did the model assign to them at that time? A logprob is the logarithm of that probability.

Keeping only text and re-tokenizing it afterward creates several problems:

• Token drift: re-tokenizing the same characters and spaces at different context boundaries can produce different token IDs. The trainer may update tokens that the model never actually sampled.
• Contamination by helpful repairs: before sending history into the next step, harnesses may clean whitespace, restore missing brackets, or repair JSON fields. Computing loss on that final text makes the model responsible for a corrected answer that it never generated.
• Invalid ratios under asynchronous training: trajectory generation and backpropagation run concurrently. By the time the trainer updates parameters, its weights may be several steps newer than those used to generate the trajectory. Importance sampling needs the current probability divided by the real probability at generation time. Without the original logprobs, recomputing with current weights can make the old/new ratio collapse to an uninformative 1, removing the information needed to correct for policy changes.

This work retains the inference engine's original token IDs and generation-time logprobs rather than reconstructing them from text. The guide also discusses OpenForgeRL, which reconstructs samples from requests and responses and reports useful results. Exact capture is the route adopted here; it does not prove that other approaches cannot train models.

Three pieces working together: OpenEnv, Harbor, and TRL

To connect unmodified existing tools with strict RL training requirements, the study combines three open-source pieces.

OpenEnv supplies interfaces and services shared by the environment, harness, and trainer. Its capture proxy is one component. Rewards come from task verifiers, and TRL updates parameters. Connecting communication channels alone does not teach the model to finish tasks.

For each task, Harbor starts a real harness in an isolated sandbox. Each model request goes through the OpenEnv capture proxy and then to vLLM or SGLang. The proxy preserves generation records and converts the response back into the format the harness expects. At completion, the trainer obtains exported trajectories from the proxy and scores from Harbor; TRL then updates the weights.

Existing tools can be retained because every harness must call a model endpoint. Each rollout receives a separate capture session. The harness uses the proxy's base URL and passes the session ID as its API key. The proxy uses this identifier to assign concurrent requests to their tasks, while the tool continues executing, retrying, and organizing context according to its own rules.

How the capture proxy works

The capture proxy is the architecture's central component: a model gateway compatible with multiple protocols.

1. Translating dialects: existing scaffolds speak Anthropic Messages, OpenAI Chat Completions, Gemini formats, or OpenAI Responses. The proxy detects the protocol from request paths, headers, and bodies. Converters integrated from NVIDIA's Polar gateway normalize them into standard completion requests, so the underlying inference engine receives a unified format.
2. Capturing complete responses, then replaying a stream: when a tool requests streaming, the proxy does not collect partial delta chunks from the inference engine. It waits for the complete response, records all token IDs and original logprobs, and then streams the result to the scaffold. The harness still receives its required protocol, but time to the first token increases.
3. Removing sampling bias: the proxy enforces top_p=1.0, sampling from the full distribution, and enables processed_logprobs. Truncating low-probability tokens distorts the generation distribution and can contribute to entropy collapse in RL. In the experiment, setting top_p to 1.0 moved importance-sampling ratios from the biased 0.985–0.993 range toward 1, to 0.9984–0.9999.
4. Organizing a network of calls into training trees: real coding work is not one question and one answer. Tools retry failures, invoke sub-Agents, and compress history when limits are reached. The proxy finds the longest common token prefix to connect model calls into branching graphs:
   • Abandoned retries form branches without successors.
   • Sub-Agents have their own system prompts and become roots of new trees.
   • Each connected root-to-leaf path is exported as a training sequence. Harness-provided context receives Mask=0, excluding it from loss; model-generated steps receive Mask=1, allowing gradients.

Connecting Harbor and TRL

• Harbor schedules task sandboxes. It natively supports more than 40 mainstream harnesses and 26 sandbox backends, including E2B, Daytona, and local Docker. Task directories provide instructions, environment configuration, and verifier scripts; the harness and sandbox backend are selected at runtime. A rollout instruction specifies the task, tool, and sandbox, separating tasks from tools.
• Records are checked against Harbor's official ATIF trajectories. The system compares the captured token sequences with ATIF step by step, including token lengths and step counts, to catch communication failures that silently remove training data. The team previously found a tool whose empty tool array caused a 400 error and silently cut a trajectory short.
• TRL runs asynchronous GRPO through AsyncGRPOTrainer. Trajectory generation and training backpropagation are decoupled. One H100 runs vLLM inference and produces trajectories; another computes gradients from training sequences. Generation and training run in parallel, reducing time spent waiting for the slowest Agent. Trajectories typically come from weights two steps old, and at most four steps old, making original generation probabilities essential.

An HTTP boundary also separates the environment server from the trainer. Previously, placing the harness inside the training process meant a failure could crash one process while others waited indefinitely for synchronization. Failures now return ok=False and reward=None for the trainer to handle. A verifier that never ran is not mistaken for an incorrect answer with reward 0.

The proxy probes inference-backend capabilities: being able to answer does not mean being able to supply training records. A hosted endpoint without exact tokens and corresponding probabilities can still evaluate tools but cannot support this training route. Calls with missing or misaligned probabilities are retained only as context, not training targets. History compression can also create new roots in the call graph.

What improved when a small model trained in four tools?

The team started with Liquid AI's LFM2.5-2.6B, a 2.6-billion-parameter model, and trained on SmolDataEnvs data-analysis tasks built from Kaggle notebooks. Tasks require inspecting data, executing calculations, reading results, and submitting verifiable answers. One complete attempt is a rollout: it includes multiple tool operations, not just a single response.

One medium-difficulty example asks: “What percentage of all football matches were draws?” The sandbox provides database.sqlite. The instructions require inspecting the data, calculating the result, writing one numeric value to /workdir/answer.txt, and stopping. The expected answer, 25.4%, is supplied to the verifier through task configuration. The model must calculate it from the input and submit a file. The test therefore measures whether it reads the right data, calls tools, follows submission formatting, and makes the answer available to the verifier—not merely whether it can state a number.

The training pool contains 1,000 tasks: 400 medium and 600 hard. Both runs follow the same task order for 1,000 steps. One uses only OpenCode; the other rotates among OpenCode, Claude Code, Codex, and Mini-SWE-Agent. All eight GRPO attempts for a given task use the same tool; rotation happens between groups, preventing differences between tools from being treated as differences between action quality.

Evaluation uses 250 held-out tasks with no notebook, question, or instruction overlap with training. Each task is attempted once in each of four tools, creating 1,000 evaluation units. Pass@1 measures whether the first scorable attempt is correct. These are data-analysis results, not success rates for repairing real code repositories.

Efficiency bonuses require a correct answer

Early Qwen experiments rewarded only correctness, and some trajectories grew increasingly long. LFM therefore received a small efficiency bonus: an incorrect answer gets 0; a correct answer gets 1 + 1.5 ÷ (15 + tool calls). The bonus is at most 0.1 and decreases as calls increase.

GRPO, Group Relative Policy Optimization, compares rewards among eight attempts at the same task, using their positions relative to the group mean to produce learning signals. If all eight are correct, a binary reward gives every attempt 1, with no reward difference. The efficiency bonus gives shorter correct trajectories higher rewards and longer correct trajectories lower rewards. An incorrect attempt still gets 0 even if it uses only three calls.

Among groups contributing to retained training updates, 97/555 OpenCode-only groups, 17.5%, and 155/686 multi-tool groups, 22.6%, had all scorable attempts correct but different call counts. The bonus was their only source of reward differences. These counts refer to groups, not optimizer steps.

Look at each tool before the overall average

The final results at step 1,000 are below. The base model produced scores for 998/1,000 evaluation units; both final runs produced scores for all 1,000.

The same base model solves roughly six in ten tasks in Mini-SWE-Agent but only three in ten in Claude Code. That is the effect of pairing model and tool. Training only in OpenCode gives the largest gain in OpenCode. Four-tool training exceeds the base model across all four interfaces, and beats single-tool training in Claude Code and Codex.

However, 54.2% versus 52.3% is only a 1.9-percentage-point gap. The guide considers it within noise, so it does not establish significantly better overall accuracy for multi-tool training. Mini-SWE-Agent's 64.8% and 66.0% are close, not identical. The multi-tool run evaluates new tasks in four interfaces that were included in training; it does not test a completely unseen harness.

The curves reveal what the final table cannot. Both runs make their main gains in the first 200 steps, then fluctuate while broadly retaining those gains through step 1,000. Each checkpoint runs only one attempt per task and tool. Fluctuations of one or two percentage points cannot directly establish changes in ability; the broader trajectory is more informative.

Call counts show an apparent contradiction: multi-tool training averages more calls during training, yet uses fewer calls than the single-tool model across all four tools at evaluation. The training average includes Claude Code, Codex, and Mini-SWE-Agent, which naturally use more calls than OpenCode. Evaluation runs both models in the same four tools, allowing comparisons within each interface. Training averages reflect different tool mixtures and cannot by themselves establish which model is more efficient.

Correct solutions use fewer operations, but “31% saved” has a specific denominator

Comparing only the same task–tool pairs solved correctly by both the base and trained models, final multi-tool training reduces total tool calls by about 31%, versus about 11% for single-tool training. The multi-tool reductions are 32.8% in OpenCode, 28.3% in Claude Code, 53.0% in Codex, and 16.2% in Mini-SWE-Agent.

Fewer calls also mean less repeatedly transmitted history, reducing input tokens. The 31% measures a relative reduction in total calls on matched tasks. It is neither an average over every task nor a reduction in bills or elapsed time. Single-tool training improves accuracy in Claude Code but ultimately increases calls and input tokens; at step 500, output tokens were around 62% higher. Accuracy and execution efficiency need separate assessment.

Calls, output, and input are three different measures of work. In Codex, calls fall by 53.0%, output tokens by 39.7%, and input tokens by 45.9%. Halving calls therefore does not mean halving all computation.

LFM has no control run without the efficiency bonus, so all savings cannot be attributed specifically to that bonus. Training exposure also differs: the single- and multi-tool runs encounter 555 and 626 distinct tasks, with 12.5M and 18.7M supervised tokens and 162M and 451M processed tokens. Claude Code rewrites history, producing around eight training rows per trajectory on average, versus around one for the other tools. Each setting has only one random seed; this is an observational comparison.

What came from imitating a larger model's successful trajectories?

The team also used Qwen3.8-27B as a teacher. It attempted each task in each tool up to three times and retained the first trajectory judged correct by the verifier, yielding 3,189 successful trajectories from 888 tasks. Supervised fine-tuning, SFT, treats preceding conversation and tool outputs as context and learns the teacher's next response. RL learns instead from rewards for the small model's own attempts.

Single-tool SFT uses 801 OpenCode trajectories. Multi-tool SFT mixes all trajectories without balancing by tool. Both train for two epochs.

A successful trajectory is not one training example. Each assistant turn becomes “preceding context → next response,” with only that response trained. The guide tokenizes using the student LFM's chat template. This does not conflict with the earlier RL requirement: SFT learns the probability of teacher answers under the student model, whereas this RL route must align the student's actually sampled tokens and generation probabilities to compare old and new policies.

Multi-tool SFT improves Claude Code, 33.2% → 42.4%; Codex, 40.0% → 46.8%; and OpenCode, 33.6% → 38.0%. But Mini-SWE-Agent drops from 62.1% to 45.2%. This offsets the other gains, leaving overall performance at 43.1%, close to the base model's 42.2%. Single-tool SFT mainly improves OpenCode, 33.6% → 51.6%. Claude Code stays at 33.2%, while Mini-SWE-Agent drops to 58.8%. The team has not explained why SFT harms Mini-SWE-Agent.

RL achieves higher accuracy here, but that does not justify “SFT merely memorizes answers; only RL learns to correct errors.” The objectives, data sources, task pools, and compute differ, and each setting runs only once. Multi-tool SFT has around seven times the supervised tokens of single-tool SFT. The table uses SFT after two epochs and RL's best checkpoint selected on the test set, slightly favoring RL. These RL checkpoints are all less than half a percentage point from the final overall scores at step 1,000.

Another observation: multi-tool SFT reduces calls by 24.2%, across all four interfaces, despite having no tool-efficiency reward. Different training routes can improve efficiency; reduced calls alone cannot establish that a particular reward caused the improvement.

Planned follow-up controls address these confounders: compare SFT followed by RL with RL directly from the base model to test whether imitation provides a useful starting point; compare mixtures on the 692 tasks solved by the teacher in all four tools to reduce task-selection differences; and equalize supervised tokens across tools to reduce sample-volume differences. These experiments have not yet been conducted and should better identify where gains come from.

Why did Qwen improve and then decline?

Early Qwen3.5-2B runs used an easier pool of 1,000 tasks and rewarded correctness alone. The multi-tool run rose from 14.6% to 37.0% at step 500. OpenCode-only reached 39.5% at step 700. Both fell back to roughly 26% by step 1,000. A separate standalone OpenCode environment run followed a different path, rising slowly to its best score, 29.8%, at step 1,000.

The trajectories expose different problems:

• Multi-tool training allows up to 16,384 output tokens, while evaluation allows only 4,096. Later responses grow longer, often giving lengthy explanations when calculation is needed. Evaluation units hitting the limit increase from 9 to 556. Evaluation cuts off a habit encouraged by the training budget. Whether a larger evaluation budget would solve these tasks was not established in these results.
• OpenCode-only keeps operating without submitting. Calls rise from around 10 to 20 during training, and from 17 to 21 at evaluation. Among units with submission records, the proportion that submits an answer falls from 69% to 41%.
• After the multi-tool restart at step 684, almost all sampled tasks are previously seen ones, changing late-stage exposure. But the decline began at step 500, so the restart bug cannot explain its onset.

Two more effects can be hidden by the phrase “rotating training.” Across 35%–58% of optimizer steps, all groups lack reward differences. The four tools each provide about a quarter of trajectories, yet Claude Code contributes 77% of training rows and 35% of supervised tokens. Equal trajectory shares do not mean equal actual training exposure. Continuing with a new pool of harder tasks also fails to restore the multi-tool peak.

The later LFM experiment changes three things together: adds the efficiency bonus, switches to medium and hard tasks, and sets both training and evaluation output limits to 4,096. LFM retains gains through step 1,000, but these simultaneous changes prevent separating their contributions. Training budgets, evaluation budgets, submission behavior, and within-group reward differences all need checking; rising training reward alone is insufficient.

What should developers take away?

First, Agent performance depends jointly on the model and the program running it. Evaluate choices in your own tools, tasks, context management, and budgets instead of relying only on model rankings detached from their harnesses. Changing either model or tool can change the entire system's behavior.

Second, this work offers an open engineering route: preserve real harness tools and loops, collect the records needed for training at the model interface, and connect different tools to one trainer. It supports programs using the relevant model APIs, but still requires compatible inference backends, configuration, and verifiable tasks. A proxy alone does not make any arbitrary program trainable.

Rewards must match the actual goal. The guide records an attempt that awarded 0.2 for submitting anything. The model learned to submit immediately. Training reward looked normal, but held-out accuracy fell from 74.0% to 17.8%. Correctness and efficiency must be encoded in the objective so that fewer steps or early submission cannot substitute for completing the task.

Each LFM run uses two H100s: roughly 32 hours for single-tool training and 46 hours for multi-tool training, excluding evaluation, queues, and restarts. Total cost was not recorded. The small-model prototype establishes feasibility on a class of data-analysis tasks; it does not establish a general ranking of all models, tasks, or tool mixtures.

Reproduction: first confirm that training records can be captured

The guide starts with four OpenEnv commands. info checks backend, task, and tool capabilities; rollout runs a task independently; serve provides the environment service; and push deploys that service. Verify a single task and its records before connecting the trainer, so tool-execution and service-layer problems can be diagnosed separately.

Dependencies include Python 3.12, OpenEnv's Harbor extension and typed TrainingTrace API, and TRL's HarnessRolloutWorker. At publication, that worker was in an experimental module on TRL's main branch rather than its then-current stable release. Reproduction should follow the accompanying example's dependencies.

vLLM needs --return-tokens-as-token-ids and --logprobs-mode processed_logprobs. The guide says SGLang supports non-streaming token returns from v0.5.17, but the team tested only a main build. These vLLM flags cannot simply be copied to SGLang. A successful ordinary chat call does not prove complete token and probability capture.

Service capacity must also accommodate a complete group. The guide's environment service defaults max_concurrent_envs to 4, while a GRPO group has 8 rollouts. Exceeding capacity returns CAPACITY_REACHED; configure it for actual concurrency before training. The capture proxy itself is a single process. The team observed stressed health checks around 200 concurrent sessions and a crash at 320. Those are observations from that run, not capacity guarantees; test your own load before scaling.

The complete SmolDataEnvs training set contains about 5,000 tasks; this experiment selects 1,000. Historical experiments use E2B sandboxes, while the current tutorial uses Daytona and includes a standalone OpenCode comparison. The current tutorial therefore does not exactly replay the historical experiment.

Code, models, and data are available through the accompanying code and original guide.

The model and data collection gathers the experimental outputs. The LFM multi-tool RL model defaults to step 1,000; its step-700 branch is the best test checkpoint. The training comparison dashboard provides original curves. The SFT dataset provides per-tool splits and pre-tokenized configurations. New runs should use Harbor's OpenCode adapter; the standalone opencode_env retained in the guide is marked as a historical comparison entry point.