Harness Learning: Letting Models Edit Harness Source Code to Adapt to New Tasks and Build a Self-Evolving Harness

When we talk about "large language model agents," they are typically composed of two parts:

1. The Model's Brain (Solver): The LLM itself, responsible for understanding problems, reasoning, and generating responses.
2. The External Scaffolding (Harness): The Python code that controls how this brain works—such as which tools to invoke first, how to format prompts, how to retry on errors, and how to aggregate retrieved documents.

What makes this difficult? Traditional model fine-tuning requires updating weights, which can be costly, and it is hard to make these updates on the fly when a new task arrives. Many practitioners manually craft prompt chains or agentic workflows; however, as soon as the task changes, these existing workflows may no longer fit and often need further debugging.

Researchers from Johns Hopkins University (JHU) and Carnegie Mellon University (CMU) introduced Harness Learning: allowing an AI agent to adapt to new tasks at test time by modifying its own execution code, without updating any model parameters.

They detail this method in their paper, Harness Learning Enables Generalizable Test-Time Adaptation.

Their solution: Keep the model's brain fixed, modify its outer code! The paper trains a model called the Proposer. When the agent tackles a new task, the Proposer inspects runtime execution logs—noting which step raised an error and which test cases were answered incorrectly—and then rewrites the harness's Python source code. For instance, it might add an interpreter loop, restructure the retrieval pipeline, or introduce an answer verification step.

The modified program is executed again, and the new results serve as feedback for the next round of modification. At test time, both the Solver and Proposer model parameters remain frozen; what gets updated is the executable code.

How Does a Round of Modification Work?

This process can be thought of as: run a trial, identify issues, revise the workflow, and re-evaluate. What is being adjusted is the execution program for an entire class of tasks—such as a general solution workflow suited for Sudoku puzzles—rather than hardcoding an answer for a single problem instance.

1. Run a small batch of problems. Use the current code on a set of problems with ground-truth answers. Record which tools were called, what reference materials were read, which answers were incorrect, and where errors occurred. The initial code version is referred to in the paper as the "seed harness," or the baseline starting program.
2. Hand the code and diagnostics to the modification model. The Proposer simultaneously inspects the task description, the current Python code, and the execution logs, and then proposes modifications. The reasoning experiments use SEARCH/REPLACE patches (finding an existing block of code and providing replacement text), whereas the question-answering experiments return a revised Python program.
3. Re-run the problems and score the candidate. The system applies the modification and executes the evaluation problems to verify whether the new program runs properly and how well it performs.
4. Select the version to carry forward. Selection criteria vary across experimental settings, detailed below.
5. Continue refining with new feedback. The selected program runs again, and the new execution report becomes feedback for the subsequent round.

• Reasoning tasks (Reasoning Gym): 1 modification candidate is attempted per round, for up to 5 rounds. The version is updated only if the new score strictly improves; otherwise, the original version is retained.
• Multi-hop QA: 8 modification candidates are attempted per round, for up to 10 rounds. The highest-scoring runnable version from the current round is selected, even if its score is lower than the original; the original version is kept only if all 8 candidates fail.

"Learning how to modify code" and "actively modifying code" occur at different stages. During training, the researchers update the Proposer's parameters so it learns how to adapt workflows. When facing a new task, it applies this learned skill to modify code, while the parameters of both the Proposer and the Solver remain frozen.

This paradigm of first learning "how to adapt" and then applying that adaptation strategy to novel tasks belongs to Meta-Learning. Because the object modified during adaptation in this work is an executable program, it is termed "meta-learning over executable programs."

What Code Does It Actually Modify?

The modifications fall into three primary categories: allowing the answering model to read more complete source material, deciding whether to perform additional searches, and delegating computation to Python. The interactive flow comparator below illustrates these workflows before and after modification.

# Conceptual illustration, not raw paper code or a standalone executable implementation
- answer = answer_from_summaries(s1, s2)
+ answer = answer_from_passages(passages1 + passages2)

Reading Raw Passages Directly for the Final Answer, Using Summaries Only for Search

Multi-hop question answering requires following multiple clues across several search steps to piece together an answer. A "hop" can be thought of as conducting a subsequent search based on clues discovered in the preceding step.

Consider an illustrative example: "Which country does the birth city of a certain author belong to?" First, search for the author to find the birth city; then, search for that city to determine the country. This requires two interconnected searches.

The initial seed program followed this workflow:

First search → Summary and next search query → Second search → Summary → Answer based on both summaries.

The issue is that a summary may leave out details needed to answer the question. For example, while the raw text from the second search mentions the country, the generated summary might preserve only its population and geographic location. As a result, the answering model relying solely on summaries misses the country entirely.

The Proposer modified this into: Summaries continue to guide "what to search next," but the final answer directly ingests all raw passage texts retrieved across both searches. This mitigates errors where crucial details are lost in summarization, but the earlier retrieval and search-query generation still have to work well.

Across three QA benchmarks, 84% of the retained programs produced by the model trained with single-step reinforcement learning (Single-step RL) adopted this direct passage-reading structure, compared to 69% for multi-step RL and 47% for the untuned base model. (Sampled across 3 benchmarks × 4 runs × 10 rounds = 120 retained versions per method; 84% represents the structural proportion among programs, not answer accuracy. The two training regimes are elaborated below.)

Adding Another Retrieval Step When Two Searches Are Not Enough

The paper also highlights two distinct approaches for expanding search coverage:

• Fixed extra search. After the second search and material synthesis, generate a third search query, execute a third retrieval step, and finally synthesize all collected materials to answer. This is one observed modification pattern; not all programs are modified to conduct three fixed searches.
• Draft first, search conditionally if absent. The model first produces a draft answer, and Python code checks whether that draft string appears verbatim in the retrieved passages. If it appears, the draft is returned; if not, the draft string is used as a new search query to retrieve additional passages before formulating the final answer.

The second pattern introduces a conditional branch deciding "whether to search further." An exact substring match in the passages does not necessarily guarantee the answer is correct; this check merely serves as a heuristic condition to trigger supplemental retrieval.

Letting the Model Write Code and Delegating Computation to Python

In algorithmic reasoning tasks such as arithmetic, Sudoku, and mazes, the seed harness called the model only once to predict the answer directly. Modified programs, by contrast, frequently instruct the model to write problem-solving code and execute it.

Take the illustrative task "calculate the sum of squares from 1 to 10000": the model can generate sum(i*i for i in range(1, 10001)) and leave the computation to Python. This divides responsibility across three entities:

• The Proposer modifies the workflow: It equips the agent with an "emit code → execute → inspect output/traceback → iterate" loop.
• The Solver writes solution steps: Given a specific problem instance, it generates the required Python code.
• The runtime environment executes the computation: It returns the evaluated result; if an exception is raised, the harness feeds the traceback back to the Solver to revise its code and try again.

This is an interpreter loop. What is synthesized is a general workflow capable of solving such problem types, rather than a hardcoded answer for a specific instance.

On unseen tasks where external tools were not explicitly requested, 77% of the programs generated by the model exposed to supervised demonstrations (SFT) adopted an interpreter loop, compared to 78% after reinforcement learning. (This is the proportion of generated program structures, not answer accuracy.) Input parsing and solver code must still be correct, and the system must supply the safety restrictions needed to execute code.

How Was the Proposer Trained to Modify Code?

Its training can be understood through two learning paradigms: learning from expert demonstrations, and trial-and-error adjustment based on performance.

• Reasoning tasks: Demonstration first, then trial and error. 432 successful modification demonstrations generated by a 35B teacher model were used to teach the Proposer modification formats and strategies via Supervised Fine-Tuning (SFT). The Proposer then proposed its own modifications. The system ran them, and the model learned through Reinforcement Learning (RL) based on their scores. The Proposer and Solver used Qwen3.5-4B, while the teacher was Qwen3.6-35B-A3B.
• Multi-hop QA: Pure trial and error. Skipping the teacher demonstration stage, RL was applied directly on the HotpotQA dataset. The Proposer used Qwen3-4B, and the Solver used Qwen3-8B.

Here, 4B denotes ~4 billion parameters and 35B denotes ~35 billion parameters. Separate modification models were trained for the two experimental domains; only the Proposer's parameters were updated during training, while the problem-solving Solver stayed frozen throughout the experiments.

What Problems Are Used for Diagnostics, and Which for Evaluating Modifications?

Analogous to a student preparing for an exam: practice questions reveal specific weaknesses, quizzes evaluate skill after practice, and an independent final exam measures generalization. The problems in the paper serve three distinct roles:

1. Diagnostic problems: Identify faults. Execute the current program to produce execution reports, allowing the Proposer to pinpoint tracebacks and incorrect answers.
2. Quiz problems: Select candidates. Execute modified candidate programs to compute scores, which are used to calculate training rewards or select the best version.
3. Independent test problems: Evaluate generalization. Held out from modification and selection, these measure the program's final performance on unseen instances.

Why this separation? Fixing only the errors encountered so far might amount to overfitting to a few cases, or inadvertently breaking other instances. Evaluating on separate problems is essential to verify whether modifications truly generalize.

These three roles do not always map to three completely disjoint problem splits. During training, both experimental domains maintained separate diagnostic and scoring sets; when adapting to new tasks at test time, the configurations were:

• Reasoning tasks: For each task category, the same batch of 25 problems provided diagnostic feedback and candidate selection, while 75 separate problems served as an independent test set.
• Multi-hop QA: Out of 500 development problems, 6 problems were sampled per round to provide feedback (with reports containing gold answers and supporting documents), while 64 disjoint problems from the same split scored candidates. Approximately 300 problems served as the independent test set.

Consequently, "adapting to new tasks" here still relies on problems with ground-truth labels and evaluation metrics to provide feedback signals; it cannot be applied off-the-shelf to tasks completely devoid of labels or validation signals.

How Is an Effective Modification Rewarded?

Given the same task specification, current code, and execution report, the Proposer generates 8 modification candidates simultaneously. The system executes each modified program and evaluates them across the same scoring set.

The training algorithm compares the performance of these 8 candidates: modifications outperforming the group average are reinforced, while below-average ones become less likely to be generated. Through repeated training, the Proposer becomes more inclined to propose effective modifications. This intra-group comparison technique is known as GRPO (Group Relative Policy Optimization).

The reward for each candidate consists of two components:

Reward = Problem-solving score of the modified program + Executability bonus.

The problem-solving score ranges from 0 to 1; reasoning tasks can also receive partial credit from evaluators. Executability is rewarded as follows:

• Reasoning tasks: 0.05 bonus if the modification format is parseable; an additional 0.10 if the patch applies cleanly; and up to an additional 0.15 proportional to the fraction of evaluation runs completed. The total executability bonus is capped at 0.30.
• Multi-hop QA: A flat 0.10 bonus if the code parses, defines a run function, stays within 200 lines, and completes the evaluation run; otherwise, the candidate receives a reward of 0.

"Executable" and "correct" are two entirely different things. A program can receive the full 0.30 executability reward while answering every question incorrectly. In reasoning experiments, failed modifications may still retain minor validity rewards accumulated in earlier checks; rewards in neither experiment drop below zero into negative values.

Can It Still Modify Effectively on Unseen Tasks?

The critical question is: did the model merely learn to fix problems it was trained on, or can it generalize its modification strategies to novel task categories? The researchers evaluated both reasoning tasks and multi-hop QA, comparing single-step modifications against multi-round iterative adaptation.

Can a Smaller Proposer Outperform a Large Teacher?

In the Reasoning Gym benchmark, the task categories seen by the Proposer during training were kept separate from the test categories: supervised demonstration training used 21 categories, and reinforcement learning used 5 categories (with overlap); evaluation was primarily conducted on 21 unseen task categories not used during training. (Termed out-of-distribution or OOD in the paper; this denotes that the task category was excluded from the Proposer's training, not that the underlying model had never encountered related concepts during pre-training.)

The chart above compares the average problem-solving scores of programs after modification by different models. Scores range from 0 to 1, where higher is better.

The unmodified seed harness scored approximately 0.49; after modification by the untuned Base model, the score dropped to 0.32; after modification by the SFT model exposed to demonstrations, it scored approximately 0.49; after modification by the 4B model trained with RL, it reached 0.62, whereas the 35B teacher scored 0.56. (Scores of ~0.49 for the Seed and SFT models are visual estimates from the paper's figures.)

This demonstrates that modifying code does not inherently guarantee improvement: an untuned model degraded performance from ~0.49 down to 0.32. After reinforcement learning, however, the modified programs averaged 0.62, surpassing the 0.56 achieved by the 35B teacher.

This comparison evaluates "whose modifications are on average more effective," rather than comparing the models' standalone problem-solving capacities. When inspecting post-hoc test results and selecting the single best candidate out of each group of 8, the teacher achieved 0.85, compared to 0.83 for the 4B model—the teacher still scored slightly higher in this comparison. (This post-hoc metric selecting the best test performer is called an oracle; in the actual experiments, versions were selected using development scores.)

Reinforcement learning also reduced the incidence of unexecutable or zero-scoring modifications. A score of zero can also occur when a program runs smoothly but yields entirely incorrect answers, so it does not solely indicate a code crash.

Another noteworthy baseline: A manually engineered, fixed Python tool loop achieved an average score of 0.69, outperforming the learned single-step modification average of 0.62. That tool loop permitted up to 5 interaction rounds for the Solver, whereas the seed harness involved only a single model call. This research demonstrates that harness modification capabilities can be learned and transferred across tasks; this result did not beat the fixed tool loop or establish better performance at the same computing cost.

The primary figure excluded 5 difficult task categories. When re-incorporating them, the average scores across all 26 task categories were: Base 0.282, SFT 0.415, 4B RL 0.528, and Teacher 0.501. The 4B model maintained an advantage on average, though the gap narrowed. The interactive chart above toggles between these two evaluation scopes. (The excluded tasks were knight_swap, letter_jumble, modulo_grid, sokoban, and ab.)

If One-Step Modification Is Modest, What About 10 Consecutive Rounds?

For multi-hop QA, the Proposer was trained exclusively on HotpotQA and then evaluated on two held-out QA benchmarks: MuSiQue and 2WikiMultihopQA.

Two experimental approaches should be distinguished:

• 80 independent attempts: Every attempt modifies the original seed code independently, yielding 80 single-step candidates.
• 10-round iterative relay: 8 modification candidates are proposed per round, one is selected via development performance, and the next round modifies that selected version using fresh execution logs.

The results are summarized below. The QA metric is exact match (EM): comparing the normalized answer text against the ground truth; MuSiQue and 2Wiki also accept listed answer aliases. (Given the different scoring rubrics, scores should not be directly compared against reasoning tasks.)

(Single-step figures from Table 23; 10-round results are final averages across 4 runs, with ~0.55 for 2Wiki estimated from figures.)

On MuSiQue, the seed harness scored 0.1438, and a single modification step yielded an average of 0.1467—virtually unchanged. However, 10 consecutive rounds pushed performance to ~0.27. On 2Wiki, the seed harness scored 0.2500, a single modification step averaged 0.3090 (with 67.5% of individual candidates outperforming the seed), and 10 rounds raised the average further to ~0.55.

The "post-hoc best" in the table reflects selecting the top-scoring program after inspecting test scores; it reveals the peak potential present within the candidate pool. In real-world deployment where test labels are unavailable, selection must rely strictly on development feedback.

Iterative adaptation driven by feedback is valuable, but comparisons must account for resource consumption. Each 10-round run explores 8 × 10 = 80 candidates, totaling 320 candidates across 4 runs, compared to a single pool of 80 candidates for the independent setting. While the candidate count per run is identical, the feedback signals and execution paths differ fundamentally; the experiments did not match total compute budgets.

Furthermore, retrieval took place within predefined corpora: HotpotQA used 5.23 million Wikipedia abstracts, MuSiQue used 21,100 passages, and 2Wiki used 54,957 passages. What was evaluated was the capacity to retrieve and answer within these specific corpora.

If Testing Uses Many Rounds, Must Training Also Span Many Rounds?

Single-step training teaches the model "how to execute one effective modification given current code and execution logs"; at test time, this recipe can be applied iteratively. It is akin to learning how to fix the bug in front of you, reading the next error trace, and proceeding to fix the next one. (Single-step does not mean training inputs are restricted to seed code: reasoning experiments also incorporated stronger existing program checkpoints.)

Multi-step training, by contrast, feeds intermediate programs generated during sequential rounds back into training, teaching the model to handle its own previously modified code.

In experiments, multi-step training did not consistently outperform single-step training:

• Final 10-round scores in multi-hop QA were nearly identical: HotpotQA ~0.62, MuSiQue ~0.27, and 2Wiki ~0.55.
• On reasoning tasks, multi-step training performed marginally better on standard problem formats, but slightly worse when the presentation format of the problems was changed.

Thus, in these experiments, a model trained to make one modification could also make several successive modifications at test time. Both training regimes award credit based on the immediate modification's score, without propagating downstream trajectory rewards back to earlier edits; whether modeling long-term trajectory credit alters this finding remains an open question.

Why Do Errors Persist Even When the Algorithm Is Correct?

Writing a correct algorithmic subroutine is only one piece of the puzzle; the harness code must also bind the problem statement, tool calls, and final outputs together seamlessly.

Parsing the Problem, Calling Tools, and Returning Answers

The researchers also explored "compositional prompts," directing the Proposer to write specialized algorithms—such as maze solvers, Sudoku engines, or prime finders—and integrate them into the agent's toolchain.

Upon analyzing 32 candidate programs, 12 successfully integrated auxiliary algorithms into the toolchain, but only 3 programs had their auxiliary algorithm actually contribute to the final answer generation. (This measures whether the tool contributed to the output, which does not guarantee an increased score.)

Why did the presence of an algorithm in the code fail to help? The paper documents three concrete points of failure:

• Failure to parse the problem into the algorithm: Parsing code missed goal markers in the problem string, failing to pass the instance to the algorithm and falling back to a raw model call.
• Failure during tool invocation: Malformed tool invocation payloads prevented the tool from executing.
• Failure to extract and return results: The final submission returned the unsolved Sudoku grid, leaving the answer incomplete.

Hence, between "authoring an algorithmic solver" and "enabling the agent to answer correctly with it," three separate bridges must hold: input parsing, tool invocation, and result extraction.

Why Do Some Tasks Plateau After One Round While Others Keep Improving?

Reasoning tasks plateaued relatively early; on problems presented in a different format, the single-step model achieved most of its gains in the very first round. Multi-hop QA, however, continued to improve across all 10 rounds, with the first round accounting for only one-third to two-thirds of the total improvement.

The initial baseline workflow may partially explain this: reasoning tasks started from a single model call, so adding computation tools in round one was already a massive structural leap; QA started with two retrieval steps, leaving ample headroom to refine passage passing, add conditional searches, or adjust retry logic.

However, training setups and version selection criteria also differed between the two domains. This explanation alone does not conclusively prove why one task plateaued quickly while the other continued to improve.

What Is Missing on the Path to Continual Learning?

Prior explorations like Harness-R1, JIT-Agent, and Ornith have investigated teaching models to construct and modify execution scaffolding. This paper demonstrates further: Learned harness modification strategies can generalize to unseen task categories and iteratively adapt via multi-round feedback.

To evolve into agents capable of accumulating experience over the long term, several challenges remain:

• Autonomous debugging. In the current experiments, an external system prepares the code and execution reports for the Proposer; the next step is enabling the model to inspect code, design tests, and benchmark candidate modifications autonomously.
• Validation without ground-truth labels. Currently, detecting failures and selecting code versions depend on labeled development sets. Real-world tasks often lack ready-made answers, demanding alternative reliable verification signals.
• Validating long-term retention. The study demonstrates adaptation over several to 10 rounds within a single task family; it has yet to test whether modification expertise accumulates or whether prior capabilities degrade when sequentially facing many diverse tasks.
• Joint training of both models. The current Solver is frozen while only the Proposer is trained; whether joint training can allow problem-solving capabilities and workflow design to reinforce each other remains a question for future work.

For agent developers, this research outlines a promising path: beyond training models to answer questions, we can train them to modify their own tool utilization and information routing based on execution feedback. Cross-task adaptation has been demonstrated here; autonomous debugging and lifelong experience accumulation await further exploration.