What Exactly Is a Harness

A harness is the entire system wrapped around a base model. It orchestrates the whole execution process — how the model thinks and plans, how it calls tools to act, how it perceives and manages context, where it stores its outputs, and how it checks whether results are correct.

Early framings of agents used the formula "agent = model + memory + tools + planning + action." Harness engineering wraps a few more layers around that: workflow design (how you design that loop), evaluation, permission control, persistent state management. It's already close to runtime and software system design — it's about how the model observes, acts, remembers, self-checks, and improves, as one whole system.

Letting the Model Plan, Remember, and Even Split Itself to Work

Weng starts with three concrete, actionable harness design patterns. They all answer the same question: what are the foundational pieces a system needs to work autonomously over long stretches of time?

Pattern One: Give the Model a Workflow It Can Iterate On

The key is first defining a process where the model can "act, test, revise." A common approach is a goal-directed loop: plan, execute, observe or test the results, improve, execute again — round and round until the goal is reached. Along the way it can also proactively ask the user to clarify the task. Karpathy's autoresearch repo is a clean example of this kind of workflow. One thing this loop especially emphasizes: the model should analyze its own past trajectories and failure cases and iterate based on those, rather than clinging to a fixed prompt template.

A simplified version of the Codex agent loop: the agent calls tools, and the tool results feed back into what the model generates next. Source: OpenAI Codex agent blog post

Pattern Two: Treat the File System as the Model's Long-Term Memory

For an agent working over long stretches, experiment logs, code diffs, paper summaries, error records, and past execution traces quickly grow longer than any context window the model saw during training. So a harness shouldn't cram the entire process and all logs into context — it should write this persistent state to files and read it back when needed. Reading, writing, and editing files (usually via the bash command) is already a basic skill for AI models, and managing long-term memory through this simplest of forms naturally improves as the model's own capability improves.

Pattern Three: Dispatch Sub-Agents to Work in Parallel — and Keep Them Under Control

A harness can spawn multiple sub-agents to execute in parallel, then watch over those background tasks. This is useful when the main agent needs to test several hypotheses at once, run experiments in parallel, or offload independent sub-tasks without polluting the main-thread context. Here, the main agent needs a lightweight process manager: something that can launch tasks, inspect logs, cancel failed runs, and merge results back into the main thread.

The most critical design choice here is making parallelism inspectable and traceable. If a sub-agent's output only lives in volatile conversation context, it quickly goes stale and gets buried. Only by saving it as files, logs, and state records can the model recover after an interruption and go back to review its own execution history.

Put these three pieces together and you get what mainstream coding agents look like today. Weng notes that the core interfaces of products like Claude Code, Codex, OpenCode, and Cursor have already converged — they all run on a similar set of tools:

With a set of tools in hand, a coding agent can develop and debug inside a code repo, much like a human developer with an IDE. Source: Lil'Log

What Gets Edited Keeps Climbing: From Prompts to Code Itself

This is the central thread of the whole piece. What gets optimized inside a harness system is climbing a ladder: prompts, structured context, workflows, harness code, and finally optimizer code. The stronger the model, the more complex and general the optimization target it can handle.

The three case studies below, arranged in escalating order, lay this ladder out in full.

ACE: Treating Context as a Living Notebook

Dumping every tool result and model generation straight into context spirals out of control once a task runs long. Agentic Context Engineering (ACE) takes a different approach: maintain context as a living notebook of ever-growing entries, rather than one fixed block of text that just keeps getting longer. Every lesson in the notebook carries a number and a one-line description. Three roles collaborate to run it:

A key design choice in ACE is that the Curator never rewrites the whole prompt — it produces a batch of "number + description" entries, merges them into the structured notebook with deterministic logic, and periodically refines and deduplicates. This avoids the old failure mode where repeated rewriting bleeds out detail (context collapse and over-compression).

Agentic Context Engineering (ACE) framework: the Generator, Reflector, and Curator roles collaborate to maintain an itemized context notebook. Source: Zhang et al. 2025

MCE: Optimizing Even the Method for Managing the Notebook

ACE can learn lessons from execution traces, but its update rules and overall process are still hand-coded by humans. Meta Context Engineering (MCE) goes one step further, splitting "mechanism" from "content": one layer manages what's actually inside a given context (the inner loop), while another manages "what method is used to manage context" (the outer loop).

Concretely, the inner loop takes a given strategy and optimizes the specific context for a task; the outer loop evolves new strategies via crossover, drawing on a skill library that records historical strategies and their evaluation scores. Unlike ACE, MCE doesn't force context into a fixed shape — it uses free-form "skills" to store the most important knowledge for a task, letting the skill and its corresponding context co-evolve. In practice, this takes the form of a dedicated directory full of files: a static skill.md alongside dynamic context and data files.

Meta Context Engineering (MCE): the outer loop runs evolutionary search over skills within the "context management mechanism," while the inner loop optimizes the context for a specific task. Source: Ye et al. 2026

Meta-Harness: Using a Coding Agent Directly to Build Harnesses

Meta-Harness goes a level deeper still: the thing being optimized is directly the code that decides what information should be stored, retrieved, and presented to the model. The "Meta-" in the name means exactly this — it's a harness for optimizing harnesses.

The proposer responsible for suggesting new harnesses is itself a coding agent, and what it ultimately produces is a set of harness candidates sitting on the Pareto frontierWhen several metrics can't all be maximized at once (say, both accuracy and speed), pick the set where "no other option beats it on every metric simultaneously," instead of forcing a single best answer.. A few details worth noting: the entire execution history lives on the file system, and the coding agent reads it with grep and cat rather than stuffing everything into one prompt; each proposed harness is a directory on the file system holding its own source code, score, execution trace, and status updates; this loop repeatedly generates new harnesses, and only the ones that meet the bar survive.

Meta-Harness's outer optimization loop: a coding agent repeatedly proposes, evaluates, and filters new harness candidates. Source: Lee et al. 2026

Across these three levels, Weng's point comes through clearly: once harness design becomes an executable search space, a sufficiently strong coding agent can mine that very same design space human engineers use.

Click for detail: Treating the Workflow Itself as a Search Problem

The "workflow" rung on the ladder can likewise be handed to an algorithm to search, rather than left entirely to manual design. ADAS (Automated Design of Agentic Systems) treats agent design itself as an optimization problem: first build an archive using simple agents like CoT and self-refine, then have a meta-agent reference the existing solutions in the archive and write new agent designs in code — designs that pass evaluation get added back to the archive, iterating repeatedly.

ADAS: a meta-agent repeatedly proposes new agent workflow designs in code. Source: Hu et al. 2025

AFlow represents an agentic workflow as a graph, where nodes are model-calling actions and edges are logic operations implemented in code, then uses Monte Carlo Tree Search (MCTS) to optimize that graph: starting from a template, selecting nodes with a mixed score-and-exploration strategy, having the model revise a new workflow based on current performance, evaluating it, and adding it back to the tree if it improves — until the score plateaus or the budget runs out. On QA, coding, and math tasks, AFlow shows solid gains over both hand-designed workflows and ADAS.

AFlow compared against manual methods and ADAS. Source: Zhang et al. 2025

There's no shortage of workflows purpose-built for automated research, too — AI Scientist (a full pipeline for proposing ideas, writing code, running experiments, writing papers, and peer review), ScientistOne (which treats verifiability as a core constraint, requiring every claim to trace back to a source of evidence), and Autodata, which plays "data scientist" by using a question-setter, a strong solver, a weak solver, and a judge to synthesize training data with just-right difficulty.

Letting the Harness Operate on Itself

The previous section was still about the "optimization target" climbing the ladder. In this section, AI can already autonomously modify its own runtime system code — not just tune parameters.

STOP: Not Solving the Problem, But Editing the Improver Responsible for Solving It

Self-Taught Optimizer (STOP) is one of the early cases of recursively improving the shell. It starts with a "seed improver" — taking an initial solution, a utility function measuring quality, and a language model used as a black box, and outputting a better solution. What STOP actually optimizes is the improver itself: it uses the previous version of the improver to generate the next version, self-optimizing one layer inside another.

In Zelikman et al.'s experiments, the evolved improvers genuinely grew a whole range of strategies on their own: genetic algorithms, splitting the problem apart and improving each piece separately, multi-armed-bandit-style prompt search, simulated annealing, temperature tuning, beam search, and tree search. This is exactly the proof point that a harness workflow can be treated as an optimization target.

Self-Taught Optimizer (STOP) algorithm: recursively generating the next version of the improver from the previous one. Source: Zelikman et al. 2023

Self-Harness: Mine Weaknesses, Propose Changes, Validate and Merge

The more recent Self-Harness has an agent improve its own harness through a "propose, evaluate, accept" loop. The whole loop breaks into three steps:

Mine weaknesses: Run tasks with the current harness and collect execution traces for analysis. Two failures can look identical in their error messages (both timeouts, both missing artifacts) while having different underlying causal mechanisms. So what's needed is an information-rich failure record — including both the terminal cause at the validator level and the causal state of related behavior and the abstract mechanism the trace exposes — used to dig out the root cause, rather than just reading the surface error.

Propose changes: The same model is used as the proposer, fed a bounded context: which parts of the current harness are editable, the validated failure patterns, the passing behaviors that should be preserved, and a summary of previously tried changes. It prioritizes errors that recur and can be fixed with a narrow edit (rather than tasks that are simply too hard), and candidates must be distinct from each other and diverse.

Validate and merge: Candidate changes run regression tests on two data sets — held-in (checking whether the weakness was actually fixed) and held-out (checking whether it introduced other unknown problems). Only if neither set regresses is the change accepted and merged into the new harness version; rejected candidates get recorded but don't touch the harness currently in use. Across three different base models — MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 — Self-Harness learned its own dedicated harness instructions for each, targeting each model's specific weaknesses and raising held-out pass rates.

Self-Harness's "mine weaknesses → propose changes → validate and merge" loop. Source: Zhang et al. 2026

Another Path: Mass-Produce Variants, Keep the Strong, Cull the Weak

Everything so far has been step-by-step editing. There's another path that works like natural selection: mass-produce variants, keep only the highest-"fitness" ones, and cull the weak. Evolutionary search comes in handy when the search space is large and weird, hard to optimize directly with gradients, but easy to score a given solution against — and harness search fits that profile exactly.

Evolutionary search has long been used for prompt optimization — Promptbreeder (which evolves even the "mutation prompt" that tells the model how to edit prompts) and GEPA (which combines reflection-based prompting with evolutionary search, using natural-language reflection on trial-and-error traces to propose changes). At the code-editing level, the following are the representative works.

AlphaEvolve: Maintaining a Program Pool and Having the Model Keep Proposing Diffs

AlphaEvolve is a coding-agent-style evolutionary search system: it maintains a pool of candidate programs, has a frozen model generate diff-style edits, repeatedly evaluates the offspring programs, keeps the successful ones, and finds better solutions over time. A few design details: prompts include the parent program, results, and instructions, sometimes with meta-information too; the coding agent can see the whole repo, but the code regions eligible for evolution are explicitly marked with # EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END; the meta-prompt itself co-evolves alongside the solving program. A follow-up, ShinkaEvolve, added three tricks to boost sampling efficiency: more sample-efficient parent sampling (balancing performance rank against offspring count), pruning candidates too similar to the existing population via embedding cosine similarity, and a meta-scratchpad that records good patterns from successful solutions to guide subsequent mutation. ThetaEvolve, meanwhile, combines evolutionary search with reinforcement learning and in-context learning.

AlphaEvolve: a candidate program pool + a frozen model generating diffs + evaluation keeping successful offspring. Source: Novikov et al. 2025

AlphaEvolve's experimental results across a range of algorithmic tasks. Source: Novikov et al. 2025

DGM: Directly Evolving a Coding Agent That Can Edit Its Own Code

Darwin Gödel Machine (DGM) differs from the methods above that optimize a "solution" — it targets evolving an editable harness code repository, where the agent is permitted to directly modify its own harness. The process resembles Darwinian evolution:

It starts with a single coding agent in the pool; each round, a parent is chosen with probability weighted toward "better performance, fewer existing offspring," and gets modified and split off into a new agent; the selected parent reviews its own evaluation logs and then proposes improvements to its own harness code, generating a new version; code edits use only two basic tools, bash and an editor (view / create / edit files); once a new agent is evaluated, only ones that perform well enough get added back to the pool. DGM evolves the harness under a fixed model — the base used in experiments was Claude 3.5 Sonnet. A follow-up, Hyperagents, introduces a meta-agent specifically to control "how to transform an existing task agent into a new agent."

Weng notes that this family of methods works best in domains where candidate solutions can be automatically evaluated and fitness is easy to quantify — matrix multiplication, GPU kernel optimization, algorithmic competitions, data center scheduling. Once you're in a domain where evaluation is slow, fuzzy, and mostly relies on human judgment, it struggles — and the compute efficiency and effectiveness of evolution itself are also concerns.

Click for detail: Can the Model Also Update Its Own Weights at the Same Time?

Harness evolution edits the non-parametric system around the model. For full self-improvement, the model could also be allowed to update its own weights at the same time, achieved through improving the training pipeline or continual learning at test time. SIA is an early attempt to put harness improvement and model parameter updates into the same optimization loop, with three roles: a Meta-Agent that proposes the initial harness, a Task-Specific Agent that executes tasks, and a Feedback-Agent that decides, based on recent trajectories, whether this round should edit the harness or the weights. That said, Weng points out that SIA's experiments have some confounds that make the results hard to interpret (for instance, the task-executing agent uses a much weaker model than the other two roles, and the baseline is also weak) — the direction is interesting, but the evidence is still only preliminary. Continual learning itself deserves its own dedicated piece.

SIA: putting harness improvement and weight updates into the same loop. Source: Hebbar et al. 2026

The 7 Walls This Path Has Hit So Far

The progress is real, but so are the limits. Weng lists 7 unresolved sticking points — a quick scan tells you exactly where things currently stand.

On the reward-hacking point specifically, Weng adds something more concrete: evaluators and permission control are best kept sitting outside the loop that evolves the harness, paired with held-out test sets, trajectory audits, and human review preserved at key decision points. Exactly how far this oversight can be automated and scaled remains an open research question in itself.

How Far Has AI Doing Its Own Research Actually Gotten

Let's pin this down with hard numbers, to close. The AI Scientist line of work proves that an expert-designed harness really can orchestrate a large chunk of the automated research loop. But writing a decent-looking paper isn't the same as making a scientific discovery: a system can produce a plausible-looking draft while underneath it sits fabricated citations, distorted implementations, or experimental results that don't hold up.

The AI Scientist pipeline: a fully automated flow from proposing ideas and running experiments to writing the paper and peer review. Source: Lu et al. 2026

The cheat sheet below shows at a glance exactly where "automated research" currently stands:

Note: On RE-Bench, human experts scored above zero in 82% of 8-hour attempts, and matched or beat the reference solution in 24%; the data comes from 71 eight-hour attempts by 61 experts.

Where Turning an Idea Into an Actual Paper Goes Wrong

Trehan & Chopra 2026 ran a more realistic experiment: giving the model only basic tools — read_file, write_file, llm_search, list_files — to see if it could go all the way from a research idea to a finished draft. They prepared 45 to 50 high-quality seed documents in each of three domains — world models, multi-agent reinforcement learning, and AI safety and alignment — to spark new ideas.

The process repeatedly exposed 6 failure modes:

• Defaulting to training-data priors: using outdated libraries, stale commands, boilerplate formats, or assumptions made without grounding in the actual repo and dataset.
• Implementation drift under pressure: once the implementation gets technically complex, the model tends to slide toward a more common, simpler approach instead of the originally proposed method.
• Memory and context degradation: in long-horizon projects, key details get lost unless they're written down as persistent artifacts.
• Overoptimism: the model declares success while the experimental results are still noise or have outright failed — the same "p-hacking, fake eureka moments" observed elsewhere, papered over with "numerical duct tape."
• Insufficient domain intuition: the model lacks that hard-to-articulate craft sense — anticipating implementation complexity, judging whether an experimental result is credible, knowing which baselines actually matter.
• Weak research taste: the experiment runs fine, but it fails to answer the question that actually mattered.

Weng's closing stance is clear: on the road toward full RSI, researchers have made real, tangible progress, but the sticking points above are still there, and most of them need human feedback and guidance. After all, we're building this technology for a better human future — not the other way around.

Weng's judgment: don't expect near-term self-improvement to come from the model directly editing its weights. It'll first land at the "deployment system" layer, and the most critical piece there is the harness — the runtime system wrapped around the model that calls tools, manages context and memory, and checks whether results are correct.

But what exactly does "editing the shell" mean, and how does AI do it itself? That's the core of the piece.

What gets edited climbs a ladder: prompts → structured context → workflow → harness code → optimizer code. The higher up, the larger the space AI can search on its own — and the more it demands of the underlying model. The most critical rung is letting AI directly edit its own runtime code. Here's how that actually turns:

The official claim is "score climbed from 20% to 50%" — meaningless to most people. Converted into the same concrete batch of problems, it's clear (longer bar = more bugs fixed).