MIT Team Open-Sources Reef: Turning Inference Servers into Continuous Learners for AI Agents
Live traffic becomes attributed experience. Recipes generate model or harness candidates, and a review-and-versioning pipeline decides which changes make it back into production.
- Most agents stop improving after deployment, not because there's no experience online, but because inference logs, feedback, training, and versioning are siloed.
- Reef's Serve, Observe, Grow, and Commit stages turn requests into attributed experience, then generate model or harness candidates. Only reviewed versions return to serving.
- Recipes wire methods like SAO, OpenClaw-RL, TTT-Discover, SkillClaw, and GEPA onto a common base; scenario scopes records and versions, but learning behavior is set by deployment config.
- Reef can start as a record-only proxy in front of external models, then shift into learning mode. The closed loop controls updates, but teams still own feedback and evaluation targets.
Every time an agent completes a real task, it leaves behind a valuable trail of experience: what the user corrected, why a tool failed, which execution path scored well, and what actually solved the problem. Most inference servers, however, discard this signal the moment they return a response. The moment a model goes live is also the moment it stops learning.
Ao Qu at MIT, along with the Human-Agent-Society team, has built a project aimed directly at this disconnect: Reef. Reef's goal is to link online inference, experience logging, candidate generation, evaluation, and versioned release into a continuously running loop, so an inference server isn't just responsible for answering queries—it also drives ongoing agent learning and self-improvement.
The system doesn't target a single training algorithm; it tackles a broader engineering gap: creating a persistent learning infrastructure that reliably extracts trustworthy experience from live traffic, determines whether to update the model or its outer operating framework, and blocks unverified changes from contaminating production. Reef calls this broader objective continual self-improvement. The system can repeatedly refine itself from experience, but that's not the same as recursive self-improvement, where a system builds successively more capable successors on its own.
Why agents stop learning after deployment
A conventional large model lifecycle is a one-way pipeline: collect data, train the model, evaluate offline, deploy, and respond to requests. The training system cares about how parameters change; the inference system cares only about returning results reliably. The two sides are cut off at deployment.
Agents make that divide increasingly costly. A long task involves multi-turn dialogue, tool calls, intermediate state, execution results, and user feedback. Together, these answer three critical questions: what the system saw, what it did, and how well it worked. This kind of experience is far closer to real-world needs than generic offline corpora, yet it rarely has a stable path back into the learning system.
There's another complication: agent behavior isn't determined by model weights alone. Prompts, memory, skills, tool permissions, and orchestration logic all shape the final outcome. If the wrong tool was selected, context organization spiraled out of control, or a rule was badly written, further training the model may be expensive and ineffective. The thing that actually needs to change might sit outside the model entirely.
Continuous learning therefore demands closing two gaps at once.
- Inference is not the endpoint. Online trajectories, execution results, user feedback, and environment rewards must feed the next round of learning.
- The model is not the only update target. Weights and the harness—the outer system handling context, memory, skills, tools, and orchestration—need to be part of the same evolution and versioning flow.
That recasts the problem from "add another training pipeline" to "manage an online system that's constantly generating experience, candidates, and new versions."
This is where Reef connects to two earlier pieces on this site: the Harness engineering article explores what else beyond the model can be optimized, and the OpenAI ARC-AGI-3 retrospective puts the harness's impact on results into a concrete experiment. Reef is the missing next step: turning those changes into a process that continuously generates improvements from online experience, evaluates them, and safely ships them as the next service version.
How a single request becomes the next update
Reef structures a single learning cycle into four stages: Serve, Observe, Grow, Commit. Serve processes requests and leaves a record. Observe maps subsequent feedback back to the specific interaction. Grow uses qualifying experience to generate candidates. Commit only releases candidates that pass evaluation as a new version.
Keeping those four stages running reliably depends on three end-to-end abilities:
| What must be controlled | Problem it solves | What happens without it |
|---|---|---|
| Experience | Connect feedback precisely to a request, context, and actual version | Samples can't be attributed, so training data quickly becomes unreliable |
| The whole agent | Identify mutable parts across both model and harness | Behavioral changes can't be located, and the wrong update target gets chosen |
| Version release | Evaluate candidates and decide when they replace live versions | Bad updates corrupt production, and stale tasks can override newer versions |
Map experience and feedback first
At its core, Reef is an inference server. It accepts OpenAI-style /v1/chat/completions and Anthropic-style /v1/messages request bodies, so applications don't need to adopt a different calling protocol. The key difference: each response includes an extra x-reef-agent-record-id header, essentially a receipt for that inference.
When user ratings, test results, or environment rewards arrive, the application calls /reef/report and uses that receipt to indicate which interaction the feedback refers to.
response = reef.post(
"/v1/chat/completions",
headers={"x-reef-scenario": "support-agent"},
json={"model": model, "messages": messages},
)
receipt = response.headers["x-reef-agent-record-id"]
reef.post(
"/reef/report",
json={
"score": 0.0,
"feedback": "没有处理空 token 的情况",
"references": [receipt],
},
)
In this example, the feedback says that the empty-token case was not handled. The receipt solves the most fundamental attribution problem in continuous learning: identifying precisely which inference, which context, and which agent version a piece of feedback is about. Reef's notion of stateful inference is broader than just saving chat history—it also means persisting structured inference records, feedback references, scenarios, artifact versions, and sample freshness.
The unified record layer also handles multi-turn session assembly, duplicate samples, and off-policy staleness—a measure of how old a policy was when an experience was generated, and whether it's still appropriate for training the current version. Each scenario keeps its own records and versions isolated. Recipes are selected by the deployment configuration; requests can name a scenario but can't switch learning methods on the fly.
Then decide: change the model, or the harness?
The model handles language, reasoning, and generation; the harness arranges those abilities into reliable actions. No matter how capable the model, incorrect tool permissions and chaotic context will undermine it. Conversely, no matter how well-crafted the harness, it can't conjure capabilities the model lacks. Reef therefore treats both as learning surfaces that can produce candidates, undergo evaluation, and be released as new versions.
- On the model side: a recipe asynchronously consumes qualifying records and trains a checkpoint or LoRA candidate. The model training backend is built on Slime. Once approved, weights can be synchronized via NCCL and the inference engine hot-updated without stopping the service.
- On the harness side: recipes modify rules, prompts, memory, skills, tools, or the orchestration tree based on execution traces, feedback, and evaluation tasks. Cordis serves as the harness evolution backend, delivering new versions as installable artifacts. Interactive clients can choose to update or skip.
Model weights
Records feed into the training backend, producing checkpoints or LoRA candidates. After evaluation, hot updates refresh the serving runtime.
Deliverable: a new model artifactHarness
Trajectories and feedback drive changes to prompts, memory, skills, tools, and the orchestration tree.
Deliverable: an installable harness artifact
Official code layering: upper entrances and recipes enter the application kernel; lower capabilities connect via adapters to the runtime, training backend, and Git LFS/SQLite.
New harness versions arrive as clear update prompts; interactive users can install or skip them.
"Agent version" thus refers to more than just a model name. Which weights, which rule set, which skills and tools were used for a given request all become part of a traceable version chain. The model and harness don't have to change together either: a recipe can update just one side, or even train a guide model while keeping the external execution model frozen.
Only let evaluated candidates reach the service
The biggest risk in continuous updating is that the system internalizes noise, personal bias, or reward loopholes and gets progressively worse. Reef draws a strict line between "new candidate" and "currently serving version": recipes generate candidates, then evaluation plugins compare them against the existing version. If a candidate is rejected, the live service remains untouched. If evaluation itself fails abnormally, the pipeline shuts down and no pending updates get released.
Rejected:The serving version stays unchanged.
Accepted:Commit artifact → load serving state → advance release head.
Only when evaluation passes explicitly does Reef commit the artifact, load the new serving state, and advance the release. Model checkpoints, LoRAs, harness trees, and routing policies are all versionable objects, stored via Git and Git LFS.
Each scenario maintains its own append-only release chain. When a publisher advances the release head, it must declare the old head it saw. If another task has since released an update, a compare-and-swap rejects the stale commit, ensuring an older training job can't overwrite a newer version.
That gating ensures updates follow a disciplined path into the service, but it can't tell a team what "better" means. When evaluation sets are too narrow, feedback is biased, or metrics are easy to game, a rigorous version chain simply executes the wrong objective more reliably.
How different learning methods share one foundation
The learning recipe determines "how to learn," while the shared foundation handles recording, batching, evaluation, and release. Online reinforcement learning, test-time training, prompt optimization, and skill evolution are algorithmically different, but each boils down to three configuration questions:
- Where does the signal come from? Environment reward, user feedback, test results, the next state, or a judge model's assessment?
- How is experience acquired? Passively waiting for real tasks, or actively exploring, searching, and self-playing?
- What gets updated in the end? Model weights, a guide model, rules, a skill pool, or the whole harness tree?
Reef already carries multiple routes through the same interface:
| Recipe | Experience and signal | Main update target |
|---|---|---|
| SAO | Consecutive tasks scored by a test or verifier | Model weights |
| OpenClaw-RL | Derives usable signal from the next state of user interactions, without requiring explicit reports | Model weights |
| TTT-Discover | Repeated attempts on the same problem, using environment reward to retain better solutions | Model weights |
| Guidance-TTT | Scores code search results while keeping the execution model frozen | Guide model weights |
| SkillClaw / Harness Evolve | Generates skill or rule candidates from failed traffic, execution traces, and evaluation tasks | Skill pool or harness tree |
| GEPA | Reflects on failures and rewrites rules, filtering through small batches and a full validation set | Harness components such as rules or skills |
The OpenClaw-RL example feeds a hypothetical student's 72 GSM8K homework problems to a personal agent in sequence, measuring adaptation speed by how many interaction rounds it takes to match user preferences. TTT-Discover takes a different route: it repeatedly attempts a scorable problem like Packing 32, using environment reward to keep the more effective solutions. One learns passively from ongoing interactions, the other actively searches on a single problem—yet both share the same underlying record and version infrastructure.
GEPA demonstrates further that continuous learning needn't mean updating weights. It selects one harness component from failed traffic, asks a stronger reflection model to propose a rewrite, and then filters through small batches and the full validation set. Reef's AIME 2025 validation improved from 26.67% to 46.67% and from 24.00% to 36.00% across two random seeds. That's only two seeds; it confirms the implementation runs and produces candidate improvements, not that it's consistently better than other methods across diverse tasks.
Getting started doesn't require training. The Basic Recipe only logs requests and doesn't learn: you can proxy an external HTTP model through a single Reef process without any local GPU, or connect to a local SGLang instance without launching training. A team can verify that requests, receipts, feedback, and the version chain all work reliably before deciding whether to turn on weight or harness updates.
One note: the reef-infra PyPI package contains the core infrastructure, but Cookbook Recipes live in the source repository and aren't distributed with the wheel. Running these methods requires a source checkout plus each one's specific dependencies. Model training typically also needs a GPU environment as documented; skill or harness evolution has paths that work without GPUs.
What Reef really changes
Reef connects three systems that were previously separated in agent development. The inference service begins accumulating attributable experience. The model and harness share a common version boundary. And training results must pass through evaluation and release control before they return to production.
Training thereby stops being just a periodic job that produces a checkpoint. Observe determines which feedback is trustworthy and still fresh. Grow proposes changes. Commit decides whether a change becomes the next service version. Continuous learning becomes a long-term release engineering discipline rather than an offline experiment.
This architecture suits scenarios where tasks recur, results can be tested or scored with reasonable reliability, and the team is willing to maintain regression evaluations and artifact lifecycles. With only scattered subjective feedback and no stable evaluation surface, wiring up a closed loop just accumulates uncertainty faster.
Reef is Apache-2.0 licensed, requires Python 3.10 and up, and depends on Git LFS for the artifact and checkpoint features. It provides a control plane for agents to learn while working in a live service, but the lessons learned—good or bad—still depend on the feedback signals, the learning methods, and the quality of the evaluations.
Members can copy or download the full Markdown, and let their AI subscribe and ask.
Found this useful? Leave your email — every new explainer, the moment it's out. Unsubscribe anytime.