Your Inference Server is Secretly a Learner: Reef Infrastructure for Continual Self-Improving Agents

Note: body below is original English text extracted from the source. Do not treat this file as a translation.

Published 2026-09-15 · https://huggingface.co/blog/quao627/your-inference-server-is-secretly-a-learner-reef · code https://github.com/Human-Agent-Society/reef

🚀 Reef is fully open source: https://github.com/Human-Agent-Society/reef. The project is actively maintained, and contributions are welcome!

1. Before all the "RSI" hype becomes real

Before today's excitement around RSI is fully realized, we want to open-source Reef, an infrastructure we've been building for the same broader problem: enabling agents (harness + model) to continuously evolve from their experience.

The goal is to make it easier for the open-source community to experiment with continual self-improvement and readily access production-grade infrastructure for it. We use continual self-improvement as the broader and more practical framing here, with RSI representing a more fully recursive form of the same idea.

2. Why does continual self-improvement need new infrastructure?

Most LLM infrastructure assumes a relatively simple lifecycle. We train a model, evaluate it, deploy it, and then use it for inference. For continually self-improving agents, two assumptions behind this setup start to break.

First, inference is no longer the end of the pipeline. Agents generate useful experience while they work, including trajectories, execution results, user feedback, and other signals that can drive future improvement. What happens at inference time is no longer something we simply serve and discard. It becomes part of the learning process itself.

Second, the model is no longer the only thing that evolves. An agent is more than a model, and continual self-improvement should not be restricted to model weights. Prompts, memory, skills, tools, and orchestration logic can all potentially improve from experience. A stronger model expands the agent's capabilities, while a better harness helps elicit those capabilities more effectively and exercise them more reliably in complex tasks.

Together, these changes turn what was once a mostly sequential pipeline into a continuous evolution loop. Agents interact, generate experience, improve different parts of themselves, evaluate whether those changes are worth keeping, and return to serving as new versions of the system.

That is also why we do not think of Reef as merely training infrastructure. Training is only one part of the loop. Infrastructure for continual self-improvement has to start from live inference and support the evolution of the whole agent.

3. What does such infrastructure need, and how does Reef implement it?

Infrastructure for continual self-improvement needs to own three things end to end: the experience, the agent, and the updates. That means (1) learning from live traffic, (2) updating both the model and the harness, and (3) properly evaluating, versioning, and controlling how updates are released.

Own the experience: learning has to be built on top of live serving

Inference and training have historically been decoupled. Some RL infrastructure (e.g., Slime, veRL) incorporates an inference engine for generating data, but these systems are designed for model training rather than model serving. We posit that a continual self-improving infra should first be an inference infra and build its training capacity around live inference: the system serves real applications, collects test-time experience, and lets learning recipes continuously consume it.

Inference is native to Reef. Reef exposes standard inference endpoints (OpenAI format). Applications can report rewards, evaluator feedback, or other signals associated with specific inference calls. Different from existing inference engines that stay static throughout their lifecycles, Reef offers stateful inference: Reef stores inference traces and feedback as a structured experience stream, handling issues such as off-policy staleness, session merging, and deduplication. Learning recipes define how this stream is processed, which learning algorithm is used, and when updates are evaluated and deployed.

Own the whole agent: both the model and the harness are evolved via stateful inference

An AI agent capable of delivering end-to-end results consists not only of a model, but also of a harness. The model provides the underlying capabilities needed to solve tasks, while the harness enables reliable execution across complex, long-horizon trajectories by managing tools, context, memory, feedback, and orchestration. The two are tightly coupled. A continual self-improving infrastructure should support the joint evolution of the entire agent stack, including not only model weights, but also prompts, memory, skills, tools, and orchestration logic.

On the harness side, Reef uses Cordis as a "training backend." A harness evolution recipe typically analyzes agent trajectories and feedback, then proposes edits to the harness. Each evolved harness version is released to the user as an installable update.

On the model side, learning recipes consume Reef records to update model weights. Training runs asynchronously with live serving using a distributed training backend, currently adapted from Slime, and produces candidate weight updates such as checkpoints or LoRA adapters. Once a candidate passes evaluation and is approved for deployment, Reef publishes it as a new version of the scenario's model artifact and hot-updates the serving engine using NCCL-based weight synchronization, without restarting the service.

Own the updates: evolved releases are evaluated and versioned

A continuously evolving agent is subject to service quality degradation especially as the evolution is not guaranteed to bring performance improvement. As such, each evolved candidate should be evaluated before release. Reef controls whether an evolved candidate is allowed to replace the artifact currently serving a scenario. If the candidate is rejected, serving remains unchanged. Otherwise Reef publishes it as a new, auditable release.

Anything Reef can evolve, such as a model checkpoint, LoRA adapter, harness tree, or routing policy, is represented as an artifact managed by a version controller. Reef adopts Git LFS for managing the artifacts. Each scenario has an append-only release chain. Reef advances the scenario's release head using compare-and-swap, so a stale publisher cannot overwrite a newer release.

4. Continual self-improving methods in Reef

The infrastructure above provides the common abstractions for continual self-improvement. The actual logic of how an agent improves is implemented through modular learning recipes.

Despite their different names and mechanisms, methods such as online reinforcement learning, test-time training, skill evolution, harness evolution, and self-play share the same basic pattern: signals generated at test time are turned into updates to the system that generates the next interaction.

These recipes differ mainly along three dimensions:

• Learning signal: What form of signal drives improvement, and where does it come from?
• Experience acquisition: How is learning experience generated? Is it proactively sought by the agent, or reactively generated from an external task or interaction?
• Evolving target: What actually changes: the model, the harness, or both?

Reef is designed so that these methods can largely be expressed through different learning recipes on top of the same infrastructure.

5. Conclusion

Reef is our attempt to turn the broad idea of continual self-improvement into a concrete systems problem. By open-sourcing Reef, we hope to make this problem easier to study and give the community a practical foundation for building agents that do not merely serve, but continually learn and evolve from experience.

We invite you to try Reef, build your own learning recipes, and integrate it with your agents. Explore the code, documentation, and example recipes at https://github.com/Human-Agent-Society/reef.

Footnote: We use continual self-improvement as a broader and more practical framing than RSI. It covers systems that repeatedly improve from experience without requiring the fully closed loop often associated with RSI, where AI itself participates in building and improving the system that produces its next version. Reef is designed for this broader problem, while leaving room for more recursive forms of self-improvement to emerge on top of it.