REEF
Continual learning infrastructure for self-improving agents
Serves agent requests, records feedback, trains updates, publishes versions
github.com/Human-Agent-Society/reef
1 / 3
Self-evolving agents must learn from the
experience
they generate at
inference time
.
Both the
model
and the
harness
need updates.
REAL USE
users and apps
requests
responses
AGENT
v1
model
weights
harness
prompts · skills · tools
memory · orchestration
1
trajectories
execution results
user feedback
experience
2
UPDATE
grow
3
model trainer
RL · distillation
harness trainer
proposed edits
evaluate, then publish
AGENT
v2
model
weights
harness
prompts · skills · tools
memory · orchestration
4
1
Serve
POST /v1/chat/completions
2
Observe
POST /reef/report
3
Grow
train weights or update harness
4
Commit
evaluate, then publish
accepted updates
are served to the next request
REEF
System design
Architecture and supported recipes
github.com/Human-Agent-Society/reef
2 / 3
Architecture
ENTRYPOINTS
method plug-ins
recipes/*
HTTP · CLI · reef-client
reef/service
KERNEL
application kernel
reef/dispatcher · reef/scenario: coordination, commit ordering, recovery
DOMAINS
serving
runtime · surface
SGLang · vLLM
evolution
train · harness
Slime · harness adapters
state
artifact · records
Git LFS · SQLite
ADAPTERS
Modules for each step of the loop
1
Serve
service · runtime
2
Observe
records · train/processors
3
Grow
recipe · train
4
Commit
train/evaluation · artifact · surface
Supported recipes
model weights
trains the model with Slime and SGLang
online RL from live traffic
SAO
math reasoning · IMO-Bench
OpenClaw-RL
personalized chat
test-time training
TTT-Discover
open-ended discovery
Guidance-TTT
program search
agent harness
works with any model endpoint, no training GPU
prompt evolution
GEPA
math reasoning · AIME 2025
skill evolution
SkillClaw
agent tasks · WildClawBench
composition search
Meta-Harness
terminal tasks · Terminal-Bench
REEF
What Reef can do next
Four research directions
github.com/Human-Agent-Society/reef
3 / 3
DAY · SERVE, RECORD
NIGHT · REPLAY, GENERATE, TRAIN
AGENT
v1
model
harness
experience
generated tasks
replays and variations
AGENT
v2
model
harness
train while idle
1
Evolve during sleep
Generate training tasks from the day's traffic
and train
while the agent is idle
.
ORGANIZATION
shared experience · shared versions
PLANNER
model
harness
CODER
model
harness
REVIEWER
model
harness
OPERATOR
model
harness
hand-off
2
Multi-agent
Multiple agents share
one learning loop
, so
their roles and hand-offs are updated together.
EXPERIENCE
GATE
learn
refuse
attributed · verified
stale · unverifiable
weights
harness
v1
v2
v1
v2
v3
v4
co-evolve
3
Control over evolution
Decide
which experience
to learn from, and
when
to update the weights or the harness.
AGENT
model
harness
best known
?
personalization
one user's preferences
scientific discovery
a better bound
social forecasting
a future event
FIXED BENCHMARK
REAL OUTCOMES
4
Open-ended tasks
Personalization, scientific discovery and
social forecasting, evaluated on
real outcomes
.