Metadata-Version: 2.4
Name: harbor-firecracker
Version: 0.1.0
Summary: A Harbor BaseEnvironment provider backed by PandaStack Firecracker microVMs — self-hosted evals + fork-based RL rollouts.
Project-URL: Homepage, https://github.com/pandastack-io/harbor-firecracker
Project-URL: Repository, https://github.com/pandastack-io/harbor-firecracker
Project-URL: PandaStack, https://pandastack.ai
Project-URL: Harbor, https://github.com/harbor-framework/harbor
Author: PandaStack
License-Expression: MIT
License-File: LICENSE
Keywords: agent-evaluation,firecracker,harbor,microvm,pandastack,rl-environments,swe-bench,terminal-bench
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.10
Requires-Dist: pandastack>=0.3.0
Provides-Extra: bench
Requires-Dist: matplotlib>=3.6; extra == 'bench'
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: harbor
Requires-Dist: harbor>=0.1; extra == 'harbor'
Description-Content-Type: text/markdown

<h1 align="center">harbor-firecracker</h1>

<p align="center">
  <b>The self-hostable Firecracker provider for <a href="https://github.com/harbor-framework/harbor">Harbor</a> — that <i>forks</i> a running agent VM in ~400&nbsp;ms.</b><br>
  Run agent evals and RL rollouts on your own KVM hosts. Warm a task image once,
  copy-on-write-branch it into thousands of rollouts that share memory.
</p>

<p align="center">
  <a href="#why">Why</a> ·
  <a href="#install">Install</a> ·
  <a href="#use-it-with-harbor">Use with Harbor</a> ·
  <a href="#fork-based-rl-rollouts">Fork rollouts</a> ·
  <a href="#benchmark">Benchmark</a>
</p>

---

[Harbor](https://github.com/harbor-framework/harbor) (from the Terminal-Bench team) is becoming the standard harness for agent evaluations and RL environments. It runs each task in an isolated environment behind a pluggable `BaseEnvironment` interface, and ships providers for Docker, Modal, Daytona, E2B, and Runloop.

There is **no open-source, self-hostable Firecracker provider.** This is it — backed by [PandaStack](https://pandastack.ai) microVMs.

## Why — isolation isn't the point, `fork()` is <a name="why"></a>

E2B and Daytona already run on Firecracker, so "hardware isolation" is *not* the differentiator. The differentiator is **forking a running VM**.

RL and eval workloads do **rollouts**: take one warmed task environment and run the agent against it N times from the *same* starting state. Every other provider makes you **cold-restore a snapshot per rollout** — you pay the full boot + state-load N times. harbor-firecracker warms the task image **once**, then `fork_tree`s it: each rollout is a copy-on-write branch of the live VM (memory *and* disk) that diverges in ~400 ms and shares pages with its siblings until it writes.

| | Managed providers | harbor-firecracker |
|---|---|---|
| Where it runs | their cloud | **your KVM hosts** |
| Per rollout | cold-restore a snapshot (linear cost) | **CoW-fork a warm VM (~400 ms, shared memory)** |
| 50th rollout | as expensive as the 1st | **nearly free** |
| Cost model | per-sandbox vendor markup | self-hosted |

The 50th rollout being as cheap as the 2nd is the whole pitch.

<!-- TODO: embed bench/out/chart.png — SWE-bench Verified, fork vs cold-restore, wall-clock + $. Generate with: python -m bench.swebench_rollout --dry-run --chart bench/out/chart.png -->

## Install <a name="install"></a>

```bash
pip install harbor-firecracker          # the provider + fork-rollout API
pip install "harbor-firecracker[harbor]" # also pulls Harbor, to run `harbor run --env firecracker`
pip install "harbor-firecracker[bench]"  # also pulls matplotlib, for the benchmark chart
```

Set a PandaStack API key (self-host PandaStack on your KVM hosts, or use the hosted control plane):

```bash
export PANDASTACK_API_KEY=pds_...
# export PANDASTACK_API=https://api.pandastack.ai   # default; point at your own control plane to self-host
```

> `harbor` is an **optional** peer dependency. `import harbor_firecracker` works without it; you only need it installed to actually run a Harbor eval.

## Use it with Harbor <a name="use-it-with-harbor"></a>

harbor-firecracker provides a `FirecrackerEnvironment` whose `type()` is `"firecracker"`. Register it with Harbor's environment factory, then select it on the command line:

```python
# conftest-style registration, or anywhere before `harbor run`:
from harbor_firecracker import register
register()   # inserts FirecrackerEnvironment into Harbor's environment factory
```

```bash
harbor run -d "<org/benchmark>" -m "<model>" -a "<agent>" --env firecracker
```

Two integration paths (see [`register.py`](src/harbor_firecracker/register.py) for the exact seam):

1. **Standalone** (this package): call `register()` to add the provider to Harbor's factory map at runtime.
2. **Upstream**: vendor `FirecrackerEnvironment` into `harbor/environments/` and add it to Harbor's `EnvironmentType` enum + factory (a PR into Harbor).

Each Harbor task gets its own microVM. `exec()` runs the agent's commands inside it (with `cwd` / `env` / `user` / `timeout` honored), `upload_*`/`download_*` move task files and results, and `stop()` tears it down.

## Fork-based RL rollouts <a name="fork-based-rl-rollouts"></a>

The reason to use this provider. Warm one environment, then fan out:

```python
from harbor_firecracker import fork_rollouts

# Warm a base sandbox (install deps, clone the task, snapshot it), then
# CoW-fork it into 64 rollout environments that share the parent's memory.
with fork_rollouts(base_sandbox, n=64) as pool:
    for rollout in pool:               # each is an isolated, diverging microVM
        rollout.exec("python agent.py ...")
        # collect reward / verifier output, then discard the fork
```

`fork_tree` is hard-capped at **16 children per call**, so `fork_rollouts(n=64)` issues batches of 16 under the hood — and going past a single host's memory means scheduling across hosts (see [`rollouts.py`](src/harbor_firecracker/rollouts.py) for the batching + the multi-host note). This is an honest constraint, not a "scale to infinity" claim.

## Benchmark <a name="benchmark"></a>

The headline graph — fork vs cold-restore on a SWE-bench-Verified-shaped workload — is reproducible, and runs **without a live host** in simulation:

```bash
python -m bench.swebench_rollout --dry-run --n 64 --chart bench/out/chart.png
```

`--dry-run` (default) models fork (~400 ms, shared-CoW) vs cold-restore (linear) from measured constants — useful for the chart and CI. `--live` runs the real workload against your PandaStack hosts (documented in [`bench/README.md`](bench/README.md)).

## License

[MIT](LICENSE). Built on [PandaStack](https://github.com/pandastack-io/pandastack-ai) (Apache-2.0) and targets [Harbor](https://github.com/harbor-framework/harbor) (its own license).
