Metadata-Version: 2.5
Name: fieldtrial
Version: 0.1.0a1
Summary: Find out whether your robot policy actually got better: statistically rigorous real-world evaluation of robot policies.
Project-URL: Homepage, https://github.com/rokbenko/fieldtrial
Project-URL: Repository, https://github.com/rokbenko/fieldtrial
Project-URL: Issues, https://github.com/rokbenko/fieldtrial/issues
Project-URL: Changelog, https://github.com/rokbenko/fieldtrial/blob/main/CHANGELOG.md
Author: Rok Benko
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: confidence intervals,hypothesis testing,lerobot,policy evaluation,robot learning,robotics,statistics,vla
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: numpy>=1.26
Requires-Dist: rich>=13.8
Requires-Dist: scipy>=1.13
Requires-Dist: typer>=0.15
Provides-Extra: dev
Requires-Dist: httpx>=0.28; extra == 'dev'
Requires-Dist: hypothesis>=6.130; extra == 'dev'
Requires-Dist: import-linter>=2.1; extra == 'dev'
Requires-Dist: mypy>=2.0; extra == 'dev'
Requires-Dist: pre-commit>=4.0; extra == 'dev'
Requires-Dist: pytest-cov>=6.0; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.16; extra == 'dev'
Requires-Dist: scipy-stubs>=1.15; extra == 'dev'
Requires-Dist: statsmodels>=0.14.4; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.6; extra == 'docs'
Requires-Dist: mkdocs<2,>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.29; extra == 'docs'
Description-Content-Type: text/markdown

# fieldtrial

**Find out whether your robot policy actually got better.**

[![CI](https://github.com/rokbenko/fieldtrial/actions/workflows/ci.yml/badge.svg)](https://github.com/rokbenko/fieldtrial/actions/workflows/ci.yml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/rokbenko/fieldtrial/blob/main/LICENSE)

fieldtrial is an open-source Python framework for statistically rigorous, real-world
evaluation of robot policies. LeRobot trains the policy; fieldtrial tells you whether it
actually got better.

> **Status: alpha.** The statistics core and the calculator commands work. Studies, the
> operator console and reports follow in 0.1.0. See the [roadmap](https://github.com/rokbenko/fieldtrial/blob/main/docs/ROADMAP.md).

## Why

Real-world evaluation is slow, noisy and ad hoc. With 40 rollouts, the 95% confidence
interval for a success rate is about ±15 percentage points, so many apparent
improvements from a fine-tuning run are just noise. fieldtrial helps at every stage:

- **Before an evaluation:** how many rollouts do I need? What is the smallest
  difference this evaluation can detect?
- **During:** a randomized, blinded schedule, plus a phone-friendly operator console
  that logs the outcome, progress stage and failure tags of each rollout.
- **After:** correct statistics (exact tests, paired designs, multiplicity control),
  honest wording, and a shareable report.

## 30-second demo

```console
$ uvx fieldtrial compare 74/80 91/120
Arm 1: 74/80 = 92.5% (95% CI 84.6%–96.5%)
Arm 2: 91/120 = 75.8% (95% CI 67.4%–82.6%)
Difference +16.7 pp, Newcombe 95% CI [+6.2 pp, +26.0 pp]
Boschloo p = 0.0020 (two-sided); rejects H0 at α = 0.05
Fisher p = 0.0022

$ uvx fieldtrial power --p1 0.76 --p2 0.90      # rollouts needed to detect 76% → 90%
112 per arm (pooled-z)

$ uvx fieldtrial mde --p1 0.76 --n 40           # what 40 rollouts per arm can detect
+21.1 pp (to 0.9707) / −30.1 pp (to 0.4588)
```

The same functions are available from Python in `fieldtrial.stats`. Every one of them is
tested against an independent reference implementation.

`fieldtrial demo`, with a simulated study, the operator console and a report, arrives in
0.1.0.

## How it fits with LeRobot and openpi

fieldtrial complements LeRobot and openpi and never forks them. It is not a training
framework, a simulation benchmark, a robot driver, a labeling platform or a cloud
service. In manual mode, fieldtrial schedules and records the trials and you run the
robot however you like. Integrations with other stacks are adapters.

## Development

See [CONTRIBUTING.md](https://github.com/rokbenko/fieldtrial/blob/main/CONTRIBUTING.md).

## License

Apache-2.0. Copyright 2026 Rok Benko.

---

<sup>The name comes from agricultural field trials. At Rothamsted in the 1920s, R. A.
Fisher developed randomized block designs for field trials, and those are the designs
fieldtrial uses to compare robot policies.</sup>
