Metadata-Version: 2.5
Name: seatrialkit
Version: 0.1.0
Summary: Simulator-independent scoring of marine autonomy trials: station keeping, wayfinding and navigation time from ground-truth trajectories.
Project-URL: Documentation, https://seatrialkit.docs.lhzn.io
Project-URL: Issues, https://github.com/lhzn-io/seatrialkit/issues
Project-URL: Source, https://github.com/lhzn-io/seatrialkit
Author-email: Long Horizon Observatory <dfry@lhzn.io>
License: MIT License
        
        Copyright (c) 2026 Long Horizon Observatory
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Requires-Dist: numpy>=1.26
Provides-Extra: scenario
Requires-Dist: jsonschema>=4.20; extra == 'scenario'
Requires-Dist: pyyaml>=6; extra == 'scenario'
Description-Content-Type: text/markdown

# seatrialkit

Scoring functions for marine autonomy trials, computed from ground-truth vehicle trajectories.

## Motivation

seatrialkit was built to score marine autonomy trials by consistent published metrics, regardless of the setting,
in any simulator or at sea.

Each autonomy trial produces a time series of the vehicle's position and heading. Station-keeping error, distance to
each waypoint and time to arrival are all properties of that time series and the trial's goal. seatrialkit
therefore defines each score as a function of exactly two inputs: the recorded track and a scenario file that
describes the trial.

Defining scores on these two inputs has three results:

- **One implementation for every source of trial data.** Any simulator, and any field trial with an independent
  position reference, can produce the track, and seatrialkit scores each track with identical code.
- **Scores that anyone can recompute.** A reported score can be reproduced from the track and the scenario file
  without access to the simulator, middleware or autonomy software used in the trial.
- **Open definitions.** Each metric has a written formula, each free parameter is an argument stored with the
  result, and each metric taken from published work is tested against its original implementation. The first
  such metrics are the station-keeping and wayfinding scores of the Virtual RobotX (VRX) simulation [1, 2].

Interpreting a score requires knowing the environmental conditions of the trial. A station-keeping error of 2 m
reflects one level of control performance in slack water and another in a 1.5 m/s tidal current. A planner given
an accurate current forecast also faces a different task from a planner given an inaccurate one. For this reason
the scenario file is part of every score: it records each environmental input to the trial (currents, wind, waves,
bathymetry), the source of each input (an observation, a hindcast or a forecast), and its role, either the
conditions acting on the vehicle or the information available to the vehicle's planner.

Long Horizon Observatory built seatrialkit for its own evaluation work. The Observatory prepares environmental
data, including ocean forecasts, hindcasts and bathymetry, for high-resolution simulation. It integrates
established open-source simulators and autonomy software with that data, and contributes to those projects, so
that existing autonomy software can be run under the recorded environmental conditions of specific days and
places. Comparing those runs with one another, and with field trials conducted with partner operators,
requires scores computed identically in every setting. The Observatory plans to use seatrialkit to score these
trials, to measure how error in forecast environmental inputs changes the scores, and to publish the definitions
and code behind every score it reports.

Out of scope: simulation, vehicle models, environmental data processing and running autonomy stacks.

## Metrics

| Task | Metrics | Origin |
| :--- | :--- | :--- |
| Station keeping | Mean combined pose error; RMS, maximum and 95th-percentile excursion; mean heading error; fraction of time inside a watch circle; propulsion energy | VRX source [2] for the mean combined pose error; the others are summary statistics of distance from the station |
| Wayfinding | Minimum combined pose error per waypoint, and the mean of those minima | VRX source [2] |
| Navigation time | Success-gated ratio of optimal to actual traversal time, with clipping bounds as parameters | BARN Challenge [4, 5] |

Formulas, units and reporting rules are in the documentation at
[seatrialkit.docs.lhzn.io](https://seatrialkit.docs.lhzn.io) (source in `docs/source/`). The combined pose error is
E = d + 0.75^d * h, where d is the distance to the goal in meters and h the heading error in radians, as in the
VRX scoring plugins [2]. The VRX paper [1] names the tasks; the formulas are stated in the code. The test suite
compares seatrialkit with line-by-line transliterations of the VRX update loops on random trajectories; the two
agree to a relative tolerance of 1e-12. Trajectories replayed through the VRX scoring plugins themselves, built
from the same commit and run in Gazebo Harmonic, give the same scores to within 1e-14 (see Validation in the
documentation).

## Scenarios

A scenario file (YAML) names the region, vehicle, mission, timing windows, scoring set and random seeds. It is
validated against a JSON Schema shipped in the package (`seatrialkit/scenario.schema.json`, version 0). Scenario
identifiers follow the convention of the CommonOcean benchmarks [3]. `scenario_hash` returns the SHA-256 of the
scenario in canonical JSON, which is independent of key order. Two idealized examples are in
[`examples/scenarios/`](https://github.com/lhzn-io/seatrialkit/tree/main/examples/scenarios).

## Installation

```bash
pip install seatrialkit              # metrics only
pip install "seatrialkit[scenario]"  # adds scenario loading and validation
```

Requires Python 3.11 or later.

## Example

```python
import numpy as np
import seatrialkit as tk

t = np.arange(0, 300, 1 / 60)
traj = tk.Trajectory(t, x=0.3 * np.sin(t / 20), y=0.2 * np.cos(t / 15), yaw=np.full_like(t, np.pi / 2))
result = tk.score_stationkeeping(traj, goal_x=0.0, goal_y=0.0, goal_yaw=np.pi / 2, watch_radius_m=2.0)
print(result.as_dict())
```

## Dependencies

- [NumPy](https://numpy.org/) [6]: all metric computations.
- [PyYAML](https://pyyaml.org/) and [jsonschema](https://python-jsonschema.readthedocs.io/) (optional extra
  `scenario`): scenario loading and schema validation.

Related projects (not dependencies):

- [osrf/vrx](https://github.com/osrf/vrx) (Apache-2.0) [2]: source of the station-keeping and wayfinding
  definitions.
- [CommonOcean](https://www.ce.cit.tum.de/cps/software/commonocean/): benchmark identifiers and scenario conventions for motion
  planning on water.

## Status

Version 0.1.0, pre-release. Metric definitions and the scenario schema may change before 1.0. Record the
seatrialkit version with every reported score. A score computed in simulation describes behavior in that
simulation, not field performance.

## References

1. B. Bingham, C. Agüero, M. McCarrin, J. Klamo, J. Malia, K. Allen, T. Lum, M. Rawson and R. Waqar, "Toward
   Maritime Robotic Simulation in Gazebo," in *OCEANS 2019 MTS/IEEE Seattle*, 2019, pp. 1-10.
   doi:[10.23919/OCEANS40490.2019.8962724](https://doi.org/10.23919/OCEANS40490.2019.8962724)
2. Open Source Robotics Foundation, *VRX: Virtual RobotX*, source code, branch `jazzy`, commit `7609d1b`, scoring plugins
   `vrx_gz/src/StationkeepingScoringPlugin.cc` and `vrx_gz/src/WayfindingScoringPlugin.cc`. Apache-2.0.
   [github.com/osrf/vrx](https://github.com/osrf/vrx), read 2026-11-12.
3. H. Krasowski and M. Althoff, "CommonOcean: Composable Benchmarks for Motion Planning on Oceans," in *2022 IEEE
   25th International Conference on Intelligent Transportation Systems (ITSC)*, 2022, pp. 1676-1682.
   doi:[10.1109/ITSC55140.2022.9921925](https://doi.org/10.1109/ITSC55140.2022.9921925)
4. D. Perille, A. Truong, X. Xiao and P. Stone, "Benchmarking Metric Ground Navigation," in *2020 IEEE
   International Symposium on Safety, Security, and Rescue Robotics (SSRR)*, 2020, pp. 116-121.
   doi:[10.1109/SSRR50563.2020.9292572](https://doi.org/10.1109/SSRR50563.2020.9292572)
5. X. Xiao et al., "Autonomous Ground Navigation in Highly Constrained Spaces: Lessons Learned From the Benchmark
   Autonomous Robot Navigation Challenge at ICRA 2022," *IEEE Robotics & Automation Magazine*, vol. 29, no. 4,
   pp. 148-156, 2022. doi:[10.1109/MRA.2022.3213466](https://doi.org/10.1109/MRA.2022.3213466). Current rules:
   [BARN Challenge 2026](https://people.cs.gmu.edu/~xiao/Research/BARN_Challenge/BARN_Challenge26.html).
6. C. R. Harris et al., "Array programming with NumPy," *Nature*, vol. 585, pp. 357-362, 2020.
   doi:[10.1038/s41586-020-2649-2](https://doi.org/10.1038/s41586-020-2649-2)

## License

MIT. Copyright Long Horizon Observatory.
