Metadata-Version: 2.4
Name: telemeval
Version: 0.3.4
Summary: Event-wise and affiliation-based evaluation metrics for spacecraft-telemetry anomaly detection, with a train/test-window leakage guard
Project-URL: Repository, https://github.com/rosscyking1115/telemeval
Author-email: Cheng-Yuan King <rosscyking1115@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
License-File: src/telemeval/metrics/_affiliation_vendor/LICENSE
Keywords: affiliation-metrics,anomaly-detection,esa-adb,evaluation,metrics,spacecraft,telemetry,time-series
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.0
Description-Content-Type: text/markdown

# telemeval

**Scoring for anomaly detectors on spacecraft telemetry — the grading side, not
the detecting side.** You bring the labelled anomalies and your model's output;
telemeval returns precision, recall and F-scores that count whole *events*
rather than individual timestamps, and refuses to return a number at all when
the inputs would produce a misleading one.

For anyone evaluating a time-series anomaly detector who wants the score to be
defensible. Anomaly-detection scoring is unusually easy to get wrong — the
widely used *point-adjusted F1* can rank random guessing above a real detector —
and the metrics the literature recommends instead have not had a maintained,
permissively licensed, pip-installable home. This is that home.

[![PyPI](https://img.shields.io/pypi/v/telemeval)](https://pypi.org/project/telemeval/)
[![Python](https://img.shields.io/pypi/pyversions/telemeval)](https://pypi.org/project/telemeval/)
[![License](https://img.shields.io/badge/license-Apache--2.0-green)](https://github.com/rosscyking1115/telemeval/blob/main/LICENSE)
[![CI](https://github.com/rosscyking1115/telemeval/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/rosscyking1115/telemeval/actions/workflows/ci.yml)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21250548.svg)](https://doi.org/10.5281/zenodo.21250548)

> **Status: v0.x, early releases.** The API may still change before v1.
> Maintained and CI-tested, but this is an evaluation library, not a certified
> tool: nothing here is airworthiness evidence or input to a flight-safety
> decision, and no metric is claimed to be novel. The value is packaging,
> correctness against reference fixtures, and maintenance.
> Released under the
> [Apache 2.0 Licence](https://github.com/rosscyking1115/telemeval/blob/main/LICENSE).

## Install

```bash
pip install telemeval
```

Requires Python 3.11+. Core dependencies are numpy and pandas, and nothing else
— a CI job installs the wheel into an empty environment and fails if anything
more appears.

## Use it

```python
import pandas as pd
from telemeval import evaluate

labels = pd.DataFrame(
    {
        "ID": ["anomaly_1"],
        "Channel": ["channel_41"],
        "StartTime": ["2024-01-01T00:02:00"],
        "EndTime": ["2024-01-01T00:03:00"],
    }
)
timestamps = pd.date_range("2024-01-01", periods=6, freq="1min")
predictions = {
    "channel_41": pd.DataFrame({"Timestamp": timestamps, "Score": [0, 0, 1, 1, 0, 0]})
}

result = evaluate(labels, predictions, dataset="my-mission")
print(result.metrics["event_wise"]["event_wise_fbeta"])     # 1.0
print(result.metrics["affiliation"]["affiliation_fbeta"])   # 1.0
result.save(json_path="report.json", markdown_path="report.md")
```

Both printed values are asserted by the test suite, so this block cannot drift
from what the library does.

## What it does

- **Event-wise precision / recall / F-beta** — every labelled anomaly weighs the
  same regardless of how long it lasted, with documented overlap semantics.
- **Affiliation-based precision / recall** (Huet et al., KDD 2022) — distance-based
  scoring that rewards being *close* to an event. The canonical MIT reference
  implementation is vendored, wrapped and kept green against its own test suite.
- **ADTQC detection-timing quality** — scores *when* each event was first caught,
  not just whether. Previously available only inside ESA-ADB's research fork.
- **Channel- and subsystem-aware F-beta** — did the detector point at the right
  *source*? Checked against the reference suite's exact expected values.
- **A validated ingestion contract** — malformed or misaligned inputs raise a
  typed, actionable error instead of quietly producing a number.
- **A train/test-window leakage guard**, on by default. See below.
- **Deterministic JSON and Markdown reports**, stamped with which metrics ran and
  what they do not cover.
- **Format readers and sklearn-style wrappers** — ESA-ADB and TimeEval layouts in,
  `score(y_true, y_pred)` functions for dropping the metrics into existing code.

## Why trust this

This library exists because an evaluation was found to be wrong, not because a
metric was missing.

A spacecraft anomaly detector was reporting a recall that looked like a weak
model. It was not the model. The scorer was counting anomalies from the
*training* period — events the detector had never been asked to predict — as
missed detections. Restricting scoring to the window the model was actually
given, which is what the benchmark's own protocol requires, raised the figure
substantially. The correction made a published result look *better*, which is
the direction nobody audits.

That audit is what produced the leakage guard here. `evaluate()` compares the
label window against the prediction window and raises `WindowLeakageError`
rather than scoring across the boundary; clipping is possible but has to be
asked for explicitly with `clip_to_window=True`, and the choice is recorded in
the report. It is a runtime check with tests behind it, not a claim — but it
guards that one leakage class, not every way an evaluation can be wrong, and it
does not apply to the thin
[sklearn-style wrappers](https://github.com/rosscyking1115/telemeval/blob/main/src/telemeval/sklearn.py),
which take raw arrays with no window to check.

The full audit and its measured figures are in the reference pipeline,
[aerospace-prognostics](https://github.com/rosscyking1115/aerospace-prognostics),
which consumes this library as a dependency.

## Documentation

- [Usage](https://github.com/rosscyking1115/telemeval/blob/main/docs/usage.md) —
  the leakage guard, continuous scores and thresholds, ESA-ADB and TimeEval
  loaders, parquet input, sklearn wrappers, the metric registry.
- [Metric definitions](https://github.com/rosscyking1115/telemeval/blob/main/specs/metric-definitions.md)
  — exact semantics, including where event-wise recall matches ESA-ADB's
  definition and where precision deliberately diverges from it.
- [Related work](https://github.com/rosscyking1115/telemeval/blob/main/docs/related-work.md)
  — an honest map of prior art and when to use something else.
- [Roadmap](https://github.com/rosscyking1115/telemeval/blob/main/specs/roadmap.md)
  · [Changelog](https://github.com/rosscyking1115/telemeval/blob/main/CHANGELOG.md)

## Scope

telemeval scores detectors; it does not build them, benchmark them, or serve
them. For detectors see PyOD, aeon or darts; for benchmark harnesses and dataset
collections see TimeEval or TSB-AD. The closest neighbours on metrics are
**TSADmetrics** (GPL-3.0, generic time-series metrics), **aeon** (range and VUS
metrics) and **Merlion** (point-adjusted F1). What is different here is the
permissive licence, the telemetry-domain ingestion contract, and maintained
affiliation and ESA-ADB metrics; where metrics overlap, the aim is to reproduce
prior-art numbers rather than invent new ones.

Not affiliated with or endorsed by ESA. The ESA Anomaly Dataset is not
redistributed here.

## Licence and citation

Apache-2.0. The vendored affiliation reference implementation stays MIT, with
attribution retained in
[NOTICE](https://github.com/rosscyking1115/telemeval/blob/main/NOTICE). Dataset
licences are separate from this code and nothing is bundled. To cite a specific
release, see
[CITATION.cff](https://github.com/rosscyking1115/telemeval/blob/main/CITATION.cff)
or the DOI badge above. Contributions:
[CONTRIBUTING.md](https://github.com/rosscyking1115/telemeval/blob/main/CONTRIBUTING.md).
