Metadata-Version: 2.4
Name: validanytime
Version: 0.1.0
Summary: Official Python SDK for ValidAnytime: anytime-valid monitoring with budgeted page-tier alarms and clearly-labeled warn-tier early warnings.
Project-URL: Homepage, https://validanytime.com
Project-URL: Documentation, https://validanytime.com/docs
Project-URL: Repository, https://github.com/compiled-intelligence/validanytime
Project-URL: Issues, https://github.com/compiled-intelligence/validanytime/issues
Project-URL: Changelog, https://github.com/compiled-intelligence/validanytime/blob/master/CHANGELOG.md
Project-URL: API, https://api.validanytime.com
Author: Compiled Intelligence, Inc.
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: anytime-valid,changepoint-detection,drift-detection,e-process,e-values,monitoring,sequential-testing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.24
Provides-Extra: detectors
Requires-Dist: validanytime-detectors>=0.1; extra == 'detectors'
Provides-Extra: dev
Requires-Dist: pyright>=1.1; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.7; extra == 'dev'
Description-Content-Type: text/markdown

# validanytime

[![PyPI](https://img.shields.io/pypi/v/validanytime.svg)](https://pypi.org/project/validanytime/)
[![Python versions](https://img.shields.io/pypi/pyversions/validanytime.svg)](https://pypi.org/project/validanytime/)
[![CI](https://github.com/compiled-intelligence/validanytime/actions/workflows/ci.yml/badge.svg)](https://github.com/compiled-intelligence/validanytime/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)

Official Python SDK for [ValidAnytime](https://validanytime.com) — anytime-valid monitoring.

> Alarms with a stated false-alarm budget, valid however often you look — plus unbudgeted early warnings, clearly labeled.

- Python >= 3.10, one runtime dependency ([`httpx`](https://www.python-httpx.org/)).
- Sync (`Client`) and async (`AsyncClient`) with identical method sets.
- Typed dataclass models, actionable errors, automatic retries, idempotent ingestion.

```bash
pip install validanytime
```

**Contents:** [Quickstart](#quickstart) · [Examples](#examples) · [The two alarm tiers](#the-two-alarm-tiers) · [Autoconfig & the backtest gate](#autoconfig--the-backtest-gate) · [CLI](#cli) · [Configuration](#configuration) · [Retries & idempotency](#retries-and-idempotency) · [Errors](#errors)

## Quickstart

```python
import validanytime

va = validanytime.Client()  # reads VALIDANYTIME_API_KEY

# 1. Create a monitor from a template. The server resolves the template at
#    create time and returns the RESOLVED config — what you see is what runs.
monitor = va.create_monitor(
    "judge-scores",
    template="llm_quality",   # also: default, ml_drift, latency, kpi — each a
                              # distinct, gate-verified config (va.use_cases())
    mu_baseline=0.92,         # the level your metric should hold (optional:
)                             # omitted, it is learned from the first values)

# 2. Stream your metric.
va.ingest(monitor.id, 0.91, retry=True)          # one event, retry-safe
va.ingest_batch(monitor.id, [0.90, 0.93, 0.92])  # or a batch

# 3. Read alarms — as often as you like.
for alarm in va.alarms(monitor.id):
    print(alarm.tier, alarm.guarantee_tag, alarm.message)
```

Async is the same surface:

```python
async with validanytime.AsyncClient() as va:
    m = await va.create_monitor("judge-scores", template="llm_quality")
    await va.ingest(m.id, 0.91, retry=True)
    alarms = await va.alarms(m.id)
```

## Examples

Runnable end-to-end scripts live in [`examples/`](examples/):

- [`quickstart.py`](examples/quickstart.py) — create, ingest, read both tiers.
- [`async_quickstart.py`](examples/async_quickstart.py) — the same flow, async.
- [`agent_autoconfig.py`](examples/agent_autoconfig.py) — shape-aware suggestion,
  ratified by the backtest gate before it goes live.
- [`ci_gate.py`](examples/ci_gate.py) — a backtest gate that exits non-zero, so it
  drops straight into CI.

## The two alarm tiers

Every alarm carries a `guarantee_tag`; `Alarm.tier` maps it to one of two tiers
(the same mapping the dashboard uses):

- **`"page"` — budgeted, anytime-valid.** Page-tier alarms come from
  e-processes whose false-alarm control holds at *every* look at once, within
  a false-alarm budget stated up front. Polling `alarms()` more often never
  degrades the guarantee. Tags: `anytime_valid_under_conditional_mean_null`,
  `average_run_length_e_detector`, `online_evalue_fdr_control`. Page-tier
  alarms carry the receipt: `e_value` and `theorem_ref`.
- **`"warn"` — early, unbudgeted, not guaranteed.** Warn-tier alarms
  (tag `heuristic_adaptive`) come from classical detectors (e.g. CUSUM) whose
  calibration is model-based (iid standardized inputs) and **not**
  anytime-valid: on realistic data the false-warning rate can exceed the
  nominal rate substantially. Treat warnings as sensitive early hints; page
  humans on the page tier.
- **`"unknown"`** — missing or unrecognized tag.

```python
pages = [a for a in va.alarms(monitor.id) if a.tier == "page"]
warns = [a for a in va.alarms(monitor.id) if a.tier == "warn"]
```

## Autoconfig & the backtest gate

Don't hand-write detector math. Bring the data; let the **backtest gate** — not
any model — ratify a config before it goes live. This is the agent-driven path:
your agent reads the metric (or asks the user), then calls the API.

```python
sample = [...]                                    # your metric's recent values

# Shape-aware suggestion, already run through the gate on your own history.
s = va.suggest_config(sample, history=sample)     # -> Suggestion
if s.backtest.passed:                             # the non-bypassable check
    m = va.create_monitor("my-metric", config=s.config)
    print(s.backtest.summary)                     # "...caught it on day X."
```

- `va.use_cases()` — the public use-case → config matrix, a **data-free**
  lookup (`UseCase` rows with the resolved `config` + a `direction`/`rationale`
  you can show the user). Pass a row's `config` to `create_monitor`, or its
  `id` as `use_case=` to `suggest_config` as a hint.
- `va.suggest_config(sample, history=..., use_case=...)` — reads the sample's
  shape (level, scale, direction, boundedness, drift), tunes a config, and
  ratifies it on `history` (defaults to `sample`). Returns a `Suggestion`
  (`.config`, `.backtest`). **Check `.backtest.passed` before going live.**
- `va.backtest(history, config=..., inject_shift=...)` — re-run the gate on any
  config. Graded on the **page tier** (`.passed`, `.caught_at_seq`); a warn-tier
  fire on the normal stretch shows up as `.warned_on_normal` and never fails
  the gate. `inject_shift=None` replays as-is and only checks it stays quiet.

A backtest is a replay of past data, not a forecast about a different stream.

### Other methods

- `get_monitor(id)` / `list_monitors()` — the returned `config` is always the
  resolved config that actually runs.
- `update_monitor(id, name=..., template=... | config=...)` — rename and/or
  reconfigure. A config change is validated like create and **resets** the
  monitor's derived engine state (carry, learned baseline, fired-alarm
  latches); the new config applies to future events only. The response's
  `state_reset` flag says whether that happened.
- `list_events(id, after_seq=0, limit=500)` — one seq-cursor page of the
  monitor's event history (ascending), as an `EventsPage` of typed `Event`
  rows (`seq`, `value`, `ts`, `label`) plus `next_after_seq` (pass it back as
  `after_seq`; `None` means done).
- `iter_events(id)` — a generator (async generator on `AsyncClient`) that
  walks the cursor transparently and yields every `Event`:

  ```python
  for event in va.iter_events(monitor.id):
      print(event.seq, event.value)
  ```

`va.fleet_status()` returns the tenant's pooled certificate: fleet-confirmed
discoveries carry FDR <= alpha under arbitrary dependence between streams
(e-LOND). Warn-tier alarms never enter that pool.

## CLI

Installing the package also installs a tiny `validanytime` command:

```sh
validanytime init                    # write a starter monitoring script (no network)
validanytime backtest metrics.csv    # replay a CSV through the OFFLINE detector
                                     # stack — `pip install 'validanytime[detectors]'`,
                                     # runs locally, nothing leaves your machine
```

`backtest` prints the same machine-readable gate verdict the hosted
`POST /v1/onboarding/backtest` returns (add `--json` for the full dict) and
exits non-zero when the gate fails, so it drops straight into CI.

## Configuration

| Setting  | Argument   | Environment variable   | Default                       |
| -------- | ---------- | ---------------------- | ----------------------------- |
| API key  | `api_key`  | `VALIDANYTIME_API_KEY` | — (required)                  |
| Base URL | `base_url` | `VALIDANYTIME_API_URL` | `https://api.validanytime.com` |

Authentication is `Authorization: Bearer va_...` — mint keys from the
dashboard (the plaintext key is shown once).

## Retries and idempotency

- The SDK retries requests that fail with **429, 502, 503, or 504** up to 3
  times, with exponential backoff plus jitter; a `Retry-After` header
  (seconds) is honored. No other 4xx is ever retried.
- Ingestion is **idempotent per `event_id`**: the server deduplicates, so an
  event with an `event_id` is safe to send twice — the second attempt returns
  `duplicate=True, accepted=0`. Pass your own `event_id` when you have a
  natural one, or pass `retry=True` to `ingest()` and the SDK mints a uuid4
  `event_id` client-side, so automatic retries (and your own re-sends of the
  same payload) can never double-count.
- Ingesting **without** an `event_id` is not deduplicated; a retried request
  could then count twice. Prefer `event_id`/`retry=True` in production.

## Errors

All SDK errors derive from `validanytime.ValidAnytimeError` and carry
`status_code` and the server's `detail`:

- `AuthError` (401) — missing/invalid API key.
- `NotFound` (404) — unknown monitor (or another tenant's).
- `ValidationError` (422) — the server's message is designed to be actionable
  (e.g. it lists valid template ids or known config keys) and is surfaced
  verbatim.
- `ServerError` (5xx) — raised after retryable statuses are exhausted.

## Development

The test suite is self-contained — it drives the client through
`httpx.MockTransport`, so no account, API key, or backend is required:

```bash
pip install -e '.[dev]'
pytest                 # tests
ruff check .           # lint
ruff format --check .  # formatting
pyright                # types
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for scope and pull-request guidelines.

## License

[Apache-2.0](LICENSE) © 2026 Compiled Intelligence, Inc. See [NOTICE](NOTICE)
for attribution.
