Metadata-Version: 2.4
Name: httpx-hedged
Version: 0.5.0
Summary: Adaptive, per-endpoint request hedging transport for httpx. Learns latency percentiles per endpoint with DDSketch, caps hedge rate with a token bucket, and stops hedging under a host/endpoint circuit breaker.
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: ddsketch>=3.0.1
Requires-Dist: httpx>=0.27.0
Description-Content-Type: text/markdown

# httpx-hedged

An [httpx](https://www.python-httpx.org/) transport that adds
adaptive, per-endpoint request hedging: fire a backup request when the
primary is running slow, take whichever finishes first, cancel the
loser. Based on Google's [The Tail at
Scale](https://research.google/pubs/pub40801/) and modeled heavily on
[hedge-python](https://github.com/sunhailin-Leo/hedge-python).

## Quick start

```python
import asyncio
import httpx
from httpx_hedged import HedgedTransport

async def main():
    transport = HedgedTransport()
    async with httpx.AsyncClient(transport=transport) as client:
        response = await client.get("https://api.example.com/data")
        print(response.json())

asyncio.run(main())
```

For a runnable, end-to-end demo (a small backend with different latency
profiles per route, driven by a client that prints a hedge/latency/circuit
breaker report), see [`examples/`](examples/README.md).

With no configuration, `HedgedTransport` learns a p90 latency estimate per
host (via a [DDSketch](https://arxiv.org/abs/2004.08604) quantile sketch)
and fires a hedge request whenever the primary exceeds it. 

### Sync clients

`SyncHedgedTransport` is the same adaptive hedging (same `HedgeConfig`,
`EndpointConfig`, stats, and circuit breaker) for a plain `httpx.Client`:

```python
import httpx
from httpx_hedged import EndpointConfig, SyncHedgedTransport

transport = SyncHedgedTransport()
transport.register("GET", "/api/v1/fast-lookup", EndpointConfig(percentile=0.90))

with httpx.Client(transport=transport) as client:
    response = client.get("https://api.example.com/data")
    print(response.json())
```

> [!TIP]
> Prefer the async `HedgedTransport` if you can. The sync version exists
> for codebases where async isn't an option.

Races run on a thread pool (`ThreadPoolExecutor`, default 200 workers,
configurable via `max_workers=`/`executor=`), and a losing thread blocked
on a socket read can't be cancelled the way an `asyncio` task can.
Limitations:

- **Threads per request**: the calling thread plus a pool worker, plus a
  third during a hedge race, which scales worse than async at high
  concurrency.
- **Losers run to completion**, doing full duplicate backend work and
  holding their pooled connection until they finish or time out.
- **`close()` blocks** until every orphaned loser finishes.
- **No shared state** with an async `HedgedTransport` hitting the same
  backend.

> [!WARNING]
> A request timeout is mandatory with `SyncHedgedTransport`: a losing
> request can't be interrupted, and hung losers pile up until the thread
> pool is exhausted.

See "Race and cancel" below for the details.

## Why per-endpoint?

A single host can host wildly different endpoints. Learning one latency
distribution per *host*, rather than per endpoint, means a handful of
calls to a slow endpoint skew the hedge trigger for a fast one sharing the
same host:

```
GET /api/v1/fast-lookup   median 10ms,  high RPS
GET /api/v1/bulk-export   median 900ms, low RPS
```

`HedgedTransport` lets you register per-endpoint config up front. Each
registered endpoint gets its own latency sketch, rate estimate, and hedge
budget, all still funneled through a **single inner transport and
connection pool**, unlike using `httpx` `mounts={}`, which would mean one
connection pool per pattern:

```python
from httpx_hedged import EndpointConfig, HedgedTransport

transport = HedgedTransport()
transport.register("GET", "/api/v1/fast-lookup", EndpointConfig(percentile=0.90))
transport.register("GET", "/api/v1/bulk-export", EndpointConfig(percentile=0.90))
```

Requests that don't match a registered pattern fall back to a default
config, tracked per host (the same behavior as hedging with no registered
endpoints at all).

Route patterns may contain `{name}` placeholders or a bare `*` for a single
path segment, e.g. `/api/v1/users/{id}`. Patterns are matched in
registration order, so register more specific patterns first.

## Hardcoded vs. adaptive delay

Most endpoints should hedge adaptively, against their own learned
percentile:

```python
transport.register("GET", "/api/v1/search", EndpointConfig(percentile=0.95))
```

For an endpoint where you already know the right delay, or want
deterministic behavior without a warmup period, hardcode it instead:

```python
transport.register("GET", "/api/v1/health", EndpointConfig(hedge_delay=0.05))
```

A hardcoded endpoint still records latency into its sketch for
observability; it just isn't consulted for the hedge-delay decision.

## Explicit endpoint override

Auto-matching not precise enough for a particular call site (or you'd
rather not register a pattern)? Tag the request directly; this bypasses
pattern matching entirely:

```python
await client.get(
    "https://api.example.com/some/path",
    extensions={"hedge_endpoint": "pinned-name"},
)
```

The name must already be registered (`register(..., name="pinned-name")`);
an unknown name raises `UnknownHedgeEndpointError` rather than silently
falling back, so typos fail loudly.

## Configuration

### `HedgeConfig` (transport-wide default)

| Parameter | Type | Default | Description |
|---|---|---|---|
| `percentile` | `float` | `0.90` | Sketch quantile used as the hedge trigger |
| `budget_percent` | `float` | `10.0` | Max hedge rate as percent of total traffic |
| `estimated_rps` | `float \| None` | `None` | Pin the expected requests/sec, or leave `None` to auto-estimate from observed traffic |
| `rps_window_duration` | `float` | `10.0` | Rolling window (seconds) for RPS auto-estimation |
| `min_delay` | `float` | `0.001` | Floor on the hedge delay in seconds |
| `warmup_requests` | `int` | `20` | Requests using a fixed delay before the sketch is trusted |
| `warmup_delay` | `float` | `0.01` | Fixed hedge delay during warmup, in seconds |
| `window_duration` | `float` | `30.0` | Latency sketch rotation interval in seconds |
| `circuit_breaker` | `CircuitBreakerConfig` | see below | Health circuit-breaker configuration |

### `EndpointConfig` (per-registered-endpoint override)

Every field mirrors `HedgeConfig` and defaults to `None`, meaning "inherit
the transport default." One extra field:

| Parameter | Type | Default | Description |
|---|---|---|---|
| `hedge_delay` | `float \| None` | `None` | Hardcode the hedge delay for this endpoint, skipping the sketch for the decision |

### `CircuitBreakerConfig`

| Parameter | Type | Default | Description |
|---|---|---|---|
| `error_rate_threshold` | `float` | `0.5` | Failure fraction that trips the breaker open |
| `min_samples` | `int` | `20` | Minimum samples in-window before the breaker can trip |
| `window_duration` | `float` | `30.0` | Rolling window (seconds) for the error-rate estimate |
| `cooldown` | `float` | `30.0` | Seconds the breaker stays open before a half-open trial |
| `half_open_max_trial` | `int` | `5` | Trial requests allowed through while half-open |
| `treat_5xx_as_failure` | `bool` | `True` | Whether an HTTP 5xx response counts as a failure |

All three config classes validate their fields at construction time (e.g.
`percentile` must be strictly between 0 and 1, delays and windows must be
non-negative/positive) and raise `ValueError` immediately on a bad value,
rather than silently misbehaving later.

## How it works

### Race and cancel

```
              ┌─ primary  ─────────── ✓ (fast) ──→ return
request ──────┤
              └─ hedge fires after estimated delay ─── ✗ (cancelled)
```

Only idempotent methods (`GET`, `HEAD`, `OPTIONS`) are ever hedged, to avoid
duplicating side effects. A request with a body is also never hedged, even
if the method is idempotent: the primary and hedge send the same
`httpx.Request` object, and a body backed by a one-shot stream can't be
safely read twice.

`SyncHedgedTransport` races primary and hedge on a thread pool instead of
`asyncio` tasks. The "cancel" step is where the two genuinely diverge:
`asyncio.Task.cancel()` interrupts a coroutine at its next `await`, almost
immediately, but there's no equivalent way to interrupt an OS thread blocked
in a socket read. So the sync scheduler returns the winner's result right
away without waiting on the loser, and the loser keeps running in the
background until its blocking call finally returns on its own, at which
point its result is discarded.

> [!WARNING]
> This makes a configured request timeout load-bearing for
> `SyncHedgedTransport` specifically: without one, a losing request can
> occupy its thread indefinitely.

### DDSketch quantile estimator

Each tracked key (an endpoint, or the per-host fallback) gets its own
sliding-window DDSketch pair that rotates every `window_duration` seconds.
DDSketch gives relative-error quantile guarantees regardless of the
underlying latency distribution's shape. The sketch itself comes from the
[`ddsketch`](https://github.com/DataDog/sketches-py) package (DataDog's
reference implementation of the paper below).

### Token bucket budget

Hedges are rate-limited by a token bucket refilling at
`estimated_rps * budget_percent / 100` tokens/second, per key. During a
genuine outage the bucket drains and hedging stops automatically,
preventing the load-doubling spiral that would deepen the incident. By
default the RPS feeding this calculation is estimated automatically from
observed traffic per key, rather than requiring a manual guess per
endpoint.

## Circuit breaker

A closed / open / half-open circuit breaker tracks request success/failure
at **two independent tiers**: one breaker per host, one breaker per
endpoint key. Either tripping open suppresses hedging for its scope: a
host-level trip disables hedging for every endpoint on that host, while an
endpoint-level trip disables hedging only for that one endpoint.

Crucially, the breaker **only ever suppresses the hedge request**. The
primary request is always sent, and its result or exception is always
returned to the caller normally. This is not a request-blocking circuit
breaker, so hedging can't pile extra load onto a backend that's already
struggling.

```
CLOSED ──(error rate ≥ threshold, samples ≥ min_samples)──▶ OPEN
OPEN ──(cooldown elapsed)──▶ HALF_OPEN
HALF_OPEN ──(trial requests mostly succeed)──▶ CLOSED
HALF_OPEN ──(trial requests mostly fail)────▶ OPEN
```

> [!NOTE]
> Health is recorded from the *winning* task's outcome only. A cancelled
> loser's real outcome is unknowable, and losers are cancelled
> deliberately (not doing so would defeat the point of reducing load on a
> struggling backend).

## Observability

### Polling stats and health snapshots

```python
transport = HedgedTransport()

# ... after running some traffic ...
for key, snap in transport.stats.all_snapshots().items():
    print(key, snap)

print(transport.stats.global_snapshot())
print(transport.health.host_state("api.example.com"))
```

`StatsSnapshot` reports `total_requests`, `hedged_requests`, `hedge_wins`,
`primary_wins`, `budget_exhausted`, `warmup_requests`, `circuit_blocked`,
and `errors` per key, plus a global aggregate.

To see the learned latency estimate itself (e.g. the current p90 driving
the hedge trigger), query `latency_quantile()` with the same key format:

```python
name = transport.register("GET", "/api/v1/search", EndpointConfig(percentile=0.90))
# ... after running some traffic ...
p90 = transport.latency_quantile(f"endpoint:{name}", 0.9)
if p90 is not None:
    print(f"p90: {p90 * 1000:.1f}ms")
```

Returns `None` if the key hasn't recorded any samples yet. You can query
any quantile, not just the one driving the hedge decision.

### Push-based hooks: metrics and alerting

Polling snapshots works for dashboards, but alerting on a circuit-breaker
trip and emitting a metric on every hedge fire are both things you want to
happen *at the moment they occur*, not on the next poll. `HedgedTransport`
takes two optional callbacks for exactly this:

```python
import logging

import httpx
from httpx_hedged import EndpointConfig, HedgeConfig, HedgedTransport

logger = logging.getLogger("myapp.hedging")


def emit_hedge_fired_metric(key: str) -> None:
    statsd_client.incr("http.hedge.fired", tags=[f"endpoint:{key}"])


def alert_on_circuit_open(scope: str, key: str) -> None:
    # scope is "host" or "endpoint"; key is the host name or endpoint key
    # that tripped. Fires exactly once per OPEN transition.
    logger.error(
        "hedging circuit breaker OPEN: %s=%s is unhealthy, hedging suspended", scope, key
    )


transport = HedgedTransport(
    default_config=HedgeConfig(),
    on_hedge_fired=emit_hedge_fired_metric,
    on_circuit_open=alert_on_circuit_open,
)

transport.register("GET", "/api/v1/fast-lookup", EndpointConfig(percentile=0.90))
transport.register("GET", "/api/v1/bulk-export", EndpointConfig(percentile=0.90))

async with httpx.AsyncClient(transport=transport) as client:
    ...
```

`on_hedge_fired` is called with the key each time a hedge request is
actually launched, after the idempotency, circuit-breaker, and budget
gates have all passed, so it only fires for hedges that were genuinely
sent. `on_circuit_open` is called once per OPEN transition (not on every
suppressed hedge while it stays open), so it's safe to wire straight into
an alerting/paging pipeline without flooding it.

> [!IMPORTANT]
> Both callbacks run synchronously on the request path, so keep them fast
> (increment a counter, log a line); don't do network I/O in them directly.

## Relationship to hedge-python

This library is modeled heavily on
[hedge-python](https://github.com/sunhailin-Leo/hedge-python), which
pioneered the DDSketch-based adaptive-hedging approach this project
borrows. hedge-python keys its sketch per host, which works well for a
single-endpoint-per-host use case; this project exists to add per-endpoint
tracking on top of the same core idea, plus a health circuit breaker,
for services that expose many differently-shaped endpoints on one host. See
the filed [upstream issue](https://github.com/sunhailin-Leo/hedge-python/issues/2)
for the motivating scenario.

## References

- [The Tail at Scale](https://research.google/pubs/pub40801/): Google's paper on tail latency
- [DDSketch: A fast and fully-mergeable quantile sketch with relative-error guarantees](https://arxiv.org/abs/2004.08604): Masson et al., VLDB 2019
- [hedge-python](https://github.com/sunhailin-Leo/hedge-python): the project this one is modeled after
- [httpx documentation](https://www.python-httpx.org/)

## License

MIT License. See [LICENSE](LICENSE) file for details.
