Metadata-Version: 2.5
Name: sonde
Version: 0.2.0
Summary: Probe any HTTP API for its rate limits, burst ceiling, and full-scrape time.
Project-URL: Homepage, https://github.com/Jartan-LLC/sonde
Project-URL: Repository, https://github.com/Jartan-LLC/sonde
Project-URL: Issues, https://github.com/Jartan-LLC/sonde/issues
Project-URL: Changelog, https://github.com/Jartan-LLC/sonde/blob/main/CHANGELOG.md
Author: Jartan LLC
License-Expression: MIT
License-File: LICENSE
Keywords: api,benchmark,cli,http,rate-limit,scraping
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: httpx<1,>=0.24
Requires-Dist: requests<3,>=2.28
Provides-Extra: dev
Requires-Dist: pyright==1.1.414; extra == 'dev'
Requires-Dist: pytest<10,>=7.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: furo>=2025.12.19; extra == 'docs'
Requires-Dist: myst-parser<6,>=5.1.0; extra == 'docs'
Requires-Dist: sphinx<10,>=9.1.0; extra == 'docs'
Description-Content-Type: text/markdown

# Sonde

[![PyPI](https://img.shields.io/pypi/v/sonde)](https://pypi.org/project/sonde/)
[![CI](https://github.com/Jartan-LLC/sonde/actions/workflows/ci.yml/badge.svg)](https://github.com/Jartan-LLC/sonde/actions/workflows/ci.yml)
[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/Jartan-LLC/sonde/badge)](https://scorecard.dev/viewer/?uri=github.com/Jartan-LLC/sonde)
[![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/Jartan-LLC/sonde/blob/main/LICENSE)

Probe any HTTP API for its rate limits, burst ceiling, and full-scrape time. Provider-pluggable, safe by default.

## Install

Requires Python 3.12+.

```bash
pip install sonde
```

From source:

```bash
git clone https://github.com/Jartan-LLC/sonde.git
cd sonde
pip install -e .
```

For Docker, see [Docker](#docker) below.

## Quick Start

Probe the Roblox asset-owners endpoint:

```bash
export ROBLOX_COOKIE="your_roblosecurity_cookie"
sonde asset-owners --asset-id 20573078 --total-items 1470000
```

Probe GitHub stargazers:

```bash
export GITHUB_TOKEN="ghp_..."
sonde github-stargazers --owner Jartan-LLC --repo sonde
```

Without `GITHUB_TOKEN`, sonde probes anonymously, at GitHub's lower rate limit.

Results are written to `sonde_report.json` by default:

```bash
sonde asset-owners --asset-id 20573078 --output my_report.json
```

## How It Works

Sonde runs five phases against the target endpoint, then combines the measurements into a safe rate estimate.

| Phase | What it does |
|---|---|
| **Sanity** | One request. Validates auth, reads rate-limit response headers (e.g. `x-ratelimit-limit`, `x-ratelimit-remaining`), and records items-per-page for the scrape-time estimate. |
| **Sequential** | Fires back-to-back requests (up to `--seq-cap`, default 150) until the first 429 or the cap. Measures baseline throughput and how many requests the API allows before throttling. |
| **Burst** | Fires N truly-concurrent requests (default sizes: 10, 20, 40, 80) via httpx on a single asyncio event loop. After the first throttled burst, measures the **recovery window** -- how long until requests succeed again -- via adaptive geometric backoff. |
| **Sweep** | Drains the rate-limit bucket, then paces requests at progressively faster intervals (default: 8s down to 0.15s) to find the fastest sustainable interval from empty. Skipped by default when authoritative rate-limit headers are present (override with `--force-sweep`). |
| **Estimate** | Combines all measurements into a recommended request interval and, if a total item count is known, a wall-clock full-scrape estimate. |

### How the estimate is produced

The estimate phase uses a priority ladder to determine the safe rate:

1. **Authoritative headers** -- If the API returned `x-ratelimit-limit` and a window, use those directly (e.g. 100 requests per 60s).
2. **Swept floor** -- If the sweep found a fastest sustainable interval, use that.
3. **Token-bucket inference** -- If burst results show a clean burst size and a measured recovery window, infer the bucket rate.
4. **Sequential fallback** -- Use the observed sequential throughput before the first 429.
5. **No-throttle fallback** -- If nothing ever throttled, no ceiling was found, so fall back to a conservative fraction of the measured sequential throughput.

Every rung applies the safety margin (default 80%, configurable with `--margin`) -- the recommended pace is ~25% slower than the measured ceiling. Rung 5 has no measured ceiling, so it applies an extra 0.5 factor on top (~40% of observed throughput at the default margin).

## Endpoints

### asset-owners

Roblox `inventory.roblox.com/v2/assets/{id}/owners` -- paginated list of owners of a collectible asset.

| Option | Required | Default | Description |
|---|---|---|---|
| `--asset-id` | Yes | -- | Asset ID to probe (e.g. `20573078`) |
| `--sort-order` | No | Asc | `Asc` or `Desc` |
| `--page-size` | No | 100 | Items per page (capped at 100) |
| `--total-items` | No | None | Known total owners, for wall-clock estimate |

**Auth:** Set `ROBLOX_COOKIE` (legacy web-session) and/or `ROBLOX_BEARER` (Open Cloud) environment variables.

### github-stargazers

GitHub `api.github.com/repos/{owner}/{repo}/stargazers` -- users who starred a repository.

| Option | Required | Default | Description |
|---|---|---|---|
| `--owner` | Yes | -- | Repository owner or organization (e.g. `Jartan-LLC`) |
| `--repo` | Yes | -- | Repository name (e.g. `sonde`) |
| `--page-size` | No | 100 | Items per page (capped at 100) |
| `--total-items` | No | None | Known stargazer count, for wall-clock estimate |

**Auth:** Set `GITHUB_TOKEN` environment variable. Without it, you get the anonymous rate limit (60 requests/hour).

## Adding an Endpoint

Write the endpoint in your own package. Once the package is installed alongside sonde, the endpoint is a `sonde` subcommand.

1. Subclass `Endpoint`, set a unique `name` (the subcommand) and a one-line `help`, and implement `build_request(cursor)` and `parse_page(response)`.
2. Override `make_provider()` to return the API's `Provider`: the generic one handles standard 200/429 and IETF headers, and `GitHubProvider` and `RobloxProvider` are built in. A provider that sends credentials in headers lists them in `credentials()`, so logs redact them; query-parameter credentials go in its `auth_params()`, not in an endpoint's request parameters.
3. Optionally implement `total_items()` for scrape-time estimates, `add_arguments()` / `from_args()` for CLI options, and `extra_headers()` for endpoint-specific headers.
4. If the endpoint is paginated, call `add_pagination_args(parser, page_max=...)` in `add_arguments()` and `pagination_from_args(args, page_max=...)` in `from_args()`, with the API's largest page size, so it gets the shared `--page-size` / `--total-items` flags (clamped to that size).
5. Declare the class in your package's `pyproject.toml`:

   ```toml
   [project.entry-points."sonde.endpoints"]
   my-endpoint = "my_package.endpoints:MyEndpoint"
   ```

Import what you need from `sonde`: its exports are the plugin API. While sonde is 0.x, a minor release can change them, and the changelog says how.

Minimal example:

```python
from typing import Any, override

from sonde import Endpoint, PageResult, RequestSpec


class MyEndpoint(Endpoint):
    name = "my-endpoint"
    help = "one-line description for --help"

    @override
    def build_request(self, cursor: Any) -> RequestSpec:
        return RequestSpec(url="https://api.example.com/items", params={"page": cursor or 1})

    @override
    def parse_page(self, response: Any) -> PageResult:
        data = response.json()
        return PageResult(count=len(data["items"]), next_cursor=data.get("next_page"))
```

A built-in endpoint lives in `src/sonde/endpoints/` instead, decorated with `@register` and imported in that package's `__init__.py`.

## CLI Reference

Common options shared by all endpoints:

| Option | Default | Description |
|---|---|---|
| `--max-requests` | 1200 | Hard global cap across all phases (safety budget) |
| `--seq-cap` | 150 | Max sequential requests before stopping |
| `--skip-burst` | off | Skip the concurrent burst phase |
| `--burst-sizes` | `10,20,40,80` | Comma-separated list of concurrent burst sizes |
| `--burst-cooldown` | 60.0 | Fallback seconds between bursts if the recovery window can't be measured |
| `--recovery-step` | 0.25 | Initial poll delay when measuring the throttle window (grows geometrically) |
| `--recovery-max` | 90.0 | Give up measuring the window after this many seconds |
| `--recovery-polls` | 15 | Max polls during recovery measurement |
| `--skip-sweep` | off | Skip the sustained-interval sweep phase |
| `--force-sweep` | off | Run the sweep even when authoritative rate-limit headers are present |
| `--sweep-intervals` | `8,5,3,2,1.2,0.6,0.3,0.15` | Inter-request intervals (seconds) to test, slow to fast |
| `--sweep-count` | 20 | Paced requests per interval after draining |
| `--sweep-drain` | 500 | Cap on rapid requests used to empty the bucket before each interval |
| `--sweep-tolerance` | 0.1 | Max fraction of 429s for an interval to count as sustainable |
| `--margin` | 0.8 | Safety margin: pace at 80% of the measured max rate (0.8 = 25% slower than ceiling) |
| `--output` | `sonde_report.json` | Path for the JSON report (use `-` for stdout) |
| `-v` / `--verbose` | off | Show per-request detail (sets log level to DEBUG) |
| `-q` / `--quiet` | off | Only show warnings and errors (sets log level to WARNING) |
| `--log-format` | `plain` | Log line format: `plain` (message-only) or `json` (structured) |

`-v` and `-q` are mutually exclusive. Logs always go to stderr; the report goes to `--output`.

**Exit codes:** `0` success, `2` precondition failure (bad arguments, an endpoint plugin that won't load, unwritable `--output`, or the endpoint returned no usable response), `1` unexpected crash, `130` interrupted.

### Piping and machine-readable output

Use `--output -` to write the JSON report to stdout instead of a file. Combine with `-q` to suppress INFO-level log noise on stderr:

```bash
sonde asset-owners --asset-id 20573078 --output - -q | jq .estimate
```

Use `--log-format json` for structured log lines on stderr (keys: `timestamp`, `level`, `logger`, `message`, plus `exc` on error lines), useful for log aggregators or CI pipelines:

```bash
sonde asset-owners --asset-id 20573078 --log-format json 2>sonde.log
```

## Docker

Build:

```bash
docker build -t sonde .
```

Run (mount current directory so the report lands on the host):

```bash
docker run --rm -v "$(pwd):/data" -e ROBLOX_COOKIE sonde \
    asset-owners --asset-id 20573078 --total-items 1470000
```

```bash
docker run --rm -v "$(pwd):/data" -e GITHUB_TOKEN sonde \
    github-stargazers --owner Jartan-LLC --repo sonde
```

The container writes `sonde_report.json` to `/data` (the mounted volume).

## Development

Setup and the checks to run before a pull request are in
[CONTRIBUTING.md](https://github.com/Jartan-LLC/sonde/blob/main/CONTRIBUTING.md).

## License

[MIT](https://github.com/Jartan-LLC/sonde/blob/main/LICENSE)
