Metadata-Version: 2.4
Name: tapbench
Version: 0.1.0
Summary: A small fixed-rate benchmark client for TAP asynchronous jobs
Project-URL: Homepage, https://github.com/mjuric/tapbench
Project-URL: Repository, https://github.com/mjuric/tapbench
Project-URL: Issues, https://github.com/mjuric/tapbench/issues
Author-email: Mario Juric <mjuric@uw.edu>
License-Expression: BSD-3-Clause
License-File: LICENSE
Keywords: ADQL,IVOA,TAP,UWS,benchmark
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Astronomy
Requires-Python: >=3.12
Requires-Dist: aiohttp<4,>=3.11
Description-Content-Type: text/markdown

# tapbench

`tapbench` is a small fixed-rate benchmark client for IVOA TAP asynchronous
jobs. It schedules one ADQL query at an offered rate, follows each admitted UWS
job to a terminal phase, optionally streams its primary result, and reports
latency and throughput. It is intentionally not a general-purpose load-testing
framework.

Python 3.12 or newer is required.

## Install

Install the command from PyPI with either `uv` or `pipx`:

```bash
uv tool install tapbench
# or
pipx install tapbench
```

From a checkout, [uv](https://docs.astral.sh/uv/) can create the development
environment and run the command:

```bash
uv sync
uv run tapbench --help
```

The service credentials must already be present in the process environment as
`TAPBENCH_USER` and `TAPBENCH_PASSWORD`. Both must be nonempty. The command does
not read dotenv files and does not accept credentials as command-line options.

## Run a benchmark

```bash
uv run tapbench run \
  --url https://example.org/tap \
  --query-file examples/fast_lookup.sql \
  --rate 20 \
  --duration 60 \
  --max-inflight 200 \
  --poll-interval 0.5 \
  --no-download-results
```

`--url` is the TAP base URL; `tapbench` appends `/async`. `--query-file` must
contain exactly the ADQL query offered at every opportunity. The query itself
is never printed or stored in JSON; the report identifies it by file name/path
and SHA-256 digest.

The `run` options are:

| Option | Meaning |
| --- | --- |
| `--url URL` | TAP base URL. Only HTTP(S) URLs without credentials, query text, or fragments are accepted. |
| `--query-file FILE` | Readable, nonempty file containing one ADQL query. |
| `--rate FLOAT` | Submission opportunities per second; must be positive and finite. |
| `--duration FLOAT` | Time during which new opportunities are scheduled; must be positive and finite. |
| `--max-inflight INTEGER` | Maximum admitted operations whose full local lifecycle has not finished; must be at least one. |
| `--poll-interval FLOAT` | Approximate phase-poll interval, with reproducible small jitter. |
| `--request-timeout FLOAT` | Timeout for one HTTP operation, including a result-body operation or cleanup request. |
| `--job-timeout FLOAT` | Time allowed after job creation to observe a terminal UWS phase. |
| `--download-results`, `--no-download-results` | Stream completed primary results, or stop after observing `COMPLETED`; downloading is the default. |
| `--keep-jobs` | Do not delete remote UWS jobs after processing. |
| `--json-output FILE` | Atomically write the detailed `tapbench.run/v1` report. |
| `--seed INTEGER` | Seed polling jitter for reproducible runs. |
| `--progress`, `--no-progress` | Enable or suppress live stderr progress; progress is enabled by default. |

Run `uv run tapbench run --help` for the installed command's defaults.

## Fixed-rate behavior

This is an open-loop offered-rate benchmark. Opportunity `k` has an absolute
deadline of `k / rate` seconds from the monotonic scheduling epoch, beginning
with opportunity zero at `t=0`. All deadlines strictly before `duration` are
offered. Thus `--rate 10 --duration 10` creates exactly 100 opportunities even
when earlier queries are slow. Polls and result requests do not count toward
the offered rate, and a late event-loop wake-up does not shift later deadlines.

An opportunity is admitted only if an in-flight slot is immediately available.
Otherwise it is recorded as `local_scheduling_drop`; it is never queued to run
later. A `--max-inflight` slot is held from submission start through job
creation, phase handling, optional result download, and attempted cleanup.
Consequently, slow downloads or cleanup can legitimately cause later drops.
This bounds local tasks and network work as well as remote nonterminal jobs.

After the scheduling interval, `tapbench` drains the bounded set of admitted
operations. The job timeout covers waiting for a terminal phase, beginning
after successful job creation. Result download and cleanup occur outside that
job deadline and are instead bounded by per-operation request timeouts.

Job creation, job start, phase polling, and remote cleanup treat an explicit
HTTP 429 response as transient server backpressure through one shared retry
policy. `tapbench` honors a valid `Retry-After` delay (seconds or HTTP date),
adds small jitter to avoid synchronized retries, and otherwise uses bounded
jittered exponential backoff. Creation retries remain within the request
timeout, start and polling retries remain within the original job deadline, and
DELETE retries remain within a separate cleanup deadline. Ambiguous POST
timeouts and transport failures are not retried. Each record and the terminal
summary report stage-specific rate-limit counts and cumulative wait times.

After observing a terminal phase, `tapbench` retrieves the standard UWS job
document before cleanup and reads `creationTime`, `startTime`, and `endTime`.
It reports query queue duration as `startTime - creationTime` and query
execution duration as `endTime - startTime`. Timing retrieval is best effort:
missing, malformed, or unavailable fields remain null and do not change the
job outcome. A service's timestamp precision limits these measurements; a
whole-second service may report zero for a subsecond query. The queue duration
also includes any time the job remained PENDING before the client sent RUN, so
it is not necessarily pure database scheduler wait.

## Results and success

By default, a `COMPLETED` job is successful only after
`{job}/results/result` has been completely consumed and no HTTP, TAP, or
VOTable result error was detected. Response-header, first-byte, complete-body,
and submission-to-download timings are recorded along with the byte count. The
body is streamed rather than retained in memory.

With `--no-download-results`, a job is successful as soon as creation succeeds
and `COMPLETED` is observed. No result-list, result-resource, redirect-target,
or size-discovery request is made, and all result timing and byte fields are
null. Cleanup failure is reported separately and does not change an otherwise
successful operation.

Result inspection is deliberately bounded and does not deserialize a complete
VOTable. `tapbench` retains only a small prefix while streaming and detects a
`QUERY_STATUS=ERROR` marker when it appears there. An error marker appearing
only beyond that inspection bound cannot be detected.

Latency summaries ignore missing observations and use nearest-rank p50, p90,
p95, and p99 values: sort the samples and select observation
`ceil(percentile * count)`. An unavailable metric is shown as `unavailable`
and serialized as JSON `null`.

Throughput labels have distinct denominators:

- achieved submission rate is submission starts divided by the configured
  scheduling duration, or by elapsed scheduler-active time after interruption;
- job-creation throughput is created jobs divided by the time from the
  scheduling epoch through the last successful creation response;
- server execution throughput is completed jobs with valid UWS timing divided
  by the server work window from the earliest `startTime` to the latest
  `endTime`; it is independent of client polling and unavailable when the
  server timestamps produce a zero-width window;
- result-completion throughput, when downloading, is successful downloads
  divided by the download window from the first download start through the
  last completion;
- download throughput is successfully downloaded bytes divided by the span
  of that same download window. Download rates are unavailable when there are
  no successful downloads or the span is zero.

No throughput denominator includes phase-poll sleeps or post-completion cleanup.
The report shows how many completed jobs supplied valid UWS timing because
server execution throughput may use fewer samples than the completed count.

## Progress and terminal output

Progress is operational telemetry; final metrics are recomputed from all
records. It is written to stderr so the final human summary remains on stdout.
On a terminal, one line refreshes about once per second:

```text
[schedule  12.0/60.0s] offered=241 started=238 dropped=3 active=87 created=235 completed=141 error=2 aborted=0 success=139 failed=5 polls=812 submission_rate_limited=4 start_rate_limited=2 rate_limited=11 cleanup_rate_limited=3 downloaded=812 MiB
```

Once scheduling ends, the label changes rather than implying submissions are
continuing:

```text
[drain  72.4s] offered=1200 started=1184 dropped=16 active=87 created=1179 completed=1068 error=9 aborted=1 success=1059 failed=26 polls=6312 submission_rate_limited=18 start_rate_limited=9 rate_limited=47 cleanup_rate_limited=12 downloaded=3.1 GiB
```

The `downloaded` field is omitted in no-download mode, and stage-specific rate
limit fields are omitted until nonzero. When stderr is not a TTY, searchable
`tapbench progress:` snapshots are emitted at most every five seconds, plus one
at the schedule-to-drain transition. `--no-progress` suppresses all snapshots
but not the final summary.

The final stdout report includes opportunity, submission, terminal, success,
failure, HTTP, poll, and byte counts; peak in-flight work; the separate
throughput values; client-observed and UWS server-side latency percentiles; and
a primary-failure breakdown. Result rows are omitted in no-download mode.

## JSON output and exit status

`--json-output` records sanitized configuration, UTC correlation timestamps,
monotonic durations, query identity, aggregates, and one structured record for
every opportunity, including local drops. Its schema id is `tapbench.run/v1`.
Enums are strings and unavailable observations are null. Output is flushed and
atomically replaces the destination, so a failed write does not leave a
partially encoded report. Per-job records retain the three raw UWS timestamp
strings alongside the derived queue and execution durations.

Exit codes are:

| Code | Meaning |
| --- | --- |
| `0` | The benchmark completed. Individual query failures and local drops are measurements, not process failures. |
| `2` | Invalid CLI/configuration or missing credentials. No benchmark was started. |
| `3` | Interrupted partial run. |
| `4` | A fatal run-level client or output error prevented a valid run. |

On the first SIGINT or SIGTERM, scheduling stops immediately, active operations
are cancelled, and already-created remote jobs receive bounded best-effort
cleanup unless `--keep-jobs` was selected. Accumulated records are retained,
the partial summary and requested JSON are produced, and the command exits 3.
A second interruption may bypass cleanup and output.

## Security

Basic authentication is applied explicitly to every TAP/UWS request. Job,
phase, error, result, cleanup, and redirect URLs must remain on the configured
TAP origin; userinfo and cross-origin redirects are rejected. Displayed and
serialized URLs omit userinfo, query strings, and fragments, and job identifiers
are redacted. Errors are bounded and scrubbed of credential values. Neither
credentials, authorization headers, query text, nor complete job URLs belong in
terminal output or JSON.

Never commit local credential files. In particular, `.env.mppdb` and backup
variants are local secrets and are ignored by this repository.

## APDB example queries

The queries in [`examples/`](examples/) use table and column names from the
[published Rubin APDB schema](https://sdm-schemas.lsst.io/apdb.html):

- `fast_lookup.sql` returns at most ten current object summaries;
- `selective_join.sql` returns at most 25 reliable r-band detections joined to
  their current object positions;
- `expensive_small_result.sql` scans and aggregates source fluxes but returns
  only one row per band.

The SQL files are included in both source and wheel distributions. In an
installed wheel they reside under the `tapbench/examples/` package directory;
the repository copies remain the most convenient paths for command-line use.

They intentionally use unqualified table names, as shown by the published APDB
schema. If a deployment exposes APDB tables only under a TAP schema qualifier,
make a local copy and qualify both table names for that deployment. `tapbench`
does not discover or rewrite schemas at runtime.

For a safe live check, first run each example once at a low rate with a single
in-flight slot and normal cleanup. Then perform a short no-download benchmark:

```bash
uv run tapbench run \
  --url https://mppdb.juriclab.org \
  --query-file examples/fast_lookup.sql \
  --rate 1 \
  --duration 3 \
  --max-inflight 1 \
  --no-download-results
```

Do not use `--keep-jobs` for routine validation. Confirm the report contains
three opportunities and no sensitive URL or credential material.

## Development

The test suite uses a deterministic local fake TAP service; live service
behavior is not a replacement for those tests. Run all checks with:

```bash
uv run ruff format --check .
uv run ruff check .
uv run mypy src
uv run pytest
```

## Releasing

Package versions are derived from Git tags. Releases use clean tags of the
form `vX.Y.Z`; for example, tag `v0.1.0` produces package version `0.1.0`.
Untagged commits produce development versions and are never published.

Pushing a release tag builds and validates the wheel and source distribution,
prints their complete file manifests, and saves them as a workflow artifact.
Publishing first pauses at the protected `testpypi` environment. After the
TestPyPI package is verified, production publishing pauses separately at the
protected `pypi` environment. Review the artifact names, hashes, and manifests
before approving either environment; the release workflow uses Trusted
Publishing and stores no PyPI token.

Before the first release, create pending Trusted Publishers on TestPyPI and
PyPI with project `tapbench`, owner `mjuric`, repository `tapbench`, workflow
`publish.yml`, and the corresponding `testpypi` or `pypi` environment. Configure
both GitHub environments with required reviewers so neither upload can begin
without explicit approval.
