Metadata-Version: 2.4
Name: tapbench
Version: 0.1.3
Summary: A small workload benchmark client for TAP asynchronous jobs
Project-URL: Homepage, https://github.com/mjuric/tapbench
Project-URL: Repository, https://github.com/mjuric/tapbench
Project-URL: Issues, https://github.com/mjuric/tapbench/issues
Author-email: Mario Juric <mjuric@uw.edu>
License-Expression: BSD-3-Clause
License-File: LICENSE
Keywords: ADQL,IVOA,TAP,UWS,benchmark
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Astronomy
Requires-Python: >=3.12
Requires-Dist: aiohttp<4,>=3.11
Requires-Dist: rich<15,>=13.9
Description-Content-Type: text/markdown

# tapbench

`tapbench` is a small workload benchmark client for IVOA TAP asynchronous
jobs. It schedules ADQL queries as a one-shot burst or at an offered rate,
follows each admitted UWS
job to a terminal phase, optionally streams its primary result, and reports
latency and throughput. It is intentionally not a general-purpose load-testing
framework.

Python 3.12 or newer is required.

## Install

Install the command from PyPI with either `uv` or `pipx`:

```bash
uv tool install tapbench
# or
pipx install tapbench
```

From a checkout, [uv](https://docs.astral.sh/uv/) can create the development
environment and run the command:

```bash
uv sync
uv run tapbench --help
```

The TAP base URL normally comes from the nonempty `TAPBENCH_URL` environment
variable. The service credentials must already be present as `TAPBENCH_USER`
and `TAPBENCH_PASSWORD`; both must be nonempty. The command does not read dotenv
files and does not accept credentials as command-line options.

## Run a benchmark

```bash
export TAPBENCH_URL=https://example.org/tap
tapbench run examples/mppdb.large_data.sql
```

From a checkout, prefix the command with `uv run`.

With no scheduling options, every positional query is submitted once and all
submissions are offered together at `t=0`. This makes the common shell-glob
case a concurrent one-shot workload:

```bash
uv run --env-file .env.mppdb -- tapbench run mppdb.*.sql
```

The shell expands the glob before `tapbench` runs. Its expansion order becomes
the query order, and the behavior applies equally to any explicit list of query
files.

Each positional file must contain exactly one nonempty ADQL query. Multiple
files may be supplied; opportunities rotate through them in argument order,
including opportunities dropped locally. The queries themselves are never
printed or stored in JSON; the report identifies each by its zero-based index,
file name/path, and SHA-256 digest.

The `run` options are:

| Option | Meaning |
| --- | --- |
| `QUERY_FILE [QUERY_FILE ...]` | One or more readable, nonempty files, each containing one ADQL query. |
| `--url URL` | TAP base URL, overriding `TAPBENCH_URL`. Only HTTP(S) URLs without credentials, query text, or fragments are accepted. |
| `--rate FLOAT` | Pace submission opportunities at this positive, finite rate. Without `--duration`, offer each query exactly once. |
| `--duration FLOAT` | Use fixed-duration scheduling for this many positive, finite seconds. The rate defaults to 1 job/s when omitted. |
| `--max-inflight INTEGER` | Maximum admitted operations whose full local lifecycle has not finished; must be at least one. |
| `--phase-wait INTEGER` | Maximum seconds requested for each adaptive UWS WAIT; defaults to `25`. `0` disables WAIT and preserves the original `/phase` polling request order exactly. |
| `--poll-interval FLOAT` | Approximate interval for fallback `/phase` polling, plus a short anti-spin backoff when WAIT returns repeatedly without a phase change. It does not delay normal detection when WAIT is effective. |
| `--request-timeout FLOAT` | Timeout for one HTTP operation, including a result-body operation or cleanup request. |
| `--job-timeout FLOAT` | Time allowed after job creation to observe a terminal UWS phase. |
| `--download-results [DIRECTORY]` | Stream every completed primary result for measurement. If an existing directory is supplied, also save one representative result per query after the run. Downloading is disabled by default. |
| `--keep-jobs` | Do not delete remote UWS jobs after processing. |
| `--json-output FILE` | Atomically write the detailed `tapbench.run/v5` report. |
| `--seed INTEGER` | Seed polling jitter for reproducible runs. |
| `--progress`, `--no-progress` | Enable or suppress live stderr progress; progress is enabled by default. |

Run `uv run tapbench run --help` for the installed command's defaults.

## Stage packaged examples

Installed wheels contain example families for the `bq` and `mppdb`
deployments. List them or copy one family into a working directory without TAP
credentials or network access:

```bash
tapbench examples list
tapbench examples stage bq queries
tapbench run queries/bq.large_data.sql
```

The destination defaults to the current directory and is created when needed.
Staging preserves the packaged names and preflights every destination; if any
would be overwritten, nothing is copied. Use `--force` to atomically replace
existing files.

## Scheduling behavior

There are three scheduling modes:

- With neither `--rate` nor `--duration`, `tapbench` offers every positional
  query exactly once at `t=0` (`burst_once`). The query count must not exceed
  `--max-inflight`; an oversized burst is rejected before credentials or
  network activity instead of being reported as local drops.
- With `--rate` but no `--duration`, `tapbench` offers every positional query
  exactly once at absolute deadlines `k / rate` (`paced_once`).
- With `--duration`, `tapbench` retains fixed-duration scheduling
  (`fixed_duration`). The rate is the explicit `--rate`, or 1 job/s when it is
  omitted. Queries rotate for every opportunity until the duration ends.

Paced and fixed-duration operation is open-loop. Opportunity `k` has an
absolute deadline of `k / rate` seconds from the monotonic scheduling epoch,
beginning with opportunity zero at `t=0`. In fixed-duration mode, all deadlines
strictly before `duration` are offered. Thus `--rate 10 --duration 10` creates
exactly 100 opportunities even when earlier queries are slow. Polls and result
requests do not count toward the offered rate, and a late event-loop wake-up
does not shift later deadlines. With multiple query files, opportunity `k` uses
query index
`k % number_of_queries`; a local drop still advances this rotation.

An opportunity is admitted only if an in-flight slot is immediately available.
Otherwise it is recorded as `local_scheduling_drop`; it is never queued to run
later. A `--max-inflight` slot is held from submission start through job
creation, phase handling, optional result download, and attempted cleanup.
Consequently, slow downloads or cleanup can legitimately cause later drops.
This bounds local tasks and network work as well as remote nonterminal jobs.

After the scheduling interval, `tapbench` drains the bounded set of admitted
operations. The job timeout covers waiting for a terminal phase, beginning
after successful job creation. Result download and cleanup occur outside that
job deadline and are instead bounded by per-operation request timeouts.

Job creation, job start, phase polling, and remote cleanup treat an explicit
HTTP 429 response as transient server backpressure through one shared retry
policy. `tapbench` honors a valid `Retry-After` delay (seconds or HTTP date),
adds small jitter to avoid synchronized retries, and otherwise uses bounded
jittered exponential backoff. Creation retries remain within the request
timeout, start and polling retries remain within the original job deadline, and
DELETE retries remain within a separate cleanup deadline. Ambiguous POST
timeouts and transport failures are not retried. Each record and the terminal
summary report stage-specific rate-limit counts and cumulative wait times.

By default, phase observation uses bounded UWS 1.1 blocking requests on the
full job resource, such as `GET {job}?WAIT=25&PHASE=EXECUTING`. `tapbench`
first observes the ordinary job document, then sends `PHASE=RUN` exactly once
when that observation is `PENDING`; job creation never includes an inline
`PHASE=RUN`. Each WAIT is shortened when necessary to fit both the remaining
job deadline and the request timeout, including a response grace margin. WAIT
responses are parsed incrementally with a 1 MiB total limit; query text,
parameters, result URLs, owners, and raw XML are never retained.

WAIT support is learned from observed behavior per actual job origin. Ignored,
rejected, malformed, repeatedly immediate/unchanged, or repeatedly timed-out
WAIT requests fall back safely to jittered `/phase` polling. Three distinct-job
negative samples temporarily suppress WAIT for an origin; suppression later
grants one generation-safe probe so transient server behavior cannot disable
WAIT permanently. `Retry-After` applies to both WAIT and fallback polling, and
the absolute job deadline remains authoritative. `HELD`, `SUSPENDED`, and
`UNKNOWN` use fallback polling. `ARCHIVED` is terminal and is reported as
`uws_archived` without requesting a result.

When WAIT reports `COMPLETED`, `tapbench` closes that response and begins result
discovery immediately in the same job coroutine, with no poll sleep, capability
probe, or timing request in between. It does not opportunistically extract a
result URL from the job document: download mode continues to request the
standard `{job}/results/result` resource. Timing fields already delivered with
the terminal phase are reused. If timing is incomplete, the best-effort job
document retrieval runs only after the complete result download and before
cleanup; in no-download mode it may run immediately after terminal observation.
The same download-before-timing ordering applies when fallback polling observes
completion.

UWS `creationTime`, `startTime`, and `endTime` yield query queue duration
(`startTime - creationTime`) and execution duration (`endTime - startTime`).
Missing, malformed, out-of-order, or unavailable timing remains null and does
not change the outcome. A service's timestamp precision limits these
measurements; a whole-second service may report zero for a subsecond query. The
queue duration includes time spent PENDING before RUN, so it is not necessarily
pure database scheduler wait.

Adaptive WAIT changes request load into connection load: an effective service
receives fewer phase requests, while more HTTP connections can be held
concurrently. Connections remain bounded by the run connector limit (about
twice `--max-inflight`), and every job has at most one acquired or queued HTTP
request at a time. Reports expose WAIT and fallback counts so runs remain
interpretable.

## Results and success

By default, a job is successful as soon as creation succeeds and `COMPLETED`
is observed. No result-list, result-resource, redirect-target, or size-discovery
request is made, and all result timing and byte fields are null.

With `--download-results`, a `COMPLETED` job is successful only after
`{job}/results/result` has been completely consumed and no HTTP, TAP, or
VOTable result error was detected. Response-header, first-byte, complete-body,
and submission-to-download timings are recorded along with the byte count. The
body is streamed rather than retained in memory.
Cleanup failure is reported separately and does not change an otherwise
successful operation.

Supplying a directory, as in `--download-results results/`, retains the first
successful job for each query until the benchmark has fully drained. After the
benchmark finish time is captured and progress reporting stops, tapbench fetches
each representative result again and saves it as
`<query-file-stem>.vot`; for example, `bq.large_data.sql` becomes
`bq.large_data.vot`. Query files must have distinct base names when saving.
These extra transfers do not alter the benchmark's per-job measurements,
aggregate byte counts, or elapsed time. The directory must already exist, and
tapbench refuses to overwrite an existing representative file. Each retained
job is then cleaned up normally unless `--keep-jobs` was selected.

Result inspection is deliberately bounded and does not deserialize a complete
VOTable. `tapbench` retains only a small prefix while streaming and detects a
`QUERY_STATUS=ERROR` marker when it appears there. An error marker appearing
only beyond that inspection bound cannot be detected.

Latency summaries ignore missing observations and use nearest-rank p50, p90,
p95, and p99 values: sort the samples and select observation
`ceil(percentile * count)`. An unavailable metric is shown as `unavailable`
and serialized as JSON `null`.

Throughput labels have distinct denominators:

- achieved submission rate is submission starts divided by the configured
  scheduling duration, or by elapsed scheduler-active time after interruption;
- job-creation throughput is created jobs divided by the time from the
  scheduling epoch through the last successful creation response;
- server execution throughput is completed jobs with valid UWS timing divided
  by the server work window from the earliest `startTime` to the latest
  `endTime`; it is independent of client polling and unavailable when the
  server timestamps produce a zero-width window;
- result-completion throughput, when downloading, is successful downloads
  divided by the download window from the first download start through the
  last completion;
- download throughput is successfully downloaded bytes divided by the span
  of that same download window. Download rates are unavailable when there are
  no successful downloads or the span is zero.

No throughput denominator includes phase-poll sleeps or post-completion cleanup.
The report shows how many completed jobs supplied valid UWS timing because
server execution throughput may use fewer samples than the completed count.

## Progress and terminal output

Progress is operational telemetry; final metrics are recomputed from all
records. It is written to stderr so the final human summary remains on stdout.
On an interactive terminal, a Rich progress display refreshes in place:

```text
⠹ Scheduling ━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━  40%  12.0/30.0s
  Completed  ━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  26/300 queries

  workload   offered 121  started 118  dropped 3   active 87  peak 91
  jobs       created 115  completed 26 error 2     aborted 0
  outcome    success 24   failed 5     downloaded 812 MiB
  UWS        polls 412    waits 97     changes 81  fallback polls 4  fallback jobs 2  WAIT disabled 0
  limits     submit 4     start 2      polling 11  cleanup 3
```

Once scheduling ends, the determinate bar becomes a pulsing drain indicator:

```text
⠸ Draining   ━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━  87 active  42.4s elapsed
  Completed  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━  205/300 queries

  workload   offered 300  started 292  dropped 8   active 87  peak 200
  jobs       created 292  completed 205 error 0     aborted 0
  outcome    success 205  failed 8     downloaded 3.1 GiB
  UWS        polls 994    waits 246    changes 246 fallback polls 0  fallback jobs 0  WAIT disabled 0
  limits     submit 0     start 0      polling 0   cleanup 0
```

The green completion bar advances only for jobs reaching UWS `COMPLETED`; its
total is the run's exact planned opportunity count. Drops, errors, and aborts
therefore leave it below 100%. Scheduling state uses cyan, draining state uses
yellow, failures use red, and drops, aborts, fallbacks, and nonzero rate limits
use yellow. Zero alert counts are dimmed, and color is never the only indication
of state. The `downloaded` metric is omitted in no-download mode. When stderr is
not a TTY, the original ANSI-free `tapbench progress:` snapshots are emitted at
most every five seconds, plus one at the schedule-to-drain transition.
`--no-progress` suppresses both forms but not the final summary.

The final stdout report includes opportunity, submission, terminal, success,
failure, HTTP, poll, WAIT, fallback, and byte counts; peak in-flight work; the separate
throughput values; client-observed and UWS server-side latency percentiles; and
a primary-failure breakdown. Its lifecycle section includes:

```text
  phase WAIT requests / changes: 123 / 97
  WAIT unchanged / timeouts:     24 / 2
  fallback jobs / polls:         2 / 4
  WAIT disabled jobs:            0
```

Fallback reasons are also summarized by stable machine value when nonzero:
`request_timeout_too_short`, `nonactive_phase`, `http_rejected`,
`malformed_job_document`, `immediate_unchanged`, `transport_timeout`, and
`origin_suppressed`. Result rows are omitted in no-download mode. Polling
rate-limit totals continue to combine WAIT and `/phase` activity because both
belong to the same lifecycle stage.

## JSON output and exit status

`--json-output` records sanitized configuration (including `phase_wait`), UTC
correlation timestamps, monotonic durations, query identities, aggregates, and
one structured record for every opportunity, including local drops. Its schema
id is `tapbench.run/v5`. The sanitized configuration records `scheduling_mode`
as `burst_once`, `paced_once`, or `fixed_duration`, plus the exact
`opportunity_count` for either once mode (null for fixed-duration mode). The
run-level `queries` array records each query index,
path, and SHA-256 digest; each opportunity record contains the selected
`query_index`. The `saved_results` array reports each post-run destination,
byte count, and any save error.
Enums are strings and unavailable observations are null. Output is flushed and
atomically replaces the destination, so a failed write does not leave a
partially encoded report. Per-job records retain the three raw UWS timestamp
strings alongside the derived queue and execution durations. They also expose
the normalized UWS version; whether WAIT was disabled; physical WAIT, fallback,
and timeout request counters; successfully parsed changed/unchanged WAIT
counters; and the first durable fallback reason. Aggregate fields total these
counters and provide a fallback-reason breakdown. Request counters include
physical 429 attempts, while phase-change/unchanged counters include only
successfully parsed non-429 responses.

Exit codes are:

| Code | Meaning |
| --- | --- |
| `0` | The benchmark completed. Individual query failures and local drops are measurements, not process failures. |
| `2` | Invalid CLI/configuration or missing credentials. No benchmark was started. |
| `3` | Interrupted partial run. |
| `4` | A fatal run-level client or output error prevented a valid run. |

On the first SIGINT or SIGTERM, scheduling stops immediately, active operations
are cancelled, and already-created remote jobs receive bounded best-effort
cleanup unless `--keep-jobs` was selected. Accumulated records are retained,
the partial summary and requested JSON are produced, and the command exits 3.
A second interruption may bypass cleanup and output.

## Security

Basic authentication is applied explicitly to requests on the configured TAP
origin. Redirects may cross origins so services can return results from object
storage, but cross-origin requests never receive the TAP Authorization header.
URLs containing userinfo or using unsupported schemes are rejected. Displayed
and serialized URLs omit userinfo, query strings, and fragments, and job
identifiers are redacted. Errors are bounded and scrubbed of credential values.
Neither credentials, authorization headers, query text, nor complete job URLs
belong in terminal output or JSON.

Never commit local credential files. In particular, `.env.mppdb` and backup
variants are local secrets and are ignored by this repository.

## APDB example queries

The queries in [`examples/`](examples/) use table and column names from the
[published Rubin APDB schema](https://sdm-schemas.lsst.io/apdb.html):

- `PREFIX.fast_lookup.sql` returns at most ten current object summaries;
- `PREFIX.selective_join.sql` returns at most 25 reliable r-band detections
  joined to their current object positions;
- `PREFIX.expensive_small_result.sql` scans and aggregates source fluxes but returns
  only one row per band.

The `mppdb.*.sql` family uses unqualified development-server table names. The
`bq.*.sql` family qualifies tables with the `ppdb` schema used by the integration
deployment. All SQL files are included in source and wheel distributions under
`tapbench/examples/`; `tapbench examples stage` accesses them through package
resources without relying on an installation-specific filesystem path.

`tapbench` does not discover or rewrite schemas at runtime. Select the family
whose qualification matches the target deployment, or stage and edit a local
copy.

For a safe live check, first run each example once at a low rate with a single
in-flight slot and normal cleanup. Then perform a short no-download benchmark:

```bash
uv run tapbench run \
  --url https://mppdb.juriclab.org \
  --rate 1 \
  --duration 3 \
  --max-inflight 1 \
  examples/mppdb.fast_lookup.sql
```

Do not use `--keep-jobs` for routine validation. Confirm the report contains
three opportunities and no sensitive URL or credential material.

## Development

The test suite uses a deterministic local fake TAP service; live service
behavior is not a replacement for those tests. Run all checks with:

```bash
uv run ruff format --check .
uv run ruff check .
uv run mypy src
uv run pytest
```

## Releasing

Package versions are derived from Git tags. Releases use clean tags of the
form `vX.Y.Z`; for example, tag `v0.1.0` produces package version `0.1.0`.
Untagged commits produce development versions and are never published.

Pushing a release tag builds and validates the wheel and source distribution,
prints their complete file manifests, and saves them as workflow artifacts. It
does not publish anything. Review the artifact names, hashes, and manifests,
then create a draft GitHub Release for that tag.

Publishing the GitHub Release is the explicit production approval. It rebuilds
and validates the tagged source, publishes the distributions to PyPI through
Trusted Publishing, and verifies a fresh installation from PyPI. The publishing
workflow stores no PyPI token and is restricted to the `pypi` environment.

Before the first release, create a pending Trusted Publisher on PyPI with
project `tapbench`, owner `mjuric`, repository `tapbench`, workflow
`publish.yml`, and environment `pypi`. Configure that GitHub environment to
permit only tags matching `v*`.
