Metadata-Version: 2.4
Name: benchloop_client
Version: 0.2.5
Summary: Benchloop CLI — command line tool for benchmark analysis data (metrics, trajectory, reports, artifacts).
Requires-Python: >=3.10
Requires-Dist: httpx>=0.25.2
Requires-Dist: tomli>=1.1.0; python_version < '3.13'
Requires-Dist: typer>=0.9.4
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# benchloop-client (`bl`, `bmcli`)

Command line tool for fetching and downloading benchmark analysis data
(run metrics, trajectory, pre-analysis reports, task/job lists, and artifacts)
by `group_id` / `job_id`.

> PyPI package name is `benchloop_client`; the installed commands are `bl` and `bmcli`.

## Install package

```
pip install benchloop_client

bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID

# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

bl overview get -j $JOB_ID
bl metrics get -j $JOB_ID

```

## Install and run from source

```bash
# with uv (recommended)
uv sync
uv run bl --help

# or editable install with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .
bl --help
```

## Auth & config

Authentication uses a personal api key sent as the `X-API-Key` header.

Precedence: CLI flag > env var > `~/.benchloop/config.toml` > default.

- Env: `BENCHLOOP_API_KEY` (or `BENCHMARK_API_KEY`), `BENCHLOOP_HOST`, `BENCHLOOP_OUTPUT_DIR`
- Default host (prod): `https://benchmark-analysis.alibaba-inc.com`
- Pre host: `https://pre-benchmark-analysis.alibaba-inc.com`

```bash
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com
```

## Usage

Two-segment `bl <resource> <action>` form. `get` prints a JSON array to
stdout (pipeable to `jq`); `download` writes the standard file layout.

```bash
bl metrics get -j J123 | jq '.[0]'
bl metrics download -g G1 --scope all -o ./out
bl trajectory download -g G1,G2 -o ./out
bl trajectory download --task-id TASK_ID -o ./out
bl report download -j J123 -o ./out
bl jobs get -g G1 --all          # get supports --all (auto-paginate)
bl jobs download -g G1           # download is always full
bl artifacts download -g G1 -o ./out
bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out
```

IDs accept repeated flags, comma-separated values, and `-` to read from stdin.
`-g` and `-j` are mutually exclusive in a single command. For trajectory
downloads, `--task-id` is a third mutually exclusive input mode.

### Download filtered trajectories by task ID

The benchmark analysis Jobs page can turn the current filters into a trajectory
download task. Copy the command shown by the page and run it locally:

```bash
bl trajectory download --task-id TASK_ID -o ./out
```

The CLI fetches the task's complete job ID list, downloads job trajectories with
fixed concurrency, and prints count/ETA progress to stderr. It creates
`./out/TASK_ID_trajectories.zip`, containing one `<jobId>_trajectory.json` per
successful job and a `manifest.json` with total, success, and failure details.
If only some jobs fail, the successful files are still archived and the command
exits with code `1`. If every job fails, no empty ZIP is created.

## Exit codes

| code | meaning |
| ---- | ------- |
| 0 | all success |
| 1 | partial failure (some ids failed) |
| 2 | usage / parameter error |
| 3 | auth failure (HTTP 401) |
| 4 | network / timeout |

stdout carries data only; logs and progress go to stderr. `--quiet` suppresses
non-error logs, `--verbose` adds debug logs.


## Release new package

```
uv build
twine check dist/benchloop_client-0.1.7-py3-none-any.whl
twine upload dist/benchloop_client-0.1.7-py3-none-any.whl --verbose
```
