Metadata-Version: 2.4
Name: dvbench
Version: 0.0.0
Requires-Dist: cryptography>=46,<47
Requires-Dist: blake3>=1,<2
Requires-Dist: cbor2>=5,<6
Requires-Dist: requests>=2,<3
Requires-Dist: tqdm>=4,<5
Requires-Dist: zkdv
Requires-Dist: flatbuffers>=25,<26 ; extra == 'jax'
Requires-Dist: jax>=0.10.2,<0.11 ; extra == 'jax'
Requires-Dist: zkdv[jax] ; extra == 'jax'
Requires-Dist: zkdv[torch] ; extra == 'torch'
Provides-Extra: jax
Provides-Extra: torch
Summary: Python client for dvbench
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# dvbench client

The dvbench Python client creates named experiments, tails a live ZKDV `records.cborl` transcript, and sends each
complete CBOR item in a CBOR transport envelope. Records above the shared chunk
threshold use multiple acknowledged messages and are reassembled before worker
validation. The client does not interpret or validate records. Record shape, cryptographic, and
protocol validation are entirely the server's responsibility.

The process uses placeholder `user_id` zero and generates a `client_id` at
startup. Each `record_id` is the BLAKE3 digest of the exact encoded record
bytes, and the first record's ID is the stable `transcript_id`. Record events
are numbered from zero. On SIGINT or SIGTERM, the client sends a `Close` event
whose sequence is the next expected value and waits for its acknowledgement.
A failed record send is retried in memory; restarting the client replays the
transcript under a new `client_id`.

The public lifecycle is:

```python
from dvbench import Client

experiment = Client().experiment("owner/model")
proof = experiment.proof(backend="jax")
# Train through the ordinary ZKDV API, then close the proof before measuring.
proof.finish()

measurement = experiment.measure(checkpoint)

@measurement.evaluate(params=0, batch=1)
def evaluate(params, batch):
    return score(params, batch)

@measurement.aggregate
def aggregate(outputs):
    return sum(outputs) / len(outputs)

measurement.submit("accuracy", measurement.run(checkpoint, batches))
```

Measurement definitions and benchmarks use the same client rather than a
separate HTTP integration:

```python
accuracy = client.define_measurement("accuracy", "Held-out exact-match accuracy.")
benchmark = client.create_benchmark(
    "Core evaluations",
    "Primary held-out evaluation suite.",
    [accuracy["id"]],
    accuracy["id"],
)
```

Checkpoint stores and mock generators that already have the compact inner
Merkle root can use `experiment.measure_opening(inner_root)`. If an experiment
has more than one run, pass the intended `transcript_id`. The projected leaves
are never sent to the API or retained in SQLite.

`measure(checkpoint)` downloads the selected run's probe-vector object in chunks,
verifies and caches its object hash for the client lifetime, then projects and
hashes one leaf at a time. The worker accepts the compact opening only when it
matches an attested update boundary. Evaluation outputs remain submitter-reported;
the client signs their trace but does not attest their execution.

`measurement.run(...)` serializes each complete input batch and output tensor
as canonical, content-addressed CBOR. `measurement.submit(...)` streams those
objects to the API before sending the small signed trace of digest/length
references. The worker, rather than the client, derives sample alignment from
the batch dimension. Uploads are bounded-concurrent and content-deduplicated
for the client session. Temporary client objects are removed after submission.

The native uploader uses these environment variables when configured directly:

- `ZKDV_RECORDS_PATH` — transcript path; defaults to
  `../zkdv/proof/records.cborl` when run from the dvbench root.
- `DVBENCH_STREAM_URL` — WebSocket destination.
- `DVBENCH_CLIENT_POLL_MS` — file polling interval; defaults to `100`.
- `DVBENCH_CLIENT_RECONNECT_MS` — reconnect delay; defaults to `1000`.

## Python development

Build and install the mixed Python/Rust package into its project environment with:

```sh
uv sync --all-extras --dev
```

The Maturin backend declared in `pyproject.toml` builds and installs the native
uploader and protocol helpers as `dvbench._core`.

Build the distributable source archive and platform wheel with the standard
PEP 517 path:

```sh
uv build
```

## Documentation

Build or preview the Zensical site through the project environment:

```sh
uv run --group docs zensical build --strict
uv run --group docs zensical serve
```

