Current Glimmer deployment comparison
Fresh matched runs of the three active deployments. Every coding task was attempted 3 times; failures and deadline hits are included in elapsed time. Model weights, serving configuration and harness settings were not tuned during the run.
Run interval: 2026-09-07T04:37:25+00:00 through 2026-09-07T04:58:01+00:00.
Highest observed coding pass count: Luyao H100, Your H100 (15/18). Fastest equal-work serving result: Your H100. These are descriptive results from this run, not a statistical ranking.
Coding performance
| Deployment | Original tasks | Holdouts | All tasks | Deadline hits | Total task time | Median task time | Output tokens |
|---|
| Luyao H100 | 12/12 | 3/6 | 15/18 | 1 | 1103.5 s | 42.0 s | 107,043 |
| Your H100 | 12/12 | 3/6 | 15/18 | 0 | 489.9 s | 22.8 s | 82,426 |
| Your A100 INT4 | 12/12 | 1/6 | 13/18 | 0 | 844.5 s | 33.7 s | 101,135 |
Task detail
| Task | Luyao H100 | Your H100 | Your A100 INT4 |
|---|
| dependency_planner | 3/3 | 3/3 | 3/3 |
| ledger | 3/3 | 3/3 | 3/3 |
| object_key_manifest | 3/3 | 3/3 | 1/3 |
| sqlite_migration | 3/3 | 3/3 | 3/3 |
| streaming_csv | 0/3 | 0/3 | 0/3 |
| ttl_cache | 3/3 | 3/3 | 3/3 |
| Deployment | Model requests | Tool calls | Reasoning tokens | Median first source edit |
|---|
| Luyao H100 | 197 | 221 | 81,730 | 25.70 s |
| Your H100 | 180 | 205 | 58,086 | 13.38 s |
| Your A100 INT4 | 209 | 228 | 71,118 | 22.73 s |
Serving speed
24 requests per deployment have identical input text, input token count and 512 generated tokens. Prefix first-use and repeat results are pooled below and separated in comparison.json. These deliberately truncated probes measure serving speed, not answer correctness.
| Deployment | Matched total time | Median latency | Median first token | Output tokens/s, including prefill |
|---|
| Luyao H100 | 125.53 s | 4.911 s | 0.317 s | 97.9 |
| Your H100 | 87.71 s | 3.181 s | 0.274 s | 140.1 |
| Your A100 INT4 | 127.80 s | 5.231 s | 0.359 s | 96.2 |
Deployment settings
| Deployment | GPU | Advertised context | Compute dtype | KV dtype | Attention | Weight quantization | Max concurrent requests |
|---|
| Luyao H100 | 7 | 196,608 | auto | fp8_e5m2 | triton | none | 2 |
| Your H100 | 0 | 32,768 | bfloat16 | fp8_e4m3 | fa3 | none | 1 |
| Your A100 INT4 | 5 | 32,768 | bfloat16 | auto | flashinfer | compressed-tensors | 1 |
| Deployment | GPU model | Used before | Used after | Total memory |
|---|
| Luyao H100 | NVIDIA H100 80GB HBM3 | 72.84 GiB | 72.84 GiB | 79.65 GiB |
| Your H100 | NVIDIA H100 80GB HBM3 | 69.95 GiB | 70.56 GiB | 79.65 GiB |
| Your A100 INT4 | NVIDIA A100-SXM4-40GB | 26.99 GiB | 28.35 GiB | 40.00 GiB |
Memory values are before/after GPU observations, not measured peaks or per-model allocations. Any sharing is visible in the saved process inventories.
Interpretation and reproducibility
Common client: released clawagents_py 6.20.79; 32,768-token context; 16,384 maximum output; temperature zero; low reasoning requested; lean tools; deadline guidance off; 240 seconds per task. Exact engine/suite hashes and settings are validated against every arm.
Advertised context is a capacity setting, not a measured context-quality score. The previous A100 45K retrieval test belongs to an earlier 48K experiment; the current A100 is restored to 32K.
Endpoints ran concurrently on distinct GPUs, with one benchmark request at a time per endpoint. Other users were not interrupted; host snapshots record observed GPU activity. This is not a claim of exclusive machines or a concurrency stress test.
A deadline hit is an explicit TimeoutError, or an unclean completion at at least 99% of the 240-second budget. All other unclean completions remain visible in the JSON.
Task time is the runner's agent_seconds metric; separate hidden-grader execution time is excluded. A faster failure is not a better task result.
3 repetitions per task show variability but do not establish statistical superiority. Deployment differences include hardware, precision, attention implementation, cache capacity and templates; this experiment does not isolate each factor's causal effect.
Serving tokens/s includes prefill and network time. Agent output tokens/s would confound model work, tool execution and cache reuse, so it is not presented as raw decoding throughput.
Agent token totals are provider/harness-reported usage; a request interrupted at the deadline can lack terminal usage. They should not be treated as an exact accounting of every generated token in timed-out runs.
Full observations: run/<deployment>/original.json, holdout.json, serving.json, traces and before/after inventories. comparison.csv is the flat deployment summary; comparison.json contains task details and exclusions.
Run another comparison
From the clawagents_py repository, use a fresh output directory so prior observations are preserved:
.venv/bin/python scripts/benchmark_deployed_models.py \
--manifest benchmarks/deployed_models_20260907/deployments.json \
--output benchmarks/deployed_models_20260907/run-new \
--engine-src ../clawagents_py-release-6.20.78/src --repeats 3
.venv/bin/python scripts/summarize_deployed_models.py \
benchmarks/deployed_models_20260907/run-new \
--output benchmarks/deployed_models_20260907/report-new
The release worktree directory retains its older name; its pyproject.toml version is 6.20.79. The benchmark accepts an explicit engine source directory and verifies engine hashes across all arms.