# Comparative benchmark pack

Cross-implementation performance bench for the three hgraph runtimes:

| mode | implementation |
|---|---|
| `upstream-py` | pip-installed `hgraph` (PyPI), Python runtime |
| `upstream-cpp` | the same package with `HGRAPH_USE_CPP=true` (the old C++ runtime) |
| `hg-cpp` | an optimized wheel built from the current repository source |

Comparative scenarios are written **once**, in standard Python hgraph syntax
(`benchmarks/scenarios.py`), and run unchanged on all three implementations.
The scenario registry gives every workload a stable command-line ID, a readable
label, a report group, a suite, and the set of runtimes that support it.
`*_std` scenarios are mostly graph/standard-operator workloads; `*_py`
scenarios deliberately put work in Python-authored nodes.

The **core** suite covers graph construction, scheduler hot loops, scalar and
compound values, dense/sparse/churning TSDs, switch and mesh nested graphs,
services, and adaptors. The **diagnostic** suite decomposes the hot paths and
adds fan-in/fan-out/conflation, Python boundary costs, TSB/TSS/TSW behavior,
large retained TSD capacity, capacity growth, clear/repopulate, multi-input
membership, explicit key sets, reducer implementation shapes, and multi-path
services.

Dynamic TSL is an hg_cpp feature with no valid upstream comparison. Its
diagnostic workload is therefore restricted to hg_cpp and appears in a
separate, explicitly non-comparative report section. Low-level native timings,
allocation counts, and additional dynamic TSL/TSW operations remain in the
`type_erasure_perf` C++ benchmark.

Terminal outputs use the implementation's native `null_sink`, keeping sink
overhead out of the Python node boundary in every mode.

## Running

```sh
# from the repo root, in the repo's environment (hg_cpp installed):
uv run python benchmarks/orchestrate.py                 # core, 3 samples
uv run python benchmarks/orchestrate.py --scale 0.1     # quick legacy shorthand
uv run python benchmarks/orchestrate.py \
  --suite core --suite diagnostic                       # all workloads
uv run python benchmarks/orchestrate.py \
  --cycle-scale 2 --size-scale 0.5                      # independent axes
uv run python benchmarks/orchestrate.py \
  --group "TSD - key lifecycle" --samples 5
uv run python benchmarks/orchestrate.py --scenario tick_std --mode hg-cpp
uv run python benchmarks/runner.py --list               # readable inventory
```

`--cycle-scale` changes the number of engine cycles without changing graph or
collection size. `--size-scale` changes graph width/depth, TSD cardinality, or
service client count without changing cycle count. `--scale` sets both and is
kept for compatibility with older commands. Explicit `--scenario` filters
override suite/group selection.

The first run for each Python/platform/architecture combination creates
`benchmarks/.venv-upstream-X.Y-PLATFORM-ARCH` (pip-installs `hgraph`) and
`benchmarks/.venv-hg-cpp-X.Y-PLATFORM-ARCH`. This prevents a repository shared
between macOS and a Linux VM from reusing an incompatible virtual environment.
The latter contains a Release wheel built from the current source and is rebuilt
whenever native, binding, or packaged Python source changes. The raw result
records both the source fingerprint and loaded native-module path so a result
cannot be mistaken for a stale editable build. All modes use the same
interpreter version. Delete the upstream directory to refresh its published
package. Results (markdown matrix + raw JSON) are written to
`benchmarks/results/`.

Each timing sample runs in a fresh subprocess. The orchestrator rotates mode
order deterministically between samples, reports the median and median absolute
deviation, and preserves every individual result in the raw JSON. A failure in
any sample makes the aggregate cell fail rather than being hidden by successful
samples. Unsupported runtimes show as `N/A`, not `FAIL`.

Before timing, the orchestrator runs the small `eval_node` workload guards in
every selected mode. These verify emitted values, service callback counts,
dense and sparse updates, key churn, and mesh dependencies. Use
`--skip-validation` only for repeated local timing after a successful
preflight.

**Timings are not CI gates.** The workload guards run in the normal Python
test suite, while timing exists for occasional controlled runs, not pass/fail
gating. Keep scenario definitions stable between runs you intend to compare.
Use the independent scale controls rather than editing scenario defaults.

The timed interval is the complete `run_graph` call, so it includes graph
construction, startup, steady-state execution, and teardown. Dedicated graph
construction scenarios expose fixed setup costs; use longer cycle scales when
the objective is steady-state throughput. The C++ microbenchmark pack is the
appropriate tool for operation-level timing and allocation counts.
