Rust S-Expression Phase 1 Corpus Measurement

Decision

The named-corpus performance and peak-memory harness is accepted as the Windows measurement baseline. It closes the implementation and local evidence portion of the Phase 1 corpus-measurement task. Release floors remain unratified until an independent Linux run reproduces the harness and the parser promotion review accepts explicit budgets.

Method

Windows Evidence

Tier Input Operation Best time Peak RSS Peak/input
Small footprint7,977 BParse0.293 ms4.35 MiBbaseline dominated
Small footprint7,977 BRoundtrip0.486 ms4.32 MiBbaseline dominated
Medium schematic996,306 BParse9.885 ms12.91 MiB13.58x
Medium schematic996,306 BRoundtrip34.635 ms24.73 MiB26.03x
Large board71,556,836 BParse584.927 ms506.58 MiB7.42x
Large board71,556,836 BRoundtrip2,247.323 ms1,136.86 MiB16.66x

These values came from Windows 11 on an AMD64 32-logical-CPU host using Rust 1.95.0. They are evidence from one run series, not portable limits.

Interpretation

Throughput is already useful, but full generic-tree memory is material. The large parse-only path peaks at about 7.4 times input size, while the two-tree oracle intentionally reaches about 16.7 times input size. This supports the existing architecture decision: the generic tree remains a compatibility and validation path, while promoted converters and typed readers must use selective spans or direct typed views rather than allocating the complete generic tree.

Select-All Sorting Evidence

A separate non-default measurement feature times form discovery and the final source-order sort independently. Five release runs per scanner and corpus tier show that sorting is material on the large board:

TierSelected formsScannerBest scanBest sortSort fraction
Small350Memory0.121 ms0.017 ms12.4%
Small350Stream0.173 ms0.018 ms9.6%
Medium41,192Memory15.438 ms1.496 ms8.8%
Medium41,192Stream19.040 ms1.554 ms7.5%
Large2,124,874Memory772.026 ms166.160 ms17.7%
Large2,124,874Stream1,021.759 ms165.172 ms13.9%

This cost is recorded in a focused backlog issue for possible source-order emission without the final O(k log k) sort. It does not block the current functional port; the parser promotion review should promote the optimization only if ratified budgets or a real consumer workload require it.

Retained Gates

2026-08-31 Native Lexer and Sparse Projection Decision

GitHub issue #75 was implemented as two private, independently measured changes without altering the public parser or projection contracts. The lexer now scans structural ASCII and token runs by byte while preserving Unicode-scalar columns. Projection frames borrow unquoted in-memory heads and derive paths from the active stack; owned public heads and paths are created only for selected forms. The streaming scanner retains only the owned active heads required after input chunks are discarded.

Same-host evidence used clean exact commits B0 72dea9973eade13034ef043dd62792006fda9722, lexer L beb40f62abab2e6f64f3daaab36f14899114bf8e, and final P 697a685ef0843dc0b6a73e8288e1599a049d93d4. Harnesses, inputs, features, release settings, locks, binaries, raw rounds, and host provenance were retained for each comparison.

GateObserved resultDecision
B0 to L lexer2.302x median; 2.292x 95% sign-interval lower boundPass
L to P sparse memory time5.053x median; 5.035x lower boundPass
L to P sparse stream time2.740x median; 2.704x lower boundPass
L to P memory allocations1,600,631 to 628 calls; 99.96% reductionPass
L to P stream allocations1,600,632 to 300,630 calls; 81.22% reductionPass
L to P select-all1.043x to 1.188x median across all tiers/scannersPass
B0 to P native Speedy7.618 s to 5.028 s median; 1.515x improvementPass

The final three-round Speedy run retained semantic parity for all 35 structured/report artifacts and all 29 SVGs, left the reviewed source tree unchanged, and measured the Rust path at 15.44x the Python median. The three paired B0-to-P Rust ratios were 1.479x, 1.515x, and 1.578x. Shared Python/Rust span vectors, exact memory/stream resource boundaries, full locked workspace tests, real Node-hosted WASM, reviewed-corpus roundtrip/determinism, Cruncher source-install behavior, and repository audits passed.

These values accept the byte lexer and allocation-light projection representation on this host; they are not portable service-level guarantees. A borrowed public tree, selector trie, final-sort removal, memchr, direct typed parsing, and a Python lexer rewrite remain separately scoped work.