Memory
~4x smaller — 28 MB vs 110 MB
Client PSS (Linux proportional set size), median of 10 cases; rustwright is smaller in every case. Browser runs remotely and is excluded.
00
Executive readout
Rustwright is a Rust CDP engine implementing Playwright’s Python API. It delivers a decisive ~4x memory reduction — replicated across three independent runs and 36 cases (controlled fixtures and live sites) — but currently trades that for higher per-operation latency: 1.1–2.6x slower on every case measured. Raw CDP round-trips are at parity, so the deficit is concentrated in connection setup and object/DOM-heavy operations, not the wire.
Latency
1.1–2.6x slower — median ~1.8x
Library latency (page waits excluded); rustwright is slower on all 36 cases across three runs.
Transport
raw CDP evaluate at parity — 55 vs 56 ms
Trivial-return evaluate RTT is identical (n=30). The wire is not the bottleneck.
Weakness
object-return evaluate ~2.6x slower
25 object-returning evals: 6.4s vs 2.5s — the sharpest per-op gap.
Weakness
connect ~2.3x slower — 2136 vs 925 ms
Session/target attach + event-stream setup (n=30). Confirms the smoke finding.
01
Methodology
The harness separates client-library work from browser hosting and excludes page-load time from the latency comparisons.
Remote execution
Both clients drive remote Skyvern cloud browsers over CDP. Browser memory is off-box, so the memory result isolates the local client process tree.
Navigation exclusion
Website and network time are excluded as a navigation band rather than attributed to either library.
Two evidence lanes
Controlled data: URL fixtures provide byte-identical work and zero render; live automation-sandbox cases exercise real navigation and interaction.
Library probes
Page-wait-free probes isolate CDP and library pipelines. Controlled click cases set page_wait=0.
02
Latency results
Lower is better. Color identifies the faster and slower side of each direct comparison; it does not imply statistical confidence.
Clean library latency (page-render-free)
milliseconds
| Operation | Rustwright | Playwright 1.59 | Finding |
|---|---|---|---|
| evaluate RTT (trivial return) | 55 ms | 56 ms | Parity — transport-bound (n=30) |
| stable-element click pipeline | 797 ms | 680 ms | Rustwright 1.17x slower |
| connect | 2136 ms | 925 ms | Rustwright 2.31x slower (n=30) |
Controlled fixtures — library latency by case (page_wait=0, 3 reps)
median milliseconds
| Case (ops) | Rustwright | Playwright 1.59 | rw ÷ pw |
|---|---|---|---|
| form checkbox click ×24 | 17993 ms | 14834 ms | 1.21x slower |
| button click ×20 | 15570 ms | 12601 ms | 1.24x slower |
| large-DOM id lookup ×100 | 7805 ms | 6740 ms | 1.16x slower |
| evaluate RTT ×100 | 7572 ms | 6694 ms | 1.13x slower |
| object-eval small ×25 | 6428 ms | 2487 ms | 2.58x slower |
| deep-DOM query ×50 | 4994 ms | 3872 ms | 1.29x slower |
| object-eval large ×10 | 4089 ms | 1978 ms | 2.07x slower |
| large-DOM all cells ×20 | 3339 ms | 2411 ms | 1.38x slower |
| dynamic wait 300 ms | 2972 ms | 1779 ms | 1.67x slower |
| goto data: URL ×20 | 2126 ms | 936 ms | 2.27x slower |
Replication — three independent Blacksmith runs
per-lane medians
| Lane (run) | Sessions | PSS rw ÷ pw | Latency rw ÷ pw | Notes |
|---|---|---|---|---|
| controlled ×10 (29465108042) | 60/60 | 0.25x | 1.80x | reps=3 · data: fixtures, page_wait=0 |
| controlled_more ×9 (29476022036) | 54/54 | 0.24x | 1.29x | reps=3 · different op mix, replicates goto 2.29x and object-eval 2.20x |
| live sites ×17 (29476022706) | 68/68 | 0.27x | 1.79x | reps=2 · the-internet sandbox; page_wait comparable across backends (0.94x) — exclusion is fair |
03
Concentrated weaknesses
Rustwright is slower on every controlled case, but the gap is far from uniform: object-return and connection paths dominate it. These are the sharpest, most actionable concentrations.
W1
CRITICAL
query_selector_all handle-returning path
~270 ms/element: N=10 → 2.8s, N=50 → 13.5s, and N=100 → 27s. Playwright is ~1 ms/element, including N=1000 → 875 ms.
locator.count() is fine and remains flat at ~150 ms, localizing the regression to the handle-returning path.
Cause: one CDP round-trip per ElementHandle, with no batching. Fix: batch handle materialization. Measured in the dedicated weakness lane (smoke stage); not re-run in the Blacksmith pass.
W2
HIGH
Object-evaluate return path
Object-returning evaluation is the sharpest per-case gap on Blacksmith: small objects ×25 run 2.58x slower (6.4s vs 2.5s) and large objects ×10 run 2.07x slower (4.1s vs 2.0s) — while trivial-return evaluate is at parity (55 vs 56 ms).
Target: profile serialization and remote-object conversion on the object-return path; the wire itself is exonerated by the trivial-return parity.
W3
HIGH
Initial CDP connection
Connect is 2.31x slower on Blacksmith: Rustwright 2136 ms vs Playwright 925 ms (median, n=30 each). This also inflates the short goto data: URL case (2.27x), where startup cost dominates.
Likely mechanism: eager target attach plus Tokio and event-stream setup. Now confirmed at n=30; profile session/target attach ordering.
These are actionable optimization targets for the team: batch handle creation first, then profile object conversion and connection initialization.
04
Memory
The strongest architectural finding is the local client footprint. Remote browser memory remains off-box and outside this measurement.
Client memory
28 vs 110 MB PSS
Peak client PSS with the browser remote: Rustwright uses ~4x less (0.24–0.27x per-lane medians, 75% less). Per-case range across all 36 cases: 25–69 MB vs 86–152 MB — smaller in every case, 91 sessions per backend.
Architectural basis
in-process Rust
An in-process Rust client replaces the Python-plus-Node-driver stack. PSS (proportional set size) counts shared pages fairly, so this is a stronger and more defensible figure than the earlier macOS RSS proxy.
05
Coverage
Functional coverage is useful smoke evidence, with controlled fixtures carrying more weight than the flaky live lane.
Linux-measured coverage
36 cases · 182/182
19 controlled fixtures + 17 live-site cases ran green on Blacksmith with zero session failures; the catalog holds 48 validated cases in total.
Live-lane stability
68/68 on Linux
The curated live lane (flaky cases culled during validation) ran without a single failure, and both backends completed identical step counts on every rep.
Adversarial review
30 findings
A WEAKNESSES.md adversarial self-review records limitations and optimization targets.
06
Caveats and next evidence
The Blacksmith pass supplies real PSS and consistent direction across all cases; the remaining caveats bound precision, not direction.
- Latency reps are modest (2–3 per case/backend); p95 tracks the median closely, but tails are not deeply characterized.
- The smoke stage’s “1.35x faster on controlled clicks” did not reproduce — Blacksmith shows clicks 1.24x slower. The Blacksmith numbers supersede all smoke latency claims.
- The
query_selector_all~300x finding is from the smoke weakness lane and has not yet been re-measured on Linux. - The two clients ran in separate remote browser sessions (by design, for parallelism); sessions come from the same Skyvern pool but are not the same instance.
- Remote CDP latency to Skyvern amplifies per-round-trip costs; local-browser gaps will be smaller in absolute terms.
- Provenance: Blacksmith 8-vCPU Ubuntu 24.04, concurrency=6. Runs
29465108042(controlled.json, reps=3, commite63087b),29476022036(controlled_more.json, reps=3, main) and29476022706(nav_the_internet.json, reps=2, main) — 182/182 sessions OK.