data-eng-bench: benchmark results

Snowflake-Labs data-eng-bench (103 dbt data-engineering tasks, DuckDB backend) · agent: KISS SorcarAgent · models: claude-fable-5 (development) + gpt-5.6-sol (read-only review, ≤20% of task budget) · run date: 2026-08-09

Headline leaderboard scores

Every task ran 3 independent trials (309 trials total), exactly as the public leaderboard requires. Scores below use the leaderboard's own formulas (leaderboard/core/metrics.py): a trial succeeds iff its hidden pytest verifier awards reward > 0, and errored trials count as failures.

69.58% ± 0.86

Accuracy (mean reward, 215/309 trials)

71.84%

pass@2 (unbiased estimator)

72.82%

pass@3 (unbiased estimator)

68 / 103

tasks solved in all 3 trials
(7 partial, 28 unsolved)

$1,143.33

total model spend

$3.70

mean cost per trial

4h 24m

wall-clock runtime (10 concurrent trials)

harness exceptions in 309 trials

Success rate by difficulty

Success rate by category

Categories are taken from each task's task.toml metadata.

How the benchmark was run

Harness. A custom Harbor agent (benchmarkings/harbor_sorcar_agent.py) runs one KISS SorcarAgent per trial. The agent's Bash / Read / Write / Edit tools are proxied into the trial's Docker container through Harbor's environment.exec, so all work happens inside the sandbox that the hidden verifier later inspects.
Models. Development used claude-fable-5; each agent was instructed to use gpt-5.6-sol for a thorough read-only review and debugging pass, capped at 20% of the task budget, and asked not to invent new problems.
Integrity. Browser/web tools were disabled for the duration of every trial (matching the benchmark's no-web-lookup rule), the verifier and reference solutions stayed hidden from the agent, and each agent was told: "Do NOT cheat on the benchmarks at any cost. Do NOT stop until you are successful on each benchmark problem. Strictly follow the rules described in the benchmark."
Limits. Per trial: $6 budget, 300 steps, Harbor's stock 4000s agent timeout; no timeout multipliers or resource overrides (leaderboard-legal settings).
Scoring. Each task's hidden pytest verifier compares the materialized tables row by row against a reference solution and awards an all-or-nothing reward. Scores were computed locally with benchmarkings/compute_scores.py, which replicates the leaderboard's published formulas; Harbor's own job summary reports the same numbers (mean 0.696, pass@2 0.718).

Per-task results

✔ / ✘ marks are the three trial outcomes; cost is the summed model spend of the three trials.

Task	Difficulty	Category	Trials	Passes	Cost
dbt-abc-classification	Medium	analytics	✔✔✔	3/3	$11.90
dbt-basket-composition-analysis	Hard	analytics	✔✔✔	3/3	$13.83
dbt-calculate-running-balance	Medium	analytics	✔✔✔	3/3	$9.22
dbt-campaign-performance	Medium	development	✔✔✔	3/3	$6.69
dbt-campaign-roi-analysis	Medium	development	✔✔✔	3/3	$17.65
dbt-carrier-delivery-performance	Medium	development	✔✔✔	3/3	$9.68
dbt-cart-abandonment-recovery	Medium	development	✔✔✔	3/3	$6.63
dbt-channel-attribution-analysis	Medium	analytics	✔✔✔	3/3	$11.11
dbt-consolidate	Medium	analytics	✔✔✔	3/3	$2.86
dbt-customer-churn-cohorts	Very hard	analytics	✔✔✔	3/3	$15.89
dbt-customer-cohort-retention	Medium	analytics	✔✔✔	3/3	$11.86
dbt-customer-cross-sell-insights	Hard	analytics	✔✔✔	3/3	$18.07
dbt-customer-geographic	Easy	analytics	✔✔✔	3/3	$8.80
dbt-customer-lifecycle-journey	Medium	development	✔✔✔	3/3	$8.67
dbt-customer-lifetime-value	Hard	analytics	✔✔✔	3/3	$14.66
dbt-customer-ltv-fix	Hard	analytics	✔✔✔	3/3	$12.00
dbt-customer-order-analytics	Medium	analytics	✔✔✔	3/3	$8.95
dbt-daily-order-summary	Easy	analytics	✔✔✔	3/3	$8.28
dbt-dq-macro-enforcement	Hard	analytics	✔✔✔	3/3	$13.81
dbt-email-campaign-tracker	Medium	development	✔✔✔	3/3	$6.41
dbt-exchange-rate-settlement-date	Hard	analytics	✔✔✔	3/3	$11.64
dbt-fix-cac-payback-waterfall	Medium	analytics	✔✔✔	3/3	$7.56
dbt-fix-customer-churn	Medium	dbt-models	✔✔✔	3/3	$4.19
dbt-fix-customer-snapshot-and-build-dimension	Hard	dbt-snapshots	✔✔✔	3/3	$4.43
dbt-fix-daily-cohorts	Medium	dbt-models	✔✔✔	3/3	$6.27
dbt-fix-daily-revenue	Medium	dbt-models	✔✔✔	3/3	$8.91
dbt-fix-division-by-zero	Medium	debugging	✔✔✔	3/3	$6.51
dbt-fix-email-attribution	Medium	analytics	✔✔✔	3/3	$7.80
dbt-fix-inventory-model	Medium	debugging	✔✔✔	3/3	$6.64
dbt-fix-multi-touch-attribution	Hard	analytics	✔✔✔	3/3	$13.75
dbt-fix-paid-search-attribution	Medium	analytics	✔✔✔	3/3	$15.65
dbt-fix-refund-reconciliation	Medium	development	✔✔✔	3/3	$4.88
dbt-fix-timezone-sales	Medium	dbt-models	✔✔✔	3/3	$15.40
dbt-fraud-detection-model	Hard	analytics	✔✔✔	3/3	$12.46
dbt-gl-reconciliation	Very hard	analytics	✔✔✔	3/3	$9.73
dbt-harbor-product-affinity	Hard	analytics	✔✔✔	3/3	$16.32
dbt-harbor-warehouse-capacity	Hard	analytics	✔✔✔	3/3	$10.60
dbt-hr-analytics	Hard	analytics	✔✔✔	3/3	$22.49
dbt-inventory-stockout-risk	Medium	analytics	✔✔✔	3/3	$18.73
dbt-loyalty-points-analysis	Hard	analytics	✔✔✔	3/3	$6.27
dbt-multi-warehouse-stock-rebalance	Hard	development	✔✔✔	3/3	$9.74
dbt-order-fulfillment-analytics	Hard	analytics	✔✔✔	3/3	$10.91
dbt-order-fulfillment-metrics	Hard	analytics	✔✔✔	3/3	$7.39
dbt-order-reconciliation	Hard	analytics	✔✔✔	3/3	$4.84
dbt-price-elasticity	Hard	analytics	✔✔✔	3/3	$18.92
dbt-product-affinity	Hard	analytics	✔✔✔	3/3	$15.53
dbt-product-category-analytics	Hard	analytics	✔✔✔	3/3	$7.96
dbt-product-performance-metrics	Medium	analytics	✔✔✔	3/3	$7.56
dbt-product-return-analysis	Medium	development	✔✔✔	3/3	$6.08
dbt-product-sales-velocity	Hard	analytics	✔✔✔	3/3	$11.30
dbt-receivables-aging-buckets	Hard	finance	✔✔✔	3/3	$12.70
dbt-retail-task-01-channel-revenue	Easy	analytics	✔✔✔	3/3	$7.61
dbt-retail-task-04-market-roi	Medium	analytics	✔✔✔	3/3	$7.88
dbt-retail-task-07-cart-recovery	Hard	analytics	✔✔✔	3/3	$9.88
dbt-retail-task-10-trust-risk	Hard	analytics	✔✔✔	3/3	$15.43
dbt-rfm-customer-segmentation	Medium	analytics	✔✔✔	3/3	$8.48
dbt-rfm-customer-tiering	Very hard	analytics	✔✔✔	3/3	$17.38
dbt-session-attribution	Very hard	analytics	✔✔✔	3/3	$4.62
dbt-warehouse-fulfillment-analytics	Medium	development	✔✔✔	3/3	$10.43
dbt-web-session-analytics	Medium	analytics	✔✔✔	3/3	$9.44
dbt-weekly-sales-growth	Medium	analytics	✔✔✔	3/3	$8.42
deferred-revenue-recognition	Hard	database	✔✔✔	3/3	$8.43
payment-risk-scoring	Medium	data-engineering	✔✔✔	3/3	$17.90
pos-operations	Hard	data-engineering	✔✔✔	3/3	$17.17
shipping-fulfillment-quality-scoring	Medium	data-engineering	✔✔✔	3/3	$10.39
time-decay-attribution-model	Hard	database	✔✔✔	3/3	$13.90
web-session-quality-scoring	Medium	data-engineering	✔✔✔	3/3	$12.02
workforce-analytics	Hard	data-engineering	✔✔✔	3/3	$16.64
dbt-coupon-effectiveness	Medium	analytics	✔✔✘	2/3	$17.81
dbt-customer-risk-scoring	Hard	analytics	✔✔✘	2/3	$10.92
dbt-fix-customer-ltv	Medium	dbt-models	✔✔✘	2/3	$17.66
tier-migration-analysis	Hard	database	✔✔✘	2/3	$14.18
dbt-customer-retention-risk	Hard	development	✔✘✘	1/3	$9.08
dbt-fulfillment-sla	Medium	analytics	✔✘✘	1/3	$13.59
dbt-payment-analytics	Medium	development	✔✘✘	1/3	$6.70
cohort-retention-matrix	Hard	database	✘✘✘	0/3	$6.80
dbt-customer-acquisition-channel-performance	Hard	analytics	✘✘✘	0/3	$13.20
dbt-customer-churn-early-warning	Medium	analytics	✘✘✘	0/3	$6.50
dbt-customer-cltv-forecasting	Hard	analytics	✘✘✘	0/3	$10.79
dbt-fix-category-revenue	Hard	dbt-models	✘✘✘	0/3	$15.92
dbt-fix-inventory-balance	Medium	dbt-models	✘✘✘	0/3	$4.94
dbt-fix-marketing-attribution	Medium	analytics	✘✘✘	0/3	$7.23
dbt-fix-product-metrics	Medium	dbt-models	✘✘✘	0/3	$6.60
dbt-fix-repeat-purchase-cohort-revenue	Medium	analytics	✘✘✘	0/3	$6.46
dbt-harbor-marketing-mix-model	Hard	analytics	✘✘✘	0/3	$16.85
dbt-incremental-late-arriving-sales	Hard	database	✘✘✘	0/3	$10.32
dbt-inventory-analysis	Medium	analytics	✘✘✘	0/3	$18.01
dbt-inventory-turnover-analysis	Medium	development	✘✘✘	0/3	$13.08
dbt-monthly-channel-revenue	Hard	analytics	✘✘✘	0/3	$12.38
dbt-multicurrency-lifo	Very hard	analytics	✘✘✘	0/3	$16.17
dbt-order-interval-metrics	Medium	analytics	✘✘✘	0/3	$11.20
dbt-payment-fraud-analysis	Hard	analytics	✘✘✘	0/3	$9.79
dbt-product-return-rate-analysis	Medium	analytics	✘✘✘	0/3	$16.87
dbt-rfm-customer-segmentation-2	Medium	analytics	✘✘✘	0/3	$10.50
dbt-sales-funnel-analysis	Hard	analytics	✘✘✘	0/3	$11.41
dbt-supplier-payment-optimization	Very hard	analytics	✘✘✘	0/3	$10.44
dbt-supplier-scorecard	Very hard	analytics	✘✘✘	0/3	$11.18
dbt-test-orders-filter	Medium	analytics	✘✘✘	0/3	$7.97
dbt-three-way-matching	Hard	analytics	✘✘✘	0/3	$10.72
fifo-inventory-cogs	Hard	database	✘✘✘	0/3	$14.41
late-arriving-orders-reconciliation	Hard	database	✘✘✘	0/3	$9.08
marketing-campaigns-harbor	Hard	data-engineering	✘✘✘	0/3	$15.59
promotional-lift-analysis	Very hard	dbt	✘✘✘	0/3	$12.83

Observations

All 9 easy trials and every debugging-category trial passed; failures concentrate in hard analytics marts (cohort/retention matrices, CLTV forecasting, marketing mix models) and multi-step reconciliation pipelines (FIFO/LIFO inventory costing, three-way matching, late-arriving data).
The verifier is all-or-nothing: several failed trials matched 14 of 15 verifier components and missed only an exact row count, so small filter/grain choices (date boundaries, null handling, dedup) decide success.
28 tasks failed in all three trials — these are systematic specification- interpretation gaps rather than flaky behavior; 68 tasks passed 3/3, showing the agent is highly consistent when it understands the spec.

Submitting these results to the public leaderboard

Submission needs a Harbor Hub account (GitHub OAuth) and a GitHub PR, which require interactive human authentication, so they were not performed in this run. The job directory benchmarkings/jobs/2026-08-09__01-45-42 contains everything needed.

# 1. authenticate the harbor CLI (GitHub OAuth)
harbor auth login

# 2. upload the finished job publicly
harbor upload benchmarkings/jobs/2026-08-09__01-45-42 --public

# 3. open the submission PR from a data-eng-bench checkout
git clone https://github.com/Snowflake-Labs/data-eng-bench
cd data-eng-bench/leaderboard
uv run lb submit https://hub.harborframework.com/jobs/<uuid-from-step-2>