spent US$ 4.893; gates {'s0': True, 's1': True, 's2': True, 's3': True, 's4': True, 's5': True, 's6': True}; fuse_reason {'length': 400}; builds {'qwen3:4b@Q4_K_M': 2530, 'typesafe/jev-1.13-20260917': 2530}
  draft_luna  calls   900 ok   900 halts 0 rl 0 US$ 0.1445 p50 1.32 p95 2.572 empty 0 off-pin 0
  draft_sol   calls   420 ok   420 halts 0 rl 0 US$ 1.3196 p50 1.626 p95 3.153 empty 0 off-pin 0
  grade       calls  3498 ok  3498 halts 0 rl 0 US$ 3.2407 p50 1.587 p95 6.069 empty 0 off-pin 0
  jev         calls  2530 ok  2530 halts 0 rl 0 US$ 0.1881 p50 0.329 p95 0.446 empty 0 off-pin 0
  local       calls  2530 ok  2530 halts 0 rl 0 US$ 0.0000 p50 0.748 p95 1.465 empty 0 off-pin 0

arm      n  wrong  esc  handoff  mean US$   by family / language
A       400     33    0        0  0.00016   {'ANS': '0/144', 'NCP': '2/112', 'NCR': '31/144'} {'en': '14/194', 'pt': '19/206'}
A2      400     35    0        0  0.00016   {'ANS': '0/144', 'NCP': '4/112', 'NCR': '31/144'} {'en': '13/194', 'pt': '22/206'}
C       384     20    0        0  0.00310   {'ANS': '0/140', 'NCP': '0/111', 'NCR': '20/133'} {'en': '11/187', 'pt': '9/197'}
B       399     21   41      210  0.00060   {'ANS': '0/144', 'NCP': '1/112', 'NCR': '20/143'} {'en': '10/194', 'pt': '11/205'}
B_decl  399     21   41       38  0.00060   {'ANS': '0/144', 'NCP': '1/112', 'NCR': '20/143'} {'en': '10/194', 'pt': '11/205'}
B@0.5   400     29    5      177  0.00028   {'ANS': '0/144', 'NCP': '2/112', 'NCR': '27/144'} {'en': '12/194', 'pt': '17/206'}
B@0.9   399     17   56      224  0.00072   {'ANS': '0/144', 'NCP': '1/112', 'NCR': '16/143'} {'en': '7/194', 'pt': '10/205'}
B2      399     22   38      210  0.00059   {'ANS': '0/144', 'NCP': '3/112', 'NCR': '19/143'} {'en': '9/194', 'pt': '13/205'}
D       398     21   16      225  0.00030   {'ANS': '0/144', 'NCP': '1/112', 'NCR': '20/142'} {'en': '9/193', 'pt': '12/205'}
D_decl  398     21   16       16  0.00030   {'ANS': '0/144', 'NCP': '1/112', 'NCR': '20/142'} {'en': '9/193', 'pt': '12/205'}
D@0.5   400     25    9      218  0.00023   {'ANS': '0/144', 'NCP': '1/112', 'NCR': '24/144'} {'en': '11/194', 'pt': '14/206'}
D@0.9   399     21   18      227  0.00032   {'ANS': '0/144', 'NCP': '1/112', 'NCR': '20/143'} {'en': '9/194', 'pt': '12/205'}
D2      398     20   13      226  0.00027   {'ANS': '0/144', 'NCP': '2/112', 'NCR': '18/142'} {'en': '8/193', 'pt': '12/205'}
L       389     22  167        0  0.00173   {'ANS': '0/143', 'NCP': '0/111', 'NCR': '22/135'} {'en': '12/191', 'pt': '10/198'}
oracle  389      0   25       15  0.00326   {'ANS': '0/141', 'NCP': '0/112', 'NCR': '0/136'} {'en': '0/189', 'pt': '0/200'}

paired     n   wrong X/A   b   c   p       p_holm  diff    Newcombe          cluster CI        cost X/A
B-A      399    21/32    11   0  0.0010  0.0020  -0.0276 [-0.048,-0.011] [-0.044,-0.013] 3.74
D-A      398    21/32    11   0  0.0010  0.0020  -0.0276 [-0.048,-0.011] [-0.044,-0.013] 1.87
L-A      389    22/25     9   6  0.6072  0.6072  -0.0077 [-0.030,+0.014] [-0.027,+0.011] 10.82
B-L      389    15/22     8   1  0.0391  0.1953  -0.0180 [-0.037,-0.002] [-0.034,-0.005] 0.34
D-L      388    15/22     9   2  0.0654  0.2617  -0.0180 [-0.039,+0.000] [-0.035,-0.003] 0.17
C-A      384    20/25    10   5  0.3018  0.6035  -0.0130 [-0.036,+0.008] [-0.033,+0.005] 19.43
B-C      384    15/20     7   2  0.1797  0.5391  -0.0130 [-0.032,+0.004] [-0.029,+0.003] 0.19
B2-A2    399    22/34    12   0  0.0005  -       -0.0301 [-0.051,-0.013] [-0.048,-0.015] 3.68
D2-A2    398    20/33    13   0  0.0002  -       -0.0327 [-0.054,-0.015] [-0.051,-0.017] 1.68
L-A2     389    22/29    14   7  0.1892  -       -0.0180 [-0.044,+0.006] [-0.041,+0.005] 10.78

ANS helpfulness vs A: arm handoffs (rate, Wilson hi) unnecessary  correct X/A  Newcombe
  B         0 (0.000, 0.026)   0  141/141  [-0.028,+0.028]
  D         0 (0.000, 0.026)   0  141/141  [-0.028,+0.028]
  L         0 (0.000, 0.026)   0  141/141  [-0.029,+0.029]
  C         0 (0.000, 0.027)   0  140/140  [-0.027,+0.027]
  B_decl    0 (0.000, 0.026)   0  141/141  [-0.028,+0.028]
  D_decl    0 (0.000, 0.026)   0  141/141  [-0.028,+0.028]
  B@0.5     0 (0.000, 0.026)   0  141/141  [-0.028,+0.028]
  B@0.9     0 (0.000, 0.026)   0  141/141  [-0.028,+0.028]
  D@0.5     0 (0.000, 0.026)   0  141/141  [-0.028,+0.028]
  D@0.9     0 (0.000, 0.026)   0  141/141  [-0.028,+0.028]
  oracle    0 (0.000, 0.027)   0  141/141  [-0.027,+0.027]

verdict (§8): {'B': {'fewer_wrong': True, 'cost_ok': True, 'handoffs_ok': True, 'eligible': True, 'opt_in': True, 'default': False}, 'D': {'fewer_wrong': True, 'cost_ok': True, 'handoffs_ok': True, 'eligible': True, 'opt_in': True, 'default': True}}
verifier jev: accept 0.44005102040816324 reject-on-wrong 0.3584905660377358 not_verifying False AUROC 0.908 Brier 0.109 ECE 0.122 (floor 0.017) lowest accepted p 0.81
verifier local: accept 0.42091836734693877 reject-on-wrong 0.2830188679245283 not_verifying False AUROC 0.931 Brier 0.116 ECE 0.123 (floor 0.011) lowest accepted p 0.8034400412072068
graders: kappa 0.763 on 1320; adjudicated 42/1304; floors {'regrade': {'n': 200, 'agreement': 0.975}, 'paraphrase': {'n': 120, 'agreement': 0.975}}
lexical L: {'majority': 222, 'no_majority': 178}
