Jev connection: 42 of 43 with-Jev runs that reached the model got a Jev answer (98%). 107 with-Jev runs never reached the local model (connection error) and are not a Jev miss.
Stop: all 150 pairs recorded
not measured (0 labeled items)
| accuracy | not measured (0 labeled items) |
|---|---|
| pairs | 43 |
| wall_s_median_difference | 52.03 |
| sign_test_p | 2.274e-13 |
| b_runs_without_jev_answer | 1 |
| run | tool | status | use gate | answer | correct | wall s | Jev calls |
|---|---|---|---|---|---|---|---|
| classify-01.A | jev_classify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.66 | 0 |
| classify-01.B | jev_classify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.66 | 0 |
| classify-02.A | jev_classify | ok | n/a (arm has no Jev) | account | False | 13.00 | 0 |
| classify-02.B | jev_classify | ok | used Jev | account | False | 67.16 | 1 |
| classify-03.A | jev_classify | ok | n/a (arm has no Jev) | final line is not JSON | False | 15.27 | 0 |
| classify-03.B | jev_classify | ok | used Jev | shipping | False | 61.12 | 1 |
| classify-04.A | jev_classify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.27 | 0 |
| classify-04.B | jev_classify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.37 | 0 |
| classify-05.A | jev_classify | ok | n/a (arm has no Jev) | product_defect | False | 13.21 | 0 |
| classify-05.B | jev_classify | ok | used Jev | product_defect | False | 54.77 | 1 |
| classify-06.A | jev_classify | ok | n/a (arm has no Jev) | test_failure | False | 17.35 | 0 |
| classify-06.B | jev_classify | ok | used Jev | test_failure | False | 69.38 | 1 |
| classify-07.A | jev_classify | ok | n/a (arm has no Jev) | flaky | False | 16.13 | 0 |
| classify-07.B | jev_classify | ok | used Jev | flaky | False | 76.06 | 1 |
| classify-08.A | jev_classify | ok | n/a (arm has no Jev) | config | False | 16.73 | 0 |
| classify-08.B | jev_classify | ok | used Jev | config | False | 100.31 | 1 |
| classify-09.A | jev_classify | ok | n/a (arm has no Jev) | infra | False | 35.32 | 0 |
| classify-09.B | jev_classify | ok | used Jev | infra | False | 237.91 | 1 |
| classify-10.A | jev_classify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.38 | 0 |
| classify-10.B | jev_classify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.63 | 0 |
| classify-11.A | jev_classify | ok | n/a (arm has no Jev) | docs_only | False | 15.99 | 0 |
| classify-11.B | jev_classify | ok | used Jev | docs_only | False | 75.08 | 1 |
| classify-12.A | jev_classify | ok | n/a (arm has no Jev) | breaking_change | False | 17.48 | 0 |
| classify-12.B | jev_classify | ok | used Jev | breaking_change | False | 70.61 | 1 |
| classify-13.A | jev_classify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.38 | 0 |
| classify-13.B | jev_classify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.32 | 0 |
| classify-14.A | jev_classify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.24 | 0 |
| classify-14.B | jev_classify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.30 | 0 |
| classify-15.A | jev_classify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| classify-15.B | jev_classify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.40 | 0 |
| compare-01.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 15.13 | 0 |
| compare-01.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.66 | 0 |
| compare-02.A | jev_compare | ok | n/a (arm has no Jev) | contradicts | False | 40.23 | 0 |
| compare-02.B | jev_compare | ok | used Jev | contradicts | False | 52.88 | 1 |
| compare-03.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.46 | 0 |
| compare-03.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.48 | 0 |
| compare-04.A | jev_compare | ok | n/a (arm has no Jev) | different_facts | False | 16.37 | 0 |
| compare-04.B | jev_compare | ok | used Jev | different_facts | False | 49.92 | 1 |
| compare-05.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 15.16 | 0 |
| compare-05.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.78 | 0 |
| compare-06.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.37 | 0 |
| compare-06.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.48 | 0 |
| compare-07.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.37 | 0 |
| compare-07.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.40 | 0 |
| compare-08.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.37 | 0 |
| compare-08.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.53 | 0 |
| compare-09.A | jev_compare | ok | n/a (arm has no Jev) | contradicts | False | 15.35 | 0 |
| compare-09.B | jev_compare | ok | used Jev | contradicts | False | 54.30 | 1 |
| compare-10.A | jev_compare | ok | n/a (arm has no Jev) | same_fact | False | 13.50 | 0 |
| compare-10.B | jev_compare | ok | used Jev | same_fact | False | 70.62 | 1 |
| compare-11.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.48 | 0 |
| compare-11.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.70 | 0 |
| compare-12.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.25 | 0 |
| compare-12.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.30 | 0 |
| compare-13.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.44 | 0 |
| compare-13.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.46 | 0 |
| compare-14.A | jev_compare | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.24 | 0 |
| compare-14.B | jev_compare | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.34 | 0 |
| compare-15.A | jev_compare | ok | n/a (arm has no Jev) | final line is not JSON | False | 16.51 | 0 |
| compare-15.B | jev_compare | ok | used Jev | different_facts | False | 51.48 | 1 |
| decide-01.A | jev_decide | ok | n/a (arm has no Jev) | ship_full | False | 19.45 | 0 |
| decide-01.B | jev_decide | ok | used Jev | ship_full | False | 79.82 | 1 |
| decide-02.A | jev_decide | ok | n/a (arm has no Jev) | ship_flagged | False | 17.59 | 0 |
| decide-02.B | jev_decide | ok | used Jev | ship_flagged | False | 87.60 | 1 |
| decide-03.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| decide-03.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.35 | 0 |
| decide-04.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| decide-04.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| decide-05.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.34 | 0 |
| decide-05.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.42 | 0 |
| decide-06.A | jev_decide | ok | n/a (arm has no Jev) | credit_20 | False | 17.35 | 0 |
| decide-06.B | jev_decide | ok | used Jev | credit_20 | False | 52.30 | 1 |
| decide-07.A | jev_decide | ok | n/a (arm has no Jev) | no_remedy | False | 21.77 | 0 |
| decide-07.B | jev_decide | ok | used Jev | no_remedy | False | 194.12 | 1 |
| decide-08.A | jev_decide | ok | n/a (arm has no Jev) | full_refund | False | 47.37 | 0 |
| decide-08.B | jev_decide | ok | used Jev | full_refund | False | 69.61 | 1 |
| decide-09.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| decide-09.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| decide-10.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.30 | 0 |
| decide-10.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.56 | 0 |
| decide-11.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.40 | 0 |
| decide-11.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.35 | 0 |
| decide-12.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.25 | 0 |
| decide-12.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.34 | 0 |
| decide-13.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.27 | 0 |
| decide-13.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.31 | 0 |
| decide-14.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.37 | 0 |
| decide-14.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.55 | 0 |
| decide-15.A | jev_decide | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.30 | 0 |
| decide-15.B | jev_decide | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.45 | 0 |
| extract-01.A | jev_extract | ok | n/a (arm has no Jev) | $5,153.50 | False | 14.77 | 0 |
| extract-01.B | jev_extract | ok | used Jev | $5,153.50 | False | 63.21 | 1 |
| extract-02.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.69 | 0 |
| extract-02.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 15.01 | 0 |
| extract-03.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.29 | 0 |
| extract-03.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.29 | 0 |
| extract-04.A | jev_extract | ok | n/a (arm has no Jev) | NF-2026-08102 | False | 15.37 | 0 |
| extract-04.B | jev_extract | ok | used Jev | NF-2026-08102 | False | 49.48 | 1 |
| extract-05.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.27 | 0 |
| extract-05.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.33 | 0 |
| extract-06.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.48 | 0 |
| extract-06.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.61 | 0 |
| extract-07.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| extract-07.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.40 | 0 |
| extract-08.A | jev_extract | ok | n/a (arm has no Jev) | 1 | False | 15.70 | 0 |
| extract-08.B | jev_extract | ok | used Jev | 1 | False | 119.17 | 1 |
| extract-09.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.26 | 0 |
| extract-09.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.33 | 0 |
| extract-10.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| extract-10.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.36 | 0 |
| extract-11.A | jev_extract | ok | n/a (arm has no Jev) | BKT-2291-0475 | False | 16.42 | 0 |
| extract-11.B | jev_extract | ok | used Jev | BKT-2291-0475 | False | 78.25 | 1 |
| extract-12.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.35 | 0 |
| extract-12.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.36 | 0 |
| extract-13.A | jev_extract | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.57 | 0 |
| extract-13.B | jev_extract | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.66 | 0 |
| extract-14.A | jev_extract | ok | n/a (arm has no Jev) | 2026-09-12 | False | 15.70 | 0 |
| extract-14.B | jev_extract | ok | used Jev | 2026-09-12 | False | 49.13 | 1 |
| extract-15.A | jev_extract | ok | n/a (arm has no Jev) | not stated | False | 15.55 | 0 |
| extract-15.B | jev_extract | ok | used Jev | not stated | False | 97.86 | 1 |
| find-01.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.34 | 0 |
| find-01.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.35 | 0 |
| find-02.A | jev_find | ok | n/a (arm has no Jev) | price-drop | False | 12.96 | 0 |
| find-02.B | jev_find | ok | used Jev | price-drop | False | 83.40 | 1 |
| find-03.A | jev_find | ok | n/a (arm has no Jev) | ship-intl | False | 16.00 | 0 |
| find-03.B | jev_find | ok | used Jev | ship-intl | False | 85.90 | 1 |
| find-04.A | jev_find | ok | n/a (arm has no Jev) | final line is not JSON | False | 14.17 | 0 |
| find-04.B | jev_find | ok | used Jev | pay-methods | False | 81.17 | 1 |
| find-05.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.27 | 0 |
| find-05.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| find-06.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| find-06.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.35 | 0 |
| find-07.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.29 | 0 |
| find-07.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.32 | 0 |
| find-08.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.25 | 0 |
| find-08.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.32 | 0 |
| find-09.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.32 | 0 |
| find-09.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.51 | 0 |
| find-10.A | jev_find | ok | n/a (arm has no Jev) | cache-bust | False | 18.31 | 0 |
| find-10.B | jev_find | ok | used Jev | cache-bust | False | 84.56 | 1 |
| find-11.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.30 | 0 |
| find-11.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.52 | 0 |
| find-12.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.26 | 0 |
| find-12.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.32 | 0 |
| find-13.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.34 | 0 |
| find-13.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.45 | 0 |
| find-14.A | jev_find | ok | n/a (arm has no Jev) | v3.1.1 | False | 16.87 | 0 |
| find-14.B | jev_find | ok | no Jev call | v3.1.1 | False | 17.43 | 0 |
| find-15.A | jev_find | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.30 | 0 |
| find-15.B | jev_find | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.39 | 0 |
| gate-01.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.29 | 0 |
| gate-01.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| gate-02.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.52 | 0 |
| gate-02.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.58 | 0 |
| gate-03.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| gate-03.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.39 | 0 |
| gate-04.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.29 | 0 |
| gate-04.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.37 | 0 |
| gate-05.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| gate-05.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.35 | 0 |
| gate-06.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.38 | 0 |
| gate-06.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| gate-07.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.42 | 0 |
| gate-07.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.51 | 0 |
| gate-08.A | jev_gate | ok | n/a (arm has no Jev) | reject | False | 19.55 | 0 |
| gate-08.B | jev_gate | ok | used Jev | reject | False | 68.35 | 1 |
| gate-09.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.29 | 0 |
| gate-09.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.36 | 0 |
| gate-10.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.35 | 0 |
| gate-10.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.50 | 0 |
| gate-11.A | jev_gate | ok | n/a (arm has no Jev) | accept | False | 19.07 | 0 |
| gate-11.B | jev_gate | ok | used Jev | accept | False | 124.79 | 2 |
| gate-12.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.26 | 0 |
| gate-12.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.31 | 0 |
| gate-13.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.98 | 0 |
| gate-13.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 15.64 | 0 |
| gate-14.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| gate-14.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.38 | 0 |
| gate-15.A | jev_gate | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.39 | 0 |
| gate-15.B | jev_gate | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| rerank-01.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.26 | 0 |
| rerank-01.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.30 | 0 |
| rerank-02.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.43 | 0 |
| rerank-02.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| rerank-03.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.26 | 0 |
| rerank-03.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.33 | 0 |
| rerank-04.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.35 | 0 |
| rerank-04.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.92 | 0 |
| rerank-05.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| rerank-05.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| rerank-06.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.32 | 0 |
| rerank-06.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.40 | 0 |
| rerank-07.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.50 | 0 |
| rerank-07.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.39 | 0 |
| rerank-08.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.26 | 0 |
| rerank-08.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.36 | 0 |
| rerank-09.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| rerank-09.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.37 | 0 |
| rerank-10.A | jev_rerank | ok | n/a (arm has no Jev) | ['rb-timeout', 'rb-manual', 'rb-oom', 'rb-disk', 'rb-registry'] | False | 22.72 | 0 |
| rerank-10.B | jev_rerank | ok | used Jev | ['rb-timeout', 'rb-manual', 'rb-oom', 'rb-disk', 'rb-registry'] | False | 73.88 | 1 |
| rerank-11.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.27 | 0 |
| rerank-11.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.32 | 0 |
| rerank-12.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| rerank-12.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.63 | 0 |
| rerank-13.A | jev_rerank | ok | n/a (arm has no Jev) | ['v4.1.0', 'v4.1.2', 'v4.2.1', 'v4.0.0', 'v4.4.0'] | False | 14.73 | 0 |
| rerank-13.B | jev_rerank | ok | used Jev | ['v4.1.0', 'v4.1.2', 'v4.2.1', 'v4.0.0', 'v4.4.0'] | False | 73.29 | 1 |
| rerank-14.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.32 | 0 |
| rerank-14.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.39 | 0 |
| rerank-15.A | jev_rerank | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| rerank-15.B | jev_rerank | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.37 | 0 |
| review-01.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| review-01.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.42 | 0 |
| review-02.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.30 | 0 |
| review-02.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.35 | 0 |
| review-03.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.25 | 0 |
| review-03.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.34 | 0 |
| review-04.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.35 | 0 |
| review-04.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.61 | 0 |
| review-05.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.41 | 0 |
| review-05.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.54 | 0 |
| review-06.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| review-06.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.40 | 0 |
| review-07.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| review-07.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.48 | 0 |
| review-08.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| review-08.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.34 | 0 |
| review-09.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.27 | 0 |
| review-09.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 15.51 | 0 |
| review-10.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.86 | 0 |
| review-10.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| review-11.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.34 | 0 |
| review-11.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.42 | 0 |
| review-12.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.29 | 0 |
| review-12.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.38 | 0 |
| review-13.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.36 | 0 |
| review-13.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.39 | 0 |
| review-14.A | jev_review | ok | n/a (arm has no Jev) | apply | False | 15.44 | 0 |
| review-14.B | jev_review | ok | used Jev | apply | False | 57.66 | 1 |
| review-15.A | jev_review | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| review-15.B | jev_review | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.43 | 0 |
| screen-01.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.25 | 0 |
| screen-01.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.37 | 0 |
| screen-02.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.34 | 0 |
| screen-02.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.39 | 0 |
| screen-03.A | jev_screen | ok | n/a (arm has no Jev) | clean | False | 15.97 | 0 |
| screen-03.B | jev_screen | ok | used Jev | clean | False | 67.10 | 1 |
| screen-04.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.32 | 0 |
| screen-04.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.43 | 0 |
| screen-05.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.54 | 0 |
| screen-05.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 15.48 | 0 |
| screen-06.A | jev_screen | ok | n/a (arm has no Jev) | injection | False | 14.55 | 0 |
| screen-06.B | jev_screen | ok | used Jev | injection | False | 53.76 | 1 |
| screen-07.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.30 | 0 |
| screen-07.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.33 | 0 |
| screen-08.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.28 | 0 |
| screen-08.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.34 | 0 |
| screen-09.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.34 | 0 |
| screen-09.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.31 | 0 |
| screen-10.A | jev_screen | ok | n/a (arm has no Jev) | injection | False | 17.36 | 0 |
| screen-10.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 26.29 | 0 |
| screen-11.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.23 | 0 |
| screen-11.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.36 | 0 |
| screen-12.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.31 | 0 |
| screen-12.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.40 | 0 |
| screen-13.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.25 | 0 |
| screen-13.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.39 | 0 |
| screen-14.A | jev_screen | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.24 | 0 |
| screen-14.B | jev_screen | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.31 | 0 |
| screen-15.A | jev_screen | ok | n/a (arm has no Jev) | final line is not JSON | False | 15.29 | 0 |
| screen-15.B | jev_screen | ok | used Jev | clean | False | 57.91 | 1 |
| verify-01.A | jev_verify | ok | n/a (arm has no Jev) | contradicted | False | 2.24 | 0 |
| verify-01.B | jev_verify | ok | used Jev | contradicted | False | 4.58 | 1 |
| verify-02.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.33 | 0 |
| verify-02.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.44 | 0 |
| verify-03.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.41 | 0 |
| verify-03.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.35 | 0 |
| verify-04.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.44 | 0 |
| verify-04.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.38 | 0 |
| verify-05.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.26 | 0 |
| verify-05.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.31 | 0 |
| verify-06.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.27 | 0 |
| verify-06.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.45 | 0 |
| verify-07.A | jev_verify | ok | n/a (arm has no Jev) | contradicted | False | 14.75 | 0 |
| verify-07.B | jev_verify | ok | used Jev | contradicted | False | 56.54 | 1 |
| verify-08.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.35 | 0 |
| verify-08.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.41 | 0 |
| verify-09.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.33 | 0 |
| verify-09.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.53 | 0 |
| verify-10.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.35 | 0 |
| verify-10.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.71 | 0 |
| verify-11.A | jev_verify | ok | n/a (arm has no Jev) | contradicted | False | 19.28 | 0 |
| verify-11.B | jev_verify | ok | used Jev | contradicted | False | 100.55 | 1 |
| verify-12.A | jev_verify | ok | n/a (arm has no Jev) | final line is not JSON | False | 15.75 | 0 |
| verify-12.B | jev_verify | ok | used Jev | unsupported | False | 87.19 | 1 |
| verify-13.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.35 | 0 |
| verify-13.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.74 | 0 |
| verify-14.A | jev_verify | failed: rc=0 subtype=error | n/a (arm has no Jev) | no final answer line | False | 14.27 | 0 |
| verify-14.B | jev_verify | failed: rc=0 subtype=error | no Jev call | no final answer line | False | 14.33 | 0 |
| verify-15.A | jev_verify | ok | n/a (arm has no Jev) | verified | False | 15.59 | 0 |
| verify-15.B | jev_verify | ok | used Jev | verified | False | 56.46 | 1 |