How the judge's yes and no become scores

The judge never gives a number; the code counts. Claims mode.

How the judge's yes and no become scores The judge never gives a number; the code counts. Claims mode. 01 / Inputs 02 / Split into claims 03 / Judge: yes or no only 04 / Code counts, not the judge 05 / Score Answer · what the system said · 01 / Inputs Answer what the system said Reference · the right answer · 01 / Inputs Reference the right answer Answer claims · single facts · 02 / Split into claims Answer claims single facts Extracts · what the system read · 02 / Split into claims Extracts what the system read Reference facts · single facts · 02 / Split into claims Reference facts single facts Backed by extracts? · claims vs extracts · 03 / Judge: yes or no only Backed by extracts? claims vs extracts In the reference? · claims vs reference · 03 / Judge: yes or no only In the reference? claims vs reference In the answer? · facts vs answer · 03 / Judge: yes or no only In the answer? facts vs answer Backed by extracts? · facts vs extracts · 03 / Judge: yes or no only Backed by extracts? facts vs extracts Faithfulness · backed / claims · 04 / Code counts, not the judge Faithfulness backed / claims Precision · matching / claims · 04 / Code counts, not the judge Precision matching / claims Recall · found / facts · 04 / Code counts, not the judge Recall found / facts Context recall · backed / facts · 04 / Code counts, not the judge Context recall backed / facts Answer correctness · F1 = 2PR / (P + R) · 05 / Score Answer correctness F1 = 2PR / (P + R) Quick mode · agreeing claims / claims · 05 / Score · one call, no context recall Quick mode agreeing claims / claims one call, no context recall split split claims claims read read facts facts yes / no yes / no yes / no yes / no P R Legend primary data Text the system read data store data flow