Download contrast set (JSONL)
Every answered probe is a labeled (original, counterfactual) pair.
Boundary sensitivity by label
Of the minimal edits intended to cross each label’s boundary, the share annotators confirmed as flips. High values mean the label sits on a knife’s edge; low values mean it is robust to small perturbations.
Annotator invariance consistency
Paraphrase probes should never flip a label. Annotators who flip on paraphrases are answering inconsistently — an attention signal collected without planting a single fake gold item.
Where labels flip
The most recent confirmed flips: the exact edit that crossed the boundary, and the annotator’s rationale for why.