{% include "admin/_page_header.html" %}

Labeling engine

{% if engine_off %}

The labeling engine is off

This study does not use the labeling engine. To turn it on, add a labeling_engine block with enabled: true, at least one source and a contract to the config, and restart the server. See the labeling engine documentation.

{% else %}

Machines label the corpus; the people's labels on a random, blind sample decide which machine labels can be accepted, and with what guarantee. Everything below is computed from those labels: nothing here is a machine grading itself.

Overview

Loading…

Open certificate Certifying with new labels spends part of each contract's error budget (each new look is tested at a stricter level), so it happens on its own as labels arrive.

Human requests

What people have been asked to label, and why. Calibration, certification and audit requests are random and blind; only those estimate accuracy.

Model calls

Every request the engine sent to a model, newest first. Prompts and replies are stored as hashes only.

{% endif %}