Overview
Annotators
Instances
Questions
Behavioral
Crowdsourcing
{% if bws_enabled %}
BWS Scoring
{% endif %}
{% if mace_enabled %}
MACE
{% endif %}
{% if embedding_viz_enabled %}
Embeddings
{% endif %}
Configuration
System Information
| User ID | Phase | Annotations | Working Time | Avg Time/Annotation | Speed (per hour) | Completion % | Max Instances | Last Activity |
|---|
| Instance ID | Text Preview | Annotations | Completion % | Most Frequent Label | Disagreement | Avg Time | Annotators | Num AI Used |
|---|
AI Assistance Usage
Quality Indicators
| User ID | Instances | Avg Time (s) | Interactions | Changes | AI Requests | AI Accept Rate | Suspicion |
|---|
Loading...
INSTANCE PREVIEW
Hover over a point to see details
SELECTION QUEUE
Use lasso or box selection to select points
Selected items will be prioritized for annotation
CONTROLS
Loading BWS scoring data...
BWS Item Scores
Compute item scores from best-worst annotations. Counting is the default method (no dependencies).
Bradley-Terry and Plackett-Luce require the choix package.
| Rank | Item ID | Text | Score | Best Count | Worst Count | Appearances |
|---|---|---|---|---|---|---|
| Click "Generate Scores" to compute BWS scores | ||||||
Loading MACE data...
Annotator Competence
MACE estimates each annotator's reliability (0 = random guessing, 1 = always correct).
| Annotator | Competence | Reliability |
|---|---|---|
| Click "Run MACE" to compute scores | ||
Predicted Labels
MACE's best estimate of the true label for each item, weighted by annotator competence.
Run MACE to see predictions.