How reliability works

Version 2.1 (18 October 2026). Controller: {{controller_name}}, {{controller_email}}.

This page explains how the site {{app_host}} assesses how reliable answers are and how the labels in the dataset are made from them. It is information; we do not ask you to accept it. The legal description of this processing, its legal basis and your rights are in the privacy notice (section 9).

The page describes the rules that stay the same. The exact thresholds and settings are tuned from time to time; their current values are in the technical description of the model in the croatian-stemmer repository on GitHub.

1. In short

Not all answers count the same. For each participant and each kind of task (word pairs, lemmas, stems) the application estimates how accurate their answers are and gives them weight accordingly. An item counts as resolved only when enough reliable answers agree on it. The estimates are recomputed regularly from all answers so far, so new answers show up in them with a short delay.

The estimate affects only three things: how much your answers count, which items you are shown, and whether your answers are set aside for review (section 7). It has no effect outside this site, and nothing about you as a participant is ever published.

2. Known-answer items

Items whose correct answer is known in advance are mixed in with the others, unannounced. Participants at the lower levels and those in a review pause get them more often, established participants less often. Checks can come from any published corpus, not only the one you chose. Some of these checks are items where the corpus suggestion is wrong. An answer on such an item is taken into account only from the following day. Which of your answers were checks on items still in use is shown neither on the site nor in the file with your data, because it would otherwise reveal those items; on request by email we retire those items and send you the details within one month.

3. Items without a suggestion

Some word-pair and lemma items are shown without the corpus suggestion: the suggestion line is not shown and the lemma field is empty. This shows us whether you judge for yourself or only confirm the suggestion. The share is the same for known-answer items and for the others, so an item shown without a suggestion does not tell you whether it is a check.

4. Knowledge check

The first items of each kind of task are an announced knowledge check; the screen says so and shows how far you are. They are always shown without the suggestion. Answers where you made a choice are counted; "Not sure" is not counted. If you answer enough of them correctly, you become qualified for that kind. The check cannot be retaken. If you do not pass it, you go on at the level "new", and you can still reach the established level through known-answer items. While the knowledge check for a kind is in use, you get items of that kind from any corpus; you can choose the corpus yourself from the qualified level up.

5. Levels

The level is set separately for each kind of task:

new: everyone at the start.
qualified: passed the knowledge check for that kind.
established: enough known-answer checks that tell knowledge apart from confirming the suggestion (items shown without the suggestion or with a wrong suggestion; for stems, all checks), high enough accuracy on them, enough answers where you made a choice, and enough right answers on items where the suggestion is wrong. You keep the level while your accuracy on those checks stays good enough. The established level is possible only once there are enough known-answer items and at least one expert for that kind. Agreeing with other participants never raises your level.
expert: people on the list of trusted experts that the controller keeps.

Each level has an upper limit on the estimate, higher for higher levels. However accurate someone is, their estimate cannot go above the limit of their level, which limits how much one account can influence the result. The lower limit is 0.50, the level of guessing: the answers of a very unreliable participant are then simply disregarded, not counted as the opposite.

6. When an item is resolved

An item is resolved when one answer is very probable, at least two different people with a good enough accuracy estimate chose it, and at least one established person or expert is among them. One expert's answer never resolves an item alone. The resolution is provisional until a sample of items confirms that items resolved this way agree with the controller's answer. On items the controller checks personally, the controller's answer is final.

7. Signs of possible abuse and the review pause

Every calculation looks for these signs:

answering very fast, faster than careful reading allows;
a long run of identical answers on word pairs or stems;
low accuracy on the latest known-answer items;
answers that repeat a wrong suggestion, while the other checks are mostly right;
on items an expert also answered, frequent disagreement with the expert's answer;
a very large number of answers in a short time, together with answering very fast.

One sign on its own does nothing. When two different signs appear in one calculation, an automatic review pause starts (status "set aside"). Your answers are then disregarded and you get more known-answer items, but you can still label, download your data and erase your profile. The pause is not announced or shown on the site, because an honest mistake can also trigger it; it can be seen, with the reasons and values, in the file you download on the "My data" page. The controller reviews the pause within 14 days and decides: end it, keep it, or block the profile. If nobody decides, the pause ends automatically after 14 days. Blocking is never automatic: a person decides, with a reason that is shown to you. Experts are never paused automatically. If you erase your profile while it is paused or blocked, your answers are deleted.

8. Objections and questions

If you disagree with an assessment or with a decision about your profile, write to {{controller_email}} and a person will review it. You can object to the assessment itself the same way. Without the assessment we cannot use your answers, so the consequence of an objection is that we no longer use them: we block your profile with the reason "objection" and show it to you as such. We will also offer you erasure of your profile.

9. What you see on the "My data" page

Under "Reliability" you see, for each kind of task, your level and an estimate in words: high, good, needs attention, or not enough data (too few known-answer checks so far). The number of checks and your agreement with resolved items are shown rounded. A review pause is not shown there; a block is, with its reason. This section is a simplified view. The complete data are in the file (JSON) you download there: every retained calculation (30 days) with level, estimate, weight, number of checks and signs of possible abuse with their values, and every change of status and level, with an explanation of each field.

10. What is public

Public releases of the dataset, published as files attached to releases of the croatian-stemmer repository on GitHub, contain only aggregated item labels. They contain no participant identifiers, individual answers, levels, weights, estimates or answer counts. Nothing is published about any individual participant.

Appendix: formulas

w(θ, K) = ln(θ · (K − 1) / (1 − θ))
P(z = c | votes) ∝ π_c · exp(Σ w(θ_a, K) over the votes for c)
θ⁽⁰⁾ = clip((α₀ + h) / (α₀ + β₀ + h + m))
s_a += λ · q(c(v)),  u_a += λ · (1 − q(c(v)))
s_a + u_a ≤ ρ · max(h + m, f)
θ⁽ᵗ⁾ = clip((α₀ + h + s) / (α₀ + β₀ + h + m + s + u))
θ_lb = μ − 1.645 · √(μ(1 − μ) / (α₀ + β₀ + n + 1)),  μ = (α₀ + hits) / (α₀ + β₀ + n)
priority = 1 − (p₁ − p₂)

θ (θ_a): the accuracy estimate of person a for one kind of task, the probability that their answer is right. clip keeps it between 0.50 and the upper limit of the level.
w(θ, K): the weight of one answer; K is the number of possible answers (2 for word pairs and stems; for lemmas the number of different lemmas given plus one, with a minimum); ln is the natural logarithm.
P(z = c | votes): the probability that the item's correct answer is c, given the answers (votes); π_c is the starting probability of answer c; the sum (Σ) runs over the weights of all answers for c; ∝ means "proportional to", and the probabilities are then scaled to add up to 1.
θ⁽⁰⁾: the starting estimate; α₀ and β₀ are starting values (higher for experts); h and m are the numbers of right and wrong answers on known-answer items.
s_a and u_a: the expected numbers of right and wrong answers of person a derived from agreement with others; λ is less than 1, so such evidence counts less than a check; q(c(v)) is the probability of the answer c(v) the person gave in answer v, computed only from other people's answers to the same item.
s_a + u_a ≤ ρ · max(h + m, f): evidence from agreement with others is capped by the evidence from checks (ρ and f are fixed settings), however many people agree with each other.
θ⁽ᵗ⁾: the estimate after pass t (s = s_a, u = u_a); the calculation has a fixed number of passes.
θ_lb: the lower bound of the estimate (about the 5th percentile) from checks that tell knowledge apart from confirming the suggestion; n is the number of such checks, hits the number right; it is used for promotion to the established level.
priority: the priority of an unresolved item when items are handed out; p₁ and p₂ are the two highest answer probabilities, so items where the difference is smaller get higher priority.

The current values of all settings are in the technical description of the model.