How reliability works

Version 1.0 (18 October 2026). Controller: {{controller_name}}, {{controller_email}}.

This page explains how the site {{app_host}} assesses how reliable answers are and how the labels in the dataset are made from them. It is information; we do not ask you to accept it. The legal description of this processing, its legal basis and your rights are in the privacy notice (section 9).

1. In short

Not all answers count the same. For each participant and each kind of task (word pairs, lemmas, stems) the application estimates how accurate their answers are and gives them weight accordingly. An item counts as resolved only when enough reliable answers agree on it. The estimates are recomputed about every 10 minutes from all answers so far, so new answers show up in them with a short delay.

2. Known-answer items

Items whose correct answer is known in advance are mixed in with the others, unannounced. Most participants get one in about every ten items shown, established participants one in every twenty. About a quarter of these checks are items where the corpus suggestion is wrong. An answer on such an item is taken into account only from the following day. Which of your answers were checks on items still in use is shown neither on the site nor in the file with your data, because it would otherwise reveal those items; on request by email we retire those items and send you the details within one month.

3. Items without a suggestion

About one in five word-pair and lemma items is shown without the corpus suggestion: the suggestion line is not shown and the lemma field is empty. This shows us whether you judge for yourself or only confirm the suggestion. The share is the same for known-answer items and for the others, so an item shown without a suggestion does not tell you whether it is a check.

4. Knowledge check

The first 12 items of each kind of task are an announced knowledge check ("Knowledge check n/12"). They are always shown without the suggestion. Answers where you made a choice are counted; "Not sure" is not counted. If you answer at least 9 of the 12 correctly, you become qualified for that kind. The check cannot be retaken. If you do not pass it, you go on at the level "new", and you can still reach the established level through known-answer items.

5. Levels

The level is set separately for each kind of task:

new: everyone at the start.
qualified: passed the knowledge check for that kind.
established: at least 30 known-answer checks that tell knowledge apart from confirming the suggestion (items shown without the suggestion or with a wrong suggestion; for stems, all checks), a high enough lower bound of the accuracy estimate from those checks (0.75), at least 100 answers where you made a choice, and at least 5 checks where the suggestion is wrong, of which at least 60% are right (for stems: at least 5 checks for each of the two possible answers, with at least 60% right for each). The level stays while the lower bound does not fall below 0.70 and you have at least 50% right on those hard items. The established level is possible only once there are enough known-answer items and at least one expert for that kind. Agreeing with other participants never raises your level.
expert: people on the list of trusted experts that the controller keeps.

Each level has an upper limit on the estimate: new 0.75, qualified 0.80, established 0.95, expert 0.98. However accurate someone is, their estimate cannot go above the limit of their level. This limits how much one account can influence the result. The lower limit is 0.50: the answers of a very unreliable participant are then simply disregarded, not counted as the opposite.

6. When an item is resolved

An item is resolved when one answer has a probability of at least 95%, at least two different people have answered it, and at least one established person or expert voted for that answer. One expert's answer never resolves an item alone. The resolution is provisional until a sample of 30 items confirms that items resolved this way agree with the controller's answer (at least 29 of 30). On items the controller checks personally, the controller's answer is final.

7. Signs of possible abuse and the review pause

Every calculation looks for these signs:

answering very fast (the median time of the last 30 answers is shorter than one second or than a quarter of the experts' median);
at least 12 identical answers in a row on word pairs or stems;
less than 60% right on the last 20 known-answer items (after at least 12);
answers that repeat a wrong suggestion: at most 40% right on at least 5 items with a wrong suggestion, while at least 90% of the other checks are right;
on items an expert also answered, more than 30% of at least 10 answers differ from the expert's answer;
at least 400 answers in one hour, together with answering very fast.

When two different signs appear in one calculation, an automatic review pause starts (status "set aside"). Your answers are then disregarded and you get more known-answer items, but you can still label, download your data and erase your profile. The pause is not announced or shown on the site, because an honest mistake can also trigger it; it can be seen, with the reasons and values, in the file you download on the "My data" page. The controller reviews the pause within 14 days and decides: end it, keep it, or block the profile. If nobody decides, the pause ends automatically after 14 days. Blocking is never automatic: a person decides, with a reason that is shown to you. Experts are never paused automatically. If you erase your profile while it is paused or blocked, your answers are deleted.

8. Objections and questions

If you disagree with an assessment or with a decision about your profile, write to {{controller_email}} and a person will review it. You can object to the assessment itself the same way. Without the assessment we cannot use your answers, so the consequence of an objection is that we no longer use them: we block your profile with the reason "objection" and show it to you as such. We will also offer you erasure of your profile.

9. What you see on the "My data" page

Under "Reliability" you see, for each kind of task, your level and an estimate:

high: accuracy estimate of at least 0.85;
good: at least 0.75;
needs attention: below 0.75;
not enough data: fewer than 12 known-answer checks.

The number of checks is rounded down to tens, and agreement with resolved items to 5% (shown once you have at least 20 resolved items). A review pause is not shown there; a block is, with its reason. This section is a simplified view. The complete data are in the file (JSON) you download there: every retained calculation (30 days) with level, estimate, weight, number of checks and signs of possible abuse with their values, and every change of status and level, with an explanation of each field.

10. What is public

Public releases of the dataset, published as files attached to releases of the croatian-stemmer repository on GitHub, contain only aggregated item labels. They contain no participant identifiers, individual answers, levels, weights, estimates or answer counts.

Appendix: formulas

w(θ, K) = ln(θ · (K − 1) / (1 − θ))
P(z = c | votes) ∝ π_c · exp(Σ w(θ_a, K) over the votes for c)
θ⁽⁰⁾ = clip((α₀ + h) / (α₀ + β₀ + h + m))
s_a += λ · q(c(v)),  u_a += λ · (1 − q(c(v)))
s_a + u_a ≤ max(h + m, 12)
θ⁽ᵗ⁾ = clip((α₀ + h + s) / (α₀ + β₀ + h + m + s + u))
θ_lb = μ − 1.645 · √(μ(1 − μ) / (α₀ + β₀ + n + 1)),  μ = (α₀ + hits) / (α₀ + β₀ + n)
priority = 1 − (p₁ − p₂)

θ (θ_a): the accuracy estimate of person a for one kind of task, the probability that their answer is right. clip keeps it between 0.50 and the upper limit of the level.
w(θ, K): the weight of one answer; K is the number of possible answers (2 for word pairs and stems; for lemmas the number of different lemmas given plus one, and at least 3); ln is the natural logarithm.
P(z = c | votes): the probability that the item's correct answer is c, given the answers (votes); π_c is the starting probability of answer c; the sum (Σ) runs over the weights of all answers for c; ∝ means "proportional to", and the probabilities are then scaled to add up to 1.
θ⁽⁰⁾: the starting estimate; α₀ and β₀ are starting values (4 and 2; for experts 18 and 2); h and m are the numbers of right and wrong answers on known-answer items.
s_a and u_a: the expected numbers of right and wrong answers of person a derived from agreement with others; λ = 0.5 (such evidence counts half as much as a check); q(c(v)) is the probability of the answer c(v) the person gave in answer v, computed only from other people's answers to the same item.
s_a + u_a ≤ max(h + m, 12): evidence from agreement with others cannot outweigh evidence from checks, however many people agree with each other.
θ⁽ᵗ⁾: the estimate after pass t (s = s_a, u = u_a); the calculation has 10 passes.
θ_lb: the lower bound of the estimate (about the 5th percentile) from checks that tell knowledge apart from confirming the suggestion; n is the number of such checks, hits the number right; it is used for promotion to the established level.
priority: the priority of an unresolved item when items are handed out; p₁ and p₂ are the two highest answer probabilities, so items where the difference is smaller get higher priority.