In some cases, rows may collect more judgments than expected. This article provides some possible answers as to why over-collection may occur.
Forgiving Test Questions
- If the correct answer for a Test Question is changed, contributors trust scores re-calculated accordingly.
- Please note: If those contributors were considered untrusted before and were actually correct on those Test Questions, they can become trusted once the test questions are forgiven. When they become trusted, the rows they judged now have an additional trusted judgment which may result in "too many" trusted judgments.
No Quiz Mode
- When there is no quiz mode, contributors can judge lots of rows before seeing enough Test Questions to become trusted. Because Figure Eight only counts trusted judgments on a row, if after a certain time contributors become trusted, those judgments that were previously untrusted become trusted, and the row can over-collect judgments.
Looking at the Data Page
- On the Data Page (jobs/{{job_id}}/units), the judgments displayed are for both trusted and untrusted judgments. A user sets how many trusted judgments they want in a job in the Job Settings menu.
Looking at an Individual Unit Page
- On the individual unit page (jobs/{{job_id}}/units/{{unit_id}}), the judgments here are total judgments. Click on "Show Worker Info" to see how many workers are trusted and for more contributor information.
Dynamic Judgments are Enabled
- Dynamic Judgments allows the user to request a range of judgments in the hopes of achieving a minimum confidence on each row.
Example: If a job is set to collect 3 judgments per row and dynamic judgments mode is turned on with max judgments per row set to 6, then rows should collect 3-6 trusted judgments.
“Rows Remain Finalized” Checked
- If a job has "Rows Remain Finalized" checked via the API Settings Page, rows in this job will not order new judgments once they are finalized, even if some of those judgments become untrusted. As a result, rows can potentially under-collect.
Example: when a job is set to collect 3 judgments/row, and a row has collected 3, if one of the contributors later become untrusted, because they fall below the trust threshold, then that row will only show 2 trusted judgments. The third judgment will not be collected again in this case.