Dynamic Judgments allows each row within the job to collect a variable number of judgments dependent upon contributor's agreement. This feature will enable the job to collect fewer judgments on rows where contributors agree on the answer and more judgments on rows where contributors have submitted answers that conflicted with one another. When the confidence on one or more of the question is below the Minimum Confidence set, the row will collect additional judgments until Max Judgments per Row is hit. This can decrease the overall costs in the job as well as improve accuracy on rows that are difficult to answer.
Steps to Enable Dynamic Judgments
- Set an amount in the Judgments per Row setting.
Fig. 1: Judgment Per Row Setting On The Launch Page
- Navigate to the Dynamic Judgments Page (Settings > Quality Control > Dynamic Judgments).
Fig. 2: Dynamic Judgments Setting Unchecked
- Enable Dynamic Judgments setting
- Max Judgments per row: determines the maximum number of Judgments that will be collected for each row
- Enabled Questions: determines which questions' agreement is measured when deciding to either finalize a row or collect an additional judgment.
- Criteria to stop collecting judgements - Minimum Confidence: sets the necessary aggrement required to determine whether a row is Finalized or whether it requires an additional Judgment. If the Confidence for this row is above this value, the row will finalize and stop collecting judgments. Take a look at this article for help with understanding how Confidence is calculated.
- Criteria to stop collecting judgements – Matching Judgements: the other way to stop collecting more judgements is by defining how many judgements need to match before finalizing a unit. When there’s not enough matching judgements, more judgements will be collected until the maximum number of judgements is reached.
Fig. 3: Dynamic Judgment Setting (Enabled)
Here is a possible scenario for the Dynamic Judgment setting:
Fig. 4: Enabled Dynamic Judgments With 5 Max Judgments, 0.7 Confidence Score For The Question Named "Sentiment"
In this example, the only Enabled Question available is "photo_category". So this will be the only question that will be considered when determining Contributors' agreement. If the Judgment Per Row Setting is set to 2, instead of just collecting 2 judgments per row, each row will collect a maximum of 5 judgments if the confidence field for question "sentiment" does not reach 0.70.
Here is an example of Dynamic Judgments in use:
- The first Contributor answers "category1" for "photo_category"
- The second Contributor answers "category2" - the Confidence measure is approximately 0.50
- The first two judgments did not reach a Confidence measure of 0.70, thus, the row will collect additional judgments.
- The third Contributor answers "category1" - the Confidence measure is approximately 0.67
- 0.67 is still less than the Minimum Confidence of 0.70 and the platform will distribute the row to a fourth Contributor:
- If the fourth Contributor answers "category1" - the Confidence measure is approximately 0.75 and the row becomes Finalized
- If the fourth Contributor answers "category2" - the Confidence measure is approximately 0.50 and the row will be sent to a fifth Contributor. The fifth Contributor will submit the last and final Judgment on the row no matter what he/she answers since Max Judgments per row is set to 5.
- Taking a step back, if both the first and second contributors both answers “category 1” for this question, the confidence would become 1.00, and the row would not continue collecting more judgements.
Note: Dynamic Judgments should be used with caution as it may increase the cost and lower throughput of the job.
Note: Dynamic Judgments setting cannot be disabled once the job is launched.