Overview

The dynamic judgments feature allows you to save time and cost by only collecting as many judgments as you need, based on contributor agreement. Dynamic judgments is an enhancement to the multiple judgments feature, that allows you to stop collecting judgments as soon as the unit has reached your pre-defined agreement threshold. Where there is greater disagreement, judgments will continue to be collected until your agreement criteria is reached, or until the max number of judgments is exhausted. 

This article describes how to use Dynamic Judgments in Quality Flow.

Job Settings

Dynamic judgment settings can be found in the JOB SETTINGS tab for your job. First, under "Rows Settings" you will indicate the minimum number of judgments you will collect in "Judgments per Row", next under "Dynamic Judgments" you will toggle Dynamic Judgments to ON and then go on to configure the remainder of your dynamic judgments settings, as described below.

 

A. "Rows Settings" > "Judgments per Row" - this will be the minimum number of judgments to collect

• 2 judgments is the recommended minimum, whether or not additional judgments will be required will be based on the level of agreement on these two judgments

B. "Dynamically Collect Judgments" - toggle to the right to turn dynamic judgment collection ON

C. "Max Judgments per Row" - this will be the maximum number of judgments you will collect for each row, if the criteria for collecting more judgments are met

• depending on the complexity of your data, or job design we recommend to set this between 3 and 5. For example, if you set Judgments per Row to 2 and max judgments to 5, you will always collect at least 2 judgments, judgments will continue to be collected until the matching criteria is met, or until you have collected 5 judgments

D. "Enabled Questions" - select which questions from your job design should be included in the matching/confidence criteria.

• matching and confidence are assessed per question, i.e. if one question in the unit doesn't match, or falls below the confidence threshold, then a new judgment will be required until the max is reached, if you are only concerned with agreement on particular questions you can include/exclude questions here
• also note that nested questions with conditional logic will not appear here and hence cannot be excluded. For example, if you are collecting minimum 3 and maximum 5, and you collect 3 matches for the top level question, but the nested questions are not matching, quality flow will continue to collect until either there are 3 nested matches, or until the maximum is reached.

E. "Criteria to Stop Collecting Judgments" - select whether to stop collecting judgments when either:

• minimum confidence is met (note it is not recommended to use confidence-based criteria when your job does not have test questions) or;
• when enough matching judgments are met, this defaults to the number you have specified in the Judgments per Row setting (if you do not enter anything here the number will remain 0, but the system will take the judgments per row number as input).

Examples

The following examples will show dynamic judgments in action, and demonstrate how choosing Matching or Confidence as criteria can affect the outcome.

Matching

In our example job, contributors are asked to categorise an animal. The dynamic judgment settings are set to collect a maximum of 5 judgments, stopping once there are two matching.

Job design

Dynamic Judgment Settings

Results

As we can see on the DATA page, for this job in the Judgment View, Unit 1_1 was final after judgments in total because both contributors selected matching answers "mammal". In Unit 1_2, three judgments were collected because contributor 1 chose "mammal" while contributor 2 chose "other"; the unit collected an additional judgment and contributor 3 chose "mammal", this matched with contributor 1 and the unit is now finalised.

Confidence

The next example has identical job design but the job includes test questions with dynamic judgements set to Confidence matching. 

Settings

Trust Scores

When a job includes test questions, contributors acquire a "trust score", which is downloadable in the Contributor Daily Report, as well as visible in the Contributor Performance Dashboard. based on their performance on those questions, which is used as a weighting when calculating confidence. Whereas in the above example, each contributor's answer has equal weight, and matching is majority agreement, in confidence the criteria isn't reached until there is agreement, and the confidence in that agreement meets the minimum threshold set.

Results

Examining the results of this job, we can see that while Unit 1_6 is final after two exact matches, the picture changes once there is any disagreement.

In Unit 1_7, again there is a 2/1 agreement and the unit is able to finalise after the third judgment, however it is important to note that the 2/1 agreement is between the contributor's with the two highest trust scores. This unit ends up with a confidence of .75 because of the weighting.

The contributor who chose "amphibian" has a trust score of 66.67, while the other two contributors, who both chose "reptile" have trust scores of 100.

The overall trust for this unit is 266.67 and confidence in the answer "amphibian" is 66.67/266.67 or .25, while the confidence in the answer "reptile" is 200/266.67, or .75, this is above the minimum threshold of .65 and so the unit finalises.

In Unit 1_8, however there is agreement between the lower trust contributor and one of the higher trust contributors, who both disagree with the other higher trust contributor. This alters the confidence as follows.

The overall trust for this unit is still 266.67 at this point in the job, however the confidence in the answer "fish" is  100/266.67, or .37 while the confidence in the answer "bird" is 166.67/266.67 or .63, which is not above the minimum threshold, and so the unit is still in Working status waiting for another judgment to bring it over the threshold.