Overview

The dynamic judgments feature allows you to save time and cost by only collecting as many judgments as you need, based on contributor agreement. Dynamic judgments is an enhancement to the multiple judgments feature, that allows you to stop collecting judgments as soon as the unit has reached your pre-defined agreement threshold. Where there is greater disagreement, judgments will continue to be collected until your agreement criteria is reached, or until the max number of judgments is exhausted. 

This article describes how to use Dynamic Judgments in Quality Flow.

Job Settings

Dynamic judgment settings can be found in the JOB SETTINGS tab for your job. First, under "Rows Settings" you will indicate the minimum number of judgments you will collect in "Judgments per Row", next under "Dynamic Judgments" you will toggle Dynamic Judgments to ON and then go on to configure the remainder of your dynamic judgments settings, as described below.

A. "Rows Settings" > "Judgments per Row" - this will be the minimum number of judgments to collect

• 2 judgments is the recommended minimum, whether or not additional judgments will be required will be based on the level of agreement on these two judgments

B. "Dynamically Collect Judgments" - toggle to the right to turn dynamic judgment collection ON

C. "Max Judgments per Row" - this will be the maximum number of judgments you will collect for each row, if the criteria for collecting more judgments are met

• depending on the complexity of your data, or job design we recommend to set this between 3 and 5. For example, if you set Judgments per Row to 2 and max judgments to 5, you will always collect at least 2 judgments, judgments will continue to be collected until the matching criteria is met, or until you have collected 5 judgments

D. "Enabled Questions" - select which questions from your job design should be included in the matching/confidence criteria.

• matching and confidence are assessed per question, i.e. if one question in the unit doesn't match, or falls below the confidence threshold, then a new judgment will be required until the max is reached, if you are only concerned with agreement on particular questions you can include/exclude questions here

E. "Criteria to Stop Collecting Judgments" - select whether to stop collecting judgments when either:

• minimum confidence is met (note it is not recommended to use confidence-based criteria when your job does not have test questions) or;
• when enough matching judgments are met, this defaults to the number you have specified in the Judgments per Row setting (if you do not enter anything here the number will remain 0, but the system will take the judgments per row number as input).