Overview
Quality Flow projects can collect between 1 - 200 judgments per row.
When collecting more than one judgment per row, there are some important notes to be aware of:
- Once a job has been set to start data routing at least once, the number of judgments to collect cannot be changed.
- It is possible to collect more that one judgment for unit groups, but in this case you only have the option to retain groupings.
- Only work jobs can collect multiple judgments
- Work jobs collecting multiple judgments are not currently available for following QA job creation. This feature is coming soon.
View Multiple Judgment Results
- On the DATASET summary there are two views: unit view and judgment view. Unit view allows you to browse and manage the source data easily. If a unit is collecting multiple judgments, all judgement-related information will be collapsed into one row of the same unit.
- After switching to Judgment view, each judgment will be visible in its own row with a unique judgment number.
- Column "Max judgments" means the max number of judgments the job is set to collect.
- Column "Judgments collected" means the number of judgments the unit has collected so far.
- In the Unit column the "Status" refers to the unit status. In the Latest Judgment column, the status is at judgment level. This means you can track each contributor's progress on different judgments.
- When a unit has not collected any judgments yet the unit status is JUDGABLE. When a unit has collected the max number of judgments, the unit status is SUBMITTED. When a unit has collected at least one judgment, but has not collected the max number of judgments yet, its status is WORKING.
- The DATASET summary displays the total number of units and total number of judgments to be collected in the top row. Click on TOTAL UNITS or TOTAL JUDGMENTS to view the relevant progress numbers.
Agreement and Confidence
For form tools (radios, ratings and checkboxes), agreement per row and confidence per judgment are available in the dataset table and report downloads. Please see this article and this article for more information about how these metrics are arrived at. Worker agreement (Wawa) can be viewed in the Contributor dashboard.
Download Multiple Judgment Reports
- In the Dataset download, by default, each judgment is given in a separate row with a unique judgment number.
- It is also possible to download aggregated results at the Job level under Job > Results.
- Four aggregation methods are supported:
- Best answer ('agg') - Returns the top 1 response.
- All answers ('all') - Returns all responses.
- Numeric average ('avg') - Returns a numeric average calculated from all results.
- Top # Answers ('agg_x') - Returns the top 'x' responses.