Guide to: Advanced Options and Quality Assurance Classes – Appen Success Center

Quality Flow Advanced Options can be used to fine-tune contributor review, rejection, and feedback handling within your following QA jobs. All the advanced options can be used together in the same job, allowing you to fully customize your quality assurance flows.

Advanced Options

Once you have created your following QA Job there are three advanced controls under QUALITY > Configure Job > Judgment Modification > Advanced Options as well as the ability to customize quality assurance classes.

Allow QA to modify: enables quality assurance contributors to directly edit the original contributor's responses. This allows you to capture a basic acceptance rate as well as more detailed quality metrics (for supported tools) such as questions and word error rate, based on a comparison of the original submission with the corrected version (see this article).
Allow send back to original contributor for rework: if a unit falls below a quality threshold (see below) it will be sent back to the original contributor to be reworked by them.
- Note this option must be used in conjunction with Quality Assurance Classes, as the threshold for triggering rework is controlled by these classes.
- Note that the same unit can be sent back for rework a maximum of two times, if after the second rework attempt (three total attempts) the unit still does not meet the quality standard it will not continue to be sent back.
Allow original contributor to dispute feedback: regardless of whether this option is selected, whenever feedback remarks are left for a contributor (see the Contributor View below) they will receive the feedback at the top of their work queue the next time they get a new unit to work on. If this dispute option is selected they will have the additional ability to dispute the feedback (see Original Contributor Views below). In this case the unit will then go into the feedback resolution process.

Quality Assurance Classes and Threshold

You can define these classes and a threshold score under the Manage Quality Assurance Classes modal. Quality Assurance Classes must be used when the Rework option is selected, but they are otherwise optional. These classes are a way to categorize QA feedback for contributors to help them see patterns in where they can improve, and give you information about patterns found in your data.

The classes are also used to drive:

Quality scores (see Results below)
Rework flows (mandatory)
Nuanced unit statuses (see Results below)

Step 1: Define a threshold value (e.g., 0.5). If the total severity score of all selected classes in a unit equals or exceeds the threshold, the unit will be considered REJECTED, and if the rework option is selected the unit will go back to the original contributor for revision.

Step 2: Click "Add Class" to create quality assurance classes and assign severities

A: choose a color for the class, this will be visible to the QA contributor as well as the original contributor if the unit is sent back to them
B: choose a title for the class, this should be a short name as it will be formatted as a button for the contributors (see below)
C: assign a severity between 0 and 1 to the class - "0" severity indicates that there is nothing wrong with the unit while "1" severity indicates that the unit is very poor quality. These severities interact with the threshold to determine unit statuses and scoring metrics (see below).
D: add a description for the class, this will be visible to the contributors upon hovering
E: decide whether the class selection should be displayed to the original contributor if the unit is sent back to them (with feedback or for rework), you may choose to employ hidden classes in so that QA contributors can make selections that will only be visible to you

Example

In the above example, with the threshold set to 0.5:

if the QA contributor selects "Perfect" or "Minor Corrections" the unit will be ACCEPTED as it falls below the threshold for rejection.
- if Rework is enabled it would not be triggered for these units
- if the QA contributor is allowed to Modify and made corrections then the status will be MODIFIED | ACCEPTED
if the QA contributor selects "Major Corrections" or "Rejected" the unit will be REJECTED as it falls above the threshold for rejection.
- if Rework is enabled these units would be sent back for rework if the QA contributor is allowed to Modify and made corrections the unit status will be MODIFIED | REJECTED
- Classes can be added, edited, or deleted as needed (however note changes will only be applicable to those units submitted after the changes).

QA Job View

Once these settings are applied, they are visible in the QA interface where the QA contributor can optionally:

select one or more quality assurance classes
edit the original annotation (if Allow Modify is enabled)
enter feedback remarks
- note that any remarks left will go back to the original contributor to review, regardless of any other settings or classes chosen
- if Feedback dispute is enabled, they will additionally be able to contest their QA result and go through the dispute resolution process.

Original Contributor Views

If any feedback is left for the contributor, they will receive this at the top of their queue before continuing with additional units. They can see:

Any classes selected by the QA contributor (unless the visible to contributor toggle is off)
Any remarks left by the QA contributor
In the example below no rework is triggered and dispute is not enabled, so the contributor simply receives and acknowledges the feedback before moving on to the next unit

In the following example, dispute feedback is enabled and the QA contributor also made changes to the original submission.

In this case the original contributor has the ability to dispute the feedback
They can also look into their original submission and compare it to any QA changes

Results

The results of the Advanced Quality Assurance process are visible in the DATASET table as well as in DASHBOARDS.

In the DATASET table, the status after QA is visible in the "STATUS" and "REVIEW RESULT" columns. The score assigned to the unit (each unit starts with a score of 1 and is penalised by the amount of the QA class severities). The classes that were chosen by the QA contributors for that unit are also presented.

Quality Dashboard

When quality assurance classes are used, there will be a Job Quality Score in the Quality Dashboard.

The quality score is calculated using the following formula:

1 - ((Sum of Severity of all Quality Assurance tags selected for all units in the Job) / Total QAed Units)

Higher severity tags across many judgments will reduce the Job Quality Score.
A Job Quality Score closer to 100% implies better quality and fewer critical errors.
This metric is derived from the error classes selected during QA.

Example:

Total Severity = 0 + 1 + 0.2 + 0.7 = 1.9
Total QA'd units = 4
Quality Score = 1 - (1.9 / 4) = 0.525 or 52.5%

In the above example, since QA also modified some units, the Acceptance score is calculated using the following formula:

(Number of Accepted Units without any QA changes / Total Judgments) * 100

Example:

the contributor submitted 4 judgments
- 1 was Accepted without modification or rejection.
- 2 were modified
- 1 was rejected
  Acceptance Score = (1 / 4) * 100 = 25%

Contributor Dashboard

The quality and acceptance scores for each contributor are also visible in the Contributor Dashboard.

Feature Combinations

Note that the Send for Rework option must be used with QA Classes in order to create the rejection condition

QA Modify	Dispute feedback	Send for Rework	QA Classes	Contributor Actions	Unit Statuses
✅	❌	❌	❌	QA contributor can make changes and leave feedback Original contributor can acknowledge feedback	ACCEPTED MODIFIED
✅	✅	❌	❌	QA contributor can make changes and leave feedback Original contributor can acknowledge feedback	ACCEPTED MODIFIED
✅	✅	❌	✅	QA contributor can make changes and leave feedback, as well as select QA Classes. Original contributor can acknowledge	ACCEPTED MODIFIED \| ACCEPTED MODIFIED \| REJECTED REJECTED (may also be PENDING until the Feedback resolution process is complete)
✅	❌	✅	✅	QA contributor can make changes and leave feedback, as well as select QA Classes. If the QA classes have high enough severity, the unit will also be sent back to the original contributor for additional rework. Original contributor can acknowledge and perform the Rework	ACCEPTED MODIFIED \| ACCEPTED MODIFIED \| REJECTED REJECTED
✅	❌	❌	✅	QA contributor can make changes and leave feedback, as well as select QA Classes. Original contributor can acknowledge	ACCEPTED MODIFIED \| ACCEPTED MODIFIED \| REJECTED REJECTED
✅	✅	✅	✅	QA contributor can make changes and leave feedback, as well as select QA Classes. If the QA classes have high enough severity, the unit will also be sent back to the original contributor for additional rework. Original contributor can dispute the feedback (and rework), or else acknowledge and perform the Rework	ACCEPTED MODIFIED \| ACCEPTED MODIFIED \| REJECTED REJECTED (may also be PENDING until the Feedback resolution process is complete)
❌	✅	✅	✅	QA Contributor can leave feedback as well as select QA Classes. If the QA classes have high enough severity, the unit will also be sent back to the original contributor for Rework Original contributor can dispute the feedback (and rework), or else acknowledge and perform the Rework	ACCEPTED REJECTED (may also be PENDING until the Feedback resolution process is complete)
❌	❌	✅	✅	QA Contributor can leave feedback as well as select QA Classes. If the QA classes have high enough severity, the unit will also be sent back to the original contributor for Rework Original contributor can acknowledge and perform the Rework	ACCEPTED REJECTED
❌	❌	❌	✅	QA Contributor can select QA Classes and leave feedback. Original contributor can acknowledge feedback.	ACCEPTED REJECTED
❌	✅	❌	✅	QA Contributor can select QA Classes and leave feedback. Original contributor can acknowledge feedback.	ACCEPTED REJECTED (may also be PENDING until the Feedback resolution process is complete)
❌	✅	❌	❌	QA Contributor can leave feedback Original contributor can acknowledge feedback	ACCEPTED