Task Types and “review-data”

We are introducing the concept of “task type” into annotation job design.

For the purpose of quality control, most annotation projects go through multiple steps. Some projects have an annotation step and a QA/audit step. Other projects might have an annotation step with an arbitration step.

To make it easy to design annotation jobs for these different steps, we are creating several different UIs in each annotation tool based on the tasks required at these steps.



On top of the default task type, which is basic annotation from scratch, the first new task type we’ve created is called “QA”.

When task-type=”qa”, the tool pre-loads annotation. Contributors would review the pre-loaded annotation and make corrections.

task-type=”qa” needs to be used in conjunction with review-data parameter. For example,

task-type=”qa” review-data=”{{annotation_column}}”

The “qa” task type is currently supported in all tools and works for jobs within Workflows.

We will be deprecating review-from parameter in 2022. If you used to use review-from="{{column}}", please start using task-type="qa" review-data="{{column}}"



Another task type we will be creating is “arbitration”. Please note that this is an upcoming feature. If you wish to discuss with us about this feature, please contact your success manager.

When task-type=”arbitration”, the tool pre-loads 2 sets of judgements, presents them side-by-side or one on top of the other for clear comparison. Contributors would choose which judgement to side with, or create a 3rd judgement.

The arbitration task type also calls for the review-data parameter in order to pre-load judgements.

The arbitration task type will be rolled out tool by tool, with Text Annotation tool being the first to adopt this task type.

Was this article helpful?
1 out of 4 found this helpful

Have more questions? Submit a request
Powered by Zendesk