Quality control in job building is an important step. When quality control mechanisms are utilized, they help attain higher quality results. There are four quality control settings under the Quality Control page in Settings: Minimum Time per Page, Maximum Judgments per Contributor, Disable Google Translate for Contributors and Answer Distribution Rules.
Fig. 1: Quality Control Settings
Minimum Time per Page
This setting is used to ensure that the contributors spend a certain amount of time on a page before they can submit.
- This is a speed trap -- it's the minimum number of seconds it should take for a contributor to complete one page of work. If a contributor is taking less time than this, they're likely not paying close enough attention in this job.
- The default value is 10 seconds.
- None of their work will be tainted and or recollected.
Max Judgments per Contributors
This sets the maximum number of judgments that an individual contributor will be able to submit on a job.
- This setting is typically used to help manage quality (e.g., prevent individual workers from over-contributing or engaging in abusive behavior), or to restrict contributions when otherwise conceptually necessary (e.g., prevent contributors from completing the same survey multiple times).
- This setting overrides the Automatic Max Judgments a contributor can submit based on the number of test questions there are in a job.
The Max Judgments Per Contributor is automatically calculated according to the following criteria:
- If Test Questions are enabled on the job, each page of work mode contains one Test Question. Therefore, the maximum number of judgments a contributor can submit is limited by the number of active test questions.
- When a job does not have Test Questions enabled, a contributor can either complete 50 rows or 10% of the rows launched, whichever amount that is higher.
Max Judgments Per Contributor is automatically calculated as follows:
- Number of rows set in Rows Per Page
- Number of active tests questions
Here's a basic formula to calculate Max Judgments per contributors
(Active Test Questions in a job - Test Questions used in Quiz Mode) x (Actual Rows per Page) = Max Judgments per contributors
Using the above equation, a job with 25 active questions is ready to launch with 84 Max Judgments Per Contributors
- Rows per Page is set to 4
- The number of test questions shown in quiz mode is the same as the number set in the Rows Per Page setting.
- In a page of work, one row will be a test question
(25 Active Test Questions - 4 Test Questions used in Quiz Mode) x (4 Rows per Page - 1 Test Questions per Page ) = 84 Judgments
Note: Max Judgments per Contributor always has a ceiling amount, or the highest number possible for the job. When Test Questions are enabled, to apply for a custom number, the number can only be set to an amount lower than the ceiling amount. If left blank, the number will be automatically set to the number of rows there is available for contributors to work on
Disable Google Translate for contributors
This feature exists on jobs with a default setting of "off", to help contributors keep context when they're working on natural language tasks or in jobs where language context matters.
- When enabled, this feature disables Google Chrome's auto-translate for contributors using Chrome browser to ensure that context and meaning are not lost in translation
- Note: By default, if a contributor is using Google Chrome and accesses a job whose instructions are written in a different language than their browser settings, the browser will offer to auto-translate the instructions and job into the contributors' language
- Most Natural language processing jobs are highly subjective and may be a good fit for disabling auto-translate.
- Image annotation jobs or internet research tasks are not recommended as contributors may need the auto-translate feature to provide the best quality work.
Contributor Answer Distribution Rules
This Setting helps monitor the distribution of answers that a contributor submits in a job. Monitoring a contributor’s Answer Distribution helps ensure that they are removed from the job if they favor a particular answer in a job and when they're deviating away from the expected answer distribution of the data.
Note: Flagging a contributor via answer distribution rules is a permanent action and cannot be reversed.
Fig. 2: Answer Distribution setting
- A. Monitor answer patterns to remove contributors whose answers are outside the expected distribution
- Check this to enable Answer Distribution monitoring on a job.
Note: Enabling Answer Distribution monitoring on a running job will enforce the Answer Distribution rules for all new judgments collected. Answer Distribution monitoring will not be calculated retroactively.
- B. Minimum rows submitted by a contributor before we start monitoring
- This is the number of rows a contributor must submit before their Answer Distribution is monitored. This can be set from 20 to 100 rows.
- C. Question
- This column displays the label associated with each CML question in a job. These are the questions that can be enabled for Answer Distribution monitoring. Enable Answer Distribution monitoring on a question by selecting "Add Rule".
The Regular Expression field requires an expected answer that will be evaluated when looking for violations of the answer distribution.
- Enter the expected answer by entering the CML value associated with the expected answer. Answer distribution will then use the Regular Expression input to enforce the maximum and minimum percentage that we expect to see that input as an answer.
Example: cml:radios- What type of category best describes this business?
The possible answers, set in the CML, are:
- <cml:radio label="Restaurant" value="restaurant" />
- <cml:radio label="Hotel" value="hotel" />
- <cml:radio label="Bar" value="bar" />
- <cml:radio label="Medical" value="medical" />
- <cml:radio label="Salon" value="salon" />
- <cml:radio label="Sport" value="sport" />
- <cml:radio label="Restaurant" value="restaurant" />
- <cml:radio label="Other" value="other" />
To enforce an answer distribution rule to check if contributors are answering 'Other' too often, add 'other' as the Regex match.
Use the sliders to set the minimum and maximum percentage that the Regular Expression answer should be submitted by the contributor.
Fig. 3: Contributors Will Be Removed If They Enter "other" More Than 15% Of The Time
- Being familiar with the data helps in determining the Acceptable Range. If no more than 15% of the source data is expected to be 'other' for the question "What type of category best describes this business", create a rule with a Regular Expression of 'other' then move the right-most slider to 15%.
- With these settings, contributors will be removed from a job if they answer 'other' on more than 15% of the judgments they submit.
- To delete a rule, click the trash can button on the right-hand side of each rule.
- Click the Add Rule button to add an additional rule for a question. In cases where there are multiple rules, all rules will be enforced.