Once your job is launched you’ll start to see contributor’s responses on your Test Questions from the Quality page. These responses are extremely valuable feedback on how contributors are understanding and completing your job. This feedback not only gives you insight into the quality of your test questions but also the effectiveness of your job design and instructions. Reviewing the responses to your test questions and updating them when needed is the most important aspect of monitoring your job and is often the deciding factor in whether job is successful or not.
Please see our Guide to: Test Question Review page for definitions and explanations of each of the features discussed in this article.
Steps to Monitoring Test Questions
- 1. Go to Quality page, test questions will be automatically listed by % Contested:
Fig. 1: Test Questions sorted by % Contested
- 2. Select Test Questions that are highly missed
Fig. 2: A highly missed test question that needs review and updating
- 3. Review the answers that are highly missed and make sure they follow your instructions
- 4. Review distribution of contributor answers on highly missed radio buttons
Note: As you can see in this example, 100% of contributors have selected ‘Negative’ when ‘Neutral’ was chosen as the correct answer. Referring to the tweet, it is indeed negative sentiment. This Test Question should be update to have the correct answer.
- 5. Also review distribution of answers on text box
Fig. 3: A highly missed text box test question that needs review and updating
Note: Here you can see 100% of contributors answered that the tweet had ‘11’ likes. This indicates that since the test question has been made there may have been likes added to this tweet. Reviewing the tweet, you can accept the answer submitted by contributors and forgive them.
- 6. After reviewing a test question you have the option to select “Pass Review” which will save any changes to your test question and also update it on the Quality page to avoid you reviewing the same question more than once:
Fig. 4: A test question that has been reviewed and “Passed”
Note: Passed Review is particularly useful for those teams with multiple individuals creating and reviewing test questions because now you can easily identify highly-contested or missed test questions and know which have already been verified, avoiding redundant reviews.
- 7. Repeat steps above until for all highly missed/contested test questions
Interpreting the Quality Page
While your job is running or after your job has completed, you are able to obtain some key information on how the contributors performed in your Test Questions.
Teal Colored Bars
Teal colored bars show acceptable answers for the Test Question. Percentage indicates the percentage of contributors that selected the acceptable answers.
Figure 10: Contributor performance on test questions
Gray Colored Bars
Gray colored bars indicate that the answer shown was not included as an acceptable answer for that Test Question. The percentage indicates the percentage of contributors that select that answer.
Figure 11: Contributor incorrect answer percentage for test question
Note: By clicking on the + sign next to an answer, you can make that answer 'acceptable'. This would mean, that contributor answers that were previously marks as 'unacceptable' will now be re-marked as 'acceptable'.