Test Questions are Appen’s quality control mechanism within jobs. By providing the answers to a small percentage of rows, you can tell how accurately a contributors is working and will automatically remove any who do not meet your accuracy threshold. This article covers the interface used to create Test Questions. If you've already created test question in your job, they will appear in a list under 'Quality' tab from the top bar. Here, you can review all of your Test Questions. For more information on reviewing Test Questions, please see the Review Test Questions article.
Fig. 1: List of existing Test Questions
Note: You will not be able to create any Test Questions if you don’t have any data.
The following options are for creating and editing Test Questions will become available after you select “Create Test Questions” or “Create More” if there are existing Test Questions in your job.
Test Question Creation
The Test Question creation page will closely resemble the interface you created in the 'Design' step with 2 exceptions:
- Radio/drop-down inputs will be replaced with checkboxes to allow multiple correct answers
- A text box area will populate under each question so you can provide a reason explaining why your answer is correct.
Selecting 'Save' will create a copy of the row and mark it as a Test Question, containing your answers and reasons.
Fig. 2: Test Question Interface
Note: As shown in this example, you can allow the job to accept multiple correct answers by selecting more than one checkbox.
Fig 4. Important Features within the Test Question Interface
A. Show Job Instructions
- This displays the job instructions within the Test Question
B. Skip
- You can choose to skip a row it does not follow along the with the Test Question’s Best Practices or the answer distributions.
C. Save and Create Another
- Save the row as a Test Question and continue with creating Test Questions. Keep in mind, the row will be copied to a Test Question row in your job's dataset.
- D. Mode
- You can select if you want the test question to appear in 'Quiz Mode', Work Mode' or 'Both'
E. Test Question Reason Textbox
- This textbox will automatically appear when an answer is selected.
Show Unit Page
This page will present you with more detailed data concerning the row selected.
Fig. 5: Important features within ‘Show Unit Page’
A. Show Data
- Test Question and Source Data in JSON format.
Fig 6. The JSON format of the selected Test Question
B. Show Statistics
- A specific CML question can be selected to display the answer distribution associated with it.
Fig 7. The answer distribution across all judgements for the CML Question ‘sentiment’
C. Show Contributor Info
- Display data belonging to each individual contributor who has submitted a judgment on the row. This data includes:
- Worker ID
- Trust Score
- If their judgment was correct
- If their judgment was tainted
- Name of the channel the worker belongs to
- If they contested the incorrect judgement
- Each answer given by the contributor, defined by the CML name
- Each answer given by the contributor, defined by the CML name
D. Edit Test Question
- Brings you back to the Test Question editing interface
Hide Test Question Answers
You can choose to hide test question answers from contributors by clicking on the toggle button on the Quality page.
Interpreting the Quality Page
While your job is running or after your job has completed, you are able to obtain some key information on how the contributors performed in your Test Questions.
Teal Colored Bars
Teal colored bars show acceptable answers for the test question. Percentage indicates the percentage of contributors that selected the acceptable answers.
Gray Colored Bars
Gray colored bars indicate that the answer shown was not included as an acceptable answer for that test question. The percentage indicates the percentage of contributors that select that answer.
Note: By clicking on the + sign next to an answer, you can make that answer 'acceptable'. This would mean, that contributor answers that were previously marks as 'unacceptable' will now be re-marked as 'acceptable'.
