Guide to: Test Question Creation Interface

Test Questions are Appen’s quality control mechanism within jobs. By providing the answers to a small percentage of rows, you can tell how accurately a contributors is working and will automatically remove any who do not meet your accuracy threshold. This article covers the interface used to create Test Questions. If you've already created test question in your job, they will appear in a list under 'Quality' tab from the top bar. Here, you can review all of your Test Questions. For more information on reviewing Test Questions, please see the Review Test Questions article.


Figure. 1: List of existing Test Questions

Note: You will not be able to create any Test Questions if you don’t have any data.

The following options are for creating and editing Test Questions will become available after you select “Create Test Questions” or “Create More” if there are existing Test Questions in your job.


Test Question Creation

The Test Question creation page will closely resemble the interface you created in the 'Design' step with 2 exceptions:

  • Radio/drop-down inputs will be replaced with checkboxes to allow multiple correct answers
  • A text box area will populate under each question so you can provide a reason explaining why your answer is correct.

Selecting 'Save' will create a copy of the row and mark it as a Test Question, containing your answers and reasons.


Figure. 2: Test Question Interface

Note: As shown in this example, you can allow the job to accept multiple correct answers by selecting more than one checkbox.


Figure 3. Important Features within the Test Question Interface

A. Show Job Instructions

This displays the job instructions within the Test Question

B. Skip

You can choose to skip a row it does not follow along the with the Test Question’s Best Practices or the answer distributions.

C. Save and Create Another

Save the row as a Test Question and continue with creating Test Questions. Keep in mind, the row will be copied to a Test Question row in your job's dataset.
D. Mode
You can select if you want the test question to appear in 'Quiz Mode', Work Mode' or 'Both'

E. Test Question Reason Textbox

This textbox will automatically appear when an answer is selected. 

Show Unit Page

This page will present you with more detailed data concerning the row selected.


Figure. 4: Important features within ‘Show Unit Page’

A. Show Data

Test Question and Source Data in JSON format.

Figure 5. The JSON format of the selected Test Question

B. Show Statistics

A specific CML question can be selected to display the answer distribution associated with it.

Figure 6. The answer distribution across all judgements for the CML Question ‘sentiment’

C. Show Contributor Info

Display data belonging to each individual contributor who has submitted a judgment on the row. This data includes:
  • Worker ID
  • Trust Score
  • If their judgment was correct
  • If their judgment was tainted
  • Name of the channel the worker belongs to
  • If they contested the incorrect judgement
  • Each answer given by the contributor, defined by the CML name
  • Each answer given by the contributor, defined by the CML name

D. Edit Test Question

Brings you back to the Test Question editing interface

Secret Test Question Feature

The primary function of Test Questions is to test the quality of contributor responses to pre-annotated data and provide feedback to them when they are annotated as incorrect, to that end Test Questions are meant to be indistinguishable from the dataset. Ideally Test Questions should not be reused across multiple jobs because long term contributors may eventually learn the specific answers they get wrong. If Test Questions will need to be reused, however, it can be useful to reserve a subset of them for which the correct answers are not revealed. 

The Secret Test Question feature allows you to hide the answer to certain Test Questions in your job. After creating Test Questions, toggle the "Hide Answer" option on the Quality page for the Test Questions that you would like to hide.

Screenshot 2023-07-18 at 5.42.37 PM.png

Figure 7: Hide Answer on Quality page


It is recommended that Secret Test Questions are used in conjunction with the gold pool functionality. Secret Test Questions cannot be used in quiz mode to qualify contributors into the job, thus if you hide Test Questions that are set to quiz and have less than 5 Test Questions allocated to quiz, quiz mode will not work. As Secret Test Questions are only meant for Work, it is recommended that they are used in addition to regular work Test Questions. It's crucial to note that Secret Test Questions do not impact contributors' accuracy in the task, nor do they trigger any flags, rejections, or affect contributors' overall performance during the job.

To assess and calculate the accuracy of contributors on Secret Test Questions, manual review via the full report is necessary at this time. To check the rows with Secret Test Questions, see Figure 8 below where column _answers_hidden equals TRUE.  Screenshot 2023-08-08 at 12.33.14 PM.png

Figure 8: Full Report featuring field _answers_hidden

Hide Test Question Answers

You can choose to hide all of the Test Question answers from contributors by clicking on the toggle button on the Quality page.


Figure 9: Hide All test questions answers on the Quality Page

Interpreting the Quality Page

While your job is running or after your job has completed, you are able to obtain some key information on how the contributors performed in your Test Questions.

Teal Colored Bars 

Teal colored bars show acceptable answers for the Test Question. Percentage indicates the percentage of contributors that selected the acceptable answers.


Figure 10: Contributor performance on test questions

Gray Colored Bars

Gray colored bars indicate that the answer shown was not included as an acceptable answer for that Test Question. The percentage indicates the percentage of contributors that select that answer.


Figure 11: Contributor incorrect answer percentage for test question

Note: By clicking on the + sign next to an answer, you can make that answer 'acceptable'. This would mean, that contributor answers that were previously marks as 'unacceptable' will now be re-marked as 'acceptable'.



Was this article helpful?
21 out of 28 found this helpful

Have more questions? Submit a request
Powered by Zendesk