Guide to: Test Question Creation Interface

Test Questions are Appen’s quality control mechanism within jobs. By providing the answers to a small percentage of rows, you can tell how accurately a contributors is working and will automatically remove any who do not meet your accuracy threshold. This article covers the interface used to create Test Questions. If you've already created test question in your job, they will appear in a list under 'Quality' tab from the top bar. Here, you can review all of your Test Questions. For more information on reviewing Test Questions, please see the Review Test Questions article.


Fig. 1: List of existing Test Questions

Note: You will not be able to create any Test Questions if you don’t have any data.

The following options are for creating and editing Test Questions will become available after you select “Create Test Questions” or “Create More” if there are existing Test Questions in your job.


Test Question Creation

The Test Question creation page will closely resemble the interface you created in the 'Design' step with 2 exceptions:

  • Radio/drop-down inputs will be replaced with checkboxes to allow multiple correct answers
  • A text box area will populate under each question so you can provide a reason explaining why your answer is correct.

Selecting 'Save' will create a copy of the row and mark it as a Test Question, containing your answers and reasons.


Fig. 2: Test Question Interface

Note: As shown in this example, you can allow the job to accept multiple correct answers by selecting more than one checkbox.


Fig 4. Important Features within the Test Question Interface

A. Show Job Instructions

This displays the job instructions within the Test Question

B. Skip

You can choose to skip a row it does not follow along the with the Test Question’s Best Practices or the answer distributions.

C. Save and Create Another

Save the row as a Test Question and continue with creating Test Questions. Keep in mind, the row will be copied to a Test Question row in your job's dataset.
D. Mode
You can select if you want the test question to appear in 'Quiz Mode', Work Mode' or 'Both'

E. Test Question Reason Textbox

This textbox will automatically appear when an answer is selected. 

Show Unit Page

This page will present you with more detailed data concerning the row selected.


Fig. 5: Important features within ‘Show Unit Page’

A. Show Data

Test Question and Source Data in JSON format.

Fig 6. The JSON format of the selected Test Question

B. Show Statistics

A specific CML question can be selected to display the answer distribution associated with it.

Fig 7. The answer distribution across all judgements for the CML Question ‘sentiment’

C. Show Contributor Info

Display data belonging to each individual contributor who has submitted a judgment on the row. This data includes:
  • Worker ID
  • Trust Score
  • If their judgment was correct
  • If their judgment was tainted
  • Name of the channel the worker belongs to
  • If they contested the incorrect judgement
  • Each answer given by the contributor, defined by the CML name
  • Each answer given by the contributor, defined by the CML name

D. Edit Test Question

Brings you back to the Test Question editing interface

Hide Test Question Answers

You can choose to hide test question answers from contributors by clicking on the toggle button on the Quality page.


Interpreting the Quality Page

While your job is running or after your job has completed, you are able to obtain some key information on how the contributors performed in your Test Questions.

Teal Colored Bars 

Teal colored bars show acceptable answers for the test question. Percentage indicates the percentage of contributors that selected the acceptable answers.


Gray Colored Bars

Gray colored bars indicate that the answer shown was not included as an acceptable answer for that test question. The percentage indicates the percentage of contributors that select that answer.


Note: By clicking on the + sign next to an answer, you can make that answer 'acceptable'. This would mean, that contributor answers that were previously marks as 'unacceptable' will now be re-marked as 'acceptable'.



Was this article helpful?
12 out of 17 found this helpful

Have more questions? Submit a request
Powered by Zendesk