Test Question Best Practices

Test questions are the most important quality control mechanism when using Appen. Creating ideal test questions is the best way to ensure high quality results from a job. This article explains the characteristics of useful test questions and best practices when creating them.

How many Test Questions do I need?

  • It is recommended to have between 50-100 test questions in a job. Because contributors can only see a test question once, the more that are created, the faster the job will complete
  • To enable Quiz Mode and test questions in a job, the total number of test question >= (Rows Per Page setting + 3). 

Characteristics of Ideal Test Questions

    • Ideally, contributors will behave exactly the same on test questions as they do on the regular rows of data in the job. To accomplish this, test questions must be indistinguishable from the overall data set.
    • Test questions should follow the rules set in the job instructions. As well, test question reasoning should explain exactly how an answer was reached. 
    • Test questions can accept multiple possible answers. This is especially useful on questions that are subjective.
      • Example: The job is to rate the sentiment of a tweet. In this type of job, what someone understands as "Slightly Positive", others may think as "Highly Positive". Accepting both answers will allow contributors to get the test question correct when either answer is selected. 


Fig. 1: Both Highly Positive and Slightly Positive are enabled as correct answers

  • Test questions should be hard enough to test a contributor’s performance but easy enough for those who are following the job instructions.
  • Every test question should be equipped with an instructive reason that explains the process of getting a correct answer.
    • Do's:
      • Write full sentences and use proper grammar
      • Refer directly to the instructions
      • Explain why the answer is correct and also why the incorrect answers are not. Find points for them to improve in their work
  • Have an even Answer Distributions of test questions will train contributors on every possible answer instead of biasing them towards one answer.
  • The method of answering test questions varies depending on how many questions there are in a job.
    • For jobs that only ask a few questions, it’s a good idea to answer every field.
    • For jobs that ask a lot of questions and include logic, the questions that are answered should vary. Varying the answer makes it easier for contributors to get the question correct because they are tested on less fields. If contributors get one question wrong they get the entire test question wrong. 

Note: Contributors will only be tested against questions where an answer has been provided. If a question isn't answered, the contributor will not be tested against that particular question

  • If the test question being submitted contains impossible logic. A warning message will appear to warn that a piece of the logic tree is not correct and the questions need to be updated before saving.


Fig. 2: Warning message in regards to answering the first question as ‘No’ and then answering the second question which should only be asked if the answer to the first question is ‘Yes’



Was this article helpful?
50 out of 79 found this helpful

Have more questions? Submit a request
Powered by Zendesk