Guide to: Test Question Generator


The Test Question Generator feature allows users to order test questions from curated groups of contributors, easing the burden of creating them on their own. This feature is currently in a Closed Beta; for access, please reach out to your Customer Success Manager.

Note: For Dedicated customers, this feature is currently not available On-Premises and only available via the cloud multi-tenant Appen Data Annotation Platform.

How it Works

Once you get to the test question creation step of your job’s lifecycle, you will now have the option to either create test questions manually – the standard way – or have them generated by a trusted group of contributors.


Figure 1: Option to create or generate test questions via the Quality Page 

If you choose the option to Generate, you will be prompted to select a channel to order from. You can choose either a pre-curated group of contributors provided by Appen or you can select one of your own custom channels.

  • If you don’t have any custom channels created yet, you will need to create one. Please see this article for more information.


Figure 2: Generating test questions

Once you place your order, a job is created in the background and a randomized number of rows corresponding to the number of test questions ordered is sent to this new job. There, contributors see your instructions and job design, with the following minor changes: 

  • The job displays only one row per page.
  • Radio buttons are converted to checkboxes to allow multiple acceptable answers to be selected (mirroring the standard test question creation interface).
  • Each form element has a corresponding text input for the contributor to enter the reason for their answer.
  • No form elements are required to be answered. This is intentional in case there are open-ended questions that you do not wish to test on.
  • Contributors can skip a row if they don’t feel it would make a good test question.

As these rows are completed, they are routed back to your job as test questions, enabled and ready to use! 

Frequently Asked Questions

  1. How much does it cost to order test questions? 
    • We apply a multiplier to the price per judgment set in your job to ensure contributors are fairly compensated for the extra time it takes to make a test question compared to answering a normal row of data. 
  2. How do I pay for these test questions?
    • You don’t! During the closed beta, we’re covering the cost of these test questions. This will change when we scale out and make this a generally available feature. 
  3. Can I order test questions from the internal channel? 
    • No, Test Question Generator is only supported with external channels.  
  4. Can I review the test questions before they’re added to the job? 
    • Currently, test questions are automatically enabled when they are routed back to your job. With that being said, we hope to add a review state in the future. 
  5. Can I order test questions through the API? 
    • We do not yet have API support for Test Question Generator, but this is coming soon! 
  6. How do I give feedback to the contributors who made the test questions? 
    • For now, you can send your feedback to your Customer Success Manager or our Platform Support team and we will pass that on to the contributors. We’re currently working on a contributor communication process, so if you’re interested in sharing your thoughts with us, please reach out to 

Was this article helpful?
0 out of 0 found this helpful

Have more questions? Submit a request
Powered by Zendesk