Follow

Guide to: Test Questions in Quality Flow

Overview

Test questions are pre-labeled questions, or gold data, that can be used in Quiz Mode to assess contributors before they enter the task, or used in Work Mode to monitor the quality of the work submitted by contributors. It is also common practice to use a combination of Quiz + Work mode in the same job; in this mode, contributors must first pass the quiz questions before receiving units to work on and while working, they will be presented with test questions at an interval set by you.

Test questions can also serve as a teaching mechanism in that they give contributors continuous feedback while they work. Importantly, this feedback should be used to learn how to annotate in the task and should align with the guidelines. Additionally, test questions serve as a method of removing low quality data and low performers, because work done by contributors who have fallen below your set quality threshold can be removed.

This article describes how test questions are created and utilized in Quality Flow. See this article for more general information and advice about test questions, but come back here for any configurations or settings specific to Quality Flow.

Note

If you are familiar with test questions in ADAP Jobs, please read the section below on uploading pre-formatted test questions before you start, as there are important differences to be aware of.

Test Questions in Quality Flow currently supports text and text area as well as form tools: radios and checkboxes.

Test Questions are not supported in projects using unit groups.

Create Test Questions from your project's dataset

Step 1: Create your Project and upload data into your Project

Step 2: Design your Job & Save, make sure you have completed your job design before creating your test questions

Step 3: Navigate to the Quality Tab or click "Next: Ensure Quality"

Step 4: In the modal that pops up, select some units to make into test questions and click "Set up Test Questions"

Step 5: Choose whether to apply settings to the whole page

At the top of the next page you will see "Overall Test Questions Settings", toggle this on to apply the Visibility and Mode settings to every test question, otherwise you can choose these settings individually for each question:

  • "Minimum Matching Accuracy (for Text and Text Area)"
    • this allows you to specify a fuzzy matching threshold for text inputs (currently cml:text and cml:text_area, while smart_text is coming soon). The fuzzy matching level is computed based on the word level Levenshtein distance. If the matching level is equal or above the threshold, the individual response will pass, otherwise it will fail.
  • "Hide Answer"
    • Contributors will not be shown the answers, they will also not be aware that it is a test question, and not a regular unit. This can be useful if you want to ensure that your test questions aren't being shared between contributors. Note that contributors will not get the benefit of learning from the answer as the answers will be hidden.
  • "Test Question Mode" 
    • Quiz+Work means the questions could be randomly distributed through the Quiz and Work portions of your job.
    • If you want to designate that some test questions should only be used in quizzes and some in regular work, you can specify the mode as depicted below. 

Step 6: Provide answers and reasons

Provide the correct answer to your questions, as well as the reason for that answer. This reason will be shown to the contributors along with the answer (if "Hide Answer" is not enabled).

Note

  • Radios will be converted to checkboxes in test questions, giving you the option to allow more than one answer
  • If you have more than one question on the same unit, you can opt to only provide an answer for one of them, contributors will only be judged on the questions that you have provided answers for

Step 7: Click Save and Continue and move on to Job Settings

Job Settings

Make your selections on the Job Settings page:

  • A: Job Mode
    • In Quiz Only mode, contributors will only be presented with quiz questions, there will be no work in this job
    • In Quiz + Work mode contributors will be presented with quiz questions first, if they pass the quiz questions they will continue on to work
    • Work Only mode is currently under development.
  • B: Minimum Accuracy
    • Set the accuracy threshold for passing the quiz and remaining in the job during work mode.
  • C: Test Question Frequency in Work Mode
    • Choose how many test questions per page, and how often test questions will be presented (only relevant for Quiz + Work)
  • D: Total Test Questions in Quiz Mode
    • Choose how many questions the contributor will be presented in Quiz Only mode or in the Quiz portion of Quiz + Work
  • E: Allow contributors to keep working in Work Mode even when no more Test Questions are available
    • If you do not turn this option on then contributors will be limited in the number of units they can do by the number of test questions you have created (for each mode).

Once you have also selected your Prices & Rows settings set your Job to "running".

Note

  • When you are running a Job in Quiz Only Mode there is no need to add any data to your Job before Running it, the Quiz Test questions are already included in your Job.
  • When you are running a Job in Work + Quiz, there is also no need to add the Golden units to your Job, filter GOLDEN units out in the dataset table before adding units to your Job.
  • Ensure that Allow Abandoned Units is not enabled prior to using test questions in your task. Test questions are not supported in Jobs where Allow Abandoned Units is enabled.

 

Monitor Responses

Once your contributors have worked on your Test Questions, you will be able to monitor their responses in various ways:

  • On the QUALITY tab of the job itself you will be able to see the number of responses received as well as the proportion that have been MISSED (i.e. got wrong by contributors) and CONTESTED (i.e. marked as unfair or incorrect by contributors)
  • Click on the CONTESTED % to see contributor responses as well as any reasons they have provided for contesting the question

  • Under the CONTRIBUTORS tab you can monitor which contributors have passed or failed any quizzes, and also see the score they achieved.

Finally, in the Dashboard section of your Project, the Contributor dashboard will display a Trust Score (along with any other metrics collected during QA).

Edit test questions

When you have a large number of CONTESTED or MISSED questions, there may be something amiss with your questions. It is recommended to monitor closely in the first instance to make sure your questions are testing what you want them to. Test questions should align with the guidelines that you have included in your task.

If you discover that you have provided wrong answers, or perhaps the test questions you have chosen are too ambiguous to be a fair test, it is possible to edit answers or disable test questions. Changing the answers, or disabling questions will have an effect on the scores of people who have attempted those questions.

If a contributor answered a test question incorrectly and the test question was subsequently disabled or edited to a different answer, their trust score will be updated accordingly. Contributors will be able to continue working in the task after failing out if the test questions they incorrectly answered are changed or disabled.

  • To bulk edit or disable:
    • select multiple test questions and choose from the actions available

  • To edit individual questions:
    • Toggle Enable/Disable
    • Click on the Question ID to Edit individual test questions as well as view details about contributors. 
    • If you wish to indicate that a Test Question has already been reviewed and is valid (despite being missed or contested a lot), you can click "Pass Review" and it will appear with a green check in the question list.

Upload pre-formatted test questions

If you are very familiar with your job's design, you can upload pre-formatted test questions in the form of a csv. 

Note

Test questions uploaded into your job must match source data that is already loaded to your Project.

Load plain source data (e.g. without gold answers) to the Project's dataset, then load the same source data, with gold columns appended, in the Job's quality page.

Do not include additional columns, other than the source columns or columns expected by test questions.

Expected columns are:

  • _golden - (TRUE)
  • {{fieldname}}_gold - (correct answer)
  • {{fieldname}}_gold_reason - (reason for correct answer)
  • quiz_work_mode - (QUIZ, WORK, QUIZ_WORK)

You do not also send units to the Job after adding test questions. Test questions are already assigned to their jobs.

It can be a good idea to create some questions in the interface, download the Test Question report from the Quality page and check the formatting to match when you want to use the upload feature.

 

If you would like to add additional test questions:

  • create new questions with the same column headings as your downloaded .csv
  • click on Add Test Questions and choose Upload Test Questions in the modal

From there you will be directed to drag and drop or browse for your file.

Note

  • To upload test questions you must match data that is uploaded to your Project already, once you are ready, append the formatted column to the same data you uploaded in your Project and load it into your Job.

To download all Test Questions click on Reports > Download Test Question Report. To download selected Test Questions only, select the test questions you wish to Download and click on Reports > Download Selected Test Questions.

Your test questions will be downloaded in the form of a csv. From here you can edit answers and re-upload (Reports > Upload Test Question Report) in order to forgive any widely missed or contested questions.

 

 


Was this article helpful?
0 out of 0 found this helpful


Have more questions? Submit a request
Powered by Zendesk