Guide to: Test Questions in Quality Flow

Overview

Test questions are pre-labeled questions, or gold data, that can be used in Quiz Mode to assess contributors before they enter the task, or used in Work Mode to monitor the quality of the work submitted by contributors. It is also common practice to use a combination of Quiz + Work mode in the same job; in this mode, contributors must first pass the quiz questions before receiving units to work on and while working, they will be presented with test questions at an interval set by you.

Test questions can also serve as a teaching mechanism in that they give contributors continuous feedback while they work. Importantly, this feedback should be used to learn how to annotate in the task and should align with the guidelines. Additionally, test questions serve as a method of removing low quality data and low performers, because work done by contributors who have fallen below your set quality threshold can be removed.

In Quality Flow, Test Questions are always created from existing units in the project dataset. Units must first be uploaded to the project level before they can be turned into test questions at the job level. That is, Quality Flow does not create new units when defining test questions, it converts existing project units into test questions for a specific job. Once units exist at the project level, test questions can be created, modified and monitored from the Quality tab at the job level.

This article describes how test questions are created and utilized in Quality Flow. 

• Create test questions in the UI
• Create test questions through CSV upload
• Job settings
• Monitor contributor responses
• Download contributor scores and answers
• Edit test questions

See this article for more general information and advice about test questions, but come back here for any configurations or settings specific to Quality Flow.

Create Test Questions

You can create test questions either directly from the UI in the Quality tab, or by using a CSV file for bulk upload. The UI is best suited for individual edits and review, while the bulk upload is recommended when you are creating test questions at scale and you are familiar with your job's design.

Create Test Questions in the UI

Step 1: Create your Project and upload data into your Project

Step 2: Design your Job & Save, make sure you have completed your job design before creating your test questions

Step 3: Navigate to the Quality Tab or click "Next: Ensure Quality"

Step 4: Click "Add or Edit Test Questions"

choose "Add Test Questions", then click "Select" this will open another modal displaying units from your project's dataset

Step 6: select units from your dataset to make into test questions and click "Next: Set up Test Questions"

Step 7: Choose whether to apply settings to the whole page

At the top of the next page you will see "Overall Test Questions Settings", toggle this on to apply the Visibility and Mode settings to every test question, otherwise you can choose these settings individually for each question.

• "Minimum Matching Accuracy (for Text and Text Area)"
  • this allows you to specify a fuzzy matching threshold for text inputs (currently cml:text and cml:text_area, while smart_text is coming soon). The fuzzy matching level is computed based on the word level Levenshtein distance. If the matching level is equal or above the threshold, the individual response will pass, otherwise it will fail.
• "Hide Answer"
  • Contributors will not be shown the answers, they will also not be aware that it is a test question, and not a regular unit. This can be useful if you want to ensure that your test questions aren't being shared between contributors. Note that contributors will not get the benefit of learning from the answer as the answers will be hidden.
• "Test Question Mode" 
  • Quiz+Work means the questions could be randomly distributed through the Quiz and Work portions of your job.
  • If you want to designate that some test questions should only be used in quizzes and some in regular work, you can specify the mode as depicted below. 

Step 8: Provide answers and reasons

Provide the correct answer to your questions, as well as the reason for that answer. This reason will be shown to the contributors along with the answer (if "Hide Answer" is not enabled).

Step 7: Click "Save and Continue" and move on to Job Settings

Job Settings

Make your selections on the Job Settings page:

• A: Job Mode
  • In Quiz Only mode, contributors will only be presented with quiz questions, there will be no work in this job
  • In Quiz + Work mode contributors will be presented with quiz questions first, if they pass the quiz questions they will continue on to work
  • Work Only mode is currently under development.
• B: Minimum Accuracy
  • Set the accuracy threshold for passing the quiz and remaining in the job during work mode.
• C: Test Question Frequency in Work Mode
  • Choose how many test questions per page, and how often test questions will be presented (only relevant for Quiz + Work)
• D: Total Test Questions in Quiz Mode
  • Choose how many questions the contributor will be presented in Quiz Only mode or in the Quiz portion of Quiz + Work
• E: Allow contributors to keep working in Work Mode even when no more Test Questions are available
  • If you do not turn this option on then contributors will be limited in the number of units they can do by the number of test questions you have created (for each mode).

Complete your other settings and run your job.

Create Test Questions through CSV upload

Test questions uploaded into a job must reference source data that already exists in the project dataset. Quality Flow does not create new units when uploading test questions.

Step 1: load plain source data (e.g. without gold answers) to the Project's dataset

Step 2: load the same source data, with gold columns appended, in the Job's quality page

Do not include additional columns other than the source columns and the columns expected by test questions.

The CSV must include the following REQUIRED test question columns:

• _golden - (set to TRUE)
• {{fieldname}}_gold - (correct answer)
• {{fieldname}}_gold_reason - (reason for the correct answer)
• quiz_work_mode - (QUIZ, WORK, QUIZ_WORK)

On the job's Quality tab, click "Add or Edit Test Questions", then select Upload Test Questions and click "Select", then browse for your file to upload.

Check on the status of your test question upload by clicking on "Processes Running in the Background" button. Once the upload is complete, refresh the page and the newly created test questions will appear in the Quality tab.

Once your test questions are loaded here, do not send the same units to the job again. Test Question units are automatically assigned to the job as part of the upload process.

Monitor Responses

Once your contributors have worked on your Test Questions, you will be able to monitor their responses in various ways:

• On the QUALITY tab of the job itself you will be able to see the number of responses received as well as the proportion that have been MISSED (i.e. got wrong by contributors) and CONTESTED (i.e. marked as unfair or incorrect by contributors)
• Click on the CONTESTED % to see contributor responses as well as any reasons they have provided for contesting the question

• Under the CONTRIBUTORS tab you can monitor which contributors have passed or failed any quizzes, and also see the score they achieved and this is also downloadable via the Reports dropdown.

Finally, in the Dashboard section of your Project, the Contributor dashboard will display a Trust Score (along with any other metrics collected during QA).

Download test question report

In addition to monitoring test question performance directly in the Quality tab, you can also download a detailed test questions report from the quality tab.

The test questions report includes comprehensive information about each test question, such as performance metrics, configuration settings and gold answers. The report can also be as an upload when:

• Bulk updating existing test questions
• Re-using test questions in duplicated jobs

The report includes the following fields:

• unit_id
• project_id
• job_id
• source_field(s)
• _judgments
• _pct_missed
• _pct_contested
• _contention
• _quiz_work_mode
• _hidden
• _reviewed
• _enabled
• {{fieldname}}_gold
• {{fieldname}}_gold_reason

Example report output:

Download contributor scores and answers

If you want to investigate in detail answers given to your test questions by specific contributors, you can download this information from the JOB > RESULTS tab.

Edit test questions

When you have a large number of CONTESTED or MISSED questions, there may be something amiss with your questions. It is recommended to monitor closely in the first instance to make sure your questions are testing what you want them to. Test questions should align with the guidelines that you have included in your task.

If you discover that you have provided wrong answers, or perhaps the test questions you have chosen are too ambiguous to be a fair test, it is possible to edit answers or disable test questions. Changing the answers, or disabling questions will have an effect on the scores of people who have attempted those questions.

If a contributor answered a test question incorrectly and the test question was subsequently disabled or edited to a different answer, their trust score will be updated accordingly. Contributors will be able to continue working in the task after failing out if the test questions they incorrectly answered are changed or disabled.

Edit test questions in the UI

• To bulk edit or disable:
  • select multiple test questions and choose from the actions available

• To edit individual questions:
  • Toggle Enable/Disable
  • Click on the Question ID to Edit individual test questions as well as view details about contributors. 
  • If you wish to indicate that a Test Question has already been reviewed and is valid (despite being missed or contested a lot), you can click "Pass Review" and it will appear with a green check in the question list.

Edit test questions through CSV upload

You can bulk edit existing test questions by downloading the test question report, making edits in the download and re-importing.

1. On the Quality tab for the job, select "Download TQ report"
2. In the report, you can update/edit the following fields:

• _hidden
• _reviewed
• _enabled
• {{fieldname}}_gold
• {{fieldname}}_gold_reason

3. Save your updated CSV

4. On Quality tab, click "Add or Edit Test Questions"

5. In the modal choose Bulk Edit Test Questions and click "Select"

6. Drag and drop or Browse for your file to upload

7. Refresh the page to see your changes reflected

If _reviewed is set to TRUE, a check mark will appear next to the corresponding question id in the table.