Note: This video tutorial displays a previous version of the Appen job user interface and may look slightly different than your current job page. Regardless, the functionality of buttons and links on the interface should remain the same.
Lesson Overview:
This lesson goes over Test Questions, which are Appen quality control mechanism.
- Test Questions are created by the job author to:
- Qualify high performing contributors and remove low performing contributors
- Continually train contributors and improve their understanding of the job
- Test Questions are used to rate contributor’s accuracy on your job
- When contributors answer a test question, their answer is checked against that of the job owner. If their answer doesn’t match, their job accuracy goes down. If their accuracy at any point falls below the minimum accuracy threshold, they will be removed from the job and their judgments will not be included in the final output. Instead, we’ll replace their judgments with the work of a contributor that has maintained an accuracy above the minimum threshold.
- Quiz Mode Test Questions:
- In order for contributors to enter your job, they must pass a quiz compiled from the test questions you created.
- All questions in Quiz Mode are test questions.
- Quiz mode is only activated if you have created enough test questions.
- Work Mode Test Questions:
- In Work Mode, contributors will begin enriching rows from your source data by completing them in ‘pages’ which consist of 5-10 source rows at a time.
- Every page in Work Mode will contain 1 Test Questions, however, contributors will not know which row it is.
- More Test Questions = Higher Throughput
- We limit the amount of work a contributor can complete by the number of test questions that are in the job.
- Once a contributor has seen all of the Test Questions in your job, they ‘max out’ and cannot submit any more work.
- The more test questions there are in a job, the more work each contributor can do which increases overall throughput A recommended number of test questions for your job will be suggested on the quality page
Test Question Report Column headers:
- _golden: This value must be set to TRUE for all test question rows.
- sentimewhat_is_the_authors_sentiment_feeling_throughout_the_post_as_it_relates_to_american_airlines_gold: Our sentiment rating for the tweet.
- what_is_the_authors_sentiment_feeling_throughout_the_post_as_it_relates_to_american_airlines_gold_reason: Our reasoning for why we chose the sentiment rating.
- is_there_an_airport_code_mentioned_in_the_tweet_gold: This is either yes or no for each test question.
- is_there_an_airport_code_mentioned_in_the_tweet_gold_reason: This is our reasoning for the above answer.
- enter_the_airport_code_gold: Our answer for the airport code.
- enter_the_airport_code_gold_reason: Our reasoning for this answer.
Steps To Complete:
- Navigate to the ‘Quality’ page
- Select ‘Create Test Questions’
- Create 3 test questions
- Download the Test Question Report here
- Navigate to the data page of your job
- Click “Add More Data”
- Upload the Test Question report
- Note that if you add data that includes extra columns, or is missing a column that already exists in the job, you will get a message similar to the one below. This is just to notify you that your column headers do not match those in the job, in case you’ve selected the wrong file to upload. If the file is indeed correct, you can click “Proceed Anyway” to continue the upload.
- Once all new rows are uploaded, click “Convert Uploaded Test Questions”
- Navigate to the ‘Quality’ page and ensure you have 20 new test questions