Guide to: Converting Finalized Rows into Test Questions

Test Questions are the most important quality control mechanism when using Appen. They not only ensure the most capable contributors are working on the job, creating more of them can lead to increased throughput.

This feature converts already completed rows into test questions quickly and effectively. It also helps to create test questions with an appropriate answer distribution, as the feature can filter for different question and answer options. Using this feature helps create more high quality test questions in significantly less time!

Note this feature is not available for all complex tools, the Convert button will be greyed out in the conversion interface in jobs where this feature is not available. Test questions may be converted from reports for all tools by following instructions here and here. 

This process is best used on a job that has successfully launched 100 rows with quality results. These 100 rows can then be converted into test questions. 

Steps to convert rows

1. Open job with at least 100 finished rows

• When a job run has completed, the Monitor page will show a new message (shown below) showing that completed rows are available to be converted into test questions.

Fig.1: Job with at least 100 finished rows showing a message to convert rows

2. Initiate conversion of test questions

• Clicking the link on the Monitor page will start the converting of the possible rows. An email will be sent to the user who selects this link when the indexing has completed. The job owner will not be notified of the conversion.

Fig. 2: The conversion has started and an email will be received when completed

3. Convert rows via the UI

• When the indexing has finished, the test question conversion interface can be accessed via the link in the email or through the Monitor page (after refreshing).

Fig. 3: Converting rows have been completed and link to access interface

Very Important: Make sure the aggregation settings in the Options Page is set to "Best Answer('agg')" for each question. This will ensure the test question conversion feature works properly.

Fig. 4: Test question conversion interface

A. Selected question filter

• • This selects the specific question to be filtered and viewed
    • Items to note:
      • It only shows one question and it’s answer at a time
      • When filtering on a logic dependent question, first select the appropriate answer to the parent question before converting

B. Answer to selected question filter

• • After selecting a question to filter, select an answer to filter by
    • Items to note:
      • Textbox answers are not able to be filtered on, but can be answered along with the other fields on the page using the ‘+’ button to select the correct answer
      • When filtering, only the top answer will be shown as a possible answer to the question
      • Filtering by the answer to the question will help create an appropriate answer distributions that ensure the job is testing for all possible cases and not biasing contributor to a particular answer

C. Next/Skip button

• • Here skips a question if its not suited to be converted to a test question

D. Convert row to test question button

• • After finding a row that is a good fit for a test question, and selecting all appropriate answers, click this button to convert into a test question

E. Number of total rows available

• • This number shows the total amount of completed rows available to be converted

F. Answer distribution

• • The answer distribution for already created test questions can be viewed here

Converted rows

After converting the desired amount of rows, they will appear in the list of test questions from the Quality page.

Indexing More Finalized Rows for Test Questions Conversion

Fig. 5: Option to Convert More Finalized Rows

If there are additional finalized rows that are ready to be converted to test questions, the Aggregated Report will need to be force regenerated via the Results page of the job.  Once the report has been regenerated, going back to the Monitor page, click the "Get New Finalized Rows" link on the Test question Convert page. This link initiates the indexing of the 1000 most recent Finalized Rows and makes them available to convert to Test Questions. 

Note: Previously converted test questions from Finalized rows in this set may appear available in this set again. The existing test questions will have no change. 

For more information on how to convert high confidence rows in a .csv please check out this article.