This is a checklist that can be used to help you calibrate the settings of your job. Completing this checklist will ensure that you’ve taken everything into consideration before you launch a job.

This checklist was built for all of the features of the Appen Enterprise Plan. Therefore, Appen Data for Everyone users may see features in this list that are unavailable on their account. To learn more about upgrading to Enterprise, please visit the Platform Plans page on the Appen website.

 

---

Settings

The following settings can be found in the "Settings" menu on the right-side navigation.

Contributors

 Level Selection: Which speed to quality performance ratio would you like for your job?

 External or Internal Contributors: Do you want the on-demand workforce or an internal team? Both? (More info)

Do you have any explicit content in your data? (More info)

Fig. 1: Contributor settings (Settings)

Contributors - Geography/Language

 Geography: Which countries should contributors come from? (More info)

 Language: Does your job require a certain language capability other than English?

Fig. 2: Geography/Language settings (Settings > Contributors > Geography/Launguage)

Quality Settings

Quality Control - Test Questions

What minimum accuracy should contributors have to keep working in your job? (More info)

Fig. 3: Test question accuracy (Settings > Quality Control > Test Questions)

Quality Control - Quality Control Settings

What is the shortest amount of time someone can spend on a page before submitting?

Do you want to monitor contributors' answer distributions? (More info)

Fig. 4: Quality Control Settings (Settings > Quality Control > Quality Control Settings)

Answer distribution enabled:

Fig. 5: Answer Distribution Selection (Settings > Quality Control > Quality Control Settings > Check "Contributor Answer Distribution Rules")

Quality Control - Dynamic Judgments

Would you like to turn on dynamic judgments in order to collect more judgments in case of contributor disagreement? (More info)

Dynamic judgments enabled:

Fig. 6: Enabled dynamic judgments (Settings > Quality Control > Dynamic Judgments)

Sharing/Visibility

Who should be notified with updates about this job?

Do you want to share this job to team members for editing and review?

Which tags will help you find your job easily?

Do you want to add this job to an existing project?

Fig. 7: Sharing & Visibility (Settings > Sharing/Visibility)

API

Do you need to use the API? (More info)

Fig. 8: API (Settings > API)

Pay Page

How many rows should be shown on each page?

How much should contributors be paid for each page of work?

Fig. 9: Pay page (Settings > Pay)

Launch page settings (More info)

How many rows do you want to launch in this run?

How much do you want to pay for each data row?

How many contributors do you want to answer each question?

Fig. 10: Launch page settings (Launch - top navigation)