What is Appen?
Our platform combines machine learning and humans-in-the-loop in a single platform for data science teams doing sentiment analysis, search relevance, or business data classification.
What is a job?
Jobs are typically sizable tasks that would be unreasonable or inefficient for one person (or even a small team of people) to complete on their own.
- They can be completed from a computer but usually cannot be fully automated, or carried out by a computer.
- They can be divided into smaller tasks that members of contributors can complete
- Jobs can include anything from getting contributors to label images, tagging and categorizing products, or helping manually transform data into a consistent format.
What are some common use cases?
- Sentiment Analysis – direct contributors to do what computers cannot: interpret context clues to extract meaning, mood, intention, or tone from a graphic, video, or piece of writing (e.g. one that contains humor, sarcasm, slang, or irony).
- Categorization – ask contributors to categorize companies, products, media, or any number of other items, based on the criteria that you provide.
- Content Moderation – ask contributors to moderate your content for guideline violations, inappropriate content, or spam. Whether it’s text, photos or video, the crowd can evaluate the content against a set of objective rules.
- Business Listing Verification – use the crowd to improve the quality of your business data by verifying company websites, addresses, phone numbers, and other important fields in your CRM system.
- Data Collection and Enhancement – ask the crowd to collect new data or enhance your current databases by finding social media handles on LinkedIn or Twitter profiles, locating parent and subsidiary companies of a business, or by flagging duplicate business records.
- Search Relevance – ask the crowd to create training data for an algorithm. For example, the crowd could help refine a search algorithm by rating how relevant search results are to some query.
- Image Transcription – direct contributors to sort through photos and transcribe information from the photos into data that you can use.
Test Questions
What are test questions and why do I need them?
Test Questions are the most important quality control mechanism in the Appen Platform. Creating accurate test questions is the best way to ensure high quality results from a job. Test questions should be hard enough to test a contributor’s performance, but easy enough for those workers who are trying to follow your instructions honestly. Not all data rows make for great test questions. Don't create test questions that take you significantly more time than an average data row to complete or that don't have a correct response in regards to the instructions.
How many test questions do I need?
Most jobs have between 50-100 test questions, however the more test questions in a job the faster the job will complete. By default there is one test question per page (with 1-19 rows per page set) which means a contributor can only do as many pages of work as there are test questions in a job. Quiz mode, which is set to the amount of rows per page in a job, will also affect the amount of test questions you need to create.
For example, if you create 15 total test questions and have 10 rows per page, quiz mode will first use 10 of these test questions. From there you will have 5 test questions remaining which means contributors can only do 5 more pages of work before maxing out. 50 total judgements are collected per contributor. However 5 of these 50 are test question judgments as there are 1 test question per page. In total, a single contributor will complete 45 judgments.
How do I upload test questions that I have made in a spreadsheet/report?
Creating test questions in bulk via a report can be done by uploading the test question data set to the job via the Data page. After formatting the report and uploading to the Data page, selecting “Convert Test Questions” will automatically create the test questions for you. This process should be used for all new test questions. Editing already existing test questions can also be done by downloading a test question report from the job, editing and uploading again.
What is quiz mode and work mode?
Before a contributor can enter your job, they must pass quiz mode which is composed entirely of test questions. Contributors that fail quiz mode are not paid and are permanently disqualified from working on your job. The length of quiz mode is set to the amount of rows per page in your job and enabled when: Total Number of Test Questions >= (Rows Per Page setting + 3). For example, if the rows per page setting is at 5, you will have 5 questions in quiz mode and need 8 total test questions to enable quiz mode.
Work mode is entered only after a contributor has passed quiz mode above the test question accuracy. They will continue to work in the job unless their accuracy falls below the set threshold. If this does happen, their judgments will become untrusted and recollected.
CML
How do I use logic?
CML logic allows you to populate questions based off a contributor’s answer to a previous question. You can use “only-if” logic to specify the name of the question and the value or answer choice that should trigger the secondary logic dependent question. As well, you can use “and”, “or” and “not” statements to connect multiple questions together.
What are validators?
CML supports a number of validation methods to ensure data quality. Some validators normalize the input, making it easier to create test questions for data that can have a varying format like phone numbers, addresses, and URLs. Our ready to use validators can be used for text and number inputs including URL’s, dates, character lengths, currency, and email. You can also use our clean validators to remove white space or capitalize inputs. Lastly, you can also create custom validators using Regular Expressions or regex to suite your needs.
Can I allow multiple answers?
There are two ways to allow multiple answers. For a “cml:text” field you can add the attribute ‘multiple=“true”’ to allow contributors to enter multiple answers in separate text fields. This is most useful when there are multiple answers such as names, dates or addresses.
This attribute can also be used with cml:group which will allow multiple answers for a set of questions, instead of just one.
What are the "name", "label" and "value" attributes?
When designing a job each question will need to have a “name” and “label” attribute. As well, each answer will need to have a “value” and “label” attribute. The “label” attribute is the text shown to the contributor when working on the job. The “name” and “value” attributes are not shown to the contributor but instead constitute the values used for the output reports and logic statements. If no name or value attributes are specified your results will automatically populate one, as seen below.
Example:
<cml:radios label="Is this posting relevant to American Airlines?" name="relevant" aggregation="agg" validates="required" gold="true">
<cml:radio label="Yes" value="yes" />
<cml:radio label="No" value="no" />
<cml:radio label="Not in English" value="non_english" />
</cml:radios>
<cml:text label="How many likes does this tweet have?" validates="required integer" name="likes" only-if="relevant:[yes]" gold="true" />Red: The “label” attributes are the question text, these can be seen from the Preview page when designing a job.
Blue: The “name” attributes are the relevant column headers that will appear in the output reports. In this example, your results file will have two new columns “relevant” and “likes” with their respective answers. The second question does not have a “name” provided, the results header will be automatically populated as “how_many_likes_does_this_tweet_have”
Green: The “value” attributes are only used for answers to a question. These will be in the results file under their respective column headers (“name”). They can also be used for logic.
Throughput
How do I make my job run faster?
- Test questions
- As mentioned in our quiz mode article, the more test questions you have the more work can be done by a single contributor. After quiz mode each additional test question allows contributors to do a full page of work. From the Contributors page on the Monitor tab, you can see if workers have seen the max number of test questions available.
- Levels
- Lowering the level down from 3 to 2 or 2 to 1 will increase the total pool of contributors who are able to access your job. Use this with caution as quality tends to drop when accessing the lower levels, while speed can increase.
- Payment
Why is my job running so slow?
- Test question accuracy
- It is extremely important to monitor test questions when running a job. If many contributor are seen failing quiz mode (shown on Monitor tab), that is a clear sign that the test questions may need some updating. As well, if many workers are also failing out of work mode, there may be other acceptable answers to test questions that need to be added or the job design/instructions need to be updated to better instruct contributors on how to complete the task.
- Geo-targeting/Language skills:
- If you are targeting a specific language or country, it is best to lower the level to 1 in order to allow all eligible contributors access to your job.
- Difficulty
- If your job has many questions or text fields, it may be difficult for contributors to finish. They will need incentive to continue working and increasing payment can help. This can either be done from the Pay page or Launch page. Alternatively, you can potentially simplify the job by splitting it up into multiple jobs.
Cost
How is my job cost calculated?
The basic estimation to estimate your job cost is outlined here:
(Judgments per row * (Pages of work * Price per page)) + buffer + transaction fee = estimated job cost
If you’re launching a job with 1,000 rows of data and you choose to display 10 rows per page; that actually means that one of those rows is a test question, while the other 9 are rows of data. This yields about 112 pages of work for the job. You decide to collect 3 judgments per row and set your payment to 10 cents per page. The basic cost estimate, then, is:
3* (112 * $0.10) = $33.60
The buffer is only necessary when the job is estimated to cost lower than $10 USD. This will make up the difference automatically. The transaction fee is based on your subscription. Data for Everyone and Trial users have a 20% markup on all job runs.
Why did my job go out of funds?
You may occasionally notice that your job costs have increased from the original estimate. The following are all reasons why a job's cost may increase after launch:
- The collection of more untrusted judgments in Work mode than expected
- Setting changes such as updating payment, rows per page or judgments per row
- The cost of untrusted work resulting from manual rejections of contributors that you make
If a job does go “Out of Funds”, an email will be sent to the job owner notifying them. The job will then show you a new estimation for the job and the original estimate. Selecting “Add Funds” will allocate the additional funds and resume the job’s progress.
Why is my estimate not updating?
When launching multiple runs in the same job, the job cost estimator will base any new estimations on the first initial job run. This can be misleading if you are running a very small amount of data, 1-100 rows at a time, and make a significant change to payment or settings. If the new estimate is above the actual amount needed, the funds will need to be added to the job and any unused funds will be placed back in the team account after job completion. If the new estimate is below the actual job cost, you will need to add more funds when prompted.
How are team funds used?
All funds are used on a team level, meaning funds added to your account can be accessed by any member of your team. The team balance appears in your account as well as any funds currently in progress. Setting up automatic payments are also based on team level. When a replenishment threshold is set on a specific team member’s account, it will trigger a payment to be added to the entire team’s account. Please note that the payment will only be triggered from the account which the setting is enabled.
How much should I be paying?
A general rule to follow is 1-3c per judgment. This can increase and decrease depending on the level of difficulty for a job. The Launch page will give an estimation of the job cost and reflect changes in price per row as it is changed for an initial job run.
API
How do I integrate with the API?
The Appen API allows you to post data to, launch, and receive results from Appen jobs in an automated fashion. The document linked provides an overview of the necessary process to complete a typical Appen API integration.
What actions can I do using the API?
A list of all commands that can be run using our API are found here. You can update settings, copy a job, launch a job, upload data and download results to name a few. Please note, not all settings found in the UI can be updated via the API.
Why is my report not accurate? What API Command can I use to regenerate it?
If a job finishes, and the report downloaded does not have all the data from the job, you’ll need to regenerate the report via the API. This action can be done with all report types and is outlined here.
To regenerate a report within a job:
curl -X POST "https://api.figure-eight.com/v1/jobs/{job_id}/regenerate?type={report_type}&key={api_key}"
To download a report from a job:
curl -o "{filename}.zip" -L "https://api.figure-eight.com/v1/jobs/{job_id}.csv?type={report_type}&key={api_key}"
Options for report_type:
- full - Returns the Full report containing every judgment
- aggregated - Returns the Aggregated report containing the aggregated response for each row
- json - Returns the JSON report containing the aggregated response, as well as the individual judgments, for the first 50 rows
- gold_report - Returns the Test Question report
- workset - Returns the Contributor report
- source - Returns a CSV of the source data uploaded to the job
Survey
How do I run a survey?
We do not recommend running surveys on Appen as the answers are typically subjective and thus bypass all of our quality control mechanisms, such as test questions. Specifically, external surveys, such as demographic or psychological, are hard to quality control as workers can easily share codes with other workers. A unique token in your survey is not adequate control to limit workers from malicious work and you will have to pay all contributors who complete the task.
If you choose to run external surveys on our platform, we recommend launching them to Level 3 crowds only and paying a small amount to complete the survey and using the manual bonusing function to bonus a large amount if the survey is correctly completed. Be sure to ask for the contributor ID on the CF and external survey to keep track of who to bonus. We cannot guarantee the quality of answers and can not refund any part of the job. As the platform is not designed for surveys the support team is not equipped to help with any implementation details or job design. We also can’t be held responsible for any discrepancies you may see in contributor numbers in your job and in your external survey.
How do I get more survey responses?
If the survey has completed and you would like to collect more responses, here are the steps to add more judgments to your survey:
- Go to the Data page and select 'Download'. Re-upload that same data set to your job.
- This will allow another set of judgments to be collect on a new data set since your first run has completed
- Set the new number of judgments (responses) you need on the Launch page
- Select 'Order more rows'
You will not collect judgments from contributors who have already accessed your survey if “Max Judgments per Contributor (Settings > Quality Control > Quality Control Settings)” was set to 1.
What do I pay?
Survey’s do not follow our general pricing standard and should be calculated based on your budget. Our more common use cases will set 1-3c per judgment. It is recommended to pay a very low amount for the initial data submission and then use our bonus function to pay those contributors who submitted quality work.