Appen's system presents an estimation of how much the job will cost before launch. This estimate is based on the settings that are enabled, as well as on the volume of data the has been uploaded. Upon launching, the job cost can differ somewhat from the initial estimate depending on certain events that occur during a job's lifespan. In the following article, we will discuss some common questions about job costs.
- How is the initial job cost estimated?
- Why would the job cost change from the original estimate while the job is running?
- Why is untrusted work paid for?
- What are the best ways to keep job costs down?
How is the initial job cost estimated?
-
Price Per Page of work
-
Number of judgments collected per row
-
Number of rows that are being ordered
-
Your Appen Transaction Fee
-
Buffer*
Here’s the basic formula used to calculate the estimated job cost:
(Judgments per row * (Pages of work * Price per page)) + transaction fee + buffer = estimated job cost
Using the above equation, a job with 1,000 rows of data is ready to launch:
- Rows per Page set to 10
- Note: 1 of these 10 rows will be a test question on every page of work
- 1000 rows/ 9 yields about 112 pages of work for the job
- Judgments per Row is set to 3
- Price Per Page is set to 10 cents
The Basic cost estimate is:
( 3 Judgments per row * ( 112 Pages of work * 0.10 Price per page)) = $ 33.60
- The plan also includes a 20% transaction fee which means the price per page is actually 12 cents
(3* (112 *($0.10 * $1.20))) = $41.63
Why does my job cost differ from the original estimate while the job is running?
- The Price Per Judgment number is increased
- The Price Per Page setting is increased
- More untrusted judgments were collected than expected. There are 3 ways judgments become untrusted:
- When test question answers are updated after the job has launched, contributors' accuracy can drop below the set Minimum Test Question Accuracy. Since their previous correct answers may no longer match the newly updated answers, they are no longer considered as trusted.
- If Minimum Test Question Accuracy is increased when the job is running, judgments from contributors who did not meet the updated accuracy will become untrusted.
- Manual rejections of contributors will turn those judgments from trusted to untrusted.
Why do I pay for untrusted judgments?
- Any judgment submitted in Work Mode is paid for. However, when a contributor becomes untrusted, they are removed from the job so they are no longer allowed to submit judgments.
Note: Rows that collected untrusted judgments will be re-released to contributors to recollect trusted judgments. As a result, new judgments will accrue additional payments. If the Minimum Accuracy Threshold in a job is set to 80%, contributors need to pass Quiz Mode with 80% accuracy. If their accuracy falls below 80% as they continue each page in Work Mode, judgments that were considered trusted become untrusted.
What are the best ways to keep my job costs down?
- Provide good examples and instructions
- Effective instructions in the job will help contributors understand and complete the task correctly.
- Include edge case examples to help contributors understand the job's more difficult rows.
- Make sure that the test questions are fair and correct.
- Poor quality test questions can cause high-quality contributors to be removed from the job, thereby causing their work to become untrusted and recollected.
- Provide test question reasoning as this help contributors lean why an answer is considered correct
- Monitor test questions
- We recommend launching a small batch of rows as poor quality test questions can be hard to spot initially. When the job is running, review test questions to ensure all answers are fair. This can help decrease untrusted judgments and increase throughput.