A job may go out of funds due to judgments becoming untrusted and additional judgments are needed in order to finalize. A contributor's judgments can become untrusted if:
- Their trust in job drops below the set accuracy threshold
- They are manually rejected from the job
For more information judgement over-collection please out this article.
As a preventative measure, a job will automatically pause if it has run out of funds instead of automatically pulling additional funds from the team account. Before resuming a paused job, it is important to review and edit the job design, test questions, and settings. This will help prevent the job from running out of funds again.
Some common issues to look for include:
- Highly missed test questions
- It can be advantageous to create tough test questions to teach a contributor how to handle rare edge cases, however, creating too many can make it difficult for a contributor to main their accuracy level.
- Incorrect test questions will lower a contributor's accuracy even if the contributor provided the correct answer. Double check highly missed test question to ensure the correct answer is provided.
- Manual rejections
- When a contributor is manually rejected from a job, all their judgments become untrusted. This can cause an increase in job costs to compensate for the additional judgments needed to replace the rejected contributor's judgments
- Dynamic judgments and high disagreement
- This setting can be used to reduce costs by only ordering more judgments for rows with high disagreement, however, it can have the opposite effect if the job id designed to use this setting incorrectly and contributors submit inconsistent responses.
Resuming the job
Once the job has been reviewed and updated, it can be resumed by adding more funds. To do so, click on the "Add Funds" button or go straight to Launch page to add more funds and manually resume the job.
Fig 1. "Add Funds" button on a paused, out of funds job
Fig 2. Launch page with more information on adding more funds to re-launch a paused job
For more information on job costs, see this article.
Platform Automatic Pauses due to Inactivity
A job will automatically be paused if it has not received any activity (new judgments) within a 14 day period.
Please note, if a job has been automatically paused due to inactivity, once the job is resumed, it must receive activity within 24 hours to avoid being automatically re-paused.
Support Related Pauses
A job may also be paused by our support staff if we received contributor feedback for problematic jobs. Our support staff will reach out to you if this occurs.