Spotcheck (Beta) Feature

The in-platform Spotcheck (beta) feature automates the selection of a random set of rows and the process of report generation for auditing job results. This feature is available in an invite-only beta for selected customers with a paid subscription. If you are interested in joining this beta program, please contact your Customer Success Manager for more details.

Initiating Spotcheck

The Spotcheck feature can be initiated on the Results page using the "Use Spotcheck" action.


Fig. 1: Results Page

This operation creates a copy of the job titled "Spotcheck:{Job_Name}" with a sample of hundred randomly selected rows and their judgements.


Fig. 2: Job's Page

You can begin reviewing results on the Quality page in a process that is similar to creating test questions. When you begin to convert a row to a test question, you will see contributor responses for the row. At this stage, you don't have to worry about those responses, just record and save your response for each row.


Fig. 3: Test Question Creation

We recommend that you create atleast 20-30 test questions to gain insight into your data quality. However, the more rows you audit, the more accurate your estimate will be.

NOTE: Spotcheck jobs already have the data required to complete spotcheck. You don't need to upload data or modify the job design in order to complete spotcheck successfully.

Downloading Spotcheck Results


Fig. 4: Result Page for Spotcheck job

Once you have created the desired number of rows you want to audit, you can download the audit summary on the Reports page in the form for the Spotcheck Report.

You may receive an email notification for job completion, but you can access the Spotcheck report as soon as test questions have been created. Other reports such as Full and Aggregated reports can be ignored as they do not contribute to the audit process.

The Spotcheck report contains your responses recorded as part of the test question creation process, the aggregated result from all contributor responses as well as the agreement between the 2.


Fig. 5: Spotcheck Result Via CSV File

Columns in the report

  • question_name:gold_answer - Your response during test question creation in the spotcheck job.
  • question_name:answer - Aggregate of contributor responses from the original job.
  • quesiton_name:confidence - Confidence for the aggregate response.
  • question_name:correct? - Determines if the aggregate response matched your response, gold_answer.

You can compute the percentage or correct / incorrect answers from the "question_name:correct?" column.

NOTE: Spotcheck currently does not support Image Annotation use cases. You can audit results from these jobs using the manual audit process.

Was this article helpful?
1 out of 1 found this helpful

Have more questions? Submit a request
Powered by Zendesk