Guide to: Quality Audit

 

Table of Contents

1. Introduction
2. Getting Started
   • Unsupported Job
   • Generate Aggregations
3. Setup Audit
   • Grid View
4. Conducting Your Audit
   • Topbar
   • Sidebar 
   • Units
5. Auditing Results
   • Saving Changes
6. Accuracy Score
7. Audit Report

-----------------

1. Introduction

Quality Audit allows you to review the results of your jobs directly on the platform. It aims to simplify and accelerate the audit process to ensure you have greater insight into your data before training a machine learning model. 

-----------------

2. Getting Started

Quality Audit requires that the job have at least one finalized row of data, at which point you can set up your Grid View and start your audit. Click on 'Audit' on the top navigation bar to access the Audit view.

When you get to the Audit page, you’ll see one of these possible screens:

 

i. Unsupported Job

If the current job does not support Quality Audit, this screen will be shown.

Aggregations will be generated automatically when a job finishes. You can still manually generate aggregations for a job that has finalized rows but has not yet finished by clicking this button.

Note: This product only supports top answer or shape aggregations (aggregation=”agg” or aggregation="box-agg", "polygon-agg", etc.), and text annotation aggregations (tagg); fields that use other aggregation types cannot be filtered.

-----------------

3. Setup Audit

Once your aggregations have been generated, you can set up your audit. In the Audit Configuration modal, you will be able to configure how Quality Audit will display Grid View in three sections:

• Sample Units Amount

  • Determine the number of units that will be available for auditing in Grid View (e.g. a value of 40% for a 1000-unit job will result in 400 units available)

  • Leaving the field blank results in all finalized units being available for auditing

• Data Source For Audit Preview

  • Up to three columns can be selected from your source data to display in Grid View, along with the data type. Grid View can render the following data types: 

    • Text

    • Image

    • Audio 

    • Video 

    • URL 

    • HTML (Click Here to learn more)

  • Be sure to select the correct data type for the column. Otherwise, the data won’t render.

  • In a Text Relationships question, when selecting the type Text for the source data column, Quality Audit will present the text from the original job:

• Question For Audit Preview

  • This section allows you to filter for answer values to specific questions. This filters the rows that are returned in Grid View. For example, if you filter for 'food' in the modal below, your Grid View will display rows of data where the top answer was 'food' for the question ‘Is this a photo of the restaurant’s menu, food, interior, or exterior?’ 

  • For image annotation or transcription jobs with only one judgment per row, be sure to select 'Include Single [shape type]' in the aggregation settings

  • You can filter by ontology class if you are auditing a question with ontology support (with the exception of Text Relationships).

    • This filter uses an or operator; images containing one or more of the selected classes will be returned in Grid View.

    • For jobs without ontology, you will not be able to filter on this field; instead, after this field is selected, each card in Grid View will show the count of total annotations in the image.

Radios, Checkbox, Checkboxes, Select, Ratings, and Taxonomy

Text, Textarea, Tools without ontology, and Text Relationships

Tools with ontology enabled: Shapes, Video Shapes, Image Transcription, Image Segmentation, Text Annotation, Audio Annotation, and Audio Transcription

 

Grid View

Upon configuring the Data Sources, the Grid View will be displayed. The Grid View shows units with their respective source data and answers distribution for the selected question.

The Grid View has three sections:

• Topbar: located at the top, it contains options for configuring which units, questions, and data sources to show;

• Sidebar: located at the right, contains information about the accuracy and other options regarding Report and Regenerating Aggregations;

• Units: the actual queried units, each unit presented as a Card containing the configured Data Sources and the selected Question.

-----------------

4. Conducting Your Audit

Once you’ve set up your Grid View, you’re ready to start auditing your job! In the Grid View, you’ll notice a few buttons: 

Topbar

• Configure Tile 

  • Customize Data Source

    • This modal is equivalent to the Data Source For Audit Preview section of Setup Audit

  • Customize Question

    • This modal is equivalent to the Question For Audit Preview section of Setup Audit

• Filters

  • Audit Status

    • Audited: this will only display the rows you’ve already audited. See the ‘Auditing Results’ section below.
    • Unaudited: This will only display the rows you’ve not yet audited.
  • Question Confidence Score
    • This filter is only enabled for the questions that support aggregation

Filters with confidence score enabled

Filters with confidence score disabled

 

• Sort by

 

• • ID: Descending Order 

    • Unit ID in descending order 

      This is the default sorting. 

  • ID: Ascending Order 

    • Unit ID in ascending order 

  • Confidence: High to Low (on supported questions)

    • Confidence for the question currently displayed in Grid View (configured via the Customize Question modal). For information on confidence scores, check out our Confidence Score article.  

  • Confidence: Low to High (on supported questions)

  • Randomize

    • Display units in a random order

 

Sidebar

• The sidebar contains information about the total rows audited and the accuracy alongside options for downloading an audit report and regenerating the aggregations.

• The accuracy is the ratio between the correct audited questions per total audited questions. It is divided into three different sections, when applicable:

  • Filtered Units: Units that match the current filters and are present in Grid View. Present when there are applied filters

  • Sampled Units: Units present in the job sample that was defined in the Setup Audit phase. Present when the sample % is lower than 100%

  • Total Finalized Units: All finalized units in the job. This section is always present

• View Details

  • When clicking on View Details in any of the three accuracy sections, the Accuracy modal for the section is opened. It shows the overall accuracy and a detailed accuracy for each question, for the scope: entire job, job sample, or filtered units

  • Note: the question accuracy is the ratio between the correct audited answers per total audited answers across all the units for the same question.

• Download Unit Report

  • You will download the CSV Report for the audited units only by clicking this button.

• Regenerate Aggregations

  • You will start a new aggregation generation when clicking this button, bringing new finalized units, if any.

 

Units

• The unit card shows the selected source data, unit id, and up to 3 aggregated answers, ontologies, or the number of annotated shapes, depending on the chosen question.

Types with Ontology

Types without ontology

Types with agg support

Text Relationships

 

• Clicking View Details will open the Detail View, which allows you to conduct an in-depth audit of your results.

-----------------

5. Auditing Results

On the Detail View, For each row of data, you can mark each field correct or incorrect by clicking the ‘X’ (incorrect) or the check (correct). If incorrect, you can choose the correct answer; no matter what, you’ll be able to provide a reason for the answer. 

If auditing an annotation tool question, you will be presented with the tool along with the judgments and aggregation if available:

Select a judgment or the aggregation from the dropdown in the top left to view the annotations.

For cml:image_annotation and cml:text_annotation, after you have marked the field as incorrect, you will be presented with some additional options to choose from:

Image Annotation (Shapes)

• Too many annotations

• Missing annotations

• Incorrect classes

• Annotations too loose

• Annotations too tight

Text Annotation

• Incorrect classes

• Incorrect spans

• Missing spans

• Missing classes

• Didn't follow guidelines

• Nonsensical Annotations

Choose as many as apply. You can still add a freeform reason to elaborate on any of the above.

For now, you cannot edit tool annotations to create the corrections.

Shapes options

Text Annotation Options

The corrected answers and the provided reasons are stored in the audit report (see ‘Audit Report’ section below) for you to discern patterns in the results and for general tracking purposes. These corrected answers do not overwrite actual answers in the aggregated report.

 

Saving Changes

Changes will only be saved when clicking on the Close button on the Detail View Modal or clicking on the Previous/Next buttons.

-----------------

6. Accuracy Score

Once you’ve audited at least one row, you’ll notice the accuracy value in the Sidebar changes. Clicking on View Details opens the Accuracy Modal, which provides a breakdown of your per-field accuracy, along with an overall job accuracy. These are calculated as follows: 

• Per-field accuracy: the number of correct answers out of total rows audited 

• Overall: the average of all the fields’ accuracies

In the example above, the first question in this job – whether the photo depicts a restaurant's menu, food, interior, or exterior – was marked correct for all five rows audited, resulting in 100% accuracy. The second question has low accuracy at 1/4 correct or 25%. The third field has pretty high accuracy, with 7 out of 8 correct, or 87.50%. The average of these three fields is 70.83%, which is shown near the top of the modal. 

These values help you pinpoint where your job is performing well and where it could be improved, whether in the instructions, test questions, job design, or any other way job accuracy is impacted. 

-----------------

7. Audit Report

In addition to an accuracy score, once you’ve audited at least one row of data, there will also be an audit report available to download. This report contains the following: 

• The unit ID 

  • The source data from your job 

• {question}_aggregated 

  • The aggregated answer for the field 

• {question}_confidence 

  • The confidence score for the field 

• {question}_correct_yn 

  • Whether each field was marked correct or incorrect 

    • A value of ‘1’ is correct, and ‘0’ is incorrect. 

• {question}_audit 

  • The correct answer provided for this field 

  • If the field were marked correct, this value would match {question}_aggregated; otherwise, it will contain the correct answer you provided during the audit. 

    • For fields marked incorrect in image annotation jobs, this column will contain any provided checkbox reasons outlined above.

• {question}_audit_reason 

  • The reason you provided for this answer 

Example: