Follow

Guide to: Quality Audit

 

Table of Content

  1. Introduction
  2. Getting Started
  3. Setup Audit
  4. Conducting Your Audit
  5. Auditing Results
  6. Accuracy Score
  7. Audit Report

-----------------

1. Introduction

Quality Audit allows users to review the results of their jobs directly on the platform. It aims to simplify and accelerate the audit process to ensure users have greater insight into their data before training a machine learning model. 

-----------------

2. Getting Started

Quality Audit requires that the job have at least one finalized row of data, at which point you can set up your Grid View and start your audit. Click on 'Audit' on the Top Navigation bar to access the Audit view.

audit.png

When you get to the Audit page, you’ll see one of these possible screens:

 

Unsupported Job

If the current job does not support Quality Audit, this screen will be shown.

unsupported.png

Currently supported tags include:

  • cml:checkbox

  • cml:checkboxes

  • cml:radios

  • cml:ratings

  • cml:select

  • cml:text

  • cml:textarea

  • cml:shapes

  • cml:video_shapes

  • cml:image_transcription

  • cml:image_segmentation

Generate Aggregations

agg.png

Aggregations will be generated automatically when a job finishes. You can still manually generate aggregations for a job that has finalized rows but has not yet finished by clicking this button.

Note: This product only supports top answer or shape aggregations (aggregation=”agg” or aggregation="box-agg", "polygon-agg", etc.); fields that use other aggregation types cannot be filtered.

-----------------

3. Setup Audit

setup.png

Once your aggregations have been generated, you can set up your audit. In the Customize Source Data modal, you’ll be able to choose up to three columns from your source data to display in your grid, along with the data type. Grid View can render the following data types: 

Be sure to select the correct data type for the column. Otherwise, the data won’t render. 

audit.gif

 

Grid View

Upon configuring the Data Sources, the Grid View will be displayed. The Grid View shows units with their respective source data and answers distribution for the selected question.

grid.gif

 

The Grid View has three sections:

  • Topbar: located at the top, it contains options for configuring which units, questions, and data sources to show;

  • Sidebar: located at the right, contains information about the accuracy and other options regarding Report and Regenerating Aggregations;

  • Units: the actual queried units, each unit presented as a Card containing the configured Data Sources and the selected Question.

-----------------

4. Conducting Your Audit

Once you’ve set up your Grid View, you’re ready to start auditing your job! In the Grid View, you’ll notice a few buttons: 

Topbar

  • Configure Tile 

    • Customize Data Source 

      • As outlined above, this is the modal you use to configure your Grid View. 

    • Customize Question 

      • This modal allows you to filter for answer values to specific questions. This filters the rows that are returned in Grid View. For example, if you filter for ‘food’ in the modal below, your Grid View will display rows of data where the top answer was ‘food’ for the question ‘Is this a photo of the restaurant’s menu, food, interior, or exterior?’ 

      • For image annotation or transcription jobs with only one judgment per row, be sure to select 'Include Single [shape type]' in the aggregation settings

      • You can filter by ontology class if you are auditing an image annotation or image transcription job with an ontology.

        • This is an or operator; images containing one or more of the selected classes will be returned in Grid View.

        • For jobs without an ontology, you will not be able to filter on this field; instead, after this field is selected, each card in Grid View will show the count of total annotations in the image.

24c15e39-d00f-4ac0-9a2a-9793eca6666e.png

Radios, Checkbox, Checkboxes, Select, and Ratings.

f894d589-6853-456a-bb6c-e7bf006bb51d.png

Text and Textarea.

54090365-ba94-44f3-ae96-d7925f76d74e.png

Shapes, Video Shapes, Image Transcription, Image Segmentation.

  • View By

    • Audited

      • This will only display the rows you’ve already audited. See the ‘Auditing Results’ section below.

    • Unaudited

      • This will only display the rows you’ve not yet audited.

  • Sort By

    • ID: Descending Order 

      • Unit ID in descending order 

        This is the default sorting. 

    • ID: Ascending Order 

      • Unit ID in ascending order 

    • Confidence: High to Low (on supported questions)

      • Confidence for the question currently displayed in Grid View (configured via the Customize Question modal). For information on confidence scores, check out our Confidence Score article.  

    • Confidence: Low to High (on supported questions)

 

Sidebar

  • The sidebar contains information about the total rows audited and the accuracy alongside options for downloading an audit report and regenerating the aggregations. Note: the job accuracy is the ratio between the correct audited questions per total audited questions.

audited.png

  • View Details

    • When clicking the View Details, the Accuracy modal is opened. Here it shows the overall job accuracy and a detailed accuracy for each question. Note: the question accuracy is the ratio between the correct audited answers per total audited answers across all the units for the same question.

view.png

 

  • Download Unit Report

    • You will download the CSV Report for the audited units only by clicking this button.

  • Regenerate Aggregations

    • We will start a new aggregation generation when clicking this button, bringing new finalized units, if any.

 

Units

  • The unit card shows the selected source data, unit id, and up to 3 aggregated answers, ontologies, or the number of annotated shapes, depending on the chosen question.

c0a18599-8915-4ebf-8eb8-53e01f01d78d.png

Types with Ontology

fe6965a9-7b61-44fc-9116-0b7fc0a24dbf.png

Types without ontology

f6ca76e8-ba14-4a0b-86fb-9b83a7716fae.png

Types with agg support

 

  • Clicking View Details will open the Detail View, which allows you to conduct an in-depth audit of your results.

-----------------

5. Auditing Results

On the Detail View, For each row of data, you can mark each field correct or incorrect by clicking the ‘X’ (incorrect) or the check (correct). If incorrect, you can choose the correct answer; no matter what, you’ll be able to provide a reason for the answer. 

886e13ba-247c-489d-9ebd-3ad135d11d98.png

If auditing an image annotation or transcription job, select a judgment or 'aggregated' from the dropdown in the top left to view the annotations. Upon marking the field incorrect in a shapes annotation job, you'll be presented with five options to choose from:

  • Too many annotations

  • Missing annotations

  • Incorrect classes

  • Annotations too loose

  • Annotations too tight

Choose as many as apply to the image. You can still add a freeform reason to elaborate on any of the above. For now, you cannot edit annotations to create the corrections.

e8085304-d675-4a05-a6dc-2df0388d90c0.gif

The corrected answers and the provided reasons are stored in the audit report (see ‘Audit Report’ section below) for you to discern patterns in the results and for general tracking purposes. These corrected answers do not overwrite actual answers in the aggregated report.

 

Saving Changes

Changes will only be saved when clicking on the Close button on the Detail View Modal or clicking on the Previous/Next buttons.

-----------------

6. Accuracy Score

Once you’ve audited at least one row, you’ll notice the accuracy value in the Sidebar changes. Clicking on View Details opens the Accuracy Modal, which provides a breakdown of your per-field accuracy, along with an overall job accuracy. These are calculated as follows: 

  • Per-field accuracy: the number of correct answers out of total rows audited 

  • Overall: the average of all the fields’ accuracies

1ea36f25-2c6f-4194-9d27-666042ffe20b.png

In the example above, the first question in this job – whether the photo depicts a restaurant's menu, food, interior, or exterior – was marked correct for all five rows audited, resulting in 100% accuracy. The second question has low accuracy at 1/4 correct or 25%. The third field has pretty high accuracy, with 7 out of 8 correct, or 87.50%. The average of these three fields is 70.83%, which is shown near the top of the modal. 

These values help you pinpoint where your job is performing well and where it could be improved, whether in the instructions, test questions, job design, or any other way job accuracy is impacted. 

-----------------

7. Audit Report

In addition to an accuracy score, once you’ve audited at least one row of data, there will also be an audit report available to download. This report contains the following: 

  • The unit ID 

    • The source data from your job 

  • {question}_aggregated 

    • The aggregated answer for the field 

  • {question}_confidence 

    • The confidence score for the field 

  • {question}_correct_yn 

    • Whether each field was marked correct or incorrect 

      • A value of ‘1’ is correct, and ‘0’ is incorrect. 

  • {question}_audit 

    • The correct answer provided for this field 

    • If the field were marked correct, this value would match {question}_aggregated; otherwise, it will contain the correct answer you provided during the audit. 

      • For fields marked incorrect in image annotation jobs, this column will contain any provided checkbox reasons outlined above.

  • {question}_audit_reason 

    • The reason you provided for this answer 

Example:

e1059c75-449c-4ab3-bdbb-a4f7396c16ec.png


Was this article helpful?
3 out of 6 found this helpful


Have more questions? Submit a request
Powered by Zendesk