Follow

Guide to: In-Platform Audit BETA

In-Platform Audit is a product that allows users to review the results of their jobs directly in the platform. It aims to simplify and accelerate the audit process to ensure users have greater insight into their data before training a machine learning model. 

This product is currently in BETA; for access, please reach out to your CSM or our Support team.

Getting Started

In-Platform Audit requires that the job have at least one finalized row of data, at which point you can build your Grid View and start your audit. When you get to the Audit page, you’ll see a button to generate aggregations: 

ipa-generate-aggs.png

Clicking this button generates aggregations on your finalized data. Note: this will only be necessary as part of the BETA product; once it’s generally available, aggregations will be generated automatically when a job finishes. You’ll still be able to manually generate aggregations for a job that has finalized rows but has not yet finished. 

Once your aggregations have generated, you can setup your audit. In the Customize Source Data modal, you’ll be able to choose up to three columns from your source data to display in your grid, along with the data type. Grid View can render the following data types: 

  • Text 
  • Image
  • Audio 
  • Video 
  • URL 

Be sure to select the correct data type for the column, otherwise the data won’t render. 

ipa-setup-audit.gif

Conducting Your Audit

Once you’ve setup your Grid View, you’re ready to start auditing your job! In the Grid View, you’ll notice a few buttons: 

  • Accuracy 
    • The overall accuracy of your job. Clicking this link will open up the Accuracy Modal. See ‘Accuracy Score’ section below. 
  • Configure Tile 
    • Customize Data Source 
      • This is the modal you use to configure your Grid View, as outlined above 
    • Customize Question 
      • This modal allows you to filter for answer values to specific questions. This filters the rows that are returned in Grid View. For example, if you filter for ‘food’ in the modal below, your Grid View will display rows of data where the top answer was ‘food’ for the question ‘Is this a photo of the restaurant’s menu, food, interior, or exterior?’ 
      • Note: This product only supports top answer or shape aggregations (aggregation=”agg” or aggregation="box-agg", "polygon-agg", etc.); fields that use other aggregation types cannot be filtered.
        • For image annotation or transcription jobs with only 1 judgment per row, be sure to select 'Include Single [shape type]' in the aggregation settings

ipa-customize-question.png

      • If auditing an image annotation or image transcription job with an ontology, you can filter by ontology class.
        • This is an or operator; images that contain one or more of the classes selected will be returned in Grid View.
        • For jobs without an ontology, you will not be able to filter on this field; instead, after this field is selected, each card in Grid View will show the count of total annotations in the image.

Screen_Shot_2021-07-20_at_11.17.21_AM.png

  • View By 
    • Audited 
      • This will display only rows you’ve already audited. See ‘Auditing Results’ section below. 
    • Unaudited 
      • This will display only rows you’ve not yet audited 
  • Regenerate Aggregations 
    • Clicking this button will regenerate aggregations on your job. This is only necessary if your job has collected more judgments since you setup your Audit.  
  • Download Report 
    • This downloads a CSV of your audit results. See ‘Audit Report’ section below for more information. 
  • Sort By
    • ID: Descending Order 
      • Unit ID in descending order 
      • This is the default sorting 
    • ID: Ascending Order 
      • Unit ID in ascending order 
    • Confidence: High to Low 
      • Confidence for the question currently displayed in Grid View (configured via the Customize Question modal) 
      • For information on confidence scores, check out our Confidence Score article  
    • Confidence: Low to High 

Auditing Results

Detail View allows you to conduct an in-depth audit of your results. For each row of data, you can mark each field correct or incorrect by clicking the ‘X’ (incorrect) or the check (correct). If incorrect, you can choose the correct answer; no matter what, you’ll be able to provide a reason for the answer. 

detail-view-audit.gif

 

If auditing an image annotation or transcription job, to view the annotations, select a judgment or 'aggregated' from the dropdown in the top left:
select-judgment.gif

Upon marking the field incorrect in an image annotation job, you'll be presented with five options to choose from:

  • Too many annotations
  • Missing annotations
  • Incorrect classes
  • Annotations too loose
  • Annotations too tight

Choose as many as apply to the image. You can still add a freeform reason to elaborate on any of the above. For now, you cannot edit annotations to create the corrections.

The corrected answers and the provided reasons are stored in the audit report (see ‘Audit Report’ section below) for you to discern patterns in the results and for general tracking purposes. These corrected answers do not overwrite actual answers in the aggregated report.

Accuracy Score

Once you’ve audited at least one row, you’ll notice the accuracy link at the top of Grid View shows a value. Clicking on this link opens the Accuracy Modal, which provides a breakdown of your per-field accuracy, along with an overall job accuracy. These are calculated as follows: 

  • Per-field accuracy: the number of correct answers out of total rows audited 
  • Overall: the average of all the fields’ accuracies
    accuracy-score.png

In the example above, the first question in this job – whether the photo depicts a menu, food, interior, or exterior of a restaurant – was marked correct for all 52 rows audited, resulting in 100% accuracy. The second question is low accuracy at 10/30 correct or 33.33% accurate. The third field has pretty high accuracy, with 18 out of 20 correct, or 90%. The average of these three fields is 74.44%, which is shown near the top of the modal. 

These values help you to pinpoint where your job is performing well and where it could use improvement, whether in the instructions, test questions, job design, or any other way job accuracy is impacted. 

Audit Report

In addition to an accuracy score, once you’ve audited at least one row of data, there will also be an audit report available to download. This report contains the following: 

  • The unit ID 
  • The source data from your job 
  • {question}_aggregated 
    • The aggregated answer for the field 
  • {question}_confidence 
    • The confidence score for the field 
  • {question}_correct_yn 
    • Whether each field was marked correct or incorrect 
    • A value of ‘1’ is correct and ‘0’ incorrect 
  • {question}_audit 
    • The correct answer provided for this field 
    • If the field was marked correct, this value will match {question}_aggregated; otherwise, it will contain the correct answer you provided during the audit 
      • For fields marked incorrect in image annotation jobs, this column will contain any of the provided checkbox reasons outlined above
  • {question}_audit_reason 
    • The reason you provided for this answer 

Screen_Shot_2021-04-16_at_5.06.37_PM.png


Was this article helpful?
1 out of 2 found this helpful


Have more questions? Submit a request
Powered by Zendesk