Follow

Guide to: In-Platform Audit BETA

In-Platform Audit is a product that allows users to review the results of their jobs directly in the platform. It aims to simplify and accelerate the audit process to ensure users have greater insight into their data before training a machine learning model. 

This product is currently in BETA; for access, please reach out to your CSM or our Support team.

Getting Started

In-Platform Audit requires that the job have at least one finalized row of data, at which point you can build your Grid View and start your audit. When you get to the Audit page, you’ll see a button to generate aggregations: 

ipa-generate-aggs.png

Clicking this button generates aggregations on your finalized data. Note: this will only be necessary as part of the BETA product; once it’s generally available, aggregations will be generated automatically when a job finishes. You’ll still be able to manually generate aggregations for a job that has finalized rows but has not yet finished. 

Once your aggregations have generated, you can setup your audit. In the Customize Source Data modal, you’ll be able to choose up to three columns from your source data to display in your grid, along with the data type. Grid View can render the following data types: 

  • Text 
  • Image
  • Audio 
  • Video 
  • URL 

Be sure to select the correct data type for the column, otherwise the data won’t render. 

ipa-setup-audit.gif

Conducting Your Audit

Once you’ve setup your Grid View, you’re ready to start auditing your job! In the Grid View, you’ll notice a few buttons: 

  • Accuracy 
    • The overall accuracy of your job. Clicking this link will open up the Accuracy Modal. See ‘Accuracy Score’ section below. 
  • Configure Tile 
    • Customize Data Source 
      • This is the modal you use to configure your Grid View, as outlined above 
    • Customize Question 
      • This modal allows you to filter for answer values to specific questions. This filters the rows that are returned in Grid View. For example, if you filter for ‘food’ in the modal below, your Grid View will display rows of data where the top answer was ‘food’ for the question ‘Is this a photo of the restaurant’s menu, food, interior, or exterior?’ 
      • Note: in BETA, this product only supports top answer aggregation (aggregation=”agg”); fields that use other aggregation types cannot be filtered.

ipa-customize-question.png

  • View By 
    • Audited 
      • This will display only rows you’ve already audited. See ‘Auditing Results’ section below. 
    • Unaudited 
      • This will display only rows you’ve not yet audited 
  • Regenerate Aggregations 
    • Clicking this button will regenerate aggregations on your job. This is only necessary if your job has collected more judgments since you setup your Audit.  
  • Download Report 
    • This downloads a CSV of your audit results. See ‘Audit Report’ section below for more information. 
  • Sort By
    • ID: Descending Order 
      • Unit ID in descending order 
      • This is the default sorting 
    • ID: Ascending Order 
      • Unit ID in ascending order 
    • Confidence: High to Low 
      • Confidence for the question currently displayed in Grid View (configured via the Customize Question modal) 
      • For information on confidence scores, check out our Confidence Score article  
    • Confidence: Low to High 

Auditing Results

Detail View allows you to conduct an in-depth audit of your results. For each row of data, you can mark each field correct or incorrect by clicking the ‘X’ (incorrect) or the check (correct). If incorrect, you can choose the correct answer; no matter what, you’ll be able to provide a reason for the answer.  

detail-view-audit.gif

The corrected answers and the provided reasons are stored in the audit report (see ‘Audit Report’ section below) for you to discern patterns in the results and for general tracking purposes. These corrected answers do not overwrite actual answers in the aggregated report.

Accuracy Score

Once you’ve audited at least one row, you’ll notice the accuracy link at the top of Grid View shows a value. Clicking on this link opens the Accuracy Modal, which provides a breakdown of your per-field accuracy, along with an overall job accuracy. These are calculated as follows: 

  • Per-field accuracy: the number of correct answers out of total rows audited 
  • Overall: the average of all the fields’ accuracies
    accuracy-score.png

In the example above, the first question in this job – whether the photo depicts a menu, food, interior, or exterior of a restaurant – was marked correct for all 52 rows audited, resulting in 100% accuracy. The second question is low accuracy at 10/30 correct or 33.33% accurate. The third field has pretty high accuracy, with 18 out of 20 correct, or 90%. The average of these three fields is 74.44%, which is shown near the top of the modal. 

These values help you to pinpoint where your job is performing well and where it could use improvement, whether in the instructions, test questions, job design, or any other way job accuracy is impacted. 

Audit Report

In addition to an accuracy score, once you’ve audited at least one row of data, there will also be an audit report available to download. This report contains the following: 

  • The unit ID 
  • The source data from your job 
  • {question}_aggregated 
    • The aggregated answer for the field 
  • {question}_confidence 
    • The confidence score for the field 
  • {question}_correct_yn 
    • Whether each field was marked correct or incorrect 
    • A value of ‘1’ is correct and ‘0’ incorrect 
  • {question}_audit 
    • The correct answer provided for this field 
    • If the field was marked correct, this value will match {question}_aggregated; otherwise, it will contain the correct answer you provided during the audit 
  • {question}_audit_reason 
    • The reason you provided for this answer 

Screen_Shot_2021-04-16_at_5.06.37_PM.png


Was this article helpful?
0 out of 1 found this helpful


Have more questions? Submit a request
Powered by Zendesk