Follow

Guide to: Pre-labeling (Beta)

Overview

The Pre-labeling feature allows users to utilize a machine learning model to provide initial "best-guess" hypotheses to an annotation project. This feature is helpful to specialized annotation projects as providing a contributor with model-predicted annotation hypotheses can dramatically cut down annotation time while maintaining and improving annotation quality. The Pre-labeling feature includes a model template library from which users can select the appropriate model to use for a given project.

Important Note: This feature is part of our managed services offering; please contact your Customer Success Manager for access or more information.

How to Pre-label Data with a Machine Learning Model

1. Copy and Customize a Base Model

blobid0.png

Fig. 1: The Model Template Library

  • Preview the predictions of the model by uploading a data file in the model preview modal.
    • The modal will display the model's predictions on the first five rows of data in the file.
  • Model preview is not available for all models at this time.

Important Note: The file must contain data column(s) with header(s) matching the headers listed under "input columns", and containing data in the indicated format accordingly.

blobid1.png

Fig. 2: Preview of Model Template

  • Click "Create" to create a copy of the base model in your team's account.
    • The copy will be accessible to all users on your team.
  • The copy can then be customized by changing its name and editing the ontology (changing class names, editing class colors, and disabling classes as desired).

blobid2.png

Fig. 3 Configure Model Page

  • Navigate to the "Evaluate" tab to preview the newly customized model's predictions on your data.
    • Upload a sample data file to view the model's predictions for the first five rows of data.
  • Model evaluation is not available for all models at this time.

blobid18.png

Fig. 4: Evaluate Model Page

2. Set-up a Job to Receive the Model's Predictions

  • Navigate to the Jobs page and create a new job.
  • To receive the model's predictions, the job must include one of the CML element(s) listed under "works before" on the model's Preview and Evaluate pages.

blobid7.png

Fig. 5: Job CML with Predicted Model Input in Review-from Parameter

  • The job CML must contain a review-from attribute formatted to read in the model's predictions from the data column containing the model output data.
    • This column will be named as indicated under "output columns" on the model Preview and Evaluate pages.
    • For example, for a job to be designed to receive predictions from the Label Pixels in Street Scene Images model, you would include the attribute review-from="{{predicted_image_url}}" in the job's CML.

Important Note: The job's ontology must match the ontology of the customized model. You can download the base model ontology for each model from the Model Template Library article.

3. Route Model Predictions to the Job Using a Workflow

  • To connect the customized model to the job, navigate to the workflows tab in the global navigation sidebar. Click "Create Workflow", then name the workflow.

blobid8.png

Fig. 6: Creating a New Workflow

  • Drag the customized model from your list of jobs to the workflows canvas as the first operator.

blobid9.png

Fig. 7: Adding Customized Model to Workflow

  • Drag the job to the workflows canvas as the second operator.

blobid12.png

Fig. 8: Adding Model Prediction Review Job to Workflow

  • In the "Select Routing Method" dropdown, select routing option:
    • "Route All Rows" is the most commonly used option. This option will allow all model predictions for each row to route to the job successfully.
    • "Route a Random Sample" is also available. This option enables routing of a random sample of the input data to the job.
    • "Route by Column Headers" is available where applicable. The option enables routing of rows according to predicted labels.
    • "Route by Column Headers" for confidence columns is not available at this time.

blobid15.png

Fig. 9: Select "Route All Rows" Routing Method to Workflow

  • Navigate to the Data page of the workflow, and upload a data file you want to annotate.

blobid16.png

Fig. 10: Uploading Data to Workflow

  • Launch the job via the Launch page of the workflow. 
    • Data rows containing the model's predictions will typically appear in the job within 1 minute or less.
  • The model's predictions can be viewed by navigating to the Data Page of the model review job. Click into individual units to review predictions for that unit.

blobid17.png

Fig. 11: Reviewing Model Predictions on Model Review Job Unit Pages


Was this article helpful?
0 out of 0 found this helpful


Have more questions? Submit a request
Powered by Zendesk