Follow

Guide to: Using custom models (Beta)

Overview

Custom models can be used to pre-label or give real-time assistance to annotators, using any machine learning model that you provide. Pre-labeling can be done for any tool, while currently only the cml:video_shapes tool is supported for assistance.

Instructions

First, determine if you plan to use your model for pre-labelling or for assistance. 

Your model is appropriate for pre-labelling when:

  • model inference takes longer than a few seconds;
  • inference does not depend on human-annotator input
  • Example:
    • Object detection: the model draws a box around all cats (or other class) on every frame in the video.

Your model is appropriate for annotation assistance when:

  • model inference takes less than a few seconds
  • inferences require one or more inputs from human annotators
  • Example:
    • Object tracking: an annotator draws a box around an object, and the model predicts where the box will be on subsequent frames

Next, follow these high-level steps. Detailed guidance is provided in the sections below.

  1. Enable Custom Models feature. Speak to your CSM to have this feature enabled
  2. Create an endpoint to serve your model
  3. Create a custom model in ADAP
  4. Set up the model for pre-labelling or assistance

Create an endpoint

The endpoint for your model must be set up as follows:

  1. Appen communicates with your model using HTTP POST requests, with JSON as the data format. Your endpoint should read the HTTP body as JSON to get the input, and send a JSON body as output. HTTP status codes should be followed as normal.
  2. The endpoint must be reachable from the Appen servers. This usually means it should be accessible via the internet.
  3. Because of point (2), we highly recommend implementing API key authentication and using HTTPS. The endpoint should restrict access to only valid API keys. Appen will send your API key to your endpoint using the X-API-Key header.

Beyond these restrictions and recommendations, you are free to host and implement your model in any way you prefer.

Each annotation tool has its own input and output data formats. Additionally, the formats are different for pre-labelling and assistance. See the sections below for documentation of these formats.

Create a Custom Model

  • Complete the following required fields:
    • Name: a name of your choice that describes the model
    • Prediction URL: the URL to your prediction endpoint. This can be either HTTP or HTTPS, but HTTPS is preferred. It can optionally include a path. For example: https://example.com/predict.
    • Prediction API key: an API key that will be sent in the X-API-Key header when Appen makes requests to your endpoint. This key is fully managed by you, including generation and verification.
  • Complete the following optional fields:
    • Description: a longer description of your model.
    • Input from CML: when used for pre-labeling, this field describes the output schema for your model. You can choose the CML tag that comes after your model in a workflow that it will use for output. Note that this field is currently unenforced.
    • Output from CML: When used for pre-labeling, this field describes the output schema for your model. You can choose the CML tag that comes after your model in a Workflow that it will use for output. Note that this field is currently unenforced.
    • Type: a category for your model, used for organization purposes on the Model Templates page. This field has no effect when running your model.
    • Tags: a set of tags for organizing and querying your model template. Press Tab or the comma key after typing the name to create the tag.

  • Your model Template will now appear under All Templates.
  • Click on your new custom Model Template, then click Create. This will create a new Model based on your Model Template.

Using a Custom Model for pre-labeling

To use a custom model for pre-labeling, follow these steps:

  1. Create a new Workflow
  2. Upload your data. If your model will be the first step in the workflow, this is the data that will be sent to your endpoint (one request per row, sent as JSON).
  3. In the "Add Operator" panel, click on the Models tab and drag your Model into the Workflow.
  4. Ensure your model's input and output matches that of the operators that come before and after it in the Workflow.
    1. Your model's input depends on the output of the operator (job/model/script) that comes before it in the Workflow. Your endpoint should accept a single row from the data of the previous operator.
    2. Your model's output depends on the input of the operator that comes after it in the Workflow. For example, if the cml for a job that follows your model has this format <cml:image_transcription source-data="{{image_url}}" name="annotation" />, then your model should return {"image_url": "some-image-url" } as its response.

See this example for creating a pre-labeling endpoint. This endpoint is designed to be used as the first operator in a workflow. It reads in urls to images and determines if they are hotdogs or not. This information can then be sent to a <cml:image_transcription> job. For example:

Input:

{ "image_url": "some-image-url" }

Output:

{ "annotation": true }

Using a Custom Model for assistance

Note: onlycml:video_shapes tool is supported for assistance.

To use a custom model for assistance, follow these steps:

  1. To start you will need a cml:video_shapes job and a custom model template configured (as described above)

  2. Go to the Model Templates page, click on your custom Model Template, then click Create. This will create a new Model based on your Model Template.

  3. Go to the Models page and note the ID of your new Model.

  4. Go to the Jobs page, click on your job, then click on the Design tab.

  5. Add the following attributes to the cml:video_shapes tag: assistant="custom" assistant-model="your_model_id". Be sure to replace your_model_id with the your Model ID from step 3. Click Save.

  6. To test, make sure your job is launched, go to the Data tab, click on a unit, and draw a shape. You should see your predictions within a few seconds.

  7. To debug, open the web browser developer tools, open the network tab, and look for a request to predict_custom. Verify that the input and output are correct.

Below are the input and output formats for cml:video_shapes assistance. Please see this Appen article which describes the Video Annotation Output Format.

Input:

{
  // The full annotation state at the time of prediction.
  // This includes the "update" changes below.
  // Uses the Video Annotation Output Format.
  "annotation": {
    "shapes": {},
    "frames": {},
    "ontologyAttributes": {}
  },

  // This object represents the change made in the annotation tool that triggered the prediction.
  // Usually this is the annotator moving or resizing a shape, and the shape/frame data of that change is reflected here.
  // Uses the Video Annotation Output Format.
  "update": {
    "shapes": {},
    "frames": {},
    "ontologyAttributes": {}
  },

  // The model ID of the prediction being made.
  // Not to be confused with the model template ID.
  "model_id": "500",

  // The job ID of the prediction being made.
  "job_id": "2000000",

  // The worker ID making the prediction.
  // This is the integer version of the worker ID, NOT the UUID version.
  "worker_id": "40000000",

  // The unit ID of the prediction being made.
  "unit_id": "3000000000",

  // The video ID of the prediction being made.
  // Each video has a unique ID.
  "video_id": "8faac105ce039efcca25b6c9d3ac2c6166cb90070c98ca1c6dbe52b4ba2341ec",

  // Frame data indexed by frame number.
  // This is the base-64 encoding of the binary image data.
  // See the code below for an example of how to use this.
  "frames": {
	"1": "dGhpc2lzYXRlc3QxCg==",
    "2": "dGhpc2lzYXRlc3QyCg==",
    "3": "dGhpc2lzYW5vdGhlcnRlc3QzCg=="
  }
}

Output:

{
  // This object should contain all the changes you want to make to the annotation state.
  // For example, if you wanted to move a box up 10 pixels on all frames, you would copy the shape from the input, then go through each of the "frames", and subtract 10 from "y" property.
  // Uses the Video Annotation Output Format.
  "update": {
    "shapes": {},
    "frames": {},
    "ontologyAttributes": {}
  }
}

Here is some example code for creating an assistance endpoint. It uses FastAPI to handle HTTP POST requests to the /tracking endpoint. It first decodes the frame data from base-64 to binary, gets the initial annotations from the input, then uses the MMTracking library to run a Single-Object Tracking (SOT) algorithm on the data.

 


Was this article helpful?
0 out of 0 found this helpful


Have more questions? Submit a request
Powered by Zendesk