Guide to: Running a Shapes Peer Review Job

Overview 

Peer review jobs allow contributors to review and edit the results from a previously run annotation job. Contributors can remove or update existing annotations, and add new annotations if needed. To upload annotations from previously run shapes jobs into the cml:shapes tool for a peer review job, you can follow these easy steps: 

Data 

• The shapes tool supports hosted image files.  
• A column including results from a previously run shapes job (either from the Aggregated or Full Report) or your own pre-made annotations (that are identically formatted as results from a cml:shapes job) is required. 
  • An example source data file is attached below for an example of the formatting required. 

Building a Job 

The following CML contains the possible parameters for a shapes job with labels: 

<cml:shapes type="['{{shape_type}}']" source-data ="{{image_url}}" name="annotation_review" review-data=”{{annotation}}” label="Review this image" validates="required" ontology="true"/> 

Note: The review-from attribute, which is required for peer review jobs, is supported in by using CML in the Code Editor only; using the Graphical Editor is not currently supported. 

Parameters 

Below are the parameters available for the cml:shapes tag in a review job. Some are required in the element, while some can be left out. 

• type 
  • The shape used in the job, set in an array. 
  • To use multiple shapes in one job, include each shape in the array, separated by commas, e.g., type="['box','dot','polygon','line']" 
    • You’ll need to include the corresponding parameters for each shape. 

• source-data 
  • The column from your source data that contains the image URLs to be annotated. 
• disable_image_boundaries: allows shapes to be drawn outside the image boundary
  • when disable_image_boundaries="true" shapes can be dragged outside the image boundaries and the output will contain negative values
  • the default setting for disable_image_boundariesis "false"

• name 
  • The results header where annotations will be stored. 

• label 
  • The question label contributors will see. 

• review-data 
  • This will read in existing annotations on an image. The format must match the output from the aggregated or full reports of a previously- run shapes job. The following components of the annotation are required: 
    • ‘type’ 
    • ‘class’ if using an ontology 
    • ‘coordinates' 
    • 'id' 
  • Example:  
    • [{“class”:“car”,“coordinates”:{“h”:330,“w”:384,“x”:1191,“y”:306},“type”:“box”,“id”:“247f2099-22e1-4825-9bc1-3e51c6019fe0”}] 

• validates (optional) 
  • Whether or not this element is required to be answered. 
  • Accepts ‘required’ 
  • Defaults to not required if absent 

• ontology (optional) 
  • The list of classes to be labeled in an image. Please see this article to learn how to create your custom ontology. 
  • Accepts 'true' or 'false'
• task-type (optional)
  • Please set task-type=”qa” when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this article for more details.

Fig. 1: Preview of a bounding box review job via Preview Page 

Shape Type Limiter

• Limit which shapes can be used with certain classes

Min/Max instance quantity

• Configure ontologies with instance limits
• Comes with the ability to mark the class as not present for long tail scenarios. This information will be added to the output as well.

Customizable Hotkeys

• Hotkeys can be assigned to classes by the user. Hotkeys cannot conflict with any other browser or tool shortcuts.

After the data is uploaded and the job is built in the CML Editor, the job will follow the same process to launch and monitor like any other job in the platform. 

Please see our other articles detailing how to for run jobs for specific cml:shapes types: 

• Bounding Boxes 
• Polygons 
• Dots 
• Lines