Guide to: Running a Shapes Peer Review Job


Peer review jobs allow contributors to review and edit the results from a previously run annotation job. Contributors can removor update existing annotations, and add new annotations if needed. To upload annotations from previously run shapes jobs into the cml:shapes tool for peer review job, you can follow these easy steps: 


  • The shapes tool supports hosted image files.  
  • A column including results from a previously run shapes job (either from the Aggregated or Full Report) or your own pre-made annotations (that are identically formatted as results from a cml:shapes job) is required. 
    • An example source data file is attached below for an example of the formatting required. 

Building a Job 

The following CML contains the possible parameters for a shapes job with labels: 

<cml:shapes type="['{{shape_type}}']" source-data ="{{image_url}}" name="annotation_reviewreview-data=”{{annotation}}” label="Review this image" validates="required" ontology="true"/> 

Note: The review-from attribute, which is required for peer review jobs, is supported in by using CML in the Code Editor only; using the Graphical Editor is not currently supported. 


Below are the parameters available for the cml:shapes tag in a review job. Some are required in the element, while some can be left out. 

  • type 
    • The shape used in the job, set in an array. 
    • To use multiple shapes in one job, include each shape in the array, separated by commas, e.g., type="['box','dot','polygon','line']" 
      • You’ll need to include the corresponding parameters for each shape. 
  • source-data 
    • The column from your source data that contains the image URLs to be annotated. 
  • name 
    • The results header where annotations will be stored. 
  • label 
    • The question label contributors will see. 
  • review-data 
    • This will read in existing annotations on an image. The format must match the output from the aggregated or full reports of a previously- run shapes job. The following components of the annotation are required: 
      • ‘type’ 
      • ‘class’ if using an ontology 
      • ‘coordinates' 
      • 'id' 
    • Example:  
      • [{“class”:“car”,“coordinates”:{“h”:330,“w”:384,“x”:1191,“y”:306},“type”:“box”,“id”:“247f2099-22e1-4825-9bc1-3e51c6019fe0”}] 
  • validates (optional) 
    • Whether or not this element is required to be answered. 
    • Accepts ‘required’ 
    • Defaults to not required if absent 
  • ontology (optional) 
    • The list of classes to be labeled in an image. Please see this article to learn how to create your custom ontology. 
    • Accepts 'true' or 'false'
  • task-type (optional)
    • Please set task-type=”qa” when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this article for more details.


Fig. 1: Preview of a bounding box review job via Preview Page 

After the data is uploaded and the job is built in the CML Editor, the job will follow the same process to launch and monitor like any other job in the platform. 

Please see our other articles detailing how to for run jobs for specific cml:shapes types: 

Was this article helpful?
0 out of 0 found this helpful

Have more questions? Submit a request
Powered by Zendesk