Follow

Guide To: Running a Job with Ontology Attributes

Overview

Our ontology attributes feature allows users to collect additional metadata on jobs using the image and video annotation tools.

Important Note: As this tool is currently in BETA, it must be enabled for your team for access; please contact your Customer Success Manager or Account Executive for more information.

Building a Job 

CML 

The ontology attributes feature can be used in any cml:shapes and cml:video_shapes type job. Please see our other articles detailing how to for run jobs for specific cml:shapes types:  

Ontology Manager 

Creating an Ontology Via the UI 

When enabled, ontology attributes can be accessed via the Ontology Manager page of a job and configured per class by clicking the pencil icon next to the desired class. 

blobid0.png

Fig. 1: Selecting a class to edit via the Ontology Manager

After selecting the pencil icon, a modal will appear with the ontology attributes section to configure questions to collect attribute information. The graphical editor follows the graphical editor used for creating questions in job designs (please see Guide to: The Graphical Editor).  

When creating questions for attribute collection, keep in mind there is a maximum of 5 and 50 options for multiple choice and dropdown items, respectively, to ensure forms are streamlined for contributors.  

blobid1.png 

Fig. 2: Ontology Attributes section and question type options

blobid2.png

Fig. 3: Example question set-up with Ontology Attributes

Creating an Ontology via JSON

An ontology can also be created by uploaded a JSON file. We recommend creating your first ontology via the graphical interface, then downloading the JSON to understand structure and continue building on.  

Here is an example JSON file that corresponds to the question created above in the graphical editor.

[ 
 { 
  "description": "", 
  "class_name": "Vehicle", 
  "questions": [], 
  "display_color": "#FF1744", 
  "children": [ 
   { 
    "description": "", 
    "class_name": "Car", 
    "questions": [ 
     { 
"id": "bd5dda2e-2cbc-4526-8500-e1b0db143f8d", 
       " customUserId": "", 
       "type": "Multiple Choice", 
       "data":  
{ 
         "questionText": "Is the car occluded?", 
         "questionChoices": 
[ 
           { 
             "id": "2a3b6c0f-5597-4863-8e48-3b8751e69b49", 
             " customUserId": "1", 
             "label": "Yes", 
             "value": " yes" 
           }, 
           { 
             "id": "ee7227c5-4667-4468-afdb-31ccf7767133", 
             " customUserId": "2", 
             "label": "No", 
             "value": " no" 
           } 
], 
         "isRequired": true, 
         "tipsHints": " Occlusion means the visibility of the car is obstructed. 
         "resultsHeader": "occluded" 
} 
} 
       ],
        "display_color": "#651FFF" 
     } 
   ] 
 } 
] 

For classes with ontology attributes, the following variables must be included unless stated otherwise: 

  • id: Platform generated id for the question 
  • Questions: This array hold the JSON objects that describe the attributes added to the class 
  • customUserId (optional)Accepts a string for a customer provided unique identifier to map to an internal ontology schema that can be applied to questions and answers.  
    • Note: This currently can only be updated via JSON upload and appears in the output.
  • Type: The type of question used for the attribute (i.e. Multiple Choice, Checkbox Group, etc.) 
  • Data: a JSON object containing information on the attribute. The variables may change depending on the question type chosen. 
    • Standard variables include:
      • questionTextAccepts a string for the question label that appears to the contributor 
      • questionChoicesAccepts a string for the options to a question (i.e. for Multiple Choice and Checkbox Group) 
        • label: Accepts a string for the option label shown to the contributor 
        • value: Accepts a string for the option value stored in the output reports 
      • defaultsCheckedAccepts a boolean value for Checkbox questions to default a checkbox as marked 
      • isRequired: Accepts a boolean value for if a question is required 
      • tipsHints:  Accepts a string for a hint to give to the contributor 
      • resultsHeader: Accepts a string for the output name of the question provided in the output reports 
    • Additional Attributes for Single Text Line Inputs: 
      • cleanOutput: Accepts a boolean value to enable clean output validators 
      • textCase: If cleanOutput is set to true, this variable accepts “UPPER CASE”, “lower case”, “title case”, or “None”  
      • trimSpaces: If cleanOutput is set to true, this variable accepts “No whitespace”, “No leading/trailing whitespace”, “No multiple whitespaces”, and “None” 

Please see the Ontology Manger article for information on standard ontology formats. 

Ontology Attributes Interpolation in Video 

When ontology attributes are added to a class in a video annotation job, contributors can take advantage of answer “interpolation” to maximize efficiency. 

When a contributor saves attributes in a job, the attributes will “interpolate” and apply to all future frames, up to the last frame where attributes were manually adjusted.  

For example, say a contributor is labeling a 15 frame clip. The contributor adds a shape to frame 1 and saves attributes. The contributor then updates the attributes in frame 10. If the contributor goes back and updates the attributes in frame 7, those attributes will then “interpolate” and apply to frames 8 and 9 since 10 is the next frame where attributes were manually adjusted. This is useful when capturing temporal tags such as occlusion or truncation. 

Peer Review

In BETA, test questions are not supported, and peer review is a method to ensure quality. To create a peer review job, follow the guidelines in our Guide to: Running a Shapes Peer Review Job article. 

Monitoring and Reviewing Results

As this is a BETA feature, aggregation is not supported. Jobs should be run either to a trusted partner or in a peer review workflow (see above). 

Results

  • The Ontology Attributes feature will append metadata per shape annotation in job results.
  • Within the metadata object per shape, there are shapeAnswers and shapeQuestions arrays to represent the attributes labeled for an object and information on the ontology available to the contributor. 
  • Example shape annotationoutput from an ontology attributes job: 
    {
    "id":"2069ebb8-3e3e-4b57-b056-00eddd1a7c78",
    "class":"Car",
    "type":"box",
    "annotated_by":"human",//video output only
    "coordinates":{
    "x":298,
    "y":450,
    "w":410,
    "h":293
    },
    "metadata":{
    "annotated_by":"human",//video output only
    "shapeAnswers":[
    {
    "type":"Multiple Choice",
    "customUserId":"",
    "name":"occlusion",
    "answer":{
    "values":"yes",
    "customUserId":"122"
    }
    }
    ],
    "shapeQuestions":[
    {
    "id":"02906416-37be-42ca-97dd-40f81b9f88ef",
    "customUserId":"",
    "type":"Multiple Choice",
    "data":{
    "questionText":"What is the level of occlusion?",
    "questionChoices":[
    {
    "id":"a06aacd5-24bf-458f-9300-1af4666f5739",
    "customUserId":"122",
    "label":"Yes",
    "value":"yes"
    },
    {
    "id":"175bdde7-fc5b-4f6a-bfa5-788024a522d9",
    "customUserId":"133",
    "label":"No",
    "value":"no"
    }
    ],
    "isRequired":true,
    "tipsHints":"",
    "resultsHeader":"occlusion"
    }
    }
    ]
    }
    }

     

Each variable can be defined as the following:

Variable Description
id UUID assigned by the platform, can be applied to a shape annotation, question, or question response
class Ontology category assigned to the shape
type The type of shape used. Can be box, dot, line, or polygon. 
coordinates

The coordinates of the shape.

metadata An object containing information on the attribute answers and questions available in the job.

 

The metadata object includes the following:

Metadata variable Description
annotated_by Applicable only in video jobs. This can be “human” if a contributor manually saved inputs on the frame or “machine” if the answers were interpolated from the last frame where attributes were manually updated
shapeAnswers

Object within metadata object containing answer information for specific attributes. Variables include:

  • type:Question form used (I.e. multiple choice or checkbox group)
  • customUserId: ID provided in ontology manager file
  • name: Value inputted for resultsHeader in ontology manager file
  • answer: Object containing information on the answer to the attribute
    • value: Value of chosen answer
    • custID: ID that corresponds to the value from the ontology manager file
shapeQuestions

Object within metadata object containing information on the attributes questions and values. Additional variables not defined above include:

  • data: JSON object containing information on the question text and choices
    • questionText: Question label shown to contributors during annotation.
    • questionChoices: An array of the available choices to the question
  • isRequired: Boolean value that determines if a question is required or not
  • tipsHints: An optional hint provided to contributor during annotation
  • resultsHeader: Value inputted for resultsHeader in ontology manager file.
  •  

Was this article helpful?
1 out of 1 found this helpful


Have more questions? Submit a request
Powered by Zendesk