Follow

Guide To: Running a Job with Ontology Attributes

Overview

Our ontology attributes feature allows users to collect additional metadata on jobs using the image and video annotation tools.

Important Note: As this tool is currently in BETA, it must be enabled for your team for access; please contact your Customer Success Manager or Account Executive for more information.

Building a Job 

CML 

The ontology attributes feature can be used in any cml:shapes and cml:video_shapes type job. Please see our other articles detailing how to for run jobs for specific cml:shapes types:   

Ontology Manager 

Creating an Ontology Via the UI 

When enabled, ontology attributes can be accessed via the Ontology Manager page of a job and configured per class by clicking the pencil icon next to the desired class. 

blobid0.png

Fig. 1: Selecting a class to edit via the Ontology Manager

After selecting the pencil icon, a modal will appear with the ontology attributes section to configure questions to collect attribute information. The graphical editor follows the graphical editor used for creating questions in job designs (please see Guide to: The Graphical Editor).   

blobid1.png 

Fig. 2: Ontology Attributes section and question type options

 

blobid2.png

Fig. 3: Example question set-up with Ontology Attributes

 

Supported Questions

Single Checkbox

ontology1.png

Fig. 4: Example question set-up with Ontology Attributes

 

Checkbox Group and Multiple Choice

In Checkbox Group users can select all possible options, whereas in Multiple Choice there’s only one option that can be selected, both of them have a maximum of 5 options per question.

ontology2.png

Fig. 5: Example question set-up with Ontology Attributes

Pulldown Menu

Follow the same UI as Checkbox Group and Multiple Choice, but with a limit of 50 possible options per question.

 

Text Box (Single Line)

This question is a single line text box, where when creating the job, it’s possible to filter the output using the clean output field below, and choosing one of the available options

ontology3.png

 

Fig. 6: Example question set-up with Ontology Attributes

 

ontology4.png

Fig. 7: Example question set-up with Ontology Attributes

 

All questions contain this field, where is possible to:

  • Show tips/hints to annotators on the tool
  • Results Header that goes to the final report of the job
  • Clean Output*

* Clean output is only for Text Box (Single Line) type of question

 

Supported Questions – Annotator Side

ontology5.png

Fig. 8: Example of questions on Annotation Side

 

Creating an Ontology via JSON

An ontology can also be created by uploaded a JSON file. We recommend creating your first ontology via the graphical interface, then downloading the JSON to understand structure and continue building on. 

Here is an example JSON file that corresponds to the question created above in the graphical editor.

[ 
  { 
   "description": "", 
   "class_name": "Vehicle", 
   "questions": [], 
   "display_color": "#FF1744", 
   "children": [ 
    { 
     "description": "", 
     "class_name": "Car", 
     "questions": [ 
      { 
       "id": "bd5dda2e-2cbc-4526-8500-e1b0db143f8d", 
          "type": "Multiple Choice", 
          "data":  
        { 
          "questionText": "Is the car occluded?", 
          "questionChoices": 
        [ 
            { 
              "id": "2a3b6c0f-5597-4863-8e48-3b8751e69b49", 
              "label": "Yes", 
              "value": " yes" 
            }, 
            { 
              "id": "ee7227c5-4667-4468-afdb-31ccf7767133", 
              "label": "No", 
              "value": " no" 
            } 
        ], 
          "isRequired": true, 
          "tipsHints": " Occlusion means the visibility of the car is obstructed."
          "resultsHeader": "occluded" 
            } 
          } 
        ], 
        "display_color": "#651FFF" 
      } 
    ] 
  } 
]

For classes with ontology attributes, the following variables must be included unless stated otherwise: 

  • id: Platform generated id for the question 
  • Questions: This array hold the JSON objects that describe the attributes added to the class 
  • Type: The type of question used for the attribute (i.e. Multiple Choice, Checkbox Group, etc.) 
  • Data: a JSON object containing information on the attribute. The variables may change depending on the question type chosen. 
    • Standard variables include:
      • questionText: Accepts a string for the question label that appears to the contributor 
      • questionChoices: Accepts a string for the options to a question (i.e. for Multiple Choice and Checkbox Group) 
        • label: Accepts a string for the option label shown to the contributor 
        • value: Accepts a string for the option value stored in the output reports 
      • defaultsChecked: Accepts a boolean value for Checkbox questions to default a checkbox as marked 
      • isRequired: Accepts a boolean value for if a question is required 
      • tipsHints:  Accepts a string for a hint to give to the contributor 
      • resultsHeader: Accepts a string for the output name of the question provided in the output reports 
    • Additional Attributes for Single Text Line Inputs: 
      • cleanOutput: Accepts a boolean value to enable clean output validators 
      • textCase: If cleanOutput is set to true, this variable accepts “UPPER CASE”, “lower case”, “title case”, or “None”  
      • trimSpaces: If cleanOutput is set to true, this variable accepts “No whitespace”, “No leading/trailing whitespace”, “No multiple whitespaces”, and “None” 

Please see the Ontology Manger article for information on standard ontology formats. 

 

Ontology Attributes Interpolation in Video 

When ontology attributes are added to a class in a video annotation job, contributors can take advantage of answer “interpolation” to maximize efficiency. 

When a contributor saves attributes in a job, the attributes will “interpolate” and apply to all future frames, up to the last frame where attributes were manually adjusted.  

For example, say a contributor is labeling a 15 frame clip. The contributor adds a shape to frame 1 and saves attributes. The contributor then updates the attributes in frame 10. If the contributor goes back and updates the attributes in frame 7, those attributes will then “interpolate” and apply to frames 8 and 9 since 10 is the next frame where attributes were manually adjusted. This is useful when capturing temporal tags such as occlusion or truncation. 

 

Peer Review

In BETA, test questions are not supported, and peer review is a method to ensure quality. To create a peer review job, follow the guidelines in our Guide to: Running a Shapes Peer Review Job article. 

 

Monitoring and Reviewing Results

As this is a BETA feature, aggregation is not supported. Jobs should be run either to a trusted partner or in a peer review workflow (see above). 

  • Answers can be read while monitoring, in a read only state.

 

Results

  • The Ontology Attributes feature will append metadata per shape annotation in job results.
  • Within the metadata object per shape, there are shapeAnswers and shapeQuestions arrays to represent the attributes labeled for an object and information on the ontology available to the contributor. 
  • Example shape annotationoutput from an ontology attributes job: 
    {
    "id":"2069ebb8-3e3e-4b57-b056-00eddd1a7c78",
     "class":"Car",
       "type":"box",
       "annotated_by":"human",//video output only 
      "coordinates":{
       "x":298,
       "y":450,
       "w":410,
       "h":293
      },
       "metadata":{
       "annotated_by":"human",//video output only
       "shapeAnswers":[
        {
         "type":"Multiple Choice",
         "name":"occlusion",
         "answer":{
         "values":"yes",
         "questionId": "02906416-37be-42ca-97dd-40f81b9f88ef"
         }
       }
     ],
       "shapeQuestions":[
        {
         "id":"02906416-37be-42ca-97dd-40f81b9f88ef",
         "type":"Multiple Choice",
         "data":{
          "questionText":"What is the level of occlusion?",
          "questionChoices":[
           {
            "id":"a06aacd5-24bf-458f-9300-1af4666f5739",
            "label":"Yes",
            "value":"yes"
            },
            {
            "id":"175bdde7-fc5b-4f6a-bfa5-788024a522d9",
            "label":"No",
            "value":"no"
           }
         ],
         "isRequired":true,
         "tipsHints":"",
         "resultsHeader":"occlusion"
        }
       }
      ]
     }
    }

Each variable can be defined as the following:

Variable Description
id UUID assigned by the platform, can be applied to a shape annotation, question, or question response
class Ontology category assigned to the shape
type The type of shape used. Can be box, dot, line, or polygon. 
coordinates

The coordinates of the shape.

metadata An object containing information on the attribute answers and questions available in the job.

 

The metadata object includes the following:

Metadata variable Description
annotated_by Applicable only in video jobs. This can be “human” if a contributor manually saved inputs on the frame or “machine” if the answers were interpolated from the last frame where attributes were manually updated
shapeAnswers

Object within metadata object containing answer information for specific attributes. Variables include:

  • type:Question form used (I.e. multiple choice or checkbox group)
  • questionId: ID from the question associated with the answer
  • name: Value inputted for resultsHeader in ontology manager file
  • answer: Object containing information on the answer to the attribute
    • value: Value of chosen answer
shapeQuestions

Object within “metadata” object containing information on the attribute’s questions and values. Additional variables not defined above include:

  • data: JSON object containing information on the question text and choices
    • questionText: Question label shown to contributors during annotation.
    • questionChoices: An array of the available choices to the question
  • isRequired: Boolean value that determines if a question is required or not
  • tipsHints: An optional hint provided to contributor during annotation
  • resultsHeader: Value inputted for resultsHeader in ontology manager file.Example annotation output from a Video Annotation ontology attributes job:

 

Example annotation output from a Video Annotation ontology attributes job:

{

  "ableToAnnotate": true,

  "annotation": {

    "frames": {

      "1": {

        "frameRotation": 0,

        "rotatedBy": "machine",

        "shapesInstances": {

          "dce": {

            "annotated_by": "human",

            "height": 10,

            "width": 10,

            "x": 40,

            "y": 20,

            "metadata": {

              "annotated_by": "human",

              "shapeAnswers": [

                {

                  "type": "Checkbox",

                  "questionId": "d2a0a51c-a060-4b0a-996d-291bfb784754",

                  "name": "checkbox",

                  "answer": { "values": false }

                }

              ]

            }

          }

        }

      }

    },

    "shapes": { "dce": { "category": "Cat", "type": "box" } },

    "ontologyAttributes": {

      "Cat": {

        "questions": [

          {

            "data": {

              "defaultsChecked": false,

              "isRequired": false,

              "questionText": "Checkbox",

              "resultsHeader": "checkbox",

              "tipsHints": "Tip on hover"

            },

            "id": "d2a0a51c-a060-4b0a-996d-291bfb784754",

            "type": "Checkbox"

          }

        ]

      }

    }

  }

}

The Video Annotation Tool Output has small differences to the Image Annotation

Metadata variable

Description

ontologyAttributes

Applicable only in video jobs. This contains the questions that on Image Annotation are inside each shape. Each question is stored inside the Ontology Name.

Was this article helpful?
13 out of 14 found this helpful


Have more questions? Submit a request
Powered by Zendesk