Guide to: Image Annotation Aggregation

The image annotation tool cml:shapes supports aggregation of shapes. Aggregation results reports can be found on a job's results page and can be previewed in tool on the data page for each individual unit. You can define aggregation settings on the job design page using cml, or the Aggregation Settings GUI on the job results page.

Fig 1. Aggregation Settings GUI

This article lays out the aggregation types for image annotation jobs.

Class aggregation

Class aggregation is done only on the shapes that can be clustered together through image aggregation (outlined below). Classes are grouped together and returned in the class dictionary for each aggregated shape.

• class-agg: all, agg, agg_x, cagg_x
  • all: returns all classes.
  • agg: returns the class with the highest confidence value.
  • agg_x: returns the X number of classes with the highest confidence value.
    • X should be a positive integer value.
  • cagg_x: returns all classes with confidence equal to or higher than X.
    • X should be a decimal with a leading 0.

Shape aggregation

Each shape type (box, ellipse, polygon, dot or line) supports aggregation through the different configuration parameters listed below. Only shapes of the same type are aggregated together.

• box-agg: number between 0.1 - 0.99, or all
  • Defines the minimum IoU (Intersection over Union) required for boxes to be clustered together. If all is selected, no clustering is done, and all boxes are returned.
• ellipse-agg: number between 0.1 - 0.99, or all
  
  • Defines the minimum IoU required for ellipses to be clustered together. If all is selected, no clustering is done, and all boxes are returned.
• polygon-agg: number between 0.1 - 0.99, or all
  • Defines the minimum IoU required for polygons to be clustered together. If all is selected, no clustering is done, and all boxes are returned.
• dot-agg: positive integer, or all
  • Defines the minimum pixel distance required for dots to be clustered together. Clustering is done using weighted arithmetic mean.

Σ (ω * x) / Σ ω
         ω - contributor trust
         x - coordinate

• • If all is selected, no clustering is done.
• line-agg: positive integer, or all
  • Defines the minimum pixel distance between lines to be clustered together. Distance is calculated using Fréchet Distance. If all is selected, no clustering is done, and all lines are returned.

Examples

Consider that a unit has received judgments from 3 different users. Each of these users have trust equal to 1. The judgments are as follows:

{"id":"1","class":"Car","number":1,"type":"box","coordinates":{"x":457,"y":240,"w":44,"h":43}}

{"id":"2","class":"Car","number":1,"type":"box","coordinates":{"x":456,"y":248,"w":48,"h":39}}

{"id":"3","class":"Car","number":1,"type":"box","coordinates":{"x":451,"y":248,"w":54,"h":38}}

If box-agg="all", the judgment results from all users are returned and no aggregation is done.

Fig 2. box-agg="all"

If the IoU is set very high, e.g. box-agg="0.99", no results are returned because the judgments are not close enough to each other for the defined IoU. Resulting aggregation: []

Fig 3. box-agg="0.99"

Reducing the IoU, to the lower end of 0.1, would result in the aggregation of all shapes in this scenario. Resulting aggregation: [{"coordinates": {"x": 454, "y": 254"w": 48, "h": 40}, "type": "box", "average_trust": 1.0, "iou": 0.79, "class": {"Car": 1.0}}]

Fig 4. box-agg="0.1"

While defining a low IoU value will most likely guarantee an aggregation result, you may end up with undesired shapes in your results. In this scenario, defining the IoU to 0.85 would still return aggregation while also excluding one of the shapes. Resulting aggregation: [{"coordinates": {"x": 453, "y": 248, "w": 51, "h": 38}, "type": "box", "average_trust": 0.67, "iou": 0.94, "class": {"Car": 1.0}}]

Fig 5. box-agg="0.85"

If a user desires to receive the shapes that did not cluster during aggregation, this can be done by selecting “Include single boxes” checkbox on the Aggregation Settings GUI on the results page. Resulting aggregation: [{"coordinates": {"x": 457, "y": 240, "w": 44, "h": 43}, "type": "box", "average_trust": 0.33, "class": {"Car": 1.0}}, {"coordinates": {"x": 453, "y": 248, "w": 51, "h": 38}, "type": "box", "average_trust": 0.67, "iou": 0.94, "class": {"Car": 1.0}}]

Fig 6.box-agg="0.85"(include single boxes)

FAQ

• Why am I getting no aggregation results?
  • Your aggregation settings have not been defined. For example, box-agg for boxes or dot-agg for dots. You can define aggregation settings on the job design page using cml, or the Aggregation Settings GUI on the job results page.
  • Your aggregation settings are too high for the judgments you have received. For example, box-agg="0.99" will require a very high similarity to return any results. Try reducing the IoU to a lower value.
  • Your dots and lines have large resolutions. Dots and lines use points distance and Fréchet Distance to define if shapes should be clustered together. Judgments may look close together but their pixel distance can be larger than expected. Try increasing the pixel distance.
  • No trusted judgments. Aggregations are trust weight based. If there are no trusted judgments there will be no aggregation results.
  • Single judgment or single shapes. Aggregation only happens when there are shapes be clustered together. If you want to see shapes that did not cluster during aggregation, you can go to the Aggregation Settings GUI on the job results page, and select the "Include single shapes" checkbox.
• Why have my aggregation results changed?
  • New judgments have been received. Whenever a new judgment is received, a new aggregation will be calculated.
  • Aggregation setting have been updated. Whenever a setting is changed, and a new report is generated, new aggregation results are created.