Guide to: Annotate With AI


"Annotate With AI" enables you to pre-annotate tasks using generative AI, thereby expediting the process for your contributors. This initial version of the feature is currently supported by the cml:radios element. 

Contributors will see pre-annotated radio buttons, which they can choose to accept or edit. Leverage this functionality to enhance efficiency and accuracy in the annotation process.


Screenshot 2024-02-06 at 3.30.01 PM.png


To utilize "Annotate With AI" you will first need to have Quality Flow and LLM enabled for your team and have models configured. More information on model configuration can be found in this article.

Building a Job

Once you have LLM enabled and at least one model configured, you can set up your job using the graphical editor or the code editor.

The Graphical Editor

Step 1: Click on the "Manage Language Models" link, located in the top right corner of the design page.

Screenshot 2024-02-06 at 3.50.08 PM.png

Step 2: Select the model you will use in this job and then click Save.

Screenshot 2024-02-08 at 12.42.41 PM.png

Step 3: Create a multiple choice question

Screenshot 2024-02-06 at 3.34.44 PM.png

Step 4: Click on the "AI" icon in the multiple choice question preview, located in the "Content" section of the design page. (If you do not see this icon, you may not have LLM enabled, please contact or reach out to your CSM for assistance.)

Screenshot 2024-02-06 at 3.39.41 PM.png

Step 5: Choose the model from the drop down. (You can also add a new model here by selecting the "Add a model" option, see this article for details on adding models. Once configured, return here to select your newly added model, and remember to also enable it for the job, see Step 1).  

Screenshot 2024-02-06 at 3.44.58 PM.png


Screenshot 2024-02-06 at 3.45.08 PM.png

Step 6: Enable/disable "Indicate to contributors that AI was used in pre-annotation". If this checkbox is ticked, contributors will see the message "Pre-annotated with AI" along with the pre-annotation. Click Save.

Screenshot 2024-02-06 at 3.47.27 PM.png

Screenshot 2024-02-08 at 12.30.27 PM.png

Step 7: Click "Preview" to see your generative AI pre-annotation in action!

Screenshot 2024-02-06 at 3.54.49 PM.png

The Code Editor

After ensuring that you have a model configured and enabled for your job (see Step 1, of graphical editor instructions above, taking note of the model ID located next to the model name), you can add the cml attribute ai-annotation="[MODEL_ID]" within the cml:radios tag. This attribute takes an array where the first field is obligatory and must contain the model ID, as shown in the following example:

Screenshot 2024-02-12 at 2.53.39 PM.png

By default the entire job design is sent to the model, however you can choose to send specific information to the model by utilizing a second optional field in the ai-annotation attribute, that will reference html elements (and their children) in the cml by a class name or ID, as shown here:

MicrosoftTeams-image (12).png

In the above image the second field of the attribute array specifies "examples" that will be sent to the model. Note, there is an additional div, that does not have any id attribute. The contents of this div will not be sent to the model.

The syntax is as follows:

  • ai-annotation="[MODEL_ID, ['div1', 'div2', ...]]

To enable or disable the "Pre-annotated with AI" label, include the following attribute:

  • show-ai-annotation (defaults to "true")
    • set show-ai-annotation="false" if you do not want the label to show to contributors


In quality flow, the output will be displayed in the judgmentRawJson column, as well as a separate field for the original model configuration.

Example of a cell in the judgmentRawJson column:


In the above example, the contributor agreed with the model pre-annotation and submitted the judgment without making any changes, hence the results in "what_is_the_current_capital_of_russia" and "what_is_the_current_capital_of_russia_model_annotation" are the same.

Was this article helpful?
6 out of 6 found this helpful

Have more questions? Submit a request
Powered by Zendesk