--- language: "en" --- # Appen Success Center * ## [Getting Started](https://success.appen.com/appen-success-center/getting-started.md) * ### [Create \& Design Jobs](https://success.appen.com/appen-success-center/create-design-jobs.md) * ### [Settings](https://success.appen.com/appen-success-center/settings.md) * ### [Data Management](https://success.appen.com/appen-success-center/data-management.md) * ### [LLM Data Products](https://success.appen.com/appen-success-center/llm-data-products.md) * ### [Enhancements](https://success.appen.com/appen-success-center/enhancements.md) * ### [Dashboards \& Reports](https://success.appen.com/appen-success-center/dashboards-reports.md) * ### [Accounts \& Admin](https://success.appen.com/appen-success-center/accounts-admin.md) * ### [Product Release Notes](https://success.appen.com/appen-success-center/product-release-notes.md) --- language: "en" --- # Accounts & Admin --- language: "en" --- # Account Management --- language: "en" --- # Team Management ## Overview The Appen Platform allows team members to collaborate on jobs between multiple accounts. The following article describes all of the team functionality in the Appen Platform. ![image-20260623-071309.png](https://success.appen.com/__attachments/a_3bd68e92a894bd3bb6db70f18da068c68dea29ba5644df2adf1c2bc85155f1bd/image-20260623-071309.png?cb=16a68316a1b7039d650cbb72105e78d8) Fig. 1: Team Management Page ## Invite Team Members ![image-20260623-071318.png](https://success.appen.com/__attachments/a_87ab1a78d5427f2641644bc33eda734f2510f993e612e9d2b9c174550f19e3b0/image-20260623-071318.png?cb=1de41c814b9cd7b1d369925409d59a32) Fig. 2: Add Team Member Team Admins may invite as many members to their team as they wish. To do so, enter the email address of the team member you would like to add to your team. Appen will send an email containing a unique token to that user and instructions on how to join. Invitees may be new or existing users. ## Edit and Remove Team Members ![image-20260623-071327.png](https://success.appen.com/__attachments/a_7ad89bdf85fabc0ed5870685faccb882cd5865102aad6f2f2ba51a5437f9b321/image-20260623-071327.png?cb=8bbfb13e1390566ac0d8367f52c14d09) Fig. 3: Remove Team Member Team Admins may resend pending invitations, set permissions, and remove members at any time. --- language: "en" --- # User Roles & Management ## Overview Appen account structure is comprised of members who belong to various Appen groups, named teams and/or organizations. Different types of members within a team or organization will have different permissions and abilities on the platform. There are four different types of permissions that members are able to have on the platform (defined below): * Read Only Member * Standard Member * Team Admin * Organization Admin ## Read-Only Member Read-only members do not have any edit permissions associated with their role. These users can not create or edit Jobs or Projects. They can: * View team members' jobs * View jobs dashboards and settings * Download results ## Standard Member Standard members have limited edit permissions. Their functionality includes: * Create and launch jobs * View and edit all team jobs * Add funds to the team ## Team Admin [Team admins](https://success.appen.com/appen-success-center/accounts-admin/account-management/team-management.md) can do everything a standard member can do as well as: * Add new members to their team and edit their roles (see below) * View standard job cost reports (and Enterprise Analytics, if enabled for the team) Note: For [Dedicated](https://success.appen.com/hc/en-us/articles/360044313511-Introduction-to-Dedicated) customers, the Enterprise Analytics feature is currently not available On-Premises and only available via the cloud multi-tenant Appen Data Annotation Platform. ## Organization Admin [Org admins](https://success.appen.com/appen-success-center/accounts-admin/organization-management/organization-admin.md) have the highest level of access within the organization (i.e., customers that need visibility into several teams) Their functionality includes those belonging to Team Admin as well as: * Invite users to any team in the organization * Enterprise Analytics views of each team in the entire org (if enabled) ## Assigning Roles in a Team Once a user has accepted an invitation to join the team, a Team or Org Admin can assign their roles by clicking on Edit Role, as in the screenshot below. ![edit_image.png](https://success.appen.com/__attachments/a_32a0fe43d8761b32b26bb600c0a70f53aaeb12f9005169abebc3a2c613de656f/3ad650ed40139f19_image-20240104-213144.png?cb=e073ae4c8a8c7950c1e4b15cb990b8d4) A modal will appear, allowing the role to be chosen for the user: ![role_image.png](https://success.appen.com/__attachments/a_3f037368afbc474918c547f5b7723ef830965785406c22851d0ca06c91d5b4aa/ec5e0ef98fb64a40_image-20240104-213222.png?cb=13a80ecad287a4eaa785ad809b7c4fe9) --- language: "en" --- # Contributors Migrated from Zendesk section **Contributors**. Articles in this section are listed below. --- language: "en" --- # Contributor Support Greetings Contributors! Thanks for tasking with us here at Appen. Appen has two sides - a side to earn money and a side to build jobs. To get the support that you need, please visit [https://contributorsupport.appen.com](https://contributorsupport.appen.com/%20) or if you can not log in please email [contributors@appen.com](mailto:communitysupport@appen.com). Please understand that the Community Support team answers over 500 tickets per day and often receives more than 1000 each day. Your ticket will be answered in the order it was received. We appreciate your patience and apologize for any inconvenience you may be experiencing. Have a great day! --- language: "en" --- # How To Enable Third Party Cookies ## **Chrome** 1. Go to the Chrome menu, select **Settings** \> **Show advanced settings** \> **Privacy** \> **Content settings**. 2. Under the "Cookies" section, ensure **Allow local data to be set (recommended)** is selected. 3. Click **Done** and refresh the browser. ### **Mozilla Firefox** * **Windows** : Click the "Open menu" icon and go to **Options**. * **Mac** : Go to **Firefox** \> **Preferences**. 1. Select the **Privacy** tab. 2. Under "History," set **Firefox will** to **Use custom settings for history**. 3. Ensure **Accept cookies from sites** is checked, and set **Accept third-party cookies** to **Always**. 4. Click **OK**. ### **Safari (Mac)** 1. Open Safari and go to **Safari** \> **Preferences** \> **Privacy**. 2. Under "Cookies and website data," select **Always allow**. 3. Close the preferences window and refresh the browser. ### **Internet Explorer** 1. In Internet Explorer, click on the "Tools" button (upper right corner). 2. Select **Internet options** \> **Privacy** \> **Advanced**. 3. Set **First-party Cookies** and **Third-party Cookies** to **Accept** or **Prompt**. 4. Click **OK** to save changes. ### **Edge (Windows 10)** 1. Open Edge and click on the "More" (...) menu. 2. Go to **Settings** \> **View advanced settings**. 3. Scroll to the "Cookies" section and choose **Don't block cookies**. ### **Brave Browser** 1. Open **Brave** and click the three horizontal lines (menu) in the upper-right corner. 2. Select **Settings** from the dropdown menu. 3. In the left sidebar, click on **Privacy and security**. 4. Under **Cookies and other site data** , find the option that says **Block third-party cookies**. 5. Toggle off **Block third-party cookies** , or select **Allow all cookies** to enable both first-party and third-party cookies. --- language: "en" --- # Guide to: Regenerating API Keys ## Overview Appen allows requestors to regenerate their API key at any time via the Account page. This guide walks through how to regenerate API keys and some considerations to keep in mind when regenerating keys. ## How to Regenerate API Keys 1. Navigate to your Account page and select the "**API**" tab. ![image-20260626-063404.png](https://success.appen.com/__attachments/a_8704ec2b72df4c89f0afaccb8a9b464457ebd7161833f47f9ec71092249720b4/image-20260626-063404.png?cb=86a2a29b52729d9d661506f45a87e522) ![Screen_Shot_2022-08-04_at_9.03.40_AM.png](https://success.appen.com/__attachments/a_33d33cf9f0792c5aefa87785ebff6f16fd9e1e2fa89ab7a34a93a1dd20294a4b/687c870219b5f0ae_Screen_Shot_2022-08-04_at_9.03.40_AM.png?cb=0c4542e10ce63609a57cae7a57ab8371) 2. Select the "Generate New" button. ![API.png](https://success.appen.com/__attachments/a_daed8c25c12f9058fe3ffde124f8a2550ad2cd086c30addb99a164eccf02ccbe/481d55f5d5fdc87c_API.png?cb=5554d1766377fc399592a4ff6ab42bbb) 3. Copy the newly generated API Key. Once you close the page, you will no longer be able to retrieve the API key and will need to re-generate another API Key. ## Impact of Regeneration and Considerations * **API Integrations** * Many customers create custom scripts to automate their data pipelines. * After regenerating an API key, be sure to update all scripts with the new key to ensure commands run successfully. * **Webhooks that use API key for webhook signature** ***(webhooks created before June 27, 2022)*** * A job owner's API key is related to the webhook signature provided by our platform. * If the job owner's API key is regenerated and webhook signature verification is in place, please update the signature verification process with the regenerated API key of the job owner. **Note:** API keys from team members can be used to post data to a job, however, only the job owner's API key can be used to verify webhook signatures. * **Cross Team Access** * [Cross Team Access](https://success.appen.com/appen-success-center/accounts-admin/organization-management/cross-team-access.md) is an organization-level feature that allows a user to utilize their API key across multiple teams. * To verify if the setting is enabled in your organization, please see the GIF below on how to toggle between teams: ![66d1399e-4e28-4e68-bb82-b1f676ff944b.gif](https://success.appen.com/__attachments/a_a6fb8cf6cf0b456d6346c1d00a899ba1b694b4c457c6bd20cda2b758b4ce1080/6bc727b22434f663_66d1399e-4e28-4e68-bb82-b1f676ff944b.gif?cb=52aadab7dc272c2bdbd55773449bdd29) Figure 1: Users with Cross Team Access enabled will have options to toggle between teams via the letter icon under Global Navigation. * Users with Cross Team Access enabled should take note of their API Team ID. After regenerating API keys, be sure to ensure the Team ID associated with API commands are executed within the correct team. Instructions for setting an API team can be found under ["API Updates"](https://success.appen.com/appen-success-center/accounts-admin/organization-management/cross-team-access.md) of our Cross Team Access article. **Note:** If an API Team is not set the platform will default to the current team selected in the UI. *** ** * ** *** --- language: "en" --- # Organization Management --- language: "en" --- # Cross Team Access ## Overview Appen's Cross-Team Access feature allows users to belong to multiple teams within an organization and across multiple organizations using just one login. This article outlines how to get started and what actions are available to whom. ### The Benefit Many users work among different types of teams. Sometimes these teams are based around objectives, sometimes they're different departments. With Cross-Team Access, Appen users can easily switch context to another team within the platform without having to create multiple accounts. ## What's Available to Users Different types of users within a team or organization will have different permissions. Below are outlined the different roles and what they can do. Users may have different roles (read-only, standard, or team admin) in each of the teams they belong to. In these cases, permissions will be dependent upon which team they are currently acting in (and the role they have in that team). ### Read-Only Member Read-only members do not have any edit permissions associated with their role. These users can not create or edit Jobs or Projects. The users can: * View team members' jobs * View jobs dashboards and settings * Download results ### Standard Member Standard users have edit permissions associated with their role. These users can do standard actions such as: * Create and launch jobs * Add funds to the team * View team members' jobs ### Team Admin [Team admins](https://success.appen.com/appen-success-center/accounts-admin/account-management/team-management.md) can do everything a standard user can do, plus the following: * Invite members to the team * View the standard job cost report * If enabled, view Enterprise Analytics for their team ### Org Admin [Org admins](https://success.appen.com/appen-success-center/accounts-admin/organization-management/organization-admin.md) have the highest level of access within the organization. Their role provides the following: * Team admin-level access to every team in the organization * If enabled, Enterprise Analytics views of each team in the entire org * The ability to consolidate team member accounts ## Accessing Multiple Teams ### Switching your platform team In order to change your current team, select from the teams you belong to from an icon in the global navigation. ![image-20260623-070706.png](https://success.appen.com/__attachments/a_ee9b732ed66388f42b66eac36e5beeab4f0bcafc088d9a70bb07b85db6c9f6d7/image-20260623-070706.png?cb=26d7343520fe31643760a49bf19a51cc) Fig. 1: Switching teams in Global Navigation Your current team can be changed while on most pages in the platform with the exception of pages within jobs (e.g., Data, Design, Settings, etc). When inside a job, the option to change your current team is disabled. Once a team is selected, all actions taken in the platform will be within the context of that team. ## Joining Multiple Teams Org or team admins can add existing users from their org to other teams. How it works: 1. If you are an org or team admin, select the team you would like to add users to by choosing the team from the global navigation (Account \> Teams). 2. Navigate to the Team page within the Account view. 3. From there, invite new users to the team or add existing users from within the organization to this team. ![image-20260623-070736.png](https://success.appen.com/__attachments/a_2cd64a57447f52b313dc77cd0f78e722231323cc7c259491d226a10d66225762/image-20260623-070736.png?cb=7d414ff4dc0d95030265739447450185) Fig. 2: How org/team admins can invite new users to a specific team ## Assigning Roles in a Team Once the user has accepted the invitation to join the team, you can assign their roles by clicking on Edit Role, as in the screenshot below. ![edit_image.png](https://success.appen.com/__attachments/a_662d07486496c0844cf97d090417038e52343a556ba26c20ba9d0e200d2108c9/3ad650ed40139f19_image-20240104-213144.png?cb=e073ae4c8a8c7950c1e4b15cb990b8d4) Fig. 3: Editing a Role A modal will appear, allowing you to choose the role: ![role_image.png](https://success.appen.com/__attachments/a_cfe46c46b4fef90262bf220d8ba7fdc22bfeaf3100d64272c06fb4bd52ec44c2/ec5e0ef98fb64a40_image-20240104-213222.png?cb=13a80ecad287a4eaa785ad809b7c4fe9) Fig. 4: Selecting a Role ## Copying a Job to Another Team Users can copy a job from one team to another team they belong to via the API or the UI. **Important note:** If the team the job belongs to is NDA-only, you will not be able to copy any data with the job (the option to copy a job with no rows will be available only). ### Copy via the API To copy a job to another team using the API, please see the "API updates" section below. ### Copy via the UI There are two places to copy a job to another team via the UI: from the jobs page and from within an individual job itself. The process for each is the same: 1. On the jobs page, click the gear icon to the right of the job. From within a job, click the copy icon in the top right. 2. Click 'Copy Job'. 3. A modal will open; select which team to copy into, whether it's the current team the job belongs to or a different team you belong to. 4. Select whether to copy with or without data. 5. Click 'Copy'. * If the job was copied into the current team, the page will redirect to the Data Page of the new job copy. * If the job was copied into a different team, you will need to switch your current team in the global navigation to view the job. ![image-20260623-070818.png](https://success.appen.com/__attachments/a_a80ef5ddee342e862eb4dc8ebb6d576598947d6f7947956aa16efd4c3019cd1b/image-20260623-070818.png?cb=5a7904e27fce72d1443e9fc13b696e4f) Fig. 5: Copying a job to another team via the UI ## Consolidating Accounts An Org admin will be able to assist if a user has multiple accounts that should be consolidated. In order to do so, the org admin can follow the steps outlined below. Note: You can view who your Org admins are by going to Account page \> Team in the global navigation. This page will display the Role type for each user. 1. Navigate to the 'Members' page within the Account view (the current team selected does not matter). 2. Find the account that you want to be your **main account** (this will be the only account after consolidation), i.e., the account you will consolidate others **into**. * They can use the search bar or paginate through the list. 3. Hover over the account row in the table and click the Consolidate icon. 4. A modal will appear. Type the emails of the accounts into the search field and select accounts that you want to be consolidated with the main account. * Multiple accounts can be selected at a time. 5. Confirm the consolidation. ***Account consolidation cannot be undone*** **.** ![image-20260623-070849.png](https://success.appen.com/__attachments/a_fb41fdabf5fe8bb9e4a8ceb7d9351ee0c30a8beb42ca0b36bfb180442418fce8/image-20260623-070849.png?cb=2333b08d545455457c46513e0eb20a74) Fig. 6: How org admins can consolidate user accounts Once accounts are consolidated, the following will occur: 1. The main account will inherit team membership from each account, along with the specified role (read-only, standard, or team admin) each account had on their respective teams. 2. The jobs from each account will remain associated with and accessible to the teams they originally belonged to, but be found within the context of the main account. 3. The original accounts will still exist but will not have any jobs attached to them. --- language: "en" --- # Guide To Enterprise Analytics Enterprise Analytics provides in-platform reporting on the organization's usage and spending. By enabling the Organization and Team Admins to view in-depth analytics, this empowers them to make data-driven decisions about allocation, resourcing and ROI. If you already have access to Enterprise Analytics you can access it in the following ways: 1. Select this option on the menu bar on the left-hand side of the screen on ADAP: ![image-20260623-071016.png](https://success.appen.com/__attachments/a_1cf15f65a4a37294e44ce267bc6581c6b8d9fa50b9ad5ed02eb9db352fd4696e/image-20260623-071016.png?cb=bf2ac4f2d0caa7bf2cda3a7c5daf1852) 2. Go to this link directly by pasting it on the URL bar of the browser: ***If you'd like to request access to this feature, contact your Customer Success Manager or Account Executive.*** Enterprise Analytics consists of four different pages * Overview * User spend report * Job insights * Contributor Stats Each of these pages presents a different set of information about the team or organization, with a date picker that can customize the timeframe to retrieve information. Note: the Contributor Stats page is not date-specific, as it contains information for a single job). ## Overview The Overview page covers the following data points: * Contributor Spend * Activity Level * Rows Finalized * Judgments Collected ### Contributor Spend The contributor spends section shows the total amount an Organization or Team has paid contributors for working on its jobs over the given timeframe along with a line chart to show historical trends. * Organization Admins have the ability to break this down by teams within their organization, or by projects the jobs are associated with. * Team admin will only see this information for their own team, so the amount automatically broken down by the project. In addition to the breakdown, there is also a line chart to show historical trends. ![Screen_Shot_2019-02-21_at_10.30.13_AM.png](https://success.appen.com/__attachments/a_1b04d65f5ab7e52c136ca15de952b45ec8cafc932a6d2d4f800abeae749ae5da/5bc39cec4d2aec2b_Screen_Shot_2019-02-21_at_10.30.13_AM.png?cb=c11f74001dfe909753612f2e299e8811) ### Activity Level - Active Jobs * The Rows Finalized section shows how many rows were finalized in the given timeframe * Note: A row is finalized when it has collected the requested number of trusted judgments ### Activity Level - Judgments collected * The Judgments collected section shows how many judgments were submitted by contributors across all jobs and does not include test question judgments. ![Screen_Shot_2019-02-21_at_10.41.24_AM.png](https://success.appen.com/__attachments/a_bb6632e619be056746c31c2cc94de8243bcd37762d4f6fff313cb6ad663bcacd/d468925240031194_Screen_Shot_2019-02-21_at_10.41.24_AM.png?cb=249185d86e73633099d27758d8364bee) ### User Spend Report * The User Spend Report covers how much team members have paid contributors over the given timeframe, along with their total lifetime spend * Note: If a user has not spent any money on jobs over the given timeframe, they won't appear in the report even if they have a lifetime spend amount. ![Screen_Shot_2019-02-21_at_10.43.44_AM.png](https://success.appen.com/__attachments/a_6ab6f3fddde66677a98e0e68c5c44f6de70ff69380a0e84cda99ede658f229ec/1350d4cc5ca92a74_Screen_Shot_2019-02-21_at_10.43.44_AM.png?cb=ca1cc237a6b77dbb76a6e16daefce1e3) Fig 3. User Spend Report page ### Job Insights The Job Insights page provides a detailed breakdown of each job run over the chosen timeframe. It contains the following information per job: * **Title:** this is the title of your job * **Job Owner:**this is the email of the user who owns the job * **Cost Over Period**: this is the amount paid to contributors over the given timeframe * **Status:**Current state of the job * **Team**: if you are a Team Admin, this will be the same for all jobs * **Project:**the project in which the job is attached to * **Tags:**A list of tags that were added to the job * **Total Cost**: this is the total amount paid to contributors for the job over all time * **First Order:**the date of the first order placed on the job (when the first set of rows was launched) * **Last Order:**the date of the last order placed on the job. In jobs that have only one order, this will be the same as first order. * **Cost Per Row**: this is the actual amount paid per finalized row, including costs for tainted judgments and test questions * **Total Rows**: the number of rows uploaded to the job * **Launched Over Period** and **Total Launched Rows**: the number of rows ordered over the given timeframe and the total number of rows ordered in the job for all time, respectively * **Finalized Rows Over Period** and **Total Finalized Rows**: the number of rows finalized over the given timeframe, then the total number of rows finalized for all time, respectively * **Total Judgments Over Period**: the total number of judgments over the given timeframe, including untrusted judgments, but not including test question judgments * **Trusted Judgments Over Period**: the number of trusted judgments over the given timeframe, not including test question judgments * **Untrusted Judgments Over Period**: the number of untrusted judgments over the given timeframe, not including test question judgments * **Total Judgments** , **Total Trusted Judgments** , and **Total Untrusted Judgments**: the total number of each type of judgment for the job for all time ![Screen_Shot_2019-02-21_at_10.46.12_AM.png](https://success.appen.com/__attachments/a_459d590cafcd6c03db0000f29c2f78875e7ba43bfd740a865271578634468e30/e22cb5c05108acaf_Screen_Shot_2019-02-21_at_10.46.12_AM.png?cb=44ef7df82d00b58d1e2a9255aa8ac697) Fig 4. Job Insight page ### Contributor Stats The Contributor Stats page on the job dashboard provides a detailed view of how contributors are performing in the job and the work they're doing. The information provided is as follows: * Trusted Judgments: The total number of trusted judgments on non-test question rows * Average Trust Score: The average test question accuracy of contributors in the job * Total Hours Active: The cumulative amount of time contributors have spent working on the job. This is not just the amount of time the job has been running. * Seconds per Trusted Judgment: This is the average time in seconds it takes contributors to judge a row of data. * Contributor ID and Channel: The contributor's account ID and the channel they're working through * Earning: How much money is in USD the contributor has earned working in the job * Total Judgments: The number of judgments the contributor has submitted, excluding test questions * Hours Active: The amount of time the contributor has spent working in the job * Time per Judgment: the average time the contributor has taken to judge a row (in hours) * TQ Accuracy: The contributor's test question accuracy in the job * TQs seen: The number of test questions the contributor has judged * Missed TQs: The number of test questions the contributor has answered incorrectly * Forgiven Count: The number of missed test questions the ![Screen_Shot_2019-02-21_at_11.02.27_AM.png](https://success.appen.com/__attachments/a_82939871fdac3efabf0fc1b0fca425e2c23e6f414514c0cd58f28d057331b052/aa0cdbf995c80766_Screen_Shot_2019-02-21_at_11.02.27_AM.png?cb=edc8114f7484af90093285d3d9bbe53e) Fig 5. Contributor Stats Page *** ** * ** *** --- language: "en" --- # Organization Admin ## Overview Organization Admin provides the ability to manage multiple teams within a single organization. An organization admin can check the status and progress of jobs across all associated teams at any time. This feature allows the user to view all teams with a single login and can monitor funds available in each team and add funds when required. This feature is only available to our Enterprise Customers. Please contact your Customer Success Manager for access to this feature. ## Organization Admin View ### Team Jobs Page ![Screen_Shot_2020-06-12_at_9.22.15_AM.png](https://success.appen.com/__attachments/a_5594dab0679b1811018937fcc2d0415fd36169273aab4b864b6e5edbac9fed6b/9a5f6f38f863611e_Screen_Shot_2020-06-12_at_9.22.15_AM.png?cb=b0f482b9218f5f6f6e5b929fa3024108) Fig. 1: Team jobs page view with Organization Admin enabled For each job in the organization, the admin can: * View all pages related to a job - Data, Design, Quality, Launch, Monitor, and Results. * Add data, update the job design, and create test questions. * View launch settings and report settings but cannot edit. **Note**: The Organization Admin cannot launch jobs that belong to other teams. ### Account Page After selecting a team to view, the Organization Admin can view each of the following tabs of the Account Page accordingly * **Team** * View a list of the selected team's members or pending invitations * Invite users into the selected team. ![Screen_Shot_2020-06-12_at_9.33.18_AM.png](https://success.appen.com/__attachments/a_b67817d700ff33928e1185a4f9c8f1c2135532bb0162acc8449ed7d06da37997/4fff654a2e6a94f1_Screen_Shot_2020-06-12_at_9.33.18_AM.png?cb=99589ca66980520fca8b93f369b53dc7) Figure 2: Account Page -\> Teams * **Funds** * Add funds to any team * View all team's Available Balances and Funds in Progress * View all row and subscription details are specific to the team selected * View purchase history for the team, which includes payments made by any user in the organization ![Screen_Shot_2020-06-12_at_9.33.09_AM.png](https://success.appen.com/__attachments/a_d2a79ee6b41a0132205f139ebfbe62fdd1919bf33876bf5e19bb0325a1118da2/2cecc5fa3819bd88_Screen_Shot_2020-06-12_at_9.33.09_AM.png?cb=3e3b9315adada3aaede9408418b3fddb) Figure 3: Account Page -\> Funds * **Jobs** * Access a list of all the jobs that are currently in progress for each team * Access [job cost report](https://success.appen.com/appen-success-center/accounts-admin/organization-management/organization-admin.md#Job-Cost-Report-Guide)for any team in the organization ![Screen_Shot_2020-06-12_at_9.33.18_AM.png](https://success.appen.com/__attachments/a_bfaecbc6eae3db316848d26c167fe7093f68c01b1f68b4cb2478ad79cc20587b/03bac0e57d6f58f2_Screen_Shot_2020-06-12_at_9.33.18_AM.png?cb=99589ca66980520fca8b93f369b53dc7) Figure 4: Account Page -\> Jobs **Data \& Security (if enabled)** * Add new [Secure Data Access](https://success.appen.com/appen-success-center/getting-started/adding-hosting-data/appen-secure-data-access-aws-integration.md) integrations to a selected team * View any Secure Data Access integrations already set up for a team **Note**: Organization Admins will only be able to view this tab if enabled. To get Secure Data Access enabled for your organization, please reach out to your Customer Success Manager. Note: The Data \& Security tab is not needed for [Dedicated](https://success.appen.com/hc/en-us/articles/360044313511-Introduction-to-Dedicated) customers to use the Secure Data Access feature. For more information, please refer to [this](https://success.appen.com/hc/en-us/articles/360054988131) article. ![Screen_Shot_2020-06-12_at_9.33.38_AM.png](https://success.appen.com/__attachments/a_5fcddc1b4985a8720f24b5cec9b388434fe18bc4917f447ef9462ba21d83f3de/37d76b2ccca89e72_Screen_Shot_2020-06-12_at_9.33.38_AM.png?cb=57148541bbec01ff6a218e7d4201346a) Figure 5: Account Page -\> Data \& Security ## Job Cost Report Guide Job Cost Reports can be generated anywhere from the first job launch to the most recent in a single account to help provide accurate information on the total amount spent and the complete list of jobs ran in over a given period. The Job Cost Report page can be accessed by clicking the 'Jobs' tab in the account details page. ![image-20260626-065134.png](https://success.appen.com/__attachments/a_a8ea26a9be9561d5c258200fa9047fdcd01de7d969b68dfc95b8786aec37e829/image-20260626-065134.png?cb=630c04cd2d7f7ca4c2eb49254038f258) Fig. 1: Team Account Page ### **Job Cost Report Page Overview** ![image-20260626-065153.png](https://success.appen.com/__attachments/a_48bdd7da651ba819ffe587e55417cad21788e42a163fdb57d3a45fa454234224/image-20260626-065153.png?cb=3538c76fbde3a12f6ab9d9d5340f01d1) Fig. 2: Job Cost Report Page As seen above, users may take the following actions: 1. Filter jobs for a given period of time (up to 12 months) 2. Filter jobs based on the project tag 3. Download a PDF of jobs over the given period 4. Download a CSV of jobs over the given period Note: The job cost report will be updated every 24 hours ### **Filter Across Months** ![image-20260626-065215.png](https://success.appen.com/__attachments/a_52ca3553d1ace209735ec3175d0c8488d106cbba84ea5d9555787a492ecbe5e0/image-20260626-065215.png?cb=8fc1dffb845816da11e0a1e3252184f1) Fig. 3: Job Cost Report Filtered Across Multiple Months When filtering the Job Cost Report across multiple months, it displays the total amount spent in that given period and the amount spent in each individual month. ### **Job Cost Report** The report lists each job as a separate row, along with other pertinent information such as job cost, units launched, judgments collected and when the job is launched. ![image-20260626-065234.png](https://success.appen.com/__attachments/a_b8d48ab3fe2bd425233f178c8dd6852c8e5628bc0236b7586a5402278d57ebcd/image-20260626-065234.png?cb=28f64af6a19c8293f5ab9183ff88117a) Fig. 4: Example CSV Job Cost Report * **Month** - Returns the month and year the job is launched * **Job ID** - Returns the Job ID of the job * **Title** - Returns the title of the job * **Project** - Returns the project tag attached to the job * **Labels** - Returns the tag attached to the job * **Units** - Returns the number of rows total * **Judgments - Returns trusted, untrusted, and test question judgments the job has accumulated** * **Cost** - Returns the cost accrued in the job * **State -** This will reflect the state of the job. The job state can be in "finished", "canceled", "paused" or "archived" --- language: "en" --- # Single Sign On ## Introduction Appen Single Sign-On (SSO) feature lets users access to Appen using one login. Customers who choose to integrate via SSO can validate usernames and passwords against their corporate user database rather than Appen managing separate passwords for each user. Federated authentication using Security Assertion Markup Language (SAML) allows you to send authentication and authorization data between Appen and your corporate network. ***To enable Single Sign-On for your team,*** don't hesitate to get in touch with your Customer Success Manager***.*** ## Benefits of Single Sign-On 1. Users have to memorize fewer passwords, thereby increasing usage and time savings. 2. All established password policies for your corporate network are in effect increasing security for users who have access to sensitive data. ## Guide to Set-up SSO Integration ### Step 1a (optional): Provide SSO Configuration Details in a QA/Test Appen Organization 1. Contact your Customer Success Manager or Account Executive to get access to set-up a QA SSO integration in a separate, testing Appen organization. * This is highly recommended to ensure the integration is successful before setting up in your Production Appen organization. 2. Let your Customer Success Manager or Account Executive know which accounts you'd like to be moved into the QA org for testing. * We recommend moving at least two users on the QA team. One user to become an Org Admin to set up the integration and one user to test SSO log in after SSO is set-up. * This move is temporary and users can be moved back to the Production organization after QA is complete. * Aside from moving existing users, we can also use test accounts if customers have a testing IDP environment of their own. Customers will need to "Sign Up" with these new accounts and provide the emails to their Appen representative to move to the test SSO org. 3. Once the QA org is created and users are moved, Appen will enable the self-service Single Sign On setting. 4. The user who is setting up the integration should now see the SSO tab under their ["Account" page](https://client.appen.com/account/profile) (also shown in Figure 1 below). 5. Provide your IdP XML metadata (per instructions below - Step 1b, 2) and add assertion details in your IDP (per instructions below - Step 2). 6. Once all metadata is successfully saved on the platform and IdP, the test user account should attempt SSO login. * SSO login can be accessed via the IdP provider if the app is added to the user's workspace * Otherwise, a user can input their email to client.appen.com with no password and the platform will redirect to the integrated IdP for sign-in. 7. If the test user can successfully sign-on, the same set-up process can be applied to the Production SSO Organization, following the steps directly below. ### Step 1b: Provide SSO Configuration Details in Production Appen Organization 1. Contact your Customer Success Manager or Account Executive to get access to the capability and specify who on the organization should have Org Admin permissions to set-up SSO. 2. In Appen, the Org Admin should navigate to their Account Page --\> SSO tab. * If you cannot find the SSO tab, please reach out to your Appen Customer Success Manager or Platform Support team. Note: Only your Organization Admin(s) have access to set up the integration. The SSO tab will not be visible to standard users. ![image-20260616-061257.png](https://success.appen.com/__attachments/a_ab9bd2acc1eb5b07c44f3791a588426795eff88a29561bbc140b67c9f68a1838/image-20260616-061257.png?cb=163a27051e4474021b9ec77d8992b59a) Figure 1: Appen Account Page 1. Provide your IdP XML metadata. You have 2 options to enter the metadata * Provide a URL with IdP metadata (ex: OR copy and paste the XML metadata in the textbox. * See the example below. Note: Replace `${certificate}` with client certificate ). * Important: HTTPs header is required * Redirect Error URL: Provide a URL the user should be redirected to when an authentication/authorization error occurs. * Redirect logout URL: Provide a URL the should be redirected to when the user is logged out of Appen. 2. Select the Org-level SSO mode. * **Org-level (default):** Only users who are in the Appenorganization associated with the customer will be able to use SSO. Other users from your company using their corporate emails to login to Appen continue to use Appen credentials, but they will not have access to the organization data or jobs. * Example: If [Dan@company.com](mailto:Dan@company.com) is part of the Appen Org named "Company", he will use SSO to access the platform. However, if [tom@company.com](mailto:tom@company.com) is not part of the Appen Org named "Company", he will not be able to use the SSO but can log in using Appen credentials. 3. Go ahead and 'Save' the settings. 4. After the set-up is completed successfully, you will receive a SAML assertion template to configure in your IdP. ![image-20260616-061403.png](https://success.appen.com/__attachments/a_3876a69b6638c0991e55424d2b59a1bb17a5a2715d2c7b78ce22ccc0bfd23d49/image-20260616-061403.png?cb=25e30256322c80eda958001316045220) Figure 3: SSO Settings - SAML Assertion Template 5. Copy the SAML assertion. 6. You can go back and edit your SSO set-up by clicking the edit button present in the "SSO Settings" tab. ### Step 2: Set-up SAML Assertion Details in IdP Appen requires the SAML assertion to follow this template provided to you after you complete SSO set-up. 1. Variables starting with $ are user-specific 2. Variables starting with # are Identity Provider (IdP) / customer-specific #{entityId} ${emailAddress} element, we will actually accept a Name value like `Name="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"`. **Once the necessary IdP configuration is done from your side, the users should be able to login to Appen via SSO.** ### Adding Requesters to SSO-Enabled Organizations * SSO must be configured on an organization before inviting new requesters. If SSO is enabled on an organization with pre-existing requesters, those requesters will be forced to log-in via SSO after the feature is enabled. ### Inviting New Requesters to SSO-Enabled Organizations * The Org Admin or Team Admin will need to invite the requester email to the desired team within an SSO-enabled organization * This can be done via the Account \> Team page * Once the invitation is sent, the requester should "Sign Up" with the invited email After signing-up, the requester will be added to the designated team and organization All subsequent logins for the requester can now be authenticated via SSO ### Additional Instructions: * Supported IdPs: Our SSO integration supports SAML and all SAML based SSO Identity Providers. * SSO login is supported for the following scenarios: * When the user logs in as a job requestor and is accessing any page on [https://client.appen.com](https://client.appen.com/) * When the user has a job requestor account and is trying to access an internal work link to work as an internal contributor. * Once a new user is invited to an SSO-enabled organization, there is no need to enter anything in the password field of that Appen login page. The invite user's email will be automatically detected and then redirected to the IDP. Alternatively, there is no need to start a new Appen session on the Appen [login page](https://client.appen.com/sessions/new) and the session can start directly from the IdP site. ![image-20260616-061438.png](https://success.appen.com/__attachments/a_a8b58b9057d15adeb08f56becae037504601712db6f34c0514c132c7810507ce/image-20260616-061438.png?cb=1ab33d6df848265b66cd2098482d3f2c) Figure 4: How to enable the Internal Channel option ## IDP Application Set-Up When setting up SSO integration, our platform will ask for either a URL to your IDP metadata or the plain text snippet of your metadata and certificate. To generate this information, proceed with adding a new application to your IDP with the following inputs: * **Single Sign On / Endpoint URL:** * \ * The organization name can be your QA or Production organization, see Step 1a and 1b below for more details. * Note: If you are using Azure Active Directory as your IDP, please use [https://make.figure-eight.com/saml/consume?customer_name=](https://client.appen.com/saml/consume?customer_name=)\ as the ReplyURL * **Entity ID / Audience URI:** * The globally unique name for a SAML entity. * ADAP's current entityID is com:figure-eight:sp * **Default Relay State** * Identifies a specific application resource in an IDP initiated Single Sign-On scenario. In most instances this is blank. * **Application username (Name ID)** * The application username will be used for the assertion's subject statement. Default value is "Okta username". Please mention if you have a special requirement. * ***Email*** * **Attribute Statements** * You can federate user profile field values to SAML attributes. Typical attributes are first name, last name, and username (email format). List all attributes you require for your application. * Please provide the exact (case-sensitive) attribute name. * ***Name = emailAddress*** ***Name format = Basic*** ***Value = user.email*** * **Name ID format** * Identifies the SAML processing rules and constraints for the assertion's subject statement. The default value is 'Unspecified' unless the application explicitly requires a specific format. * ***EmailAddress*** ![image-20260616-061529.png](https://success.appen.com/__attachments/a_b9ffbf987d5518678e38af67fefa21f57e325ba277aa95881c045bd32a68d8b9/image-20260616-061529.png?cb=0363735abed13e4d5442e84ff131d7c7) --- language: "en" --- # Create & Design Jobs --- language: "en" --- # CML Elements Migrated from Zendesk section **CML Attributes**. Articles in this section are listed below. --- language: "en" --- # cml:checkbox - Single Checkbox Renders a single checkbox. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md). ![a3b55c1dbe331877_CMLCheckbox.jpg](https://success.appen.com/__attachments/a_d26dbbaa2d10dda108032ff82723c4a465d185d2df26d4e3fbfa96203bdda58a/a3b55c1dbe331877_CMLCheckbox.jpg?cb=b70df3001bea6b3cdede6b5f2f31034b) ## Additional attributes `default` If supplied, the value of this attribute will be submitted if the contributor does not check the checkbox. Otherwise, when the contributor checks the checkbox, the checkbox's `value` attribute will be submitted (or, in the absence of the `value` attribute, the `label` attribute). `review-data (optional)` Add a column for `review-data` in the uploaded dataset to reference in the cml for that element. The column contents should match the `value` attribute. Add `review-data="{{DATASET_COLUMN}}"` and `task-type="qa"` in the cml snippet. --- language: "en" --- # cml:checkboxes - Multiple Related Checkboxes ## cml:checkboxes Renders a group of checkboxes. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md). ![92c6558c105df5b7_CMLCheckboxes.jpg](https://success.appen.com/__attachments/a_42a2fa6725bf2ad3bac4e984b3415386e1158cec6b8e66092b7adc723a8f8058/92c6558c105df5b7_CMLCheckboxes.jpg?cb=271f32020b1cbf6fe7ff31210fce62ba) The `` children elements accept the following attributes: `checked` If this attribute is "true", the child checkbox will be pre-checked when the page loads. ![047fff27b46b4fef_CMLCheckboxesChecked.jpg](https://success.appen.com/__attachments/a_ad59b3d5c43fd707bc4427166387ed20994fa5d5da21da5f72708f81b8f0f754/047fff27b46b4fef_CMLCheckboxesChecked.jpg?cb=2b47c5aa308900cf53da4217e71539a7) `label` The visible label for the child checkbox. This is different than the parent ` `element's `label` attribute, which is the label for the entire group of checkboxes. `value` The value that gets submitted if the contributor selects the corresponding child checkbox. If this isn't present, the value of the `label` attribute is submitted. `review-data (optional)` Add a column for `review-data` in the uploaded dataset to reference in the cml for that element. The column contents should match the `value` attribute. Add `review-data="{{DATASET_COLUMN}}"` and `task-type="qa"` in the cml snippet. Checkboxes also allow you to specify how test question answers should be matched to contributor inputs. To learn more about test question matching for `cml:checkboxes`, please visit . --- language: "en" --- # cml:file\_upload ## Overview The File Upload Tool `cml:file_upload`, allows contributors to upload files from their machine, such as images, audio, video, documents etc. ## Adding a File Upload Component You can incorporate the file upload tool into any jobs, on its own, or in combination with other cml objects. Use the cml below to add a file upload widget to your job design. ## Parameters * `allowed-extensions` this parameter allows you to specify which types of file extensions are accepted. * `min-size`this parameter allows you to specify the minimum size of the files accepted. This parameter takes in a number. For example: min-size="0.5" would set the minimum accepted file size to 0.5MB. * `max-size`this parameter allows you to specify the maximum size in megabytes of the files accepted. The greatest accepted value is 5120 (5GB). **Notes** 1. If neither `min-size` nor `max-size` is specified, the tool will set the allowed file size to 0-5MB by default. 2. This tool does not support test questions or aggregation. 3. You will need to upload some data in order to launch a file upload job, even though you may not need source data in your job. ### Results The file upload tool will output the file's URL and metadata in the job report. This tool has full SDA support, which means the files uploaded by contributors can be written directly to your cloud storage, for your easy access. Please contact your success manager for any help setting up SDA. --- language: "en" --- # cml:group - Group Related Form Elements Together ## cml:group Group related questions together so that they can easily be hidden or shown with [CML logic](https://success.appen.com/appen-success-center/enhancements/guide-to-cml-logic.md) without having to write only-if attributes for every field. In the above example, both the `cml:text` and `cml:checkboxes` fields will be hidden/shown depending on whether the sentiment field passes the `cml:radios` logic. ### Allowing multiple answers for cml:group There may be occasions where multiple answers are needed for elements with `cml:group`. Using 'multiple=true' can help keep your UI looking clean and allow for as many answers as you'd like: ![image-20260623-055300.png](https://success.appen.com/__attachments/a_08cc9d6b0dc051ff7cbf4e05bd1453df138ffecce3dc105a8676898875587d72/image-20260623-055300.png?cb=9ee38559e6270354144aa9b2c7ce46af) **⭐ NOTE** The following information is supported in Quality Flow and NOT Adap Legacy: * You can create multiple nested groups with no maximum depth. However, we recommend limiting nesting to a maximum of **three** levels for a better user experience for contributors. * Logic can be triggered both outside and inside a `cml:group`. * **Limitation -- Radio Buttons in Groups** : You cannot trigger logic using radio button elements either to show a group or from within a group. * ✅ Recommended alternative: Use `cml:select` instead. * **Limitation --** `cml:text`**with** `multiple="true"`: `cml:text` elements with `multiple="true"` do not work inside a `cml:group`. * ✅ Recommended alternative: Nest a `cml:group` and place the `cml:text` inside that group. ### QF Output: * When using `cml:group` with `multiple="true"`, all element outputs within the group appear in a **single column** , separated by **pipes (** `|`**)** . For example, if `cml:text` is inside a `cml:group` and a contributor creates 3 groups, the output will look like: `Response 1 | Response 2 | Response 3` * If a `cml:text` element is **optional** or **conditionally triggered** , and the contributor **does not fill it out** , the output will include a `NULL`. For example, if the second group was skipped or left blank, the output would be: `Response 1 | NULL | Response 3=` Example output from the previous code snippet - Within the job: ![image-20260623-055331.png](https://success.appen.com/__attachments/a_2bcb53b575ec795d121709f7d461e8695cfb1591b9af83c3cf5490b02c830309/image-20260623-055331.png?cb=02df450870508df7b0a91423d42ae547) In the output: ![image-20260623-055349.png](https://success.appen.com/__attachments/a_1eac639cb43b5fe5f344609ff7f42adc9b690263b3ef066609a346a63a1af391/image-20260623-055349.png?cb=a742fcc196464b88f1ae373ce3ad463d) --- language: "en" --- # cml:group and multiple=“true” Aggregation ## Overview `cml:group` paired with `multiple="true` aggregation allows requesters to aggregate data across multiple text fields and submissions within a `cml:group.` The feature generates a new column in the aggregate report to display linked entities in a `cml:group` and the confidence of each linked response. ## Building a Job The following CML contains the possible parameters for a `cml:group` with `multiple="true"` aggregation job: **Important Note** : This feature is applicable as a possible aggregation method within text fields (`cml:text`) in aggregated reports only. It does not apply to test question units. ## Parameters Below are the required parameters to enable `cml:group` and `multiple ="true"` aggregation: * `multiple` * When set to `"true"` contributors can input a dynamic number of entries of the group. * `group_name` * The name of the output aggregation column. * `aggregation` * When set to `"true"`, an aggregated output is generated. ## Aggregation Aggregation for `cml:group` and `multiple="true"` functions as follows: * Aggregation for `cml:group` with `multiple="true"` enabled aggregates linked instances, defined as `cml:text` fields within a `cml:group`, and provides confidences per instance. * Confidence is calculated by dividing the total number of unique submissions from a linked instance by the total number of judgments on the unit. * Example: Given the example CML above, if one out of two contributors submitted "Johnny" and "Appleseed" for `first_name` and `last_name` respectively, the confidence would be 0.5 ## Reviewing Results The output of the aggregated column is an array of JSON objects containing the linked entities and confidences ordered by decreasing confidence. For example, if two contributors submitted the following responses: Contributor 1: ![image-20260626-061649.png](https://success.appen.com/__attachments/a_70bc658f117953819c8d949a80d9fef85a732696dbcf2e10a1f248eae1151350/image-20260626-061649.png?cb=1f603165193f1211f19a6fbdc8ea5cea) Contributor 2: ![image-20260626-061701.png](https://success.appen.com/__attachments/a_680a7c7dc8ee75bc5c7b85e989ec7c2434d26465c1ed8c68007e9c8a656a8aa7/image-20260626-061701.png?cb=6058bfdc523ee45a9e5404ad65dba502) The aggregated output for that row in the aggregated report would be: [{"first_name":"Foo","last_name":"Bar","confidence":1.0},{"first_name":"John","last_name":"Apple","confidence":0.5},{"first_name":"Johnny","last_name":"Apple","confidence":0.5}] where: * `"first_name"` and `"last_name"` are columns of the text fields in the group * `"confidence"` is the level of agreement of the linked instances --- language: "en" --- # multiple="true": Allow Multiple Answers You can allow contributors to submit more than one answer in the following CML fields by using the attribute `multiple="true"`: * `cml:text` * `cml:textarea` (when in a `cml:group`) * `cml:select` (when in a `cml:group`) ## **cml:text** For `cml:text`, the attribute `multiple="true"` adds an '+' icon next to the text box, that allows contributors to submit as many answers as they need. ![image-20260616-070258.png](https://success.appen.com/__attachments/a_922f11ba00aa4958268ed2b1a44766fe5ec0fddcf36198a6c259b30977c8a240/image-20260616-070258.png?cb=84298583bbf84e541e7a9209d17b911a) Fig 1. CML for cml:text ![image-20260616-070248.png](https://success.appen.com/__attachments/a_b54ed43485f1362b420de0f3e78a1590d8475070c6dd5911dd0e2f8d251328e7/image-20260616-070248.png?cb=5e111e6569873e6b2d01130743641866) Fig 2. Preview for cml:text **cml:textarea** and **cml:select** Unlike `cml:text`, use the attributes `multiple="true"` and `name=""` for either `cml:textarea` or `cml:select` . The attributes will need to be included within the `cml:group` field (see examples below). This attribute adds an 'Add Another" icon on the top right-hand corner of the box, and allows contributors to submit as many answers as they need. ![image-20260616-070334.png](https://success.appen.com/__attachments/a_39922c4a2d6f046f4d7c273a67021fd859a9b65e50041cf4e5704ef0d9c416d6/image-20260616-070334.png?cb=83a1578ae3567a11386bb81274a43bcc) Fig 3. CML for cml:textarea ![image-20260616-070352.png](https://success.appen.com/__attachments/a_911a811ea5e2502f194609205ccfcb27c27ccc9c86ca79e69983311a69431988/image-20260616-070352.png?cb=6434f04ce193f604cbf1cdf26425a218) Fig 4. CML for cml:select ![image-20260616-070518.png](https://success.appen.com/__attachments/a_4e7effc1a7e0d30d95dabb0978d0dcacb11e50948afe75c87ae29b3950120e0d/image-20260616-070518.png?cb=296429248c1587c574cf8342f2e05666) Fig 5. Preview for cml:textarea and cml: select **Keyboard Shortcuts:** First instance: To select the "+" button, select the text box and press "Tab" once and "Return": ![image-20260616-070630.png](https://success.appen.com/__attachments/a_955490e903173b391bf831826d3ccedca36aec5275f8de91a711e475a0b7f324/image-20260616-070630.png?cb=f41dda673bbc082e22476883f18e334c) Other instances: To select the "+" button, select the text box and press "Tab" twice and "Return": ![image-20260616-070640.png](https://success.appen.com/__attachments/a_fbfcfbe98a041e7031612cc6e4e91b99ccc69550b7c38bd1a759cbce14fbfdde/image-20260616-070640.png?cb=018974429552c89b0f5808947e1317bb) To delete a text box press "Tab" once and "Return": ![image-20260616-070649.png](https://success.appen.com/__attachments/a_cb9fe3fc41b3dc1b13198193f71c859cf48afddefb0d28e2271b73935562c4f7/image-20260616-070649.png?cb=a70dbfc9df332cf7652ae7b0941c24bd) --- language: "en" --- # cml:hidden - Hidden Form Field ## cml:hidden Renders a hidden form input. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md). The following CML will output "true" in a column called "secret_sauce" in your generated reports: This is best used for submitting additional hidden information, such as the contributor's browser information with the `user_agent` validator: --- language: "en" --- # cml:hours- Hours of Operation Input Tool ## cml:hours Renders a widget for inputting hours of operation for every day of the week. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md) except for `value` and `default`. ![6f76808065fa484e_CMLHours.jpg](https://success.appen.com/__attachments/a_8a6075848702954d03ecc3aaaa2da42e288e5b9d2f790bd5b574fbb8343c733f/6f76808065fa484e_CMLHours.jpg?cb=df745a2ae1636109335b480674b1742f) Note that `cml:hours` is always required. If you do not want the field to be required, use [CML Logic](https://success.appen.com/appen-success-center/enhancements/guide-to-cml-logic.md) to hide the field if the contributor does not need to input anything. ### Additional attributes `allowunlisted` If set to "true", the hours dropdowns will include a "Not listed" option that will allow contributors to specify that the hours of operation were not listed for that given day. Defaults to "false". *(optional)* #### Returned columns `cml:hours` returns the following columns for each day of the week (`mon`, `tue`,`wed`, `thu`, `fri`, `sat`, `sun`): * `{field_name}_{day}_openallday` - "TRUE" if the contributor selected "Open all day", "FALSE" otherwise. * `{field_name}_{day}_closedallday` - "TRUE" if the contributor selected "Closed all day", "FALSE" otherwise. * `{field_name}_{day}_open_1` - The first open time. One of several values: * "" - Nothing is submitted if the contributor chose "Open all day" or "Closed all day" * "not_listed" - If `allowunlisted` is set to "true", contributors will have the option of selecting a "Not listed" option. This is the value that is submitted when this occurs. * "##:##" - A time is submitted in 24-hour format. For example: "17:30" (5:30pm) * `{field_name}_{day}_close_1` - The first close time. Same potential output values as the first open time. * `{field_name}_{day}_open_2` - The second open time. Will be blank unless the contributor clicked "Add extra time range". Same potential output values as the first open time. * `{field_name}_{day}_close_2` - The second close time. Will be blank unless the contributor clicked "Add extra time range". Same potential output values as the first open time. --- language: "en" --- # cml:instructions- Specify Instructions for a Field ## cml:instructions Can only be a child tag of a form element tag. This tag allows you to include a set of instructions along with the form element, similar to the `instructions` attribute. If both an `instructions` attribute and a `` tag are specified, only the value of the `instructions` attribute will be used. Check one of these. ![image-20260616-085930.png](https://success.appen.com/__attachments/a_997f0c749211ccba0ea1637ea67000ebae958b463610d7a96b84715ac66d4e00/image-20260616-085930.png?cb=f3f62e30da34cf15551b7f18e8ca726b) --- language: "en" --- # cml:radios - Multiple Radio Buttons ## cml:radios Renders a group of radio buttons. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md). ![db5aebd46b91115a_CMLRadio.jpg](https://success.appen.com/__attachments/a_de86c5e15470589f4e213a349f044ef989118def25247f117f468c3d27337910/db5aebd46b91115a_CMLRadio.jpg?cb=efdf4840dc8f676e0f245f393d7cebad) The child `` elements accept the following attributes: `checked` If this attribute is "true", the child radio button will be pre-select when the page loads. `label` The visible label for the child radio button. This is different than the parent ` `element's `label` attribute, which is the label for the entire group of radio buttons. `value` The value that gets submitted if the contributor selects the corresponding child radio button. If this isn't present, the value of the `label` attribute is submitted. `review-data (optional)` Add a column for `review-data` in the uploaded dataset to reference in the cml for that element. The column contents should match the `value` attribute. Add `review-data="{{DATASET_COLUMN}}"` and `task-type="qa"` in the cml snippet. --- language: "en" --- # cml:ratings - Multiple Radios in a Single Line ## cml:ratings Renders a set of radio buttons in a single line for performing ratings. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md). This type of rating can be used with numerical values or word values. Here we have an example of sentiment related rating with values on each radio: cml:ratings can also accept 'from' and 'to' attributes. These will add beginning and ending labels as opposed to addig every label. ``` ``` `review-data (optional)` Add a column for `review-data` in the uploaded dataset to reference in the cml for that element. The column contents should match the `value` attribute. Add `review-data="{{DATASET_COLUMN}}"` and `task-type="qa"` in the cml snippet. --- language: "en" --- # cml:select - Drop-down Menu ## cml:select Renders a drop-down menu. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md). ![65390cd41a0aa277_CMLSelect.jpg](https://success.appen.com/__attachments/a_0f3240093085e09f79fb2d80d56ce8e9ae31b6e083e9265350c8bc38abcb333b/65390cd41a0aa277_CMLSelect.jpg?cb=06512b3031c3a03c60d70bd1024d20dc) **Note:** You can allow contributors to submit more than one answer by using the attribute [multiple="true"](https://success.appen.com/appen-success-center/create-design-jobs/cml-elements/cml-group-group-related-form-elements-together/multiple-true-allow-multiple-answers.md). The `` children elements accept the following attributes: `label` The visible label for this option. This is different than the parent `` element's `label` attribute, which is the label for the entire drop-down menu. `value` The value that gets submitted if the contributor selects the corresponding option. If this isn't present, the value of the `label` attribute is submitted. `selected='true'` The default selected option. If no changes are made to the select drop-down then that option will be used as the answer. Example below: In the example above Yes is the default answer and will show as preselected in the job Preview. `review-data (optional)` Add a column for `review-data` in the uploaded dataset to reference in the cml for that element. The column contents should match the `value` attribute. Add `review-data="{{DATASET_COLUMN}}"` and `task-type="qa"` in the cml snippet. *** ** * ** *** --- language: "en" --- # cml:shapes - Bounding Box, Polygon, Dot Annotation, and Line Annotation tool Appen's cml:shapes tool offers a solution to a variety of different image annotation use cases. Please see the below articles for more information: * [Bounding Boxes](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-bounding-box-job-with-labels.md) * [Polygons](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-polygon-job-design-and-aggregation.md) * [Dots](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-dots-job-with-labels.md) * [Lines](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-polylines-job-design-and-aggregation.md) * [Ellipses](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-an-ellipse-or-circle-annotation-job.md) * [Shapes Peer Review](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-shapes-peer-review-job.md) --- language: "en" --- # cml:summary\_table - Summary Table Tool ## Overview The `cml:summary_table` CML tag provides a comprehensive overview of all form and text elements within a job, automatically populating as contributors fill out the job. This feature allows contributors to spot-check their work before submission and enables QA contributors to quickly review the work of previous contributors, ensuring accuracy and efficiency. **Form \& Text elements supported:** * Checkbox * Checkbox group * Multiple choice * Dropdown * Ratings * Smart text * Text * Textarea ![image-20260615-083610.png](https://success.appen.com/__attachments/a_a5d5c7401271cd0b61545bc473358e561f2aa29c69fb5a135ae5b55be9de70f4/image-20260615-083610.png?cb=f06934a98f0e12f86279feb17b0573cd) ## How to configure the tool Step 1: Create and design your job either through the CML or through the graphical editor. ![image-20260615-083628.png](https://success.appen.com/__attachments/a_8cb28b87b58e3caa8fe70361fd889acf3a79c12b59667c42e528c6f864d2aba5/image-20260615-083628.png?cb=d7103fe710fe015ba68c590c3b10da13) Step 2: Switch over to the CML if you're not already, and add `` tag wherever you want to place the table in your job. ![image-20260615-083847.png](https://success.appen.com/__attachments/a_64bfff2dc661a280ad45888c6625ab2520c9f1bd0990fd5e11520e410784465e/image-20260615-083847.png?cb=91066e5922405d1c7cf0f78bfc45a717) ![image-20260615-083959.png](https://success.appen.com/__attachments/a_c53f877aa27e38e57abafd8d0f7b6a125061879329e2eb43dcd2fdf3ba597b20/image-20260615-083959.png?cb=eabb2c85bbd7407a5af64c7901060f64) Step 3 (Optional) - Customize the table title using the `label` parameter and/or utilize the `elements` parameter to specify which specific form/text elements you want to include in the table.Unknown Attachment ![image-20260615-084021.png](https://success.appen.com/__attachments/a_03908e0e41880ee1a2bd2cfb16f40aa0ac14b1f422d1ac5b59b4c1f54c69708d/image-20260615-084021.png?cb=33aea8a22cb2eb3313670050248caf33) ![image-20260615-084037.png](https://success.appen.com/__attachments/a_97e4e4ba373a77aa2139cd1c3f56c0cf1ac0080e15aa76a67fb74780014cc455/image-20260615-084037.png?cb=e5cf0fd80fa82bd4719b2448cefa4722) ## Parameters `label` (optional) * Customizes the table title. * Defaults to "Summary" if not included `elements` (optional) * Represents the form and text elements you want to include in the summary table, specified as an array of name references from the `name` parameter in each element tag. * If the `elements` parameter is not included in the CML tag, all form and text elements configured in the job design are included in the summary table. * The title of each element cell references the `name` parameter in each element tag. --- language: "en" --- # cml:taxonomy\_tool - Tree Search and Input Tool ## Overview `` renders a widget that allows contributors to search and browse through a hierarchical list of items (a taxonomy) and select an item (or multiple) to be submitted. Taxonomy data must be formatted according to the [Taxonomy File Formats](https://success.appen.com/appen-success-center/create-design-jobs/cml-elements/cml-taxonomy_tool-tree-search-and-input-tool.md#_Taxonomy_File_Formats) section below. ![Screen_Shot_2022-04-28_at_12.29.05_PM.png](https://success.appen.com/__attachments/a_11505a67bb628267438b570e3aae210e5a56ee502c53a0e2579616ddedc7f8f5/6e6a47d29fa7137d_Screen_Shot_2022-04-28_at_12.29.05_PM.png?cb=ab1c4bafa7b8365c6c2e9f485cc83f21) *Figure 1: cml:taxonomy_beta in Preview* ### Creating a taxonomy job and using the Taxonomy Manager On the design page when a taxonomy job is created and the job design is saved, a selection with a link to the Taxonomy Manager link will be displayed as shown in Figure 2. The CML tag will look something like this and can be placed directly in the CML Field: `` ![Screen_Shot_2022-09-23_at_11.16.06_AM.png](https://success.appen.com/__attachments/a_dc93986bab3eb671cf89922be3f7bb444139dde6b3de1b4ee5aced7c04f560c8/dd3b12de862c29db_Screen_Shot_2022-09-23_at_11.16.06_AM.png?cb=a55c0fbf16bb4b34499d48b95fa75053) On the Taxonomy Manager page, the requestor can upload the taxonomy file (json or csv). If an existing taxonomy exists on the job, then the download link will be available as shown in Figure 3. ![Screen_Shot_2022-04-28_at_12.32.33_PM.png](https://success.appen.com/__attachments/a_e834959e87099a6d8cd47c0a4dd94e79ff4aeb96489c508d00575ff5c833783e/832566a89fce53ce_Screen_Shot_2022-04-28_at_12.32.33_PM.png?cb=3115e4590a117775affa43e0364a4634) *Figure 3: Taxonomy Manager* ### Test Questions You can use test questions in your jobs with `cml:taxonomy_tool` to ensure data and contributor quality. For units with multiple possible correct answers, the job design can be configured to support different logic for correct answers. Please refer to [this article](https://success.appen.com/appen-success-center/create-design-jobs/cml-elements/cml-taxonomy_tool-tree-search-and-input-tool.md#Test-Question-Matching) for information on supported test question matching logic. ### Note: Only-If Dependency [Only-If logic](https://success.appen.com/appen-success-center/enhancements/guide-to-cml-logic.md)in OTHER cml elements that reference the taxonomy tool is currently not supported. *** ** * ** *** ### Additional Attributes for item selection #### multi-select Accepts Boolean values, 'true or 'false'. If set to "true", the taxonomy tool will allow contributors to select multiple items. By default, a contributor can only select one item. An example of a multi-select option is shown in Figure 4. ![Screen_Shot_2022-04-28_at_12.35.29_PM.png](https://success.appen.com/__attachments/a_0eaec095a8739e9f0dc2352d1387943fb95e86fa8491530adf5361ff8e8252da/9a9d054a6ca01216_Screen_Shot_2022-04-28_at_12.35.29_PM.png?cb=fde9190887caeea4df855ab80912a1df) *Figure 4: Example of multi-select=true. User can select more than one item (only leaf/endpoint items)* #### select-all Accepts Boolean values, 'true or 'false'. If set to "true", every taxonomy item will be selectable (normally only taxonomy endpoints are selectable). An example of a select-all option is shown in Figure ![Screen_Shot_2022-04-28_at_12.37.50_PM.png](https://success.appen.com/__attachments/a_09c31d8f1bb4629094545ba8269528db9a375b964243667763064a6648dab16f/7d2112ff4d71965a_Screen_Shot_2022-04-28_at_12.37.50_PM.png?cb=12f6b9744093e37450f98055cda19be4) *Figure 5: User can select parent items when select-all is set 'true'* Users can set both \`select-all\` and \`multi-select\` to true to enable multiple selections of items at each level as shown in Figure 6. ![Screen_Shot_2022-04-28_at_12.42.09_PM.png](https://success.appen.com/__attachments/a_39a24f17e14a44a479f937fc63d70796d7ee97d3237fed1bc2bff6f6b9257a48/06dc3bdad8af3d23_Screen_Shot_2022-04-28_at_12.42.09_PM.png?cb=41f769c66a8de51a1b01cf255e2b1775) *Figure 6: Both parent and child items can be selected when both select-all and multi-select is set to true.* In addition to selecting items in the taxonomy, the tool also includes a search field as shown in Figure ![Screen_Shot_2022-04-28_at_12.43.07_PM.png](https://success.appen.com/__attachments/a_9c29abfd2d8996f1b667702edbbeb985ffd1f0a5bfc940c487fb8ed518cf9ac6/d1639207e229e823_Screen_Shot_2022-04-28_at_12.43.07_PM.png?cb=6c4707a4435b5a26a9932a57ce504d24) *Figure 7: Search bar with matching results* #### Sorting Nodes can be displayed alphabetically to workers by setting \. If not set or set to "false", by default they will display in the order uploaded." *** ** * ** *** ### Taxonomy File Formats #### 1. CSV Flat Each row represents a category and its parent. Headers must include: * **category_1** * **category_2** * **description** (optional): any information you want displayed in the taxonomy to help the user understand what it is. If a file is uploaded, the Taxonomy Manager will convert to JSON format that is available for downloads. *E.g., flat csv file:* ![Picture1.png](https://success.appen.com/__attachments/a_415c7bf356980166c0713a04967995032cdf4fa92b3fa5e1f9d29f3964ff61e3/4691fc27de410dfd_Picture1.png?cb=de3c92b5081a061a4751730f8cf30742) *Taxonomy view for the above file:* ![Picture2.png](https://success.appen.com/__attachments/a_80a8647d970f7b7dbf7c918e4c25ba020b9960afc77163421ad3b055cc97bcc4/cbd87820819650f4_Picture2.png?cb=4a230fc64530605bd2ca204ad0277828) #### 2. Nested CSV In nested taxonomies, each row describes only a single node and the relationship with the parent and children are done spatially, so ordering matters. Headers must include: * **category_1** through **category_n** : category_1 is the top-level category, followed by any number of sub-categories. ***Required fields***. * **id:**(optional). * **description**: any information you want displayed in the taxonomy to help the user understand what it is (optional). *Example nested csv:* ![f6491efd-f989-4934-ac7b-5a58339281e1.png](https://success.appen.com/__attachments/a_be1aa3c43014d60fadb0a7c98b89dacebbd74155ab495521a545bd3c7a1dd4d6/bf745b4a25f35a48_f6491efd-f989-4934-ac7b-5a58339281e1.png?cb=fa532355c5aa4eb828b3878513447cc6) #### 3. Path by Row CSV In path-by-row csv format, each row describes a full path so there may be repeats in the parent columns. Headers must include the same as those for Nested csv format (see above). *Example: path by row csv* ![5b298441-21dd-40af-8329-6852f3873af6.png](https://success.appen.com/__attachments/a_9dba1b5eed8067c641e53ac772c4e660f9237107e2568895219829b54e6ffa7f/fc0ef09d85b840d4_5b298441-21dd-40af-8329-6852f3873af6.png?cb=fd3716af99d02616fd560c5e05582af3) #### 4. JSON file format *An example JSON file format:* ![Screenshot_2022-05-27_at_6.40.49_PM.png](https://success.appen.com/__attachments/a_97ab1f2d710752b6ddfe13a619355b0875aa9c465dfbacc6c6ed0e8c6e9453b7/e18c2866156e6429_Screenshot_2022-05-27_at_6.40.49_PM.png?cb=bc1682c3b2c01cb87ad8194bb3d7a004) #### 5. JSON format to support Directed Acyclic Graph (DAG) Taxonomy Manager will also support a graph that include a DAG like the below diagram. ![Picture6.png](https://success.appen.com/__attachments/a_ac4c40476677e61360cedd4bc25c1056d2839d95d63def44ced0d2fe9df40221/8e477b2c9429a28c_Picture6.png?cb=0dd7e626a33221cf1b2d2640415cb9f2) The JSON format to support the above DAG example will be: ![Picture7.png](https://success.appen.com/__attachments/a_13e160c7e81f184e23db2804123c92e265d146ce2892926fa87033cbb6491ced/a2073604512416f2_Picture7.png?cb=891a290f19abeaaf9c35a8edcb1bbc2c) The same example can be supported in CSV using the path-by row format as shown below: ![Picture8.png](https://success.appen.com/__attachments/a_c22f18858c7512d1818169bb1f9a562291492015dfe582c990f2acf43411245b/354f2d8c32ce0a90_Picture8.png?cb=32bcfdc02d7a400d0344ae3ee057aa67) *** ** * ** *** ### CSV file validations In the Taxonomy Manager, before uploading a CSV file, the file must be formatted as follows to avoid bad parses: * CSV must have a header row. The headers must be exactly *category_1,category_2,...,category_N,description, id* , in that order. The*description* and*id*fields are both optional and can be present only after the first N category level header names. * The required delimiter for the CSV is a comma "," **Do not have spaces around this delimiting comma in your data rows**, otherwise the parsed results may not be correct. * If a field value includes a comma, you must wrap that entire field value with double quotes, e.g. "I have, comma" **There must be no spaces before the starting quote nor after the closing quote.** *The quotes must be immediately adjacent to the delimiter (,) to indicate a quoted field.* * Each category path must be entirely on one row. Category paths with category names extended into multiple lines are not yet supported. Category paths that extend to a new line will be parsed as new category paths. *** ** * ** *** ### Multiple Taxonomies You can upload multiple taxonomies to a single job, so that each row/unit can render multiple instances of taxonomy tool with different taxonomy data and/or rows requiring different taxonomies can be uploaded to the same job. #### Uploading Multiple Taxonomies ##### **UI** Via the taxonomy manager, you can upload one or more taxonomies. Taxonomies can only be uploaded one at a time. Each taxonomy should have a unique name. You can use any characters you'd like to name your taxonomies, but we recommend something easy to remember. Taxonomies can be removed from jobs as well. When you copy a job, all of its taxonomies will also be copied. ##### **API** `https://api.appen.com/v2/jobs//taxonomy?key=` You need to provide job id as part of the URL and API key as "key" get parameter. API requires next parameters: 1. `file` - taxonomy file(supports `.csv` and `.json` formats same as for UI) 2. `name` - taxonomy name(taxonomy unique name) **Example:** curl --location --request PUT 'https://api.appen.com/v2/jobs//taxonomy?key=' \--form 'name="taxonomy-unique-name"' \--form 'file=@"/path/to/file/taxonomy.csv"' #### Selecting a Taxonomy Using the cml attribute `source`, you can indicate which taxonomy you want to use by its name (as created in the Taxonomy Manager), a url string starting with `http://` or `https://`, a ref string, or using liquid reference to the unit data column if you've indicated the correct taxonomy in the input data. If you've uploaded multiple taxonomies but have neglected to specify `source`, the tool will fail to initialize and you'll be reminded to specify a taxonomy. ++**Note**++ **:** Tool can only read the taxonomy data in internal json format. Taxonomies are only parsed when uploaded via the Taxonomy Manager or the API (Taxonomy Manager converts taxonomy in path-by-row CSV and nested graph CSV formats into the internal json format while uploading). Therefore, if you are using a liquid reference or an external url as the source, you must ensure that that source contains taxonomy in internal json format, i.e. it cannot be a url to a csv file for example. #### Using One Taxonomy You can still upload only one taxonomy if only one is required. In that case, you don't need to indicate which taxonomy to use via `source`. If you've already been using the `cml:taxonomy_beta` tool, your jobs with a single taxonomy will still work without adding the source attribute to your jobs. If you upload more than one taxonomy, you will receive an error in the tool until you add `source` parameter to your cml. #### Removing a Taxonomy You can remove a taxonomy via API using DELETE. Include the taxonomy name in your call. Example: curl --location --request DELETE 'https://api.appen.com/v2/jobs//taxonomy?key=' \ *** ** * ** *** ### Uploading Taxonomies CSVs must be formatted as follows to avoid bad parses: * Your CSV must have a header row. The headers **must be exactly** `category_1,category_2,...,category_N,description,id`, in that order. The description and id fields are both optional and can be present only after the first N category level header names. * The required delimiter for the CSV is a comma `,`. Do not have spaces around this delimiting comma in your data rows, otherwise the parsed results may not be correct. * If a field value includes a comma, you must wrap that entire field value with double quotes, e.g. "I have, comma". **There must be no spaces before the starting quote nor after the closing quote** . The quotes must be immediately adjacent to the delimiter (`,`) to indicate a quoted field. * Each category path must be entirely on one row. Category paths with category names extended into multiple lines are not yet supported. Category paths that extend to a new line will be parsed as new category paths. * If using multiple taxonomies, specify the taxonomy you would like to use via the `source` attribute in the `cml:taxonomy_tool`. The Taxonomy Title is the value that you pass to the source attribute - Ex. `source="myTaxonomy"` **Note**: After uploading your taxonomy, check the Preview page to ensure all category paths are displaying as expected. ## Test Question Matching ### Note Information in this article relates to test questions in ADAP Jobs, for information about test questions in ADAP Quality Flow, please see [++this page++](https://success.appen.com/appen-success-center/settings/guide-to-test-questions-in-quality-flow.md). The default test question setting will pass a contributor as long as **one** of their answers matches one or more of your answers even if they've also selected a wrong answer. For example, if you select A and B as the correct answer, for a multiple checkbox question, a contributor will pass the test question if they select "A" **or** "B". In this example, the contributor will be marked correct even if he/she selected a wrong answer. If you decide to be less lenient with test questions, you will want to use either of the following attributes: `exact` If set to,`"true"(exact="true")` the set of contributor responses must be identical to the set of pre-defined correct responses on a test question in order for that contributor's judgment to be considered correct. A contributor's response will not be considered correct if they include a response not found in the set of correct responses. For example, if responses "a", "b", and "c" are defined as correct in the test question data, then a contributor will only be correct if they submit "a", "b", and "c". Order does not matter. Note: this setting will also attempt to match a test question even if it is left blank. For example, if you choose to not provide an answer for a set of checkboxes, the correct response with exact="true" will be **none** of the possible answers. **Example CML:** `strict` If set to `"true"(strict="true")`, every response submitted by a contributor must be included in the set of pre-defined correct responses on a test question in order for that contributor's judgment to be considered correct. Unlike `exact`, `strict` allows contributors to omit responses that are part of the test question set. For example, if responses "a", "b", and "c" are defined as correct in the test question data, then a contributor will be considered correct if they submit only "a" and "b". `matcher` Allows you to redefine how a contributor's responses to test questions are evaluated. Normally, a contributor's responses must exactly match that of the stored test question answers. This can be especially helpful when used with numeric or alpha test questions. This attribute can be set to the following: * `not` - This will reverse the normal behavior so that a contributor must answer anything but the pre-defined test question answer to be considered correct. For example, if answer "a" was stored in the test question data and the contributor selects answer "b" (or anything other than "a"), the contributor would get the test question correct. * `range` - This allows you to specify a range that a contributor response can fall between. Specifying a minimum and maximum range will allow a greater threshold of acceptability when contributors answer test questions that may have some variances. To define the minimum and maximum, this can be set within the test question interface, or within a data file with two headers named "column_name_min" and "column_name_max". Use of a number validator is recommended in conjunction with this matcher. `fuzzy` this allows you to specify a fuzzy matching threshold for text inputs (including cml:text and cml:text_area). This options accepts a value between 0 and 1. The fuzzy matching level is computed based on the word level Levenshtein distance. If the matching level is equal or above the threshold, the response passes, otherwise it fails. For example: `fuzzy="0.7"`. **Example CML:** ![tq_matching.png](https://success.appen.com/__attachments/a_cea3200c13e31df251c2cc9ce744cfdd070c7d665245bed87c8424859c00f0df/7c85f8ccd0ddafa0_360053406091.bin?cb=0e93e296281a754194a5c17064038698) Fig. 1: Examples of how different test question matching parameters evaluate contributors --- language: "en" --- # cml:text - Single Line Text Input ## cml:text Renders a single line text field. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md). ![248f068ecdf28a4f_CMLText.jpg](https://success.appen.com/__attachments/a_2008ea908e9236383ec58970e3e042224f974a0f3c3f62d71c678d132479cf00/248f068ecdf28a4f_CMLText.jpg?cb=98e640211a716232de8004230a9534be) #### Additional attributes `default` If supplied, the value of this attribute will be pre-filled in the text box when the page is loaded. It will not be submitted and will fail the `required` validator until the contributor enters text into it. ![f83bd2b547a76cfa_CMLTextDefault.jpg](https://success.appen.com/__attachments/a_d99bc6269c95baa9991b19a684dde61cf03e87d1b5e4f2cf2f58d4661475bd66/f83bd2b547a76cfa_CMLTextDefault.jpg?cb=b9f38aa06ec9f0732e8e996092bb2dc3) Note: You can allow contributors to submit more than one answer by using the attribute[multiple="true"](https://success.appen.com/appen-success-center/create-design-jobs/cml-elements/cml-group-group-related-form-elements-together/multiple-true-allow-multiple-answers.md). *** ** * ** *** ### Validators We recommend using some of the following validators to clean contributor inputs so that the inputs are uniform and will aggregate more easily: * `clean:['trim']` - Removes leading and trailing whitespace. * `clean:['titlecase']` - Capitalizes all words that are not all uppercase nor most conjunctions. * `clean:['uppercase']` - Replaces all lowercase letters with uppercase letters. * `clean:['lowercase']` - Replaces all uppercase letters with lowercase letters. To use multiple validators, add a comma after each clean validator like so: `clean:['trim','titlecase']"` For a list of more input cleaning validators, please visit [this article](https://success.appen.com/appen-success-center/enhancements/guide-to-validators.md). --- language: "en" --- # cml:textarea - Multi-line Text Input ## cml:textarea Renders a multi-line text area. Accepts all [common attributes](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor/guide-to-cml-attributes.md). Note: You can allow contributors to submit more than one answer by using the attribute[multiple="true"](https://success.appen.com/appen-success-center/create-design-jobs/cml-elements/cml-group-group-related-form-elements-together/multiple-true-allow-multiple-answers.md). ### Additional attributes `default` If supplied, the value of this attribute will be pre-filled in the text area when the page is loaded. It will not be submitted and will fail the `required` validator until the contributor enters text into it. *** ** * ** *** ### Validators We recommend using some of the following validators to clean contributor inputs so that the inputs are uniform and will aggregate more easily: * `clean:['trim']` - Removes leading and trailing whitespace. * `clean:['titlecase']` - Capitalizes all words that are not all uppercase nor most conjunctions. * `clean:['uppercase']` - Replaces all lowercase letters with uppercase letters. * `clean:['lowercase']` - Replaces all uppercase letters with lowercase letters. To use multiple validators, add a comma after each clean validator like so: `clean:['trim','titlecase']"` For a list of more input cleaning validators, please visit [this article](https://success.appen.com/appen-success-center/enhancements/guide-to-validators.md). --- language: "en" --- # Guides: Annotation Tools --- language: "en" --- # Guide to: Building an Image Annotation Job in the Graphical Editor 1. Start with the bounding box, polygon, dot, or line template, or start from scratch. 2. Upload data with a column of image URLs. 3. On the design page, select 'Add Question'. 4. In the sidebar, select 'Image Annotation'. 5. Select the column corresponding to images from the 'Data Column' drop down. 6. Determine the shapes you want to use in the job. Each shape will have its own configurable test question parameters. For information on those parameters, please refer to the following articles: * [Bounding boxes](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-bounding-box-job-with-labels.md) * [Polygons](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-polygon-job-design-and-aggregation.md) * [Dots](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-dots-job-with-labels.md) * [Lines](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-polylines-job-design-and-aggregation.md) ![ge-shapes.gif](https://success.appen.com/__attachments/a_50366e3fce24a1ebee21f6477cf8d84a41c87d9b6f2bb5309d991844c78e21a1/9a1c2a695b4b558e_ge-shapes.gif?cb=2bc6cd5c8d95a6bc630e1b74af1b375f) --- language: "en" --- # Guide to: LiDAR ## Overview The `cml:lidar_box` tag allows users to create a LiDAR annotation job for bounding box. The `cml:lidar_segmentation` tag allows users to create a LiDAR annotation job for point cloud semantics segmentation. ## Building a LiDar Job The following CML contains the possible parameters for a lidar annotation job: ## Parameters Below are the parameters available for `cml:lidar_box` and `cml:lidar_segmentation` tag. Some are required in the element; some can be left out. * `name` (Required) * The results header where the results links will be stored * `base_url` (Required) * URL pointing to the base folder containing point cloud data * `color-mode` (Optional) * Defines point cloud color mode * **Options:** 'speed', 'elevation', 'reflection', 'elevation:\[x, y\]\[z, w\]', 'reflection:\[x, y\]\[z, w\]', where x \& y define the elevation/reflection range, z \& w define the color ramp proportion range, \[z, w\] is optional. For example, 'elevation:\[0,5\]', 'elevation:\[0,2\]\[0.25,1\]', 'reflection:\[0.25,1\]', 'reflection:\[0.25,1\]\[0.25,1\]' are all acceptable * `color-mode(new)` (Optional) * Define preset color mode, if color_config is provided will replace color_mode * Options: 'Intensity:0:#0000ff,1:#00ffff', 'Elevation:0:#0000ff,1:#00ffff', can get options string by double click the color mode custom label with ALT key down * `rotate-mode` (Optional) * Defines tool rotation mode * **Options:** 'yaw' - will only allow rotation of the bounding box in the direction parallel to the ground * `range-indicators` (Optional) * Defines circle ranges around the point cloud sensor center. * **Options:** 'x, y, z, ...', 'x:colorInHex, y:colorInHex, ...', where *x, y, z* defines the distance and *colorInHex* defines the circle color. If *colorInHex* is not provided, the default color red is used * `project-async` (Optional) * United Annotation - Allows 2D annotation to be updated independently * **Options:** true or false * `project-rect` (Optional) * United Annotation - Projects 3D annotation into 2D annotation * **Options:** true or false * `auto_save` (Optional) * Flag to enable or disable autosave (frame switch, interpolate, delete from all frames) * **Options:** true or false * `tracking_mode` (Optional) * Flag to enable or disable tool-tracked events * **Options:** true or false * `validate_from` (Optional) * URL pointing to annotation to be used as ground truth for in tool validation. * The format must match the output of the lidar annotation tool (JSON in a hosted URL) * `review-data` (Currently not supported) * `show-grid` (Optional) * To show grid in 3d space. * `default-add-mode` (Optional) * Defines the behavior of Cuboid. If it is DRAG, users can change the size of the Cuboid when they add Cube. If it is Click, the size is fixed. * `label_config` (Optional) * This parameter can be used to define additional 3D Cuboid Attributes. * `label_config_2d` (Optional) * This parameter can be used to define additional 2D Shape Attributes. ## Ontology LiDAR annotation for bounding box and point cloud semantics segmentation share the same ontology structure. The [Ontology Manager](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) allows job owners to create and edit the ontology within a LiDAR Annotation job. LiDAR Annotation Jobs require an ontology to launch. When the CML for a text annotation job is saved, the Ontology Manager link will appear at the top of the Design page. ### Ontology Manager Best Practices * The limit of ontology is 1,000 classes, however, as best practice, we recommend not exceeding 16 classes in a job to ensure contributors can understand and process the different classes. * Choose from 16 colors pre-selected or upload custom colors as hex code via the CSV ontology upload. * If you uploaded model predictions as JSONs, the predicted classes should also be added to the ontology. ## Upload Data We need to first convert client data to the schema supported by our platform. Since there is no standard format in the industry, we work with our clients to understand their format and provide conversion scripts for each request. For more information on secure hosting, check out [this article](https://success.appen.com/hc/en-us/articles/115005120343-Guide-to-Hosting-Secure-Images). Below are example files on how to structure source data. ### Results **LiDAR bounding box annotation:** { "baseUrl": "https://cf-83774kd99dl.s3.amazonaws.com/Q12944/", // base_url for the scene "frames": [ { "frameId": 0, // frame number, start from 0 "frameUrl": "/points/pc_001950.bin", // file path for the frame "items" : [ // objects in the frame { "id": "13f222fd-065c-4745-b441-44dd25566cbb", // object uuid "category": "Car", "number": 8, // object number shows on tool. If the template is set to based on category, then it starts from 1 for each category. If the template is set to global then it starts from 1 for all the objects in a frame "position": { // x,y,z position of center of cuboid, note this is in the coordinate system provided by the customer "x": 66.49787120373375, "y": -37.28758690422451, "z": -4.426572264322248 }, "rotation": { // rotation of object, value in radian, rotate lidar coordinate +X to annotation object coordinate +X, if it's clockwise, the value is negative, otherwise positive around center of cuboid. "x": 0, "y": 0, "z": -1.5804235113355598 }, "dimension": { // need to clarify how cuboid is calculated based on position information. this the full height, width, depth of a cuboid in meters "x": 1.86, // the meaninig of XYZ depends on the definitation of the source data and the point format xyz setup as mentioned above. By default, X is length, Y is width, Z is height. Unit is also depends on the definition of the source data. Normally for autonomus driving, the unit is meter. "y": 4.43, "z": 1.86 }, "locked": null, // not used, please ignore "interpolated": true, // if the cuboid is a interpolated cuboid and never manually adjusted, the value is true. Otherwise the value is false. You. can say is the value is false, it's a key frame to the object. "labels": null, // This field is used to store the attribute form. Usually a json string containing key-value pairs, say '{ "attribute1": "value1", "attribute1": "valueX" }' "isEmpty": false, // for specific client to indicate if the cuboid is an empty cuboid, ignore if not needed "pointCount": 120 // count of points inside the cuboid }, ... ], "isValid" : true, // To mark a frame is valid or not "images" : [ { "image" : "/image_00/image_001950.png", //file path to image "items" : [ //array of 2D annotations { "id": "13f222fd-065c-4745-b441-44dd25566cbb", //UUID of object, matches UUID of cuboid annotation "number": 1, //object instance "category": "Car", //object class "type": "RECT" //format if proj_rect is set to rectangle (or enabled if in GAP Stage) "position": { //top left corner of box "x": 7.678934984761854, "y": 151.025760731091 }, "dimension": { //full width and height of the rectangle in relation to the position "x": 311.2317472514253, "y": 184.7346955567628 }, "labels": {\"testing\":\"Yes\"}, // This field is used to store the attribute form. Usually a json string containing key-value pairs, say '{ "attribute1": "value1", "attribute1": "valueX" }' "isManual": true // the same meaning as the interpolated for 3D cuboid, i.e. true means a labeler adjusted the 2D annotation }, ... ], "relations" : [ //list of linked objects { "id": "6711aebb-e9db-4917-8f6d-5ca2a01861c9", //relationship uuid "relation": "stopping", //relationship category "type": "cube", //type of annotations being related "from": "13f222fd-065c-4745-b441-44dd25566cbb",//uuid of object beginning the relation "to": "72162541-bcac-4fd4-ae43-e0460b6d4c16" //uuid of object ending the relation } ] } ] }, ... ] } **LiDAR point cloud semantics segmentation:** { "auditId": "c4b332f4-bdda-48e0-a395-a8a814f87fa2.157.audit", // for QA "results": [ { "frameId": 0, // frame id, start form 0 "frameUrl": "/haomo_3d/segmentation/CDXYC20210930/61506756316f405f23528861/point_cloud/bin_v1/1626944760095249.bin", // file path for data "totalPointCount": 183031, // number of point in the frame "items": [ // objects in the frame { "id": "128a61d8-87a9-4db2-a174-629c3ce9db92", // object uuid "category": "lane marking", "number": 1, // object number "points": [ // corresponding index of point in the point cloud for the object, start from 0 50657, 56142, 88959, 106743, 123995, 131539, 130796, 134074 ], "labels": "{\"ef-ontology\":\"\u8f66\u9053\u7ebf\",\"vecline_type\":\"Road_Edge\",\"occlusion_edge\":0,\"edge_type\":\"Physical\",\"current\":true,\"edge_index\":-1}", // object attributes, it's json string or null if the form is not configurated "type": "polyline", // object type is "points" or "polyline", if empty then it's "points" "pointCount": 8 // number of point in the object }, ... }, ... ] } **Note:** This report may take a while to generate and download due to the large nature of all its data files. However, the download will still be much faster compared to running scripts to scrape the results. ## Additional Reference **Training Guide for LiDAR annotators:** **Guide to Workflows for project managers:** --- language: "en" --- # Guide to: Ontology Manager The Ontology Manager allows job owners to create and edit the ontology within an image or text annotation job. The Ontology Manager supports up to 1,000 unique classes for image shapes, video annotation, and text annotation. For pixel labeling jobs, the class limit is 250 unique classes. When using `cml:shapes` and `cml:video` in conjunction with the parameter `ontology="true"`, you will be prompted to create an ontology from the design page of the job. In contrast, all text annotation jobs (`cml:text_annotation`) will require an ontology and you will receive the prompt automatically when adding a text annotation form element. There are two ways to create an ontology: ## **Creating an ontology via the UI** You can create and edit an ontology using the Ontology Manager within the job. On this page, you'll see three columns: color, title, and description. ![image-20260615-084715.png](https://success.appen.com/__attachments/a_4cd2f1cc30843315443d7bbed9ccb52ba24e9b0058d68d8ef25b175d929fbf81/image-20260615-084715.png?cb=9b42122abd039f1705776f7976161e07) Fig. 1: Ontology Manager * **Color** defaults to a randomly selected value but can be customized using a selector by clicking on the arrow next to the color. * **Title** is a required field. This is the name of the class in the ontology; it is what contributors will see when annotating. Be aware, each title must be unique and differ from any other class names. * **Report Value** is ***optional***. This is a UUID value that will be placed in the 'report_value' column of the results to be used as a key for the classes in the ontology. This is especially useful for longer, more complex ontologies. * Any report values filled in will be displayed in the results JSON. * Example: `[{"id":"11c91068-73b5-4c32-bc25-d059fe9d4fc4","class":"Car non-occuluded","report_value":"nonoccluded_car","type":"box","coordinates":{"x":263,"y":438,"w":513,"h":368}}]` * **Important note:** Report value is currently only supported in ontologies of video annotation, image annotation, and image transcription jobs. * **Description** is ***optional***. This gives contributors a brief, inline description of a class. We highly recommend providing a clear and concise description of each class for the best quality results. *** ** * ** *** ### **Actions within the Ontology Manager:** 1. To add a new class, click 'Add Class'. This will insert another row below the last class created. 2. To delete a class, hover over its row in the table and click the trash icon to the right. 3. If desired, you can nest ontologies (create a hierarchical tree of classes to label) by dragging and dropping. 4. Once you've created your ontology, click 'Save'. 5. To edit the color, title, description, and output value for each class click on the pencil icon Important Note: Ontologies cannot be updated while a job is running/paused. ![image-20260615-084747.png](https://success.appen.com/__attachments/a_c53ba7082fc46c05d6d7e3e57201490d60dc666e16e0e59680cc63ce93da3d31/image-20260615-084747.png?cb=396ec94713ef8687c81ca9b73f73b128) Fig 2: Nesting Ontologies with the Ontology Manager ![image-20260615-084800.png](https://success.appen.com/__attachments/a_353d9562ca1504d7e15710f6aa32741952a42fa3b9cd11c3fb5792ccc8120598/image-20260615-084800.png?cb=5d2fd8e830cb4bad7e1c0a0dc563fb51) Fig. 3: Nested Ontology *** ** * ** *** ### **Ontology Manager Settings:** The settings allow you to configure how nesting should behave in the ontology. If you have nesting in your ontology and would like parent classes to act as directories (for an organization, rather than as an annotatable class), you can turn that on under the settings button. ![image-20260615-084806.png](https://success.appen.com/__attachments/a_2b25e8cf8360e3f8ec6661c6bd0a8b796c63800ad86c309d758e1faf48e60fe9/image-20260615-084806.png?cb=c4308390159ed4ff7a6f829274c2a0f8) Fig. 4: Ontology Settings *** ** * ** *** ### **Creating an ontology via CSV/JSON upload** You can also create an ontology by uploading a CSV (for flat ontologies) or JSON (for nested ontologies). To do so, you can follow these steps: 1. Create a UTF-8 encoded CSV with your ontology. You'll need the following column headers: * display_color (requires a hex code) * description * class_name 2. If you would like to upload a nested ontology as a JSON, the JSON format should follow the example of the file attached at the bottom of this article. 3. Click 'upload' from the Ontology Manager. 4. You can either drag your file in or upload it using the file selector. Once you've chosen your file, your ontology will be reflected in the UI. 5. If you encounter any file upload errors while uploading the ontology, please reach out to Appen Support (via chat or [help@appen.com](mailto:help@appen.com)) and attach the file you are trying to upload. ![image-20260615-084820.png](https://success.appen.com/__attachments/a_ebc6433407835e5be5ab2fbb6186b321a416c514af6279770515a52dcec1ac75/image-20260615-084820.png?cb=3a18c32c62a22d4646d42403064d7ebb) Fig. 5: CSV with Ontology Data To see how this ontology will look for contributors, preview the job: ![image-20260615-084828.png](https://success.appen.com/__attachments/a_80cfc67d92787933ed99e87d758ff395976751bb09b9f3a0df980576ae290e9c/image-20260615-084828.png?cb=b3cf377be83d0c824e6b3ef46ffe022a) Fig. 6: Preview of the bounding box tool ### **Moving Ontologies Between Jobs** In some cases, you may need to re-use the same ontology across multiple jobs. The fastest way to achieve this is: 1. Copying the job containing the ontology with no rows. 2. Make any edits to the ontology in the copy as needed. 3. Upload and run new data through the copy. You may also download the ontology (it will download as a .json file) and upload it to any other job. Uploaded ontologies should always overwrite existing ontologies. --- language: "en" --- # Guide to: Polygon Job Design and Aggregation The cml:shapes tag allows users to create an image annotation job for polygons in conjunction with a custom ontology and the use of test questions and aggregation. ## **Building a job** The following CML contains the possible parameters for a polygon job with labels: `` Note: There are parameters for test questions and aggregation that apply to both the polygons and the labels. ### **Parameters** Below are the parameters available for the cml:shapes tag. Some are required in the element, some can be left out. * `type` * The shape used in the job, set in an array. * To use multiple shapes in one job, include each shape in the array, separated by commas, e.g., 'type="\['box','dot','polygon','line'\]"' * You'll need to include the corresponding parameters for each shape * `source-data` * The column from your source data that contains the image URLs to be annotated. * `name` * The results header where annotations will be stored. * `label` * The question label contributors will see. * `validates` (optional) * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if not present * `ontology` (optional) * The list of classes to be labeled in an image - view [this article](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) to learn how to create your custom ontology. * Accepts a boolean * Defaults to 'false' if not present * `review-data` (optional) * This will read in existing annotations on an image. The format must match the output shown in the aggregation section below, with the exception of the class attribute (see example). All that's needed is the following: * 'type' * 'class' if using an ontology * 'coordinates' * 'id' * Example: \[{"class":"car","coordinates":\[{"x":724,"y":359},{"x":1098,"y":244},{"x":1273,"y":495},{"x":903,"y":753}\],"type":"polygon","id":"247f2099-22e1-4825-9bc1-3e51c6019fe0"}\] * `polygon-threshold` * The minimum overall polygon IoU required for a contributor to pass a test question. * Accepts a decimal value between 0.1 and 0.99. * `class-threshold` * The minimum percentage of correct classes applied to polygons in a test question for a contributor to be considered correct. * Accepts a decimal value between 0.1 and 0.99. * The formula is *correct / (correct + incorrect)* * Example: the class-threshold is set to 0.7 and a test question contains 10 ground truth shapes. A contributor gets 8 out of 10 classes correct for a score of 80% and they're marked correct on the test question. * `polygon-agg` * The minimum IoU required for result polygons to be clustered together. * Accepts a decimal between 0.1 and 0.99, or the value 'all'. * If 'all' is selected, no clustering is done on the polygons. * `class-agg` * The aggregation applied to the class for a given cluster of shapes. * Accepts standard aggregation types: * `agg` * `all` * `agg_x` * `cagg_x` * `output-format` (optional) * Accepts 'json' or 'url' * If 'json', the report column containing contributors' annotation data contains the annotation data in stringified JSON format. The JSON format is as follows (this is the legacy JSON format): * [ { "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "box", "coordinates": { "x": 416, "y": 243, "w": 125, "h": 95 } } ] * If 'url', the report column containing contributors' annotation data contains links to files. Each file contains annotation data for a single data row in JSON format. With this new output option, we have updated the JSON structure to allow inclusion of more data fields. The new JSON format is as follows: * * * { ableToAnnotate: true, imageRotation: 30, annotation: [{ "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "box", "coordinates": { "x": 416, "y": 243, "w": 125, "h": 95 } }]} * In the case where the tool was unable to load the input data and the contributor was unable to annotate, `ableToAnnotate` will be set to `false`. * Defaults to 'json' if attribute not present. * This parameter is available within the ***CML only***; it is not yet supported in the Graphical Editor. * `allow-image-rotation` (optional) * Accepts `true` or `false` * If `true`, contributors can rotate the image within the image annotation tool. Contributors click a toolbar icon to turn on a rotation slider that can be used to adjust rotation angle from 0 to 359 degrees. The degrees rotated are exported in the `imageRotation` field. This feature is only compatible with export option `output-format=url`; this attribute must be added to the job cml before launch. * **Important note:** Test questions and aggregation are not currently available for this annotation mode. * If `false`, contributors cannot rotate the image. * Defaults to `false` if attribute not present. * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. ### **Aggregation** #### **Polygons** Aggregation for polygons works as follows: * Polygons are clustered based on the IoU set in the polygon-agg parameter. * After clustering, each pixel in the overlapping polygons is given the trust score of each contributor, then each of these scores is summed for the total trust per pixel. * The average area of all the polygons in the cluster is calculated. * A shape is drawn around all the pixels with the highest total trust score. * The shape is expanded to lower trust pixels until the area of the aggregated polygon comes as close to the average area of the clustered polygons as possible. #### **Classes/labels** The class-agg parameter accepts the following [standard aggregation methods](https://success.figure-eight.com/hc/en-us/articles/203527635-CML-Attribute-Aggregation): * `agg` * `all` * `agg_x` * `cagg_x` Labels (or classes) are aggregated **per returned polygon** . This means, for example, if you choose to aggregate polygons - as opposed to selecting 'all' - and you choose `class-agg="agg"`, for each aggregated polygon you'd receive the **most confident** label out of the constituent polygons in the cluster. If you choose `class-agg="all"`, you'd receive every label applied to the cluster of polygons, but still just one polygon, and so on. For `polygon-agg="all"`, you'd receive every polygon and every label in the image, no aggregation. Labels will always be grouped with the shape they were applied to and will be returned in a dictionary. Example output of a job with `polygon-agg="0.6"` and `class-agg="agg"`: `[{"average_trust":0.9375,"iou":0.87,"class":{"car":1.0},"coordinates":[{"x":724,"y":359},{"x":1098,"y":244},{"x":1273,"y":495},{"x":903,"y":753}],"type":"polygon"}]` Example output of a job with `polygon-agg="0.6"` and `class-agg="all"`: `[{"average_trust":0.9375,"iou":0.87,"class":{"car":0.33,"person":0.33,"tree":0.33},"coordinates":[{"x":724,"y":359},{"x":1098,"y":244},{"x":1273,"y":495},{"x":903,"y":753}],"type":"polygon"}]` ### **Reviewing results** To review the results of your job: 1. Go to the Data page. 2. Click on a unit ID. 3. In the sidebar of the annotation tool, select an option from the drop down menu. 1. You'll see different contributor IDs, which allow you to view individual annotations. 2. You'll also see an "aggregated" option, which shows you the result you'll get based on your aggregation settings in the CML or report options page of your job. --- language: "en" --- # Guide to: Polylines Job Design and Aggregation The `cml:shapes` tag allows users to create an image annotation job for polylines in conjunction with a custom ontology and the use of test questions and aggregation. ## **Building a job** The following CML contains the possible parameters for a polylines job with labels: `` Note: There are parameters for test questions and aggregation that apply to both the polylines and the labels. ### **Parameters** Below are the parameters available for the cml:shapes tag. Some are required in the element, some can be left out. * `type` * The shape used in the job, set in an array. * To use multiple shapes in one job, include each shape in the array, separated by commas, e.g., 'type="\['box','dot','polygon','line','ellipse'\]"' * You'll need to include the corresponding parameters for each shape * `source-data` * The column from your source data that contains the image URLs to be annotated. * `name` * The results header where annotations will be stored. * `label` * The question label contributors will see. * `validates` (optional) * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if not present * `ontology` (optional) * The list of classes to be labeled in an image - view [this article](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) to learn how to create your custom ontology. * Accepts a boolean * Defaults to 'false' if not present * `review-data` (optional) * This will read in existing annotations on an image. The format must match the output shown in the aggregation section below, with the exception of the class attribute (see example). All that's needed is the following: * 'type' * 'class' if using an ontology * 'coordinates' * 'id' * Example: \[{"class":"car","coordinates":\[{"x":724,"y":359},{"x":1098,"y":244},{"x":1273,"y":495},{"x":903,"y":753}\],"type":"line","id":"95a0b08c-b621-4dda-b983-967fe11e384e"}\] * `line-distance` * The maximum pixel distance between a golden line and a contributor line. We use the Fréchet Distance. * Accepts an integer. * `line-threshold` * The minimum overall line IoU required for a contributor to pass a test question. * Accepts a decimal value between 0.1 and 0.99. * The formula is *correct / (correct + incorrect)* * Example: the line-threshold is set to 0.7 and a test question contains 10 ground truth shapes. A contributor gets 8 out of 10 classes correct for a score of 80% and they're marked correct on the test question. * `class-threshold` * The minimum percentage of correct classes applied to polylnes in a test question for a contributor to be considered correct. * Accepts a decimal value between 0.1 and 0.99. * The formula is *correct / (correct + incorrect)* * Example: the line-threshold is set to 0.7 and a test question contains 10 ground truth shapes. A contributor gets 8 out of 10 classes correct for a score of 80% and they're marked correct on the test question. * `line-agg` * The maximum pixel distance between result polylines to be clustered together, again this is the Fréchet Distance. * Accepts an integer or the value 'all'. * `class-agg` * The aggregation applied to the class for a given cluster of shapes. * Accepts standard aggregation types: * `agg` * `all` * `agg_x` * `cagg_x` * `output-format` (optional) * Accepts 'json' or 'url' * If 'json', the report column containing contributors' annotation data contains the annotation data in stringified JSON format. The JSON format is as follows (this is the legacy JSON format): * [ { "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "box", "coordinates": { "x": 416, "y": 243, "w": 125, "h": 95 } } ] * If 'url', the report column containing contributors' annotation data contains links to files. Each file contains annotation data for a single data row in JSON format. With this new output option, we have updated the JSON structure to allow inclusion of more data fields. The new JSON format is as follows: * { ableToAnnotate: true, imageRotation: 30, annotation: [{ "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "box", "coordinates": { "x": 416, "y": 243, "w": 125, "h": 95 } }]} * In the case where the tool was unable to load the input data and the contributor was unable to annotate, `ableToAnnotate` will be set to `false`. * Defaults to 'json' if attribute not present. * This parameter is available within the ***CML only***; it is not yet supported in the Graphical Editor. * `allow-image-rotation` (optional) * Accepts `true` or `false` * If `true`, contributors can rotate the image within the image annotation tool. Contributors click a toolbar icon to turn on a rotation slider that can be used to adjust rotation angle from 0 to 359 degrees. The degrees rotated are exported in the `imageRotation` field. This feature is only compatible with export option `output-format=url`; this attribute must be added to the job cml before launch. * **Important note:** Test questions and aggregation are not currently available for this annotation mode. * If `false`, contributors cannot rotate the image. * Defaults to `false` if attribute not present. * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. **Shape Type Limiter** * Limit which shapes can be used with certain classes ![image-20260622-102446.png](https://success.appen.com/__attachments/a_d556465ef6ff7beecf485134ebbb7f17dc0007ec1439f9c4c8964c9ec62e7d2a/image-20260622-102446.png?cb=ba7c38e7ed63b6cdb46aea6fc8f00b35) **Min/Max instance quantity** * Configure ontologies with instance limits ![image-20260622-102456.png](https://success.appen.com/__attachments/a_89eeabe4571f531a8a21cd3eb87eeeb8b23a1a17105ee5feafbfcff89fe51f20/image-20260622-102456.png?cb=643507c589cd80217a8437ed530dcc32) * Comes with the ability to mark the class as not present for long tail scenarios. This information will be added to the output as well. ![image-20260622-102511.png](https://success.appen.com/__attachments/a_54de8209ad88d420b0c63e669e6733529b6261bf7e912e2b3f04dc2d58f1534b/image-20260622-102511.png?cb=72139db81d21742325f819cf5b150b5c) **Customizable Hotkeys** * Hotkeys can be assigned to classes by the user. Hotkeys cannot conflict with any other browser or tool shortcuts. ![image-20260622-102521.png](https://success.appen.com/__attachments/a_327a3d95899fd26ac0f766bfb13f955b695013a4eb5a0f4d2c214cfc68264a50/image-20260622-102521.png?cb=2949d6816ad05ed5942bdd5a773d8caa) ### **Aggregation** #### **Polylines** Aggregation for polylines works as follows: * Polylines are clustered based on pixel distance set in the 'line-agg' parameter. * Each polyline in the cluster is divided into the same number of segments, creating the same number of anchor points for each polyline. * The anchor points are weighted based on contributor trust scores, and a new anchor point is returned * The averaged anchor points are connected to create a polyline. #### **Classes/labels** The class-agg parameter accepts the following [standard aggregation methods](https://success.figure-eight.com/hc/en-us/articles/203527635-CML-Attribute-Aggregation): * `agg` * `all` * `agg_x` * `cagg_x` Labels (or classes) are aggregated **per returned polyline** . This means, for example, if you choose to aggregate polylines - as opposed to selecting 'all' - and you choose `class-agg="agg"`, for each aggregated polyline you'd receive the **most confident** label out of the constituent polylines in the cluster. If you choose `class-agg="all"`, you'd receive every label applied to the cluster of polylines, but still just one polyline, and so on. For `line-agg="all"`, you'd receive every polyline and every label in the image, no aggregation. Labels will always be grouped with the shape they were applied to and will be returned in a dictionary. Example output of a job with `line-agg="0.6"` and `class-agg="agg"`: `[{"average_trust":0.9375,` `"class":{"car":1.0},"coordinates":[{"x":724,"y":359},{"x":1098,"y":244},{"x":1273,"y":495},{"x":903,"y":753}],"type":"line"}]` Example output of a job with `line-agg="0.6"` and `class-agg="all"`: `[{"average_trust":0.9375,` `"class":{"car":0.33,"person":0.33,"tree":0.33},"coordinates":[{"x":724,"y":359},{"x":1098,"y":244},{"x":1273,"y":495},{"x":903,"y":753}],"type":"line"}]` ### **Reviewing results** To review the results of your job, you can either use ourfeature (recommended), or the following: 1. Go to the Data page. 2. Click on a unit ID. 3. In the sidebar of the annotation tool, select an option from the drop-down menu. 1. You'll see different contributor IDs, which allow you to view individual annotations. 2. You'll also see an "aggregated" option, which shows you the result you'll get based on your aggregation settings in the CML or report options page of your job. --- language: "en" --- # Guide to: Running a Bounding Box Job with Labels The cml:shapes tag allows users to create an image annotation job for bounding boxes in conjunction with a custom ontology and the use of test questions and aggregation. ## **Building a Job** The following CML contains the possible parameters for a bounding box with labels job: `` Note: There are parameters for test questions and aggregation that apply to both the bounding boxes and the labels. ### **Parameters** Below are the parameters available for the `cml:shapes` tag. Some are required in the element, some can be left out. * `type` * The shape used in the job, set in an array. * To use multiple shapes in one job, include each shape in the array, separated by commas, e.g., `type="['box','dot','polygon','line']"` * You'll need to include the corresponding parameters for each shape * `source-data` * The column from your source data that contains the image URLs to be annotated. * `disable_image_boundaries`: allows shapes to be drawn outside the image boundary * when `disable_image_boundaries="true"` shapes can be dragged outside the image boundaries and the output will contain negative values * the default setting for `disable_image_boundaries`is "false" * `name` * The results header where annotations will be stored. * `label` * The question label contributors will see. * `validates` (optional) * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if not present * `ontology` (optional) * The list of classes to be labeled in an image - view this [article](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) to learn how to create your custom ontology. * Accepts 'true' or 'false' * Defaults to 'false' if not present * `review-data` (optional) * This will read in existing annotations on an image. The format must match the output shown in the aggregation section below. All that's needed is the following: * 'type' * 'class' if using an ontology * 'coordinates' * 'id' * Example: * `[{"class":"car","coordinates":{"h":330,"w":384,"x":1191,"y":306},"type":"box","id":"247f2099-22e1-4825-9bc1-3e51c6019fe0"}]` * `box-threshold` * The minimum overall bounding box IoU required for a contributor to pass a test question. * Accepts a decimal value between 0.1 and 0.99. * `class-threshold` * Example: the class-threshold is set to 0.7 and a test question contains 10 ground truth shapes. A contributor gets 8 out of 10 classes correct for a score of 80% and they're marked correct on the test question. * The minimum percentage of correct classes applied to boxes in a test question for a contributor to be considered correct. * Accepts a decimal value between 0.1 and 0.99. * The formula is *correct / (correct + incorrect)* * `box-agg` * The minimum IoU required for result boxes to be clustered together. * Accepts a decimal between 0.1 and 0.99, or the value 'all'. * If 'all' is selected, no clustering is done on the boxes. * `class-agg` * The aggregation applied to the class for a given cluster of shapes. * Accepts standard aggregation types: * `agg` * `all` * `agg_x` * `cagg_x` * `min-box-height`(optional) * Each box drawn by a contributor must be at least this height in pixels * Accepts a positive integer - must be at least 2 * `max-box-height`(optional) * The maximum possible height in pixels each box can be * Accepts a positive integer - must be at least 2 * Please note: If using the Graphical Editor, moving the slider all the way to 1000+ does not set a maximum height. To set the maximum larger than 1000px, you will need to set the max-box-height in the [Code Editor](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor.md). * `min-box-width`(optional) * Each box drawn by a contributor must be at least this width in pixels * Accepts a positive integer - must be at least 2 * `max-box-width`(optional) * The maximum possible width in pixels each box can be * Accepts a positive integer - must be at least 2 * Please note: If using the Graphical Editor, moving the slider all the way to 1000+ does not set a maximum width. To set the maximum larger than 1000px, you will need to set the max-box-width in the [Code Editor](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor.md). * `allow-box-rotation`(optional) * Will enable bounding boxes to be rotatable * Accepts 'true' or 'false' * Defaults to 'false' if not present * `crosshair`(optional) * Will enable crosshair location indication * Accepts 'true' or 'false' * Defaults to 'false' if not present * `box-aspect-ratio` (optional) * Will enable crosshair location indication * Controls the aspect ratio of the box * The ratio is width:height * Setting this to 1:1 enforces a square * Accepts ratio of integers, e.g., 2:1 * `output-format` (optional) * Accepts 'json' or 'url' * If 'json', the report column containing contributors' annotation data contains the annotation data in stringified JSON format. The JSON format is as follows (this is the legacy JSON format): * [ { "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "box", "coordinates": { "x": 416, "y": 243, "w": 125, "h": 95 } } ] * If 'url', the report column containing contributors' annotation data contains links to files. Each file contains annotation data for a single data row in JSON format. With this new output option, we have updated the JSON structure to allow inclusion of more data fields. The new JSON format is as follows: * { ableToAnnotate: true, imageRotation: 30, annotation: [{ "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "box", "coordinates": { "x": 416, "y": 243, "w": 125, "h": 95 } }] } * In the case where the tool was unable to load the input data and the contributor was unable to annotate, `ableToAnnotate` will be set to `false`. * Defaults to 'json' if attribute not present. * This parameter is available within the CML only; it is not yet supported in the Graphical Editor. * `allow-image-rotation` (optional) * Accepts `true` or `false` * If `true`, contributors can rotate the image within the image annotation tool. Contributors click a toolbar icon to turn on a rotation slider that can be used to adjust rotation angle from 0 to 359 degrees. The degrees rotated are exported in the `imageRotation` field. This feature is only compatible with export option `output-format=url`; this attribute must be added to the job cml before launch. * **Important note:** Test questions and aggregation are not currently available for this annotation mode. * If `false`, contributors cannot rotate the image. * Defaults to `false` if attribute not present. * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. ### **Ontology Configuration** The shapes tool supports ontologies. In addition, validators can be configured for each class in order to limit the number of shape instances contributors can create in that class. ![image-20260622-100537.png](https://success.appen.com/__attachments/a_0b48d739397dae660364cc7aa567cafe5f55a098024b5713768812d59275c4e8/image-20260622-100537.png?cb=e40ca2eb4c2883c28ac8f21b67fcf2e3) *Fig. 1 Ontology Manager Configuration* **Customizable Hotkeys** * Hotkeys can be assigned to classes by the user. Hotkeys cannot conflict with any other browser or tool shortcuts. ![image-20260622-100607.png](https://success.appen.com/__attachments/a_4ded3267e13c46c00d33df6313bf2bcbcaa4eec5ff9565ef1d4a53b0c3cd18c3/image-20260622-100607.png?cb=2949d6816ad05ed5942bdd5a773d8caa) *Fig. 2 Hotkeys Configured in Ontology Manager* **Ontology Attributes** * Our ontology attributes feature allows users to collect additional metadata on shape objects. Visit this link to learn more about ontology attributes: [Guide To: Running a Job with Ontology Attributes](https://appen-external.atlassian.net/wiki/spaces/ASC/pages/6062254/Guide+to+Running+a+Job+with+Ontology+Attributes+keep+archive). ![image-20260622-100628.png](https://success.appen.com/__attachments/a_3869bd079347cf7f40ed9160e1a39b5b72d31dd8cfe3f171511ab8b3a19fb44c/image-20260622-100628.png?cb=20bdbfa40ce28633d284bed875338919) *Fig. 3 Ontology Attributes Configured in Ontology Manager* **Creating test questions** When using the cml:shapes tag, the behavior of test questions and aggregation will change based on the shapes chosen and whether or not the job includes an ontology. 1. On the quality page, click "Create Test Questions" 2. Add annotations around the objects in the way specified in the job's instructions 3. If no annotations are needed, make sure the job includes an option, such as a single checkbox, to hide the annotation tool 4. Save Test Question ![image-20260622-100653.png](https://success.appen.com/__attachments/a_b14c3eb230562098e4bca93ed50f2981d4b1523e6f04aa919e48243fad418cb6/image-20260622-100653.png?cb=1a7097211a75b549e92794c7567b50fa) *Fig. 4 GUI View Shapes Tool* ### **Reviewing test questions** * Select a test question from the quality page. * From the image annotation sidebar, click 'Find a Judgment' and choose a contributor ID from the drop-down. * The overall contributor IOU will show in the left panel for that test question. This displays the average of all valid box IOUs and unmatched box IOUs (i.e. including *both* missed contributor and correctly matched gold boxes). If a contributor single box's IOU with a gold box is below the set threshold, the platform calculates the box as an automatic fail and treats it as 0% IOU to contribute to the overall IOU. * Example: Say there is a job with a box threshold of .7 and one test question has two objects to box. If one object has an IOU of 75% and the other box has an IOU of 45%, the overall IOU would be as follows: * (.75 + 0 (from the missed contributor box since it is below the 70% threshold) + 0 (from the missed gold box) / 3 (the total number of missed/drawn boxes)) = 0.25. * Edit, create or remove the test question annotations based on feedback. Judgments are color-coded based on if they match the gold responses. * Each shape will have its own matching metrics, which can be seen by hovering over a contributor judgment or golden shape. A notification will appear in the top left corner of the image. A score from zero to one is displayed based on the [intersection over union formula](https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/). If using an ontology, the class match is also displayed. * All scores on the image are averaged and compared to the test question threshold set in the job design. The overall matching score is then displayed in the left sidebar of the tool. * Save any edits that are made to update the evaluation of the existing contributors' work and ensure any future attempts to answer the test question will be properly evaluated. ![image-20260622-100728.png](https://success.appen.com/__attachments/a_69a67a9ad0c3337d04b0a09057e91b75a081b7579af06d56af33c516a556e765/image-20260622-100728.png?cb=d4cab96a9eaea09f618a2bc0d5468914) *Fig. 5 Test Question Scores* ### **Aggregation** #### **Boxes** * Aggregation for bounding boxes using cml:shapes works as follows: * Set the `box-agg` parameter in the CML, which is the IoU (Intersection over Union) used for clustering boxes prior to aggregation. * IoU: Intersection over Union is the metric that evaluates the similarity between multiple bounding boxes. * For example, if the `box-agg` is 0.6, boxes that overlap each other by at least 60% will be clustered together. * The corners of the constituent boxes are taken as dots and weighted according to the trust scores of the contributors who drew them. * The 'dots' are then aggregated and a new box is drawn. * When using rotated bounding boxes: * The calculation of the IoU (Intersection over Union) takes into account the following parameters: * **Height**: the height of the bounding box * **Width**: the width of the bounding box * **Coordinates**: the X and Y coordinates of the top left corner of the bounding box * **Rotation**: the rotation angle of the bounding box. The rotation degrees are from the box center. * The aggregation method is the average of all these parameters, while also taking into account the contributor's overall confidence score. * Note: the `"allow-box-rotation"` parameter needs to be set in order for this aggregation to occur. * Example output when using rotated boxes: * Full Report: `[{"id":"a5528478-fe30-4f03-8e97-130bb29fda5c","class":"Boat","type":"box","angle":341,"coordinates":{"x":37,"y":147,"w":100,"h":27}},{"id":"c541f90d-e63e-4968-9e68-1b37936239c2","class":"Boat","type":"box","angle":316,"coordinates":{"x":313,"y":213,"w":62,"h":19}}]` * Aggregated Report:`[{"angle":38.11111111111111,"average_trust":0.6,"class":{"eye":1.0},"coordinates":{"h":80,"w":68,"x":781,"y":219},"iou":0.93,"type":"box"}]` * For a more in-depth guide for image annotation aggregation visit this link: [Guide to: Image Annotation Aggregation](https://success.appen.com/appen-success-center/dashboards-reports/results/guide-to-image-annotation-aggregation.md) #### **Labels (Classes)** The `class-agg` parameter accepts the following [standard aggregation methods](https://success.appen.com/appen-success-center/dashboards-reports/reports-in-quality-flow/guide-to-aggregation.md): * `agg` * `all` * `agg_x` * `cagg_x` Labels (or classes) are aggregated **per returned box**. This means, for example, if you choose to aggregate boxes - as opposed to selecting 'all' - and you choose `class-agg="agg"`, for each aggregated box you'd receive the **most confident** label out of the constituent boxes in the cluster. If you choose `class-agg="all"`, you'd receive every label applied to the cluster of boxes, but still just one box, and so on. For `box-agg="all"`, you'd receive every box and every label in the image, no aggregation. Labels will always be grouped with the shape they were applied to and will be returned in a dictionary. Example output of a job with `box-agg="0.6"` and `class-agg="agg"`: `[{"average_trust":0.7857,"class":{"car":1.0},"coordinates":{"h":330,"w":384,"x":1191,"y":306},"iou":0.9628,"type":"box"}]` Example output of a job with box-agg="0.6" and class-agg="all": `[{"average_trust":0.7857,"class":{"car":0.33,"person":0.33,"tree":0.33},"coordinates":{"h":330,"w":384,"x":1191,"y":306},"iou":0.9628,"type":"box"}]` ### **Reviewing results** To review the results of your job, you can either use ourfeature (recommended), or the following: 1. Go to the Data page. 2. Click on a unit ID. 3. In the sidebar of the annotation tool, select an option from the drop-down menu. 1. You'll see different contributor IDs, which allow you to view individual annotations. 2. You'll also see an "aggregated" option, which shows you the result you'll get based on your aggregation settings in the CML or report options page of your job. --- language: "en" --- # Guide to: Running a Dots Job with Labels The cml:shapes tag allows users to create an image annotation job for dots in conjunction with a custom ontology and the use of test questions and aggregation. ## Building a job The following CML contains the possible parameters for a dot with labels job: `` There are parameters for test questions and aggregation that apply to both the dots and the labels. ### Parameters Below are the parameters available for the cml:shapes tag. Some are required in the element, some can be left out. * `type` * The shape used in the job, set in an array. * To use multiple shapes in one job, include each shape in the array, separated by commas, e.g., 'type="\['box','dot','polygon','line'\]"' * You'll need to include the corresponding parameters for each shape * `source-data` * The column from your source data that contains the image URLs to be annotated. * `name` * The results header where annotations will be stored. * `label` * The question label contributors will see. * `validates` (optional) * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if not present * `ontology` (optional) * The list of classes to be labeled in an image - [view this article](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) to learn how to create your custom ontology. * Accepts a boolean. * Defaults to 'false' if not present. * `review-data` (optional) * This will read in existing annotations on an image. The format must match the output shown in the aggregation section below. All that's needed is the following: * 'type' * 'class' if using an ontology * 'coordinates' * 'id' * Example: \[{"class":"car","coordinates":{"x":903,"y":753},"type":"dot","id":"95a0b08c-b621-4dda-b983-967fe11e384e"}\] * `dot-distance` * The maximum pixel distance between a test question dot and a contributor's dot in order for the dot to be considered correct. * Accepts an integer. * `dot-threshold` * The minimum percentage of correct dots in a test question for a contributor to be considered correct. Accepts a decimal value between 0.1 and 0.99. * The formula is *correct / (correct + incorrect)* * Example: the dot-threshold is set to 0.7 and a test question contains 10 ground truth dots. A contributor gets 8 out of 10 dots correct for a score of 80% and they're marked correct on the test question. * `dot-agg` * The maximum distance between result dots to be clustered together. Accepts an integer or the value 'all'. * If 'all' is selected, no clustering is done on the dots. * `class-threshold` * The minimum percentage of correct classes applied to dots in a test question for a contributor to be considered correct. Accepts a decimal value between 0.1 and 0.99. * The formula is *correct / (correct + incorrect)* * Example: the class-threshold is set to 0.7 and a test question contains 10 ground truth shapes. A contributor gets 8 out of 10 classes correct for a score of 80% and they're marked correct on the test question. * `class-agg` * The aggregation applied to the class for a given cluster of shapes. Accepts standard aggregation types: * `agg` * `all` * `agg_x` * `cagg_x` * `output-format` (optional) * Accepts 'json' or 'url' * If 'json', the report column containing contributors' annotation data contains the annotation data in stringified JSON format. The JSON format is as follows (this is the legacy JSON format): * [ { "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "dot", "coordinates": { "x": 416, "y": 243, } } ] * If 'url', the report column containing contributors' annotation data contains links to files. Each file contains annotation data for a single data row in JSON format. With this new output option, we have updated the JSON structure to allow inclusion of more data fields. The new JSON format is as follows: * { ableToAnnotate: true, imageRotation: 30, annotation: [{ "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "dot", "coordinates": { "x": 416, "y": 243, } }]} * In the case where the tool was unable to load the input data and the contributor was unable to annotate, `ableToAnnotate` will be set to `false`. * Defaults to 'json' if attribute not present. * This parameter is available within the ***CML only***; it is not yet supported in the Graphical Editor. * `allow-image-rotation` (optional) * Accepts `true` or `false` * If `true`, contributors can rotate the image within the image annotation tool. Contributors click a toolbar icon to turn on a rotation slider that can be used to adjust rotation angle from 0 to 359 degrees. The degrees rotated are exported in the `imageRotation` field. This feature is only compatible with export option `output-format=url`; this attribute must be added to the job cml before launch. * **Important note:** Test questions and aggregation are not currently available for this annotation mode. * If `false`, contributors cannot rotate the image. * Defaults to `false` if attribute not present. * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. **Shape Type Limiter** * Limit which shapes can be used with certain classes ![image-20260622-102639.png](https://success.appen.com/__attachments/a_1fa1d6aa86de05b56b1c51e69128afa18bc326bf57f16689871e90b41a6fe390/image-20260622-102639.png?cb=ba7c38e7ed63b6cdb46aea6fc8f00b35) **Min/Max instance quantity** * Configure ontologies with instance limits ![image-20260622-102648.png](https://success.appen.com/__attachments/a_e282aef2a2903f42d1244d6b18303751547cc850328c1345049490e62dadc888/image-20260622-102648.png?cb=643507c589cd80217a8437ed530dcc32) * Comes with the ability to mark the class as not present for long tail scenarios. This information will be added to the output as well. ![image-20260622-102655.png](https://success.appen.com/__attachments/a_65c410234cf4360dbb853fe0cf2e8cff3fce0fec36925eeaeb9ef440118c25f3/image-20260622-102655.png?cb=72139db81d21742325f819cf5b150b5c) **Customizable Hotkeys** * Hotkeys can be assigned to classes by the user. Hotkeys cannot conflict with any other browser or tool shortcuts. ![image-20260622-102748.png](https://success.appen.com/__attachments/a_8c5f40912c6b247dc170e32dc625dfd4191eaf352b7302e73d548e437743e841/image-20260622-102748.png?cb=2949d6816ad05ed5942bdd5a773d8caa) ### Creating test questions When using the cml:shapes tag, the behavior of test questions and aggregation will change based on the shapes chosen and whether or not your job includes an ontology. 1. On the quality page, click "Create Test Questions" 2. Add dots on the objects in the way you specified via your job's instructions. If no annotations are needed, make sure your job includes an option to hide the annotation tool. 3. Save Test Question. ### Reviewing test questions 1. Select a test question from the quality page. 2. From the image annotation sidebar, click 'Find a Judgment' and choose a contributor ID from the drop-down. 3. Edit, create or remove your own annotations based on feedback. Judgments are color coded based on if they match the gold responses. * Each dot will have its own matching metrics, which you can see by hovering over a contributor judgment or golden dot. A notification will appear in the top left corner of the image. The pixel distance between the contributor and the golden dot is shown. If using an ontology, the class match is also displayed. * The overall matching score as described above is displayed in the left sidebar of the tool. 4. Save any edits that are made to update the evaluation of the existing contributors' work and ensure any future attempts to answer the test question will be properly evaluated. ![unnamed.gif](https://success.appen.com/__attachments/a_a250672c63c1252029a35d1a13589e8269481c37b78660b32c7817f194107924/4274bae2ea0f96ff_unnamed.gif?cb=9fbf50f03c0c99116fac20979382dfd6) Fig. 1 test question scores ### Aggregation #### Dots Aggregation for dots using cml:shapes works as follows: * You'll set the `dot-agg` parameter in the CML, which is the pixel distance used for clustering dots prior to aggregation. * For example, if the `dot-agg` is 10, dots that are within 10 pixels of each other will be clustered together. * Each dot has a contributor trust score associated with it. * The dots are weighted by the trust score, then aggregated, and a new dot is returned. The aggregated dot includes the average trust score of all contributor trust scores in the cluster. #### Classes/labels The `class-agg` parameter accepts the following standard aggregation methods: `agg`, `all, agg_x`, `cagg_x` Labels (or classes) are aggregated per returned dot. This means, for example, if you choose to aggregate dots - as opposed to selecting 'all' - and you choose, `class-agg="agg"` for each aggregated dot you'd receive the most confident label out of the constituent dots in the cluster. If you choose, `class-agg="all"` you'd receive every label applied to the cluster of dots, but still just one dot, and so on. For, `dot-agg="all"` you'd receive every dot and every label in the image, no aggregation. Labels will always be grouped with the shape they were applied to and will be returned in a dictionary. Example output of a job with `dot-agg="10"` and `class-agg="agg"`:`[{"average_trust":0.7857,"class":{"car":1.0},"coordinates":{"x":1191,"y":306},"type":"dot"}]` Example output of a job with `dot-agg="10"` and `class-agg="all"`:`[{"average_trust":0.7857,"class":{"car":0.33,"person":0.33,"tree":0.33},"coordinates":{"x":1191,"y":306},"type":"dot"}]` ### **Reviewing results** To review the results of your job, you can either use ourfeature (recommended), or the following: 1. Go to the Data page. 2. Click on a unit ID. 3. In the sidebar of the annotation tool, select an option from the drop-down menu. 1. You'll see different contributor IDs, which allow you to view individual annotations. 2. You'll also see an "aggregated" option, which shows you the result you'll get based on your aggregation settings in the CML or report options page of your job. --- language: "en" --- # Guide to: Running a Job with Ontology Attributes ## Overview Our ontology attributes feature allows users to collect additional metadata on jobs using the image and video annotation tools. Important Note: As this tool is currently in BETA, it must be enabled for your team for access; please contact your Customer Success Manager or Account Executive for more information. *** ** * ** *** ## Building a Job CML The ontology attributes feature can be used in any cml:shapes and cml:video_shapes type job. Please see our other articles detailing how to for run jobs for specific cml:shapes types: * [Bounding Boxes](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-bounding-box-job-with-labels.md) * [Polygons](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-polygon-job-design-and-aggregation.md) * [Dots](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-dots-job-with-labels.md) * [Lines](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-polylines-job-design-and-aggregation.md) * [Ellipses](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-an-ellipse-or-circle-annotation-job.md) * [Shapes Peer Review](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-shapes-peer-review-job.md) *** ** * ** *** ## Ontology Manager ### Creating an Ontology Via the UI When enabled, ontology attributes can be accessed via the [Ontology Manager](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) page of a job and configured per class by clicking the pencil icon next to the desired class. ![blobid0.png](https://success.appen.com/__attachments/a_8a4954dffe5cf7a24ed905e613424a09a8df4729f1b89994f335763bd669183c/1fc44f869958a9ab_blobid0.png?cb=0cc9b6222377ab80c8105f1ddeb893ac) *Fig. 1: Selecting a class to edit via the Ontology Manager* After selecting the pencil icon, a modal will appear with the ontology attributes section to configure questions to collect attribute information. The graphical editor follows the graphical editor used for creating questions in job designs (please see [Guide to: The Graphical Editor](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-graphical-editor.md)). ![blobid1.png](https://success.appen.com/__attachments/a_d9e9ad09af920416115feee2fde76a868d06b70824961590caae9975937bc183/f84ad724b94cad39_blobid1.png?cb=a5784ec8550061170b5d1ff2d10c1359) *Fig. 2: Ontology Attributes section and question type options* ![blobid2.png](https://success.appen.com/__attachments/a_eef2013e65f0a979b8f25933a5bbfc7694f28933bc352f40e85d417f76ada2bc/b6a378d3098455e8_blobid2.png?cb=13e06c34256872f3c2b129337fd7a555) *Fig. 3: Example question set-up with Ontology Attributes* *** ** * ** *** ## Supported Questions ### Single Checkbox ![ontology1.png](https://success.appen.com/__attachments/a_32a26b76c1ee4349c088c3001c48a57920d212fa052d410474aba3a0910ee952/f542f5ddcb6561b3_ontology1.png?cb=84deab1e2ea8c6c18d1179aff4e6292b) *Fig. 4: Example question set-up with Ontology Attributes* ### Checkbox Group and Multiple Choice In Checkbox Group users can select all possible options, whereas in Multiple Choice there's only one option that can be selected, both of them have a maximum of 5 options per question. ![ontology2.png](https://success.appen.com/__attachments/a_f6bd53ef25e9821b25d48e63d7ea7a256c1ddc07ec8444930a6ec038b43b10e2/06e78013ecffe9b9_ontology2.png?cb=e6a1f2d26662164fb210dd050f439da4) *Fig. 5: Example question set-up with Ontology Attributes* ### Pulldown Menu Follow the same UI as Checkbox Group and Multiple Choice, but with a limit of 50 possible options per question. ### Text Box (Single Line) This question is a single line text box, where when creating the job, it's possible to filter the output using the clean output field below, and choosing one of the available options ![ontology3.png](https://success.appen.com/__attachments/a_1c1d677ff4c29b73413e417829265ec6d4adb633eea1a4fb199abbc133a3e72a/e015ad9146e99203_ontology3.png?cb=e26f2aa447994e66e1cdd6b2511599aa) *Fig. 6: Example question set-up with Ontology Attributes* ![ontology4.png](https://success.appen.com/__attachments/a_0e664c674c9adb6fac9ea336e93af664f186c649e8d30d23f991f746c523ffba/dec4f6211669f1cb_ontology4.png?cb=ac825bac6cecd7896dc9b387e4e1cb5d) *Fig. 7: Example question set-up with Ontology Attributes* All questions contain this field, where is possible to: * Show tips/hints to annotators on the tool * Results Header that goes to the final report of the job * Clean Output\* **\* Clean output is only for Text Box (Single Line) type of question** *** ** * ** *** ## Supported Questions -- Annotator Side ![ontology5.png](https://success.appen.com/__attachments/a_382f590573ef69c9a2b61162943653d286fda62c4a60533a708257ab69f29e7a/3e335c8a664045a9_ontology5.png?cb=c9ae996c7a67a26f3f6344026673bebe) *Fig. 8: Example of questions on Annotation Side* *** ** * ** *** ## Creating an Ontology via JSON An ontology can also be created by uploaded a JSON file. We recommend creating your first ontology via the graphical interface, then downloading the JSON to understand structure and continue building on. Here is an example JSON file that corresponds to the question created above in the graphical editor. [ { "description": "", "class_name": "Vehicle", "questions": [], "display_color": "#FF1744", "children": [ { "description": "", "class_name": "Car", "questions": [ { "id": "bd5dda2e-2cbc-4526-8500-e1b0db143f8d", "type": "Multiple Choice", "data": { "questionText": "Is the car occluded?", "questionChoices": [ { "id": "2a3b6c0f-5597-4863-8e48-3b8751e69b49", "label": "Yes", "value": " yes" }, { "id": "ee7227c5-4667-4468-afdb-31ccf7767133", "label": "No", "value": " no" } ], "isRequired": true, "tipsHints": " Occlusion means the visibility of the car is obstructed." "resultsHeader": "occluded" } } ], "display_color": "#651FFF" } ] } ] For classes with ontology attributes, the following variables must be included unless stated otherwise: * `id`: Platform generated id for the question * `Questions`: This array hold the JSON objects that describe the attributes added to the class * `Type`: The type of question used for the attribute (i.e. Multiple Choice, Checkbox Group, etc.) * `Data`: a JSON object containing information on the attribute. The variables may change depending on the question type chosen. * Standard variables include: * `questionText`: Accepts a string for the question label that appears to the contributor * `questionChoices`: Accepts a string for the options to a question (i.e. for Multiple Choice and Checkbox Group) * `label`: Accepts a string for the option label shown to the contributor * `value`: Accepts a string for the option value stored in the output reports * `defaultsChecked`: Accepts a boolean value for Checkbox questions to default a checkbox as marked * `isRequired`: Accepts a boolean value for if a question is required * `tipsHints`: Accepts a string for a hint to give to the contributor * `resultsHeader`: Accepts a string for the output name of the question provided in the output reports * Additional Attributes for Single Text Line Inputs: * `cleanOutput`: Accepts a boolean value to enable clean output validators * `textCase`: If `cleanOutput` is set to true, this variable accepts "UPPER CASE", "lower case", "title case", or "None" * `trimSpaces`: If `cleanOutput` is set to true, this variable accepts "No whitespace", "No leading/trailing whitespace", "No multiple whitespaces", and "None" Please see the [Ontology Manger article](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) for information on standard ontology formats. ### Ontology Attributes Interpolation in Video When ontology attributes are added to a class in a video annotation job, contributors can take advantage of answer "interpolation" to maximize efficiency. When a contributor saves attributes in a job, the attributes will "interpolate" and apply to all future frames, up to the last frame where attributes were manually adjusted. For example, say a contributor is labeling a 15 frame clip. The contributor adds a shape to frame 1 and saves attributes. The contributor then updates the attributes in frame 10. If the contributor goes back and updates the attributes in frame 7, those attributes will then "interpolate" and apply to frames 8 and 9 since 10 is the next frame where attributes were manually adjusted. This is useful when capturing temporal tags such as occlusion or truncation. ### Peer Review In BETA, test questions are not supported, and peer review is a method to ensure quality. To create a peer review job, follow the guidelines in our [Guide to: Running a Shapes Peer Review Job](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-shapes-peer-review-job.md) article. ### Monitoring and Reviewing Results As this is a BETA feature, aggregation is not supported. Jobs should be run either to a trusted partner or in a peer review workflow (see above). * Answers can be read while monitoring, in a read only state. ### Results * The Ontology Attributes feature will append metadata per shape annotation in job results. * Within the `metadata` object per shape, there are `shapeAnswers` and `shapeQuestions` arrays to represent the attributes labeled for an object and information on the ontology available to the contributor. * Example shape annotationoutput from an ontology attributes job: {"id":"2069ebb8-3e3e-4b57-b056-00eddd1a7c78", "class":"Car", "type":"box", "annotated_by":"human",//video output only "coordinates":{ "x":298, "y":450, "w":410, "h":293 }, "metadata":{ "annotated_by":"human",//video output only "shapeAnswers":[ { "type":"Multiple Choice", "name":"occlusion", "answer":{ "values":"yes", "questionId": "02906416-37be-42ca-97dd-40f81b9f88ef" } } ], "shapeQuestions":[ { "id":"02906416-37be-42ca-97dd-40f81b9f88ef", "type":"Multiple Choice", "data":{ "questionText":"What is the level of occlusion?", "questionChoices":[ { "id":"a06aacd5-24bf-458f-9300-1af4666f5739", "label":"Yes", "value":"yes" }, { "id":"175bdde7-fc5b-4f6a-bfa5-788024a522d9", "label":"No", "value":"no" } ], "isRequired":true, "tipsHints":"", "resultsHeader":"occlusion" } } ] }} #### Each variable can be defined as the following: |---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Variable** | **Description** | | `id` | UUID assigned by the platform, can be applied to a shape annotation, question, or question response | | `class` | Ontology category assigned to the shape | | `type` | The type of shape used. Can be box, dot, line, or polygon. | | `coordinates` | The coordinates of the shape. * See specific[Success Center Shapes guides](https://success.appen.com/appen-success-center/create-design-jobs/cml-elements/cml-shapes-bounding-box-polygon-dot-annotation-and-line-annotation-tool.md) for coordinate details. | | `metadata` | An object containing information on the attribute answers and questions available in the job. | The **metadata** object includes the following: |-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Metadata variable** | **Description** | | `annotated_by` | Applicable only in video jobs. This can be "human" if a contributor manually saved inputs on the frame or "machine" if the answers were interpolated from the last frame where attributes were manually updated | | `shapeAnswers` | Object within metadata object containing answer information for specific attributes. Variables include: * `type`:Question form used (I.e. multiple choice or checkbox group) * `questionId`: ID from the question associated with the answer * `name`: Value inputted for `resultsHeader` in ontology manager file * `answer`: Object containing information on the answer to the attribute * `value`: Value of chosen answer | | `shapeQuestions` | Object within "metadata" object containing information on the attribute's questions and values. Additional variables not defined above include: * `data`: JSON object containing information on the question text and choices * `questionText`: Question label shown to contributors during annotation. * `questionChoices`: An array of the available choices to the question * `isRequired`: Boolean value that determines if a question is required or not * `tipsHints`: An optional hint provided to contributor during annotation * `resultsHeader`: Value inputted for resultsHeader in ontology manager file.Example annotation output from a Video Annotation ontology attributes job: | ### Example annotation output from a Video Annotation ontology attributes job: { "ableToAnnotate": true, "annotation": { "frames": { "1": { "frameRotation": 0, "rotatedBy": "machine", "shapesInstances": { "dce": { "annotated_by": "human", "height": 10, "width": 10, "x": 40, "y": 20, "metadata": { "annotated_by": "human", "shapeAnswers": [ { "type": "Checkbox", "questionId": "d2a0a51c-a060-4b0a-996d-291bfb784754", "name": "checkbox", "answer": { "values": false } } ] } } } } }, "shapes": { "dce": { "category": "Cat", "type": "box" } }, "ontologyAttributes": { "Cat": { "questions": [ { "data": { "defaultsChecked": false, "isRequired": false, "questionText": "Checkbox", "resultsHeader": "checkbox", "tipsHints": "Tip on hover" }, "id": "d2a0a51c-a060-4b0a-996d-291bfb784754", "type": "Checkbox" } ] } } }} The Video Annotation Tool Output has small differences to the Image Annotation |-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Metadata variable** | **Description** | | `ontologyAttributes` | Applicable only in video jobs. This contains the questions that on Image Annotation are inside each shape. Each question is stored inside the Ontology Name. | --- language: "en" --- # Guide to: Running a Shapes Peer Review Job ## Overview Peer review jobs allow contributors to review and edit the results from a previously run annotation job. Contributors can remove or update existing annotations, and add new annotations if needed. To upload annotations from previously run shapes jobs into the `cml:shapes` tool for a peer review job, you can follow these easy steps: ## Data * The shapes tool supports hosted image files. * A column including results from a previously run shapes job (either from the Aggregated or Full Report) or your own pre-made annotations (that are identically formatted as results from a `cml:shapes` job) is required. * An example source data file is attached below for an example of the formatting required. ## Building a Job The following CML contains the possible parameters for a shapes job with labels: ` ` Note: The `review-from` attribute, which is **required for peer review jobs**, is supported in by using CML in the Code Editor only; using the Graphical Editor is not currently supported. ## Parameters Below are the parameters available for the cml:shapes tag in a review job. Some are required in the element, while some can be left out. * `type` * The shape used in the job, set in an array. * To use multiple shapes in one job, include each shape in the array, separated by commas, e.g., `type="['box','dot','polygon','line']"` * You'll need to include the corresponding parameters for each shape. * `source-data` * The column from your source data that contains the image URLs to be annotated. * `disable_image_boundaries`: allows shapes to be drawn outside the image boundary * when `disable_image_boundaries="true"` shapes can be dragged outside the image boundaries and the output will contain negative values * the default setting for `disable_image_boundaries`is "false" * `name` * The results header where annotations will be stored. * `label` * The question label contributors will see. * `review-data` * This will read in existing annotations on an image. The format must match the output from the aggregated or full reports of a previously- run shapes job. The following components of the annotation are required: * 'type' * 'class' if using an ontology * 'coordinates' * 'id' * Example: * `[{"class":"car","coordinates":{"h":330,"w":384,"x":1191,"y":306},"type":"box","id":"247f2099-22e1-4825-9bc1-3e51c6019fe0"}] ` * `validates` (optional) * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if absent * `ontology` (optional) * The list of classes to be labeled in an image. Please see this [article](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) to learn how to create your custom ontology. * Accepts 'true' or 'false' * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. ![image-20260622-084845.png](https://success.appen.com/__attachments/a_73c318cad17950e5b6466a9c785a166745dc7eb183dc6cae3ca3c893a6709c33/image-20260622-084845.png?cb=22cec01d5fedaf7509b1ec2ae5bdae96) Fig. 1: Preview of a bounding box review job via Preview Page **Shape Type Limiter** * Limit which shapes can be used with certain classes ![image-20260622-084914.png](https://success.appen.com/__attachments/a_4858ec01f6c359048215b017e31952ab64b2ac5002e67fac1ccd06e5ca0f8367/image-20260622-084914.png?cb=ba7c38e7ed63b6cdb46aea6fc8f00b35) **Min/Max instance quantity** * Configure ontologies with instance limits ![image-20260622-084932.png](https://success.appen.com/__attachments/a_78b386762e4e71c764bf1b17998ad2dcbad04620b6d44481427fbed01a0db10a/image-20260622-084932.png?cb=643507c589cd80217a8437ed530dcc32) * Comes with the ability to mark the class as not present for long tail scenarios. This information will be added to the output as well. ![image-20260622-084945.png](https://success.appen.com/__attachments/a_71cb22fb160b854d0ae968069d925f403cfd2e29a5acc9532f607adbea7c9525/image-20260622-084945.png?cb=72139db81d21742325f819cf5b150b5c) **Customizable Hotkeys** * Hotkeys can be assigned to classes by the user. Hotkeys cannot conflict with any other browser or tool shortcuts. ![image-20260622-085000.png](https://success.appen.com/__attachments/a_efaf19e1293d046fe82a4e048734a62c6f2745afe9ca943b8f5c9e3239f66a9a/image-20260622-085000.png?cb=2949d6816ad05ed5942bdd5a773d8caa) After the data is uploaded and the job is built in the CML Editor, the job will follow the same process to launch and monitor like any other job in the platform. Please see our other articles detailing how to for run jobs for specific `cml:shapes` types: * [Bounding Boxes](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-bounding-box-job-with-labels.md) * [Polygons](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-polygon-job-design-and-aggregation.md) * [Dots](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-running-a-dots-job-with-labels.md) * [Lines](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-polylines-job-design-and-aggregation.md) --- language: "en" --- # Guide to: Running a Smart Text Collection Job ## Overview Smart Text includes a number of features to ensure high quality, customized and original data, including disabling copy/paste, minimum and maximum word counts, robust spelling and grammar checks, rich text and [detecting AI content](https://success.appen.com/appen-success-center/llm-data-products/guide-to-ai-detector.md). Smart Text will smooth the contributors' writing experience and ensure high-quality output, especially for jobs related to LLMs, such as creative writing prompt/response pairs and response improvement. In addition to disable pasting, outlined below, Smart Text is also compatible with the Basic Validators such as word and character counts, described in [this article](https://success.appen.com/appen-success-center/enhancements/guide-to-validators.md) and the Smart Validators, such as regex and spelling \& grammar, described in [this article](https://success.appen.com/appen-success-center/enhancements/guide-to-smart-validators.md). Smart Text autosaves every ten seconds, ensuring nothing is lost if contributors leave their task or encounter a crash. ## Job design From the side bar choose "Smart Text". ![image-20260622-065652.png](https://success.appen.com/__attachments/a_e2fc8fb511973d7c733c702f47752a23b3a562bd5ed5016ec2094a55d29e343c/image-20260622-065652.png?cb=11b916c3b3f8b71ac942792405aa1a4f) ## Disable Pasting Once you have chosen Smart Text you will see a checkbox "Disable Pasting". When pasting is disabled (`disable-pasting="true"`), contributors will not be able to paste information in the input text box, regardless of the origin of the information (another judgment, another document on their desktop, from their browser...). Copy/paste is disabled for right click, hotkeys, and keyboard shortcuts. ![image-20260622-065756.png](https://success.appen.com/__attachments/a_29826445d1ac899c8886a503535dea29b71090816281c8cb3aa37eeb98bdb425/image-20260622-065756.png?cb=bed953b5f16e585ef0888997fcb3442a) ## Rich Text Editor You are now able to design jobs using a Rich Text Editor (RTE). Using our RTE will enable your contributors to format their input text with the following: * Tables * Code blocks with syntax highlighting for HTML, SQL, Java, Javascript, and more * Math/science equation formatting using syntax for [LaTeX](https://www.cs.princeton.edu/courses/archive/spr10/cos433/Latex/latex-guide.pdf) * **Bold text** * Underlined text * *Italicized text* * Bulleted lists * Numbered lists When using Smart Text, Rich Text is enabled by default. Disable Rich Text by unticking the checkbox in the graphical editor. You can also edit the default cml attribute to `rich="false"`. ![image-20260622-065827.png](https://success.appen.com/__attachments/a_f3e89b1257a4d385f3ebc874fd5d991c5408ab9753e33ecb2ab0fccf3353afd4/image-20260622-065827.png?cb=880373a8017efe13279d987d1897d8df) Note: The "Undo" capability is currently only supported when `rich="true"` is enabled. To undo, contributors can click the back arrow button or `command+z` on their keyboard. **Note:** We have a known limitation - the AI Chat Feedback Tool should NOT be configured and used in the same job design as a Smart Text element that utilizes review data. This will cause issues in the AI Chat Feedback Tool's prompt input box. *** ** * ** *** ### Parameters * `rich="true"` (optional, defaults to "true"): * this will include rich text in your smart text * `review-data="{{review_data_column}}"` and `task-type="qa"` (optional): * This parameter enables the loading of an annotation within the smart text tool. When the contributor loads the judgment, they will see a pre-annotation in the tool and have the option to make changes before submitting. * For the smart text tool, `review-data` must reference data in specific formats. Supported formats include: * Raw text, HTML, or Markdown within the dataset column * `.txt` files * `.html` files * `.md` files * A CDS reference pointing to plain text or one of the above file formats * `equation="true"` (optional, defaults to "false") * This parameter allows contributors to type in LaTeX syntax using a dollar sign (`$`) as a wrapper, which will render the LaTeX automatically within the input box of the tool. Contributors can also copy and paste content correctly into the text box, with the content rendering automatically. * **Note:** to include a column in your output that translates everything in the smart text box to LaTeX syntax, refer to the `raw-output` parameter below. * `raw-output="true"` (optional, defaults to "false"): * if `raw-output="true"`, the output will include HTML, Markdown and LaTeX columns along with raw output. * if `raw-output="false"`, the output will only include raw text * `raw-output="['HTML', 'Markdown', 'Latex']"`: * use one or multiple options to configure only certain columns to be included in the output * **Note:** for LaTeX to output make sure `equation="true"` * `model-annotation="CML_MODEL_NAME"` (optional): * This parameter allows you to present a model response within the `cml:smart_text` element, learn more in [this article](https://success.appen.com/appen-success-center/llm-data-products/llm-data-products-user-guides/guide-to-model-mate-flexible-model-integration.md) * `read-mode="true"` (optional, defaults to "false"): * When enabled, contributors will not be able to edit the content within the text box. This mode is intended for presenting information to contributors using the `review-data` parameter. * By default, `read-mode`is set to `false` . ### Rich Text Output Format { ableToAnnotate: , annotations: { text: "...", rawContent: "...", contentType: "html" }, metadata: { ... } } When using the results report, you will also be able to visualize the raw text without html markup for readability. In [Quality Flow](https://success.appen.com/appen-success-center/getting-started/guide-to-quality-flow-project-set-up.md), any input text formatted with the Rich Text Editor will be displayed in subsequent jobs as formatted by the initial contributor. The reviewer will be able to modify the formatting as needed to improve the output quality. ## Job Report Refer to [this article](https://success.appen.com/appen-success-center/dashboards-reports/annotation-tools-job-report.md) for information on Annotation Tools Job Reports. --- language: "en" --- # Guide to: Running a Text Relationships Job ## Overview The Text Relationships tool (`cml:text_relationships`) allows users to create a job that annotates relationships between spans of text with a custom ontology, please contact your Appen representative to gain access. ![Screen_Shot_2020-05-29_at_12.41.29_PM.png](https://success.appen.com/__attachments/a_747fd8ebc0bfc1d919a4fadda568c40b1af0cccb0249d2a5c7d438896602404e/def9044c16c04824_Screen_Shot_2020-05-29_at_12.41.29_PM.png?cb=969947caea8b434943939289b91722b4) Figure 1: Text Relationships Tool via Preview Page Glossary * **Span** - a string of text with an assigned class label; the output of a model or contributor judgment. * **Relationship** - consists of two spans (from a span and to a span) and the relation between them. * **Relation** - the name/type of the relation between two spans as defined in the job's ontology. * **From span**- the starting span in a relationship * **To span** -the ending span in a relationship ## Upload Data The source data of a Text Relationships job can come from two different sources: * The output of a previous text annotation job on the Appen platform * The output of a previously ran text annotation job on the Appen platform can be uploaded to a text relationships job directly without any modification. * Data created externally * A source file containing data created externally can be uploaded to a text relationships job, but an identical JSON format (hosted in a URL and a CORS configured bucket) as the output of a text annotation job on the Appen platform will be required. **Note:** There is an example file attached below on how to format source data. ## Build a Job ### Parameters Below are the parameters available for the text relationships tool. Some are required in the element, some can be left out. * `source-data` (required) * The name of the source data column containing the data to be annotated * `name` (required) * The result header where the result links will be stored * `context-column`(optional) * The name of the source data column containing the context of each data row * `review-data` (optional) * The name of the source data column containing pre-annotated text relationships * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. * `direction` (optional): * Renders text in a specific direction * Accepts `rtl` and `ltr` for right-to-left and left-to-right scripts respectively * Defaults to left-to-right if not set ### Ontology * The [Ontology Manager](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) allows job owners to create and edit the ontology within a Text Relationships job. Text Relationships Jobs require an ontology to launch. * When the CML for a text relationships job is saved, the Ontology Manager link will appear at the top right of the Design page. * The ontology of Text Relationships jobs allows you to create relationship restrictions to each span class. * The ontology of a Text Relationships job can be copied from a Text Annotation job or another Text Relationships job, via download and upload. ![2020-06-09_08.58.56.gif](https://success.appen.com/__attachments/a_396b32ac066dc02e0aa654c777688ef55415edbe898d21ad1057f12a35e95437/c90e3544246f9109_2020-06-09_08.58.56.gif?cb=f2ad37e49945cb159cbdd807d26ffcdd) Figure 2: Ontology Manager for Text Relationships Ontology Manager Best Practices * The limit of ontology is 1,000 classes, however, as best practice, we recommend not exceeding 16 classes in a job to ensure contributors can understand and process the different classes. * Choose from 16 colors pre-selected or upload custom colors as hex code via the CSV ontology upload. **Important Note**: If there is no relationship restriction defined to a class, the class will not able to relate to any other classes in the job. ### Nested Spans Text Relationships tool (`cml:text_relationships`) supports nested spans. However, there is one restriction for creating relationships for nested spans. **Consider this example:** Here we have one root span ![rootspan.png](https://success.appen.com/__attachments/a_c54d3b118c102db25ed990586666c89da3ed07137794eb42fd5f94f3ca882fab/b9e8006e98e7e668_rootspan.png?cb=105198870502660d3159c8b10e4dbbdf) and two sub spans. ![sub_span.png](https://success.appen.com/__attachments/a_e50f0100d7c7252840310e3db753bc526cc45bab8ccd55e367573be951451523/51df0668210ff925_sub_span.png?cb=99335312668c58ae05d3857b726db2e7) We can create Relationship between sub spans ![relationship-spans.png](https://success.appen.com/__attachments/a_35b249af582787a6ed85c4fa03553bccf710edca931747d94b95a0ed684dbe1a/16233bb2ca6e959c_relationship-spans.png?cb=9f2b71efe205e8df9367ef956d63a0c8) But between the root span and the sub span we cannot create a relationship. ## Results * Results are links to a JSON file that contains a list of relationships. * The links are found in the Full or Aggregated report under the column header that was specified as the value for the name attribute. * Result links will expire 15 days after generation; to access results after the links have expired, you will need to re-generate the result report. * Each relationship instance is an array of five attributes: * `id`: the unique ID of each relationship instance * `name`: the class name of the relation * `from_span`: contains the details of from_span * `to_span`: contains the details of to_span * `annotated_by`: indicates whether a relationship instance is pre-loaded into the job or manually added by an annotator --- language: "en" --- # Guide to: Running a Tiled Image Annotation Job ## Overview The new tiled image annotation tool allows users to request annotation of tiled imagery data according to a custom ontology. If you would like to set up a tiled image annotation job, please contact your Customer Success Manager. ![image-20260626-051219.png](https://success.appen.com/__attachments/a_41973b050523af347b139522d76a738d2df10b1783f3d07a982a91d6b8ec397a/image-20260626-051219.png?cb=077bd7831d99a0457ca1deeabeee1250) The tiled image annotation tool enables contributors to annotate Slippy Map tiles of the earth at various zoom levels. A 'Slippy Map' is a modern web map that allows users to zoom and pan to display different parts of the map at different levels of resolution. In order to use the tiled image annotation tool, the map data must already be converted into map tiles and hosted using a tile server. ## Building a Job ### CML Currently, there is no Graphical Editor support for this tool. Here is sample CML to build the job: ![image-20260626-051255.png](https://success.appen.com/__attachments/a_b5a1e2c37e5d89dd04072d4862e8a6bfc8501da3cbda1c9eabd47252309029b1/image-20260626-051255.png?cb=5e368ba266655602817c24c26ad36ca1) ### **Parameters** Below are the parameters available for the tiled image annotation tool. Some are required in the element, while others are optional. * `type` * This parameter allows you to add various types of annotation modes to the tool, including "box", "polygon", "line", and "dot". * `source-data` * The column from your source data that contains the URLs to the hosted XYZ image tiles. The URLs must be in the following format: [https://c.tile.openstreetmap.org/{z}/{x}/{y}.png](https://c.tile.openstreetmap.org/%7Bz%7D/%7Bx%7D/%7By%7D.png). * To add multiple tile layers, supply an array containing the URLs in the following format: * \[{"url": "https://...", "name": "Layer1"}, {"url": "https://...", "name": "Layer2", "options": {"max_zoom": 20}}\] * See below for additional information about using `source-data` with Cloud-Optimized GeoTiff (COG) files. * The following options can be configured independently for each layer: min_zoom, max_zoom, max_native_zoom, min_native_zoom, subdomains * * ![image-20260626-051345.png](https://success.appen.com/__attachments/a_343b366d4e0207df943d28fa31e526ec4c2bd2c79bdc9f41be7e2bd70fd2f2b9/image-20260626-051345.png?cb=f75ec1a9eb7637fd3ff04dd8e1d7ef5d) Figure 2. Contributors will be able to toggle between layers in the tool. * `canvas-options` * The column from your source data that contains canvas configuration options. * The following options can be configured independently for each data row: * center (required): the latitude-longitude coordinates at the map center upon tool load * zoom (required): the initial map zoom level upon tool load * maxZoom: the maximum zoom level of the map * minZoom: the minimum zoom level of the map * maxBounds: When this option is set, the map restricts the view to the given geographical bounds. Supply the bottom-left and top-right latitude-longitude coordinates of the rectangular boundary region. * Provide configuration options for each data row as an object with the following format: * {"center":\[30.3321,-81.6556\],"zoom":11,"min_zoom":4,"max_zoom":18,"max_bounds":\[\[30.074618,-81.959666\],\[30.509033,-81.41833\]\]} * `name` * * The results header where annotations will be stored. * `label` * * The question label the contributors will see. * `validates` * * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if not present * `review-data ` * This will read in existing annotations. For this job type, they will need to be URLs or refs, each linked to the annotation file for the original annotation task. ### **GeoTiff support** The tiled image annotation tool supports loading Cloud-Optimized GeoTIFF (COG) files remotely. * GeoTIFF files are split into bands, each of which has a matrix of data values. The tool can either display a single band of values by mapping the values to a specified color map (e.g. grayscale). Or, three bands can be provided, representing red, green, and blue, which results in a true-color visualization. * To configure GeoTIFF support, set the `source-data` parameter. Below is an example configuration in which Layer 1 displays a single band, Layer 2 displays RGB using separate band files, and Layer 3 displays RGB using a single file containing multiple bands. Possible values for colormap can be found [here](https://github.com/gka/chroma.js/blob/main/src/colors/colorbrewer.js#L18). ### `source_data`**Format Guide** #### **Option 1: Simple URL String (Auto-Detection)** When `source_data` is a plain URL string, the type is auto detected by file extension: // GeoTIFF - extensions: .tif, .tiff, .gtiff, .cog source_data: 'https://example.com/image.tif' // NITF - extension: .nitf source_data: 'https://example.com/image.nitf' // Tile Layer - any other URL (treated as tile server) source_data: 'https://example.com/tiles/{z}/{x}/{y}.png' #### **Option 2: JSON Array of Layers** For more control, use a JSON array. Each layer type has different required/optional properties. ##### **GeoTIFF / COG Layer** source_data: JSON.stringify([ { layer_type: 'geotiff', // or 'cog' name: 'Layer Name', // Required: display name url: 'https://...', // Required: URL to the tiff file // Optional properties: index: 1, // Band index (1-based), default: 1 resolution: 256, // Tile resolution, default: 256 min_value: 0, // Min value for normalization max_value: 255, // Max value for normalization empty_value: 0, // NoData/empty value colormap: 'viridis', // Colormap name (for single-band) } ]) ##### **Multi-band as separate layers:** source_data: JSON.stringify([ { layer_type: 'geotiff', name: 'Band 1 - Red', url: 'https://example.com/multiband.tif', index: 1, }, { layer_type: 'geotiff', name: 'Band 2 - Green', url: 'https://example.com/multiband.tif', index: 2, }, { layer_type: 'geotiff', name: 'Band 3 - Blue', url: 'https://example.com/multiband.tif', index: 3, }, ]) **Note:** RGB files with exactly 3 bands are automatically rendered as RGB when no `index` is specified. ##### **NITF Layer** source_data: JSON.stringify([ { layer_type: 'nitf', name: 'NITF Image', url: 'https://example.com/image.nitf', // Required } ]) **Note:** NITF layers auto-calculate center from corner coordinates if `canvas_options.center` is not set. ##### **Tile Layer** source_data: JSON.stringify([ { layer_type: 'tiles', // or omit layer_type name: 'Base Map', url: 'https://example.com/{z}/{x}/{y}.png', // Required // Optional layer options: options: { min_zoom: 0, max_zoom: 18, min_native_zoom: 0, max_native_zoom: 18, } } ]) ##### `canvas_options`**Format** canvas_options: JSON.stringify({ center: [lat, lng], // Required for tiles, optional for geotiff/nitf zoom: 10, // Initial zoom level min_zoom: 2, // Minimum zoom allowed max_zoom: 20, // Maximum zoom allowed max_bounds: [ // Optional: restrict panning [south, west], [north, east] ], }) **Customizable Hotkeys** * Hotkeys can be assigned to classes by the user. Hotkeys cannot conflict with any other browser or tool shortcuts. ![image-20260626-051427.png](https://success.appen.com/__attachments/a_fa80092d4efa997250e85239c8ca1a68ae3e43017d5a45f5844267d9597fa9a8/image-20260626-051427.png?cb=2949d6816ad05ed5942bdd5a773d8caa) ## Ontology Ontology is mandatory for a tiled image annotation job. Similar to our other image annotation tools, you will be able to create a customized ontology with 4 levels maximum of nesting. Please see [this](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) article for more information on the Ontology Manager. ## Output Format The tiled image annotation tool output is in standard GeoJSON format. See below for an example: [**output-tiled-imagery.json**](https://success.appen.com/hc/en-us/article_attachments/4422796543245/output-tiled-imagery.json) ## Additional Notes This product is in BETA, so please consider the following important notes: 1. The job must be set-up in the Code Editor; the tool is not supported in the Graphical Editor yet. 2. Tiled image annotation jobs do not support test questions or aggregation at this stage. As a result, aggregated reports are not supported at this stage. 3. Launching this type of job requires one of our trusted tiled image annotation contributor channels. Please reach out to your Customer Success Manager to set this up. --- language: "en" --- # Guide to: Running an Audio Transcription Job ## Overview The `cml:audio_transcription` tag allows the users to create an audio transcription job with custom labels and tag sets. ![image-20260622-064754.png](https://success.appen.com/__attachments/a_fe7af0d83c05808b63f487333c2197d3c73586e307b11a7320de9bceb14c2ec5/image-20260622-064754.png?cb=df2bb430cf1ca4b6bd24443e9260a896) *Fig. 1: Audio Transcription tool interface for Contributors* ## Job design The graphical editor now supports audio transcription. Select the Audio Transcription option from the sidebar. ![image-20260622-064817.png](https://success.appen.com/__attachments/a_e8806a26ded605e999ba73396c24d6a6b808e1544d9ca14be252a42203931bc1/image-20260622-064817.png?cb=7837dca24973a17d6489052d962f0163) ![image-20260622-064833.png](https://success.appen.com/__attachments/a_c9be8798e791af0797848852eabf1403463ec4d8b1c135cd9e36172779b49199/image-20260622-064833.png?cb=1a905c51321bafc50b7add7c49d569e4) ### Data * The audio transcription tool supports the transcription of .wav, .mp3, and .ogg file types, as well as .mp4 and .mov (see video parameter, below). * Your data must be [CORS configured](https://success.appen.com/appen-success-center/getting-started/adding-hosting-data/guide-to-cors-configuring-an-s3-bucket.md). * All the data access control features are supported for this tool, including [Secure Data Access](https://success.appen.com/appen-success-center/getting-started/adding-hosting-data/appen-secure-data-access-aws-integration.md). ### Parameters The parameters for the job design are listed below. While some are optional, some are necessary for the element. * `type` (optional, default `play-only`) * `transcription`- enables the transcription field, as well as the tags (to be configured in ontology) and timestamps. * timestamps still also require `allow-timestamping`. * `labeling`- enables the labels, as configured in the ontology. * `segmentation`- Allows contributors to create or modify segments, to subsequently be transcribed and/or labeled. * NOTE: If contributors like to delete a segment, they can use the `CTRL + Delete` hotkey (compatible with Mac and Windows) * `play-only`- Contributors will only have access to the audio player (and, if enabled, the video).. * This type can only be used by itself. For example: `type="['play-only']"` * `none`- if no type is configured, play-only will be the default mode and only the audio player (and video, if configured) will be available to contributors. * Contributors will only have access to the audio player (and video, if enabled) if no type is set, and play-only will be the default option. * **Examples:** * `type="['labeling', 'transcription']"`Allows labeling and transcription (including tags/timestamps). * `type="['labeling', 'segmentation']"`Allows labeling and segmentation (including tags/timestamps). * A note about `type` and its interaction with review-data (see below): When you load data for review into the tool using `review-data`, all annotations provided will be visible, including transcriptions, tags, and labels. Use type to control which parts of the data a contributor can edit. For example, if you use` type="['transcription']"` and your review data contains both transcription and labels, contributors will be able to see both the labels and the transcriptions, but they will only be able to edit the transcriptions. * `source-data` (required) * The column header from your source data containing the audio URLs to be annotated. * `name` (required) * The results header where the annotations will be stored. * `segments-data` (optional) * The column header from your source data containing the audio segmentation data (the start and end timestamps of each segment). * The tool uses this data to create the transcription box for each segment. * The tool expects the data to be in the format found below. * If you do not have segmentation data, omit this parameter. * `label `(optional) * The question label that is visible to contributors. * `validates` (optional) * `validates="required"` * Defines whether or not the element is required to be answered * Defaults to not required if not present * Defaults to "required" if this is the only CML tag present in the job design, as there must be at least one required element * `validates="timestamp_direction"` * Checks whether the timestamps are in the right order upon submission * Regardless of the specified text direction or language, the waveform always runs right to left, therefore if timestamps are not placed in left-to-right order, submission will be blocked and contributors will encounter an error * `validates="minTimestamps:1"` * Checks whether the contributor has placed the minimum number of specified timestamps * Defaults to `minTimestamps:0` if not present * to use multiple validators, separate them with spaces: * example: `validates="required timestamp_direction"` * `review-data (optional)` * This will read in existing transcriptions on an audio file. * If used, a source column with links to the transcriptions formatted as is outputted by the audio transcription tool is required (format as seen below in the 'Results' section). * This parameter may be used to do a peer review or model validation job. * Please see the "Review Mode" section for more details * You can use raw text input as your `review-data` (e.g. for prompt-audio validation) as long as you have no other annotation data as input. * As mentioned above, when you load data for review into the tool using review-data, all annotations provided will be visible, including transcriptions, tags, and labels. Use `type` to control which parts of the data a contributor can edit. For example, if you use `type="['transcription']"` and your review data contains both transcription and labels, contributors will be able to see both the labels and the transcriptions, but they will only be able to edit the transcriptions. * `subset` (optional) * This parameter allows you to set up the tool to display only a subset of all the segments in each unit * Only use this if the "review-data" parameter is present * Accepts value from 0 to 1 * Defaults to 1 * See the "Review Mode" section for more details * `force-fullscreen` (optional) * Accepts 'true' or 'false'. * If 'true', a page of work contains a preview of each data row. When a contributor clicks to open a data row, the audio transcription tool loads into a fullscreen view. * If 'false', a page of work contains a view of the audio transcription tool for each data row. The contributor can open a fullscreen view of the tool at their discretion by clicking an icon in the top right corner of the tool or using a hotkey. * Defaults to 'false'. * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. * `listen-to` (optional) * This parameter allows a you to configure specific amounts of audio that the contributor must listen to in order to submit the task. The parameter is optional but if not configured, it will default to requiring the contributor to listen to 100% of the audio in order to submit the task. The parameter accepts an array of triplets. Each triplet specifies, in order: * the beginning point of a range of audio (described as out of 1) * e.g., \[0.3,0.6,0.5\]: beginning at 30% of the way through the audio * the closing point of a range of audio (described as out of 1) * e.g., \[0.3,0.6,0.5\]: ending at 60% of the way through the audio * how much of that range of audio the contributor must listen to (described as out of 1) * e.g., \[0.3,0.6,0.5\]: listen to 50% of the audio in the specified range * The contributor can listen to any audio within the range so long as it cumulatively sums to the required quantity. * If a contributor listens to the same portion of audio more than once, it only counts towards validation once. * The sum of the differences between all of the beginnings and endings specified in the array must sum to 1. That is, the entire audio must have a listening requirement specified: * Invalid: "\[\[0.3,0.6,0.5\],\[0.6,1,0.25\]\]" Because the range from 0-0.3 is unspecified * Valid: "\[\[0,0.3,0.75\],\[0.3,0.6,0.5\],\[0.6,1,0.25\]\]" Because the ranges sum to 1 * If you want to turn off validation, set the parameter as: listen-to="\[\[0,1,0\]\]" (requires the contributor to listen to 0% of the audio between 0 (beginning) and 1 (end). * Listening validation makes the "validation" checkbox that used to appear during task-type="qa" obsolete since we can now require contributors to listen to audios before submitting. * `speed` (optional) * This parameter allows you to specify the permitted playback speeds for the audio according to the following syntax: speed="\[0.5,1,1.5,3\]" * If I only provide 1 speed, then all contributors can listen only at that speed * If multiple speeds are provided, then contributors can select which speed(s) they want to listen at from the dropdown in the tool UI * When the tool opens, it defaults the speed control to the value closest to 1 and larger than 1 if there is a tie, e.g. \[0.5,1.5\] * If the `speed` configuration is not provided, the default behavior will provide the contributor with the following options: ![image-20260622-064913.png](https://success.appen.com/__attachments/a_f58c6d88dd972f76f6abcfccb0cb5482e2e8bafa7f4f5ed31b7f138fded59b3d/image-20260622-064913.png?cb=d11bf8edb88736ea192b87def402af5e) **When Type Includes**`transcription`**:** * `allow-timestamping` (optional) * set `allow-timestamping="true"` To enable the timestamping functionality in your transcription task. * Contributors will see the **add timestamp**button in each transcription box that allows them to insert timestamps within their transcription. * Timestamps can be used to generate more granular text/audio alignment and/or to allow contributors to correct and improve the segmentation points in the source data. * Timestamps appear in the output data like this: this is a \<12.345/\> transcription. * **Note:** This parameter defaults to "false" if not declared. * `text-direction` (optional) * Set `text-direction="rtl"` to specify that the language you are transcribing is written from right-to-left (e.g. Arabic). This will ensure that any tags and timestamps are placed in the correct sequential location in the text. * It is recommended to use this in combination with the timestamp-direction validator described above * If the tags themselves are in English or another left-to-right language, (e.g. \) they will continue to be displayed in the right direction * **Note:** this parameter defaults to "ltr" if not explicitly defined. **When Type Includes**` segmentation`**:** * `overlap` (optional): Job designer can set "overlapping segments" to allow / not allow using `overlap="true"|"false".` * `overlap` only influences the tool when `type=segmentation` * Defaults to `true` if not specified. * If `overlap="true",` creating overlapping segments is allowed. * If `overlap="false",` creating overlapping segments is NOT allowed. Segments can be created or modified only to be end-to-end. 1. New segments can't begin before the end of / after the beginning of an existing segment 2. Segment boundaries can still be changed but not to cross the boundary of another segment's current position * When review-data contains overlapping segments but `overlap="false"` (and `type=segmentation`) in QA, tool will load with the overlapping segments BUT any NEW segments created, cannot be overlapped. If moving the segments that were part of the review-data, segment cant overlap. * `segmentation-required` (optional): Requestor can specify that a minimum X% of audio is segmented * e.g. `segmentation-required="0.5"` requires that at least 50% of the audio has to be segmented as either a single segment or multiple segments; * `segmentation-required="1"` requires that all of the audio has to be segmented as either a single segment or multiple segments" * default, `if segmentation-required` is not specified, is `0` * by default, including `segmentation` in type will require N\>=1 segments, independent of % coverage, unless contributor selects "nothing to annotate" **When Type Includes**`play-only`**:** * waveform (optional): job designer can set waveform to be displayed to by using waveform="true"\|"false". * Defaults to `true` if not specified * When `waveform="true"`waveform will be presented on the element * When `waveform="false"` waveform will be replaced with a simpler audio player: 1. one line, a play button, the playback speed, and listening required (if configured) 1. when `video="true"` we should also see the video * `video `(optional) * Set `video="true" `and `beta="true"` to enable display of video along with the audio. * Ensure that your data is in one of the supported formats: **.mp4** or **.mov** * Video data must include an audio track to ensure the tool is usable. * **Note:** this parameter defaults to "false" if not explicitly defined. If your data is .mp4 or .mov, the tool will play audio, but the video will not be displayed. ![image-20260622-064931.png](https://success.appen.com/__attachments/a_fae1d10514376ee6f65ba26ab77ca9888af45c74c0b34d82a10c3c9376194c8e/image-20260622-064931.png?cb=d5ddf3089bd54414d73c4f220c0680e5) ## Ontology The audio transcription ontology is where you define the metadata that transcribers will use to label and tag audio files or segments. You can access the ontology by clicking the link to 'Manage Audio Transcription Ontology' that appears on the right corner of the job's Design Page. ![image-20260622-064949.png](https://success.appen.com/__attachments/a_107dc0873c9e8c962056ee4c92d90aacb347dc8d4b01d8c06c03a05db38a055b/image-20260622-064949.png?cb=0802dc23beef9f15badcc97821edb751) *Fig 2: Audio Transcription Ontology Manager* The top-level metadata defined in the audio transcription ontology consists of **Segment Identifier**, **labels**, **event tags**, and **span tags**. * **Segment Identifier** * Contributors can apply identifiers at the segment level. * Identifiers are defined at Design level and require below details: * Color Code * Title * Description * This Identifier can used as below: * Create an Identifier with a color and a speaker name (title). * In the job, contributors can create a segment and assign above identifier to the segment to that they color code the segments of that particular speaker. * **Labels** * Contributors will apply your labels at the segment level. * Labels are defined as members of label groups. Groups are just a way to keep related labels together according to common attributes and rules; groups themselves are not metadata to be labeled. * Label groups require: * a name * at least one label inside them. * Labels must be unique, even between groups. * At the "label group" level, we can define: * if selecting a label from the group is mandatory * if users can select multiple labels from the group * if the group is not transcribable * By default, we assume a segment is transcribable and show the transcribable labels. * In this case, mandatory\|transcribable applies. * If a segment is marked as "nothing to transcribe", only non-transcribable labels should be available. * In this case, mandatory\|non-transcribable applies. * The transcription box is not available. * You do not need to create any labels (i.e. you needn't create any groups if you need no labels), but if you want to include labels, they must be inside a group. * **Event Tags** * Event tags are optional. * Event tags require a name. * Event tags are initially displayed in alphabetical order. However, you have the ability to customize their order of presentation by dragging and arranging the tags as per you prefer. * You may also provide a description for each tag, which will be visible to the contributor in the tool when they click on the info icon for that tag. * **Event Groups** * Event groups are optional * Event groups require a name * Event groups are initially displayed in alphabetical order. However, you have the ability to customize their order of presentation by dragging and arranging the groups as per you prefer. * You can assign event tags to event groups using a drop down within the event tag OR by dragging an existing event tag inside an event group. * **Span Tags** * Span tags are optional. * Span tags require a name. * Span tags are initially displayed in alphabetical order. However, you have the ability to customize their order of presentation by dragging and arranging the tags as per you prefer. * You may also provide a description for each tag, which will be visible to the contributor in the tool when they click on the info icon for that tag. * **Span Groups** * Span groups are optional * Span groups require a name * Span groups are initially displayed in alphabetical order. However, you have the ability to customize their order of presentation by dragging and arranging the groups as per you prefer. * You can assign event tags to span groups using a drop down within the span tag OR by dragging an existing span tag inside an span group. *** ** * ** *** ## Segments List All the segments created on an audio are listed on left panel names Segments list. Contributor can perform few actions on segments using options in this panel. * **Hide/Unhide Segment** * Segments can now be hidden and/or unhidded, deleted. * Contributors cannot play or change transcription of a hidden segment * Use eye icon next to each segment to hide or unhide a segment (it's a toggle option) ![image-20260622-065126.png](/__attachments/a_1ab43a6c7d8448f242f85c249b977c5ee0c4bd9722ac27e85007372cc89123b5/image-20260622-065126.png?cb=3facec4f0b605f75a45e5a9738854564) ![image-20260622-065133.png](/__attachments/a_b8c7ad46b0db6b9e55980df69b7a79987e32062397ebf95562e3cd323f7beb1b/image-20260622-065133.png?cb=1bc7ed7d336d18d0e5af59d747916fc2) * Contributors can use the eye icon at top of left panel to either hide all segments/unhide all segments * **Delete Segment** * Use the delete option next to each segment to delete the segment * To see the delete option, one has to select the segment * **Filter Segments** * Once the segments are color coded by assigning a Segment Identifier, contributors can use the left side panel of segments to filter the segments. * Click on filter icon to open the filter panel * All existing Identifier will be listed and Identifiers assigned to any segment will be enabled for filtering. **Note:***when a filter is applied, new segments cannot be added. However existing filtered segments can be modified.* ![image-20260622-065158.png](/__attachments/a_208bce52aba72ce302764be9695fa4a4c35d8f91d04b39e1bf3e7f44f36301af/image-20260622-065158.png?cb=73b2434bfcfcae04e23acf3bbd6b6a5a) * Once a filter is applied, the filter can be opened again to modify, or the contributor can use the "Clear" option to clear all the filters. ![image-20260622-065211.png](/__attachments/a_8cbdbc22fa7081872bbcf7ceb97b3b9c2d14c1de2e3f333164ade5a87b1bd7ee/image-20260622-065211.png?cb=852a56ab0e2230744434ac2022cccf9e) *** ** * ** *** ## Reviewing Results Results will be provided as a secure link to a JSON file describing the annotations. **Important note:** Due to security reasons, JSON result links will expire 15 days after generation. To receive non-expired result links, please re-generate the result reports. The objects in the JSON include the following: * **For each segment:** * `id` * The universally unique identifier of every segment. * `startTime` and `endTime` * This will be displayed in seconds, to the millisecond. * These fields are inherited from the segmentation data. * `labels` * This field will contain the labels as indicated by the contributor. * `transcription` * This field will contain the transcription text as entered by the contributor. Tags and timestamps will also appear in the transcription field. * **For the entire audio file:** * `nothingToTranscribe` * This will be Boolean `true` or `false` * This will be true if the contributor has indicated they were unable to transcribe the entire audio file. * `abletoAnnotate` * This will be Boolean `true `or` false` * This will be false if the tool was unable to load the audio file. **Annotation Schema** interface AudioToolNewOutput { annotation: { segments: { // main information about the segment id: string; //prefixed with segment so as not to confuse this as annotation id startTime: number; endTime: number; // Kept the following one as extra info until the layers feature is removed from the tool entirely, // otherwise users would see layers in first load and if they load judgment or autosaved data (on page refresh) subsequently, they wouldn't. // this inconsistency would bring confusion layerId: string; labels: string[]; //comes if 'type' includes labelling & 'task-type' is labelling or qa transcription: string[]; // comes if 'type' includes transcription & 'task-type' is labelling or qa metadata: { // all other info for the segment comment?: string; // can be present in any qa judgment feedbackAcknowledged: boolean; // comes from acknowledgment task }; }[]; } ableToAnnotate: boolean; speaker: {name:"speaker name", color: "hexacode"}; hidden: boolean; nothingToTranscribe: boolean; // did not change it to nothingToAnnotate since this field is not being used for anything else so it was not useful to update *** ** * ** *** ## Review Mode We have created an experience specially designed for the purpose of quality management. ### For job creators On the job creator's side, when using the `task-type="qa`" and `review-data` CML parameters, you can also specify a `subset`, so that the tool display only a random sample of all the segments in each audio file. The reviewers can review each audio file, but much quicker. This feature is ideal if your goal is to get an idea of the transcription quality of each audio file. In the review job's output, you will see two additional fields under "metadata": 1. "original_text": in case the transcription is changed by the reviewer, this field records the original transcriptions that are loaded as input data. This field makes it easy to calculate the word error rate of each unit. 2. "review_status": for all the segments that have been randomly selected for the review job, they will have this attribute set to "reviewed". Note that this feature is designed to review jobs with only 1 judgment per row. ### For contributors We have added a "reviewed" button for review jobs, which ensures that the reviewer must go through every single segment before being able to submit. As a reviewer working on a review job, for each segment, they will need to click on the "reviewed" button after reviewing the transcription. *** ** * ** *** ### Accepted Segmentation Input Schemas **OLD:** SegmentInput { id: string; startTime: number; endTime: number; ontologyName?: string; layerId?: string; } SegmentsDataInput { annotation: SegmentInput[][]; nothingToAnnotate: boolean; } **NEW:** interface SegmentInput { id: string; startTime: number; endTime: number; ontologyName?: string; layerId?: string; } interface AudioAnnotationData { annotation: SegmentInput[][]; nothingToAnnotate: boolean; } type SegmentsDataInput = SegmentInput[] | AudioAnnotationData; * Segments input data is not required * The old input format (cml:audio_annotation output) is partially supported by the tool. Segment boundaries will be respected but ontology classes and names will **not** work in the new tool or ontology format. * If you have segments or other pre-annotations to display, they should be in the above format. * [example-ontology.csv](https://success.appen.com/hc/en-us/article_attachments/360069045592) (465 Bytes) * [example_transcription_results.csv](https://success.appen.com/hc/en-us/article_attachments/4402586008333) (276 Bytes) * [sample_segmentation_sourcefile.csv](https://success.appen.com/hc/en-us/article_attachments/4402573394061) (168 Bytes) --- language: "en" --- # Guide to: Running an Ellipse or Circle Annotation Job The `cml:shapes` tag allows users to create an image annotation job for ellipses or circles in conjunction with a custom ontology and the use of test questions and aggregation. ## **Building a Job** The following CML contains the possible parameters for an ellipse/circle job: `` Note: There are parameters for test questions and aggregation that apply to both the ellipses and the labels. **Parameters** Below are the parameters available for the `cml:shapes` tag. Some are required in the element, some are optional. * `type` * The shape used in the job, set in an array. * `source-data` * The column from your source data that contains the image URLs to be annotated. * `disable_image_boundaries`: allows shapes to be drawn outside the image boundary * When `disable_image_boundaries="true"` shapes can be dragged outside the image boundaries and the output will contain negative values * The default setting for `disable_image_boundaries is "false" ` * `name` * The results header where annotations will be stored. * `label` * The question label contributors will see. * `validates` (optional) * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if not present * `ontology` (optional) * The list of classes to be labeled in an image - view this [article](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) to learn how to create your custom ontology. * Accepts a boolean * Defaults to 'false' if not present * `review-data` (optional) * This will read in existing annotations on an image. The format must match the output shown in the aggregation section below. All that's needed is the following: * 'type' * 'class' if using an ontology * 'coordinates' * 'id' * Example: * `[{"class":"car","coordinates":{"rx":50,"ry":110,"x":100,"y":150},"type":"ellipse","id":"247f2099-22e1-4825-9bc1-3e51c6019fe0"}]` * `ellipse-threshold` * The minimum overall ellipse/circle IoU required for a contributor to pass a test question. * Accepts a decimal value between 0.1 and 0.99. * `class-threshold` * The minimum percentage of correct classes applied to ellipses/circles test question for a contributor to be considered correct. * Accepts a decimal value between 0.1 and 0.99. * The formula is *correct / (correct + incorrect)* * Example: the class-threshold is set to 0.7 and a test question contains 10 ground truth shapes. A contributor gets 8 out of 10 classes correct for a score of 80% and they're marked correct on the test question ![Screen_Shot_2022-06-13_at_3.08.20_PM.png](https://success.appen.com/__attachments/a_78b16660acb2986f329807b7d4384ed0edf65b22952f2274852892dd96549df5/c51acf1f488dfd3e_Screen_Shot_2022-06-13_at_3.08.20_PM.png?cb=0651f73b6d96f07f761829e3fcdb2c49) *Fig 1. General Options of Ellipse/Circle Tool in Graphical Editor* * `ellipse-agg` * The minimum IoU required for result ellipses/circles to be clustered together. * Accepts a decimal between 0.1 and 0.99, or the value 'all'. * If 'all' is selected, no clustering is done on the ellipses/circles. * `class-agg` * The aggregation applied to the class for a given cluster of shapes. * Accepts standard aggregation types: * `agg` * `all` * `agg_x` * `cagg_x` * `min-ellipse-height`(optional) * Each ellipse/circle drawn by a contributor must be at least this height in pixels * Accepts a positive integer - must be at least 2 * `max-ellipse-height`(optional) * The maximum possible height in pixels each ellipse/circle can be * Accepts a positive integer - must be at least 2 * Please note: If using the Graphical Editor, moving the slider all the way to 1000+ does not set a maximum height. To set the maximum larger than 1000px, you will need to set the max-box-height in the [Code Editor](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor.md). * `min-ellipse-width`(optional) * Each ellipse/circle drawn by a contributor must be at least this width in pixels * Accepts a positive integer - must be at least 2 * `max-ellipse-width`(optional) * The maximum possible width in pixels each ellipse/circle can be * Accepts a positive integer - must be at least 2 * Please note: If using the Graphical Editor, moving the slider all the way to 1000+ does not set a maximum width. To set the maximum larger than 1000px, you will need to set the max-box-width in the [Code Editor](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-the-code-editor.md). * `allow-ellipse-rotation`(optional) * Will enable contributors to rotate ellipses on the image * Accepts 'true' or 'false' * Defaults to 'false' if not present * `crosshair`(optional) * Will enable crosshair location indication * Accepts 'true' or 'false' * Defaults to 'false' if not present * `ellipse-aspect-ratio` (optional) * Controls the aspect ratio of the ellipse * The ratio is width:height * Setting this to 1:1 enforces a perfect circle * Accepts a ratio of integers, e.g., 2:1 * `output-format` (optional) * Accepts 'json' or 'url' * If 'json', the report column containing contributors' annotation data contains the annotation data in stringified JSON format. The JSON format is as follows (this is the legacy JSON format): * [ { "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "box", "coordinates": { "x": 416, "y": 243, "w": 125, "h": 95 } } ] * If 'url', the report column containing contributors' annotation data contains links to files. Each file contains annotation data for a single data row in JSON format. With this new output option, we have updated the JSON structure to allow inclusion of more data fields. The new JSON format is as follows: * * { ableToAnnotate: true, imageRotation: 30, annotation: [{ "id": "4bc1ba1d-ede9-4b80-9892-95fced615441", "class": "Car", "type": "box", "coordinates": { "x": 416, "y": 243, "w": 125, "h": 95 } }]} * In the case where the tool was unable to load the input data and the contributor was unable to annotate, `ableToAnnotate` will be set to `false`. * Defaults to 'json' if attribute not present. * This parameter is available within the ***CML only***; it is not yet supported in the Graphical Editor. * `allow-image-rotation` (optional) * Accepts `true` or `false` * If `true`, contributors can rotate the image within the image annotation tool. Contributors click a toolbar icon to turn on a rotation slider that can be used to adjust rotation angle from 0 to 359 degrees. The degrees rotated are exported in the `imageRotation` field. This feature is only compatible with export option `output-format=url`; this attribute must be added to the job cml before launch. * **Important note:** Test questions and aggregation are not currently available for this annotation mode. * If `false`, contributors cannot rotate the image. * Defaults to `false` if attribute not present. * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://appen-external.atlassian.net/wiki/display/ASC/Task+Types+and+%E2%80%9Creview-data%E2%80%9D) for more details. ![Screen_Shot_2022-06-13_at_3.09.18_PM.png](https://success.appen.com/__attachments/a_3260a02472e8e532f26227783e6e856967d996043873e3046264209a5ec4a017/c285ff2398afbc53_Screen_Shot_2022-06-13_at_3.09.18_PM.png?cb=bd0dd1f2e0d3d6b89b385ddcfd051ce7) *Fig 2. Additional Ellipse/Circle Options in Graphical Editor* **Shape Type Limiter** * Limit which shapes can be used with certain classes ![image-20260622-085317.png](https://success.appen.com/__attachments/a_a2846460a925fe0ce6f7355b584cf9c277a1f776cd9b61bc92d5bc5866fc2de1/image-20260622-085317.png?cb=ba7c38e7ed63b6cdb46aea6fc8f00b35) **Min/Max instance quantity** * Configure ontologies with instance limits ![image-20260622-085804.png](https://success.appen.com/__attachments/a_32553955e495f48fad9bf75c0a5570ea6cb95253fc3fac6e851d5f389934af80/image-20260622-085804.png?cb=643507c589cd80217a8437ed530dcc32) * Comes with the ability to mark the class as not present for long tail scenarios. This information will be added to the output as well. ![image-20260622-085815.png](https://success.appen.com/__attachments/a_bcbc5d396f3c50f27ac7a84fbd6deaa4ec5d79ebae1816b334a42c895ec88914/image-20260622-085815.png?cb=72139db81d21742325f819cf5b150b5c) **Customizable Hotkeys** * Hotkeys can be assigned to classes by the user. Hotkeys cannot conflict with any other browser or tool shortcuts. ![image-20260622-085827.png](https://success.appen.com/__attachments/a_348662d1f58f5bfdfc6bded1926f407697bd24903ab43d183c2a64833333ca58/image-20260622-085827.png?cb=2949d6816ad05ed5942bdd5a773d8caa) ### **Aggregation** #### **Ellipses/Circles** * Aggregation for ellipses/circles using `cml:shapes` works as follows: * Set the `ellipse-agg` parameter in the CML, which is the IoU used for clustering ellipses/circles prior to aggregation. * For example, if the `ellipse-agg` is 0.6, boxes that overlap each other by at least 60% will be clustered together. * The average ellipse area is then calculated by taking the average center coordinates and average X and Y radii. * Each of these is weighted by the contributor trust score. * The union area of all the ellipses in the cluster is calculated. * Finally, the average ellipse area is divided by the union area of ellipses (I / U) * When using rotated ellipses: * The calculation of the IoU (Intersection over Union) takes the degree of rotation into account as an added parameter. * 'angle' will be returned in the output as an integer between 0 and 360, representing the clockwise degree of rotation #### **Classes/Labels** The `class-agg` parameter accepts the following [standard aggregation methods](https://success.appen.com/appen-success-center/dashboards-reports/reports-in-quality-flow/guide-to-aggregation.md): * `agg` * `all` * `agg_x` * `cagg_x` Labels (or classes) are aggregated **per returned ellipse/circle** . This means, for example, if you choose to aggregate boxes - as opposed to selecting 'all' - and you choose `class-agg="agg"`, for each aggregated box you'd receive the **most confident** label out of the constituent boxes in the cluster. If you choose `class-agg="all"`, you'd receive every label applied to the cluster of boxes, but still just one box, and so on. For `ellipse-agg="all"`, you'd receive every ellipse/circle and every label in the image, no aggregation. Labels will always be grouped with the shape they were applied to and will be returned in a dictionary. Example output of a job with `box-agg="0.6"` and `class-agg="agg"` with `allow-ellipse-rotation` set to `'true'`: ` [{"average_trust":0.7857,"class":{"car":1.0},"coordinates":{"rx":50,"ry":110,"x":100,"y":150},"iou":0.9628,"type":"ellipse","angle":45}]` Example output of a job with ellipse-agg="0.6" and class-agg="all": `[{"average_trust":0.7857,"class":{"car":0.33,"person":0.33,"tree":0.33},"coordinates":{"rx":50,"ry":110,"x":100,"y":150},"iou":0.9628,"type":"ellipse"}]` ### Reviewing Results To review the results of your job: 1. Go to the Data page. 2. Click on a unit ID. 3. In the sidebar of the annotation tool, select an option from the drop-down menu. 1. You'll see different contributor IDs, which allow you to view individual annotations. 2. You'll also see an "aggregated" option, which shows you the result you'll get based on your aggregation settings in the CML or report options page of your job. --- language: "en" --- # Guide to: Running an Image Segmentation Job (PLSS) ## Overview The new image segmentation tool allows users to create an image segmentation ("pixel labeling") job in conjunction with a custom ontology. If you would like to set up an image segmentation job, please contact your Customer Success Manager. ![Screen_Shot_2020-01-23_at_3.36.57_pm.png](https://success.appen.com/__attachments/a_0aeec48ecdcb0de11504b25b0fc38c74ea49381f0611fd6d818331ec652469f7/6ada170dde36d823_Screen_Shot_2020-01-23_at_3.36.57_pm.png?cb=6d427ad365b2b77cd2d2df275eedf38d) Figure 1: Preview of Image Segmentation Tool ## Building a Job **Parameters** Below are the parameters available for the image segmentation tool. Some are required in the element, while others are optional. * `type` * This parameter allows you to add various types of annotation modes to the tool, including `"brush"`, `"polygon"`, `"fill"`, `"magic-wand"` and now `"superpixels"`. **Important note:** The superpixels feature is part of our managed services offering; contact your Customer Success Manager for access or more information. * `source-data` * The column from your source data that contains the image URLs to be annotated. * Please note: If you would like to be able to update brightness and contrast within the tool using a secure bucket, images must also be CORS configured. * `name` * The results header where annotations will be stored. * `labe` * The question label the contributors will see. * `validates`(optional) * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if not present * `review-data`(optional) * This will read in existing annotations on an image. For this job type, they will need to be image URLs, each linked to the annotated raster file for the original image. * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. * `superpixels-mask` (optional) * When the job is part of a Workflow including the Superpixels Model, this attribute reads in the superpixel grid calculated by the model. * Accepts `"{{superpixel_mask}}"` ## Ontology Ontology is mandatory for an image segmentation job. Similarly to our other image annotation tools, you will be able to create a customized ontology with 4 levels maximum of nesting. Please see [this](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) article for more information on the Ontology Manager. ## Additional Notes This product is in BETA, so please consider the following important notes: 1. The job must be set-up in the Code Editor; the tool is not supported in the Graphical Editor yet. 2. Image segmentation jobs do not support test questions or aggregation at this stage. As a result, aggregated reports are not supported at this stage. 3. Launching this type of job requires one of our trusted image segmentation contributor channels. Please reach out to your Customer Success Manager to set this up. 4. Results will be in the form of URLs that contain the annotations. The result links will expire 15 days after generation; to receive non-expired result links, please re-generate the result report. --- language: "en" --- # Guide to: Running an Image Transcription Job This image transcription tool allows you to combine image annotation and transcription in one job, simplifying your workflow. ## Building a Job ### Data This tool accepts images and PDF files (single and multipage) for annotation. If using our OCR assistance feature, images must be CORS configured to allow predictions to be made on the text. PDF files must also be CORS configured in order to be supported. Please view our [Guide to CORS configuring an s3 bucket](https://success.appen.com/appen-success-center/getting-started/adding-hosting-data/guide-to-cors-configuring-an-s3-bucket.md) for assistance. ### CML Currently, there is no Graphical Editor support for this tool. Here is sample CML to build the job: ![image-20260626-054510.png](https://success.appen.com/__attachments/a_190b94285ee969a9e70aa7292aa7974885aea41cda8d7c3a262b325068c35147/image-20260626-054510.png?cb=448af2128e452e515118707e48ea2c66) ### Parameters Below are the parameters available for the job design. * `type` (required) * The shape used in the job; currently 'box' and 'polygon' * N-sided polygon annotation type is now supported, use `polygon-max-vertices` to specify the maximum number of vertices allowed. * `type="['polygon']" polygon-max-vertices="7"` ![image-20260626-054534.png](https://success.appen.com/__attachments/a_6b02efa6c551f5346650d31620a40824dfbf97753a2fd356e1bf5d6f4d45efc0/image-20260626-054534.png?cb=1c8c2d9ea187b78a563ed3218c5d781a) * `source-data` (required) * The column from your source data that contains the image or PDF file URLs to be annotated. * `disable_image_boundaries`: allows shapes to be drawn outside the image boundary * when `disable_image_boundaries="true"` shapes can be dragged outside the image boundaries and the output will contain negative values * the default setting for `disable_image_boundaries`is "false" * `name` (required) * The results header where annotations will be stored. * `label` (required) * The question label contributors will see. * `validates` (optional) * Whether or not this element is required to be answered. * Accepts 'required' * Defaults to not required if not present * `ontology` (optional) * The list of classes to be labeled in an image - view [this article](https://success.appen.com/appen-success-center/create-design-jobs/guides-annotation-tools/guide-to-ontology-manager.md) to learn how to create your custom ontology. * Accepts a boolean * Defaults to 'false' if not present * `review-data` (optional) * This will read in existing annotations on an image. The format must match the output shown below. The following is required: * '`id`' * A randomly-generated, 32-character UUID * '`class`' * The class from the ontology * '`type`' * This is the shape type, which is 'box' or 'polygon' * '`instance`' * The shape class instance, which loads in the ontology sidebar * '`coordinates`' * The coordinates for the bounding box or polygon * '`metadata`' * '`shapeTranscription`' * This includes the following: * '`inputType`' * For transcription, this will always be 'text' * '`text`' * This is the transcription for the box or polygon * '`type`' * This is the shape type, which is 'box' or 'polygon' * Metadata example (JSON format for image or single page source file): [ { "id": "677706c8-f405-4a2c-9be1-1b6f4c5042a2", "class": "Business Name", "instance": 1, "metadata": { "shapeTranscription": { "inputType": "text", "text":"Figure Eight" } }, "type": "box", "coordinates": { "x": 250,"y": 177,"w": 26,"h": 15 } } ] Multi-page example Note: field names "ableToAnnotate", "imageRotation" and "groups" are required fields. `{ "ableToAnnotate": true, "imageRotation": 0, "groups": [], "annotation": { "pages": { "1": { "shapesInstances": { "b7e0b005-e077-4d85-b5ba-7fa2c02da453": { "visible": true, "x": 48, "y": 200, "height": 72, "width": 520, "angle": 0, "coordinates": { "x": 48, "y": 200, "w": 520, "h": 72 }, "metadata": { "shapeTranscription": { "label": "Annotation", "modelType": "ocr", "modelDataType": "block", "inputType": "text", "annotatedBy": "machine", "text": "Terrorism (Protection of Premises) Bill - Standard Tier", "output": "v2" } } }, "bc853181-2084-4cec-a599-f2d6742047b8": { "visible": true, "x": 53, "y": 281, "height": 31, "width": 303, "angle": 0, "coordinates": { "x": 53, "y": 281, "w": 303, "h": 31 }, "metadata": { "shapeTranscription": { "label": "Annotation", "modelType": "ocr", "modelDataType": "block", "inputType": "text", "annotatedBy": "machine", "text": "Government consultation", "output": "v2" } } } } }, "2": { "shapesInstances": { "fa877670-658b-4e3e-a354-c3803e673064": { "visible": true, "x": 224, "y": 256, "height": 85, "width": 318, "angle": 0, "coordinates": { "x": 224, "y": 256, "w": 318, "h": 85 }, "metadata": { "shapeTranscription": { "label": "Annotation", "modelType": "ocr", "modelDataType": "block", "inputType": "text", "annotatedBy": "machine", "text": "The proposed Bill would impose requirements in relation to certain premises and events to increase their preparedness for, and protection from, a terrorist attack by requiring them to take proportionate steps, depending on the size and nature of the activities that take place at their premises.", "output": "v2" } } }, "58791eac-1dca-496b-b50a-64c2ddf6f7c6": { "visible": true, "x": 226, "y": 345, "height": 38, "width": 306, "angle": 0, "coordinates": { "x": 226, "y": 345, "w": 306, "h": 38 }, "metadata": { "shapeTranscription": { "label": "Annotation", "modelType": "ocr", "modelDataType": "block", "inputType": "text", "annotatedBy": "machine", "text": "The proposed requirements would apply to those responsible for qualifying public premises and qualifying", "output": "v2" } } } } }, "3": { "shapesInstances": { "160e6e91-14ae-48bf-ad52-ebf7809d6fa0": { "visible": true, "x": 51, "y": 379, "height": 30, "width": 117, "angle": 0, "coordinates": { "x": 51, "y": 379, "w": 117, "h": 30 }, "metadata": { "shapeTranscription": { "label": "Annotation", "modelType": "ocr", "modelDataType": "block", "inputType": "text", "annotatedBy": "machine", "text": "Response paper:", "output": "v2" } } }, "bdf123cc-989f-426e-9143-d0646dd8189c": { "visible": true, "x": 402, "y": 235, "height": 17, "width": 87, "angle": 0, "coordinates": { "x": 402, "y": 235, "w": 87, "h": 17 }, "metadata": { "shapeTranscription": { "label": "Annotation", "modelType": "ocr", "modelDataType": "block", "inputType": "text", "annotatedBy": "machine", "text": "18 March 2024", "output": "v2" } } } } } }, "shapes": { "b7e0b005-e077-4d85-b5ba-7fa2c02da453": { "id": "b7e0b005-e077-4d85-b5ba-7fa2c02da453", "number": 1, "color": "#FF1744", "pageNumber": 1, "type": "box", "class": "Person" }, "bc853181-2084-4cec-a599-f2d6742047b8": { "id": "bc853181-2084-4cec-a599-f2d6742047b8", "number": 1, "color": "#651FFF", "pageNumber": 1, "type": "box", "class": "Event" }, "fa877670-658b-4e3e-a354-c3803e673064": { "id": "fa877670-658b-4e3e-a354-c3803e673064", "number": 3, "color": "#FF1744", "pageNumber": 2, "type": "box", "class": "Person" }, "58791eac-1dca-496b-b50a-64c2ddf6f7c6": { "id": "58791eac-1dca-496b-b50a-64c2ddf6f7c6", "number": 2, "color": "#651FFF", "pageNumber": 2, "type": "box", "class": "Event" }, "160e6e91-14ae-48bf-ad52-ebf7809d6fa0": { "id": "160e6e91-14ae-48bf-ad52-ebf7809d6fa0", "number": 2, "color": "#FF1744", "pageNumber": 3, "type": "box", "class": "Person" }, "bdf123cc-989f-426e-9143-d0646dd8189c": { "id": "bdf123cc-989f-426e-9143-d0646dd8189c", "number": 1, "color": "#00E676", "pageNumber": 3, "type": "box", "class": "Organisation" } } } }` * Note: Old metadata format is still supported for using in review-data: [{"id":"677706c8-f405-4a2c-9be1-1b6f4c5042a2","class":"Business Name", "instance":1,"metadata":[{"inputType":"text","text":"Figure Eight"}], "type":"box","coordinates":{"x":250,"y":177,"w":26,"h":15}}] * `box-threshold` (optional) * The minimum overall bounding box IoU required for a contributor to pass a test question. * Accepts a decimal value between 0.1 and 0.99. * `class-threshold` (optional) * The minimum percentage of correct classes applied to boxes in a test question for a contributor to be considered correct. * Accepts a decimal value between 0.1 and 0.99. * The formula is classes correct / (total classes correct + incorrect). * Example: the `class-threshold` is set to 0.7 and a test question contains 10 ground truth boxes. A contributor gets 8 out of 10 classes correct for a score of 80% and would be considered correct for that test question. * `crosshair` (optional) * Will enable crosshair location indication * Accepts a boolean * Defaults to 'false' if not present * `ocr` (optional) * When set to 'true', this enables OCR transcription assistance in the tool. * This feature must be enabled for your team for access and is not included in every subscription plan; please contact your Customer Success Manager or Account Executive for more information. * `output-format` (optional) * For supporting multipage PDF, `output-format="url"` is required. * Otherwise, accepts 'json' or 'url'. * If 'json', the report column containing contributors' annotation data contains the annotation data in string JSON format. * If 'url', the report column containing contributors' annotation data contains links to files. Each file contains annotation data for a single data row in JSON format. * Defaults to 'json'. * `allow-image-rotation` (optional) * Accepts `true` or `false` * If `true`, contributors can rotate the image within the image annotation tool. Contributors click a toolbar icon to turn on a rotation slider that can be used to adjust rotation angle from 0 to 359 degrees. The degrees rotated are exported in the `imageRotation` field. This feature is only compatible with export option `output-format=url`; this attribute must be added to the job cml before launch. Test questions and aggregation are not currently available for this annotation mode. * If `false`, contributors cannot rotate the image. * Defaults to `false` if attribute not present. ![2021-02-23_16.38.06.gif](https://success.appen.com/__attachments/a_1d5195514e237208d98d576f73cbd8a9f6117fbea2ffd0925dd4a100de369e01/ccff25570cbfa3fd_4406998655757.bin?cb=cd80e3ee0c2fad14ddece47d7675a493) * `allow-box-rotation`(optional) * Will enable bounding boxes to be rotatable * Accepts 'true' or 'false' * Defaults to 'false' if not present ![image-20260626-054620.png](https://success.appen.com/__attachments/a_84606b085ebcc2e98a7de2ccb79b02ca690cb2f55858d326f1c2ccac88e88c92/image-20260626-054620.png?cb=d256bad0b9d51e0be634cf4a5fc25770) * `require-transcription-review` (optional) * Requires contributors to review every bounding box. Typically used in a QA/Peer review job to ensure QA contributor is verifying every bounding box created from work task. * Only available in jobs that require data to be reviewed; thus, requires review-from to be configured * Accepts 'true' or 'false' * Defaults to 'false' if not present * `language` (optional) * This can only be used when `ocr="true"` * Accepts a liquid variable; the column in your source data must contain an ISO 639-1 code * The supported languages and their codes are the following: * '`af`': 'Afrikaans', '`ar`': 'Arabic', '`cs`': 'Czech', '`da`': 'Danish', '`de`': 'German', '`en`': 'English', '`el`': 'Greek', '`es`': 'Spanish', '`fi`': 'Finnish', '`fr`': 'French', '`ga`': 'Irish', '`he`': 'Hebrew', '`hi`': 'Hindi', '`hu`': 'Hungarian', '`id`': 'Indonesian', '`id`': 'Italian', '`jp`: 'Japanese', '`ko`': 'Korean', '`nn`': 'Norwegian', '`nl`': 'Dutch', '`pl`': 'Polish', '`pt`': 'Portugese', '`ro`': 'Romanian', '`ru`': 'Russian', '`sv`': 'Swedish', '`th`': 'Thai', '`tr`': 'Turkish', '`zh`': 'Chinese', '`vi`':'Vietnamese', '`zh-sim`': 'Chinese (Simplified)', '`zh-tra`': 'Chinese (Traditional)' * If an invalid or unsupported ISO code is passed in from the source data, the in-tool OCR will default to English and will not recognize non-English letters or diacritics. * Also supports right-to-left languages. * `task-type` (optional) * Please set task-type="qa" when designing a review or QA job. This parameter needs to be used in conjunction with review-data. See this [article](https://success.appen.com/appen-success-center/create-design-jobs/job-design-universals/guide-to-task-types-and-review-data.md) for more details. ![image-20260626-054642.png](https://success.appen.com/__attachments/a_086bf2d20ed495041fab3012e7f7b7304e948d8b10a390f01e9700ef849a530d/image-20260626-054642.png?cb=82b8618aab7a4ce05f5557dadc2c4472) ## Ontology Configuration **Note:** Test Questions, Aggregation and Pre-labelling are yet not fully supported for 'polygon' type of shape The image transcription tool supports ontologies. In addition, validators can be configured for each class in order to: A) limit the number of shape instances contributors can create in that class B) limit the text characters contributors can use when creating transcriptions in that class Validators can be configured in the class configuration modal on the ontology manager page under "Transcription Settings". ![image-20260626-054711.png](https://success.appen.com/__attachments/a_f50e7efefda2238c553cfa06630a5a2bd5023d34d674d27a49862ca6c9569af5/image-20260626-054711.png?cb=a945f8a9b748033474db634b0bb2a2ac) The following validators are available: * Date * Only valid dates can be submitted * Date format returned is YYYY-MM-DD * Alpha * Only letter characters can be submitted * Additional options include character number limitations and the ability to allow symbols and whitespaces * AlphaNum * Only letter or number characters can be submitted * Additional options include character number limitations and the ability to allow symbols and whitespaces * Numeric * Only float numbers with a single dot can be submitted * Additional options include character number limitations and the ability to allow symbols * Digit * Only integers can be submitted * Additional options include character number limitations and the ability to allow symbols * Custom Regex Validator * Users can now specify text-based validations using Regex for each class individually in the ontology manager for image transcription. * ![image-20260626-054729.png](https://success.appen.com/__attachments/a_fd5d97dac72029e63e52ee6bcaef3d44e7a3946757fae0ba4d134781e1407fce/image-20260626-054729.png?cb=67dd9a301c2900063ab38e7c26f60882) * Shape Type Limiter * Limit which shapes can be used with certain classes ![image-20260626-054746.png](/__attachments/a_0f3f9665105b56c8d94e62ee074cba303751d2067c39ecb307ea1adf317d6af7/image-20260626-054746.png?cb=ba7c38e7ed63b6cdb46aea6fc8f00b35) * Min/Max instance quantity * Configure ontologies with instance limits ![image-20260626-054800.png](/__attachments/a_8d4fb261a2e5486148844ed4db986b16ad647ef22983cb23b7a8e290209c8bad/image-20260626-054800.png?cb=643507c589cd80217a8437ed530dcc32) * Comes with the ability to mark the class as not present for long tail scenarios. This information will be added to the output as well. ![image-20260626-054811.png](/__attachments/a_8f1f36b89d4a83d8c3b888f46f8f6234dcd7f0a26dff8d63e2b481bd4c7f4859/image-20260626-054811.png?cb=72139db81d21742325f819cf5b150b5c) * Customizable Hotkeys * Hotkeys can be assigned to classes by the user. Hotkeys cannot conflict with any other browser or tool shortcuts. ![image-20260626-054827.png](https://success.appen.com/__attachments/a_454e553a8267e954bc06a834bbe020f412fb63139c02a68a1dc347bbca4f057e/image-20260626-054827.png?cb=2949d6816ad05ed5942bdd5a773d8caa) *** ** * ** *** ## Test Questions ### Creating Test Questions In BETA, test questions are only partially supported. You may test on the boxes and the classes, but not the transcriptions. 1. On the Quality Page, click 'Create Test Questions'. 2. Add boxes around the text in the way specified in the job's instructions. 3. If no annotations are needed, make sure the job includes an option, such as a single checkbox, to hide the annotation tool. 4. Save the test question. ### Reviewing Test Questions 1. Select a test question from the Quality Page. 2. From the image annotation sidebar, click 'Find a Judgment' and choose a contributor ID from the drop-down menu. 3. Edit, create or remove the test question annotations based on the feedback. Judgments are color-coded based on if they match the gold responses. * Each box will have its own matching metrics, which can be seen by hovering over a contributor judgment or golden shape. A notification will appear in the top left corner of the image. A score from zero to one is displayed on the intersection over union formula. If using an ontology, the class match is also displayed. * All scores on images are averaged and compared to the test question threshold set in the job design. The overall matching score is then displayed in the left sidebar of the tool. 4. Save any edits that are made to update the evaluation of the existing contributors' work and ensure any future attempts to answer the test question will be properly evaluated. ![image-20260626-054842.png](https://success.appen.com/__attachments/a_c0198d5ebbaca47c28a381e137476093bbd5595a59f1e43247dc6bcdb588df46/image-20260626-054842.png?cb=dcf53144e576c20c8137fcd0a350283f) ## Monitoring and Reviewing Results As this is a BETA feature, aggregation is not supported. Jobs should be run either to a trusted partner or in a peer review workflow. To set that up, you simply use the review-from parameter outlined above. ### Results #### Reviewing Results To review the results of your job, you can either use our[**Quality Audit**](https://appen-external.atlassian.net/wiki/spaces/ASC/pages/5800221/Guide+to+Quality+Audit)feature (recommended), or the following: 1. Go to the Data page. 2. Click on a unit ID. 3. In the sidebar of the annotation tool, select an option from the drop-down menu. * You'll see different contributor IDs, which allow you to view individual annotations. * Click on a box or polygon to view its transcription. ##### **Example output from an Image Transcription job** { "ableToAnnotate": true, "imageRotation": 0, "groups": [], "annotation": { "pages": { "1": { "shapesInstances": { "4db2c949-23bf-4fdd-b9ba-d0e84628e58f": { "visible": true, "x": 121, "y": 991, "height": 50, "width": 188, "angle": 0, "coordinates": { "x": 121, "y": 991, "w": 188, "h": 50 }, "metadata": { "shapeTranscription": { "label": "Transcription", "modelType": "ocr", "modelDataType": "block", "inputType": "text", "annotatedBy": "machine", "text": "some text" } } } } } }, "shapes": { "4db2c949-23bf-4fdd-b9ba-d0e84628e58f": { "id": "4db2c949-23bf-4fdd-b9ba-d0e84628e58f", "number": 1, "color": "#00B0FF", "pageNumber": 1, "type": "box", "class": "Car" } } } } **ableToAnnotate** : A boolean value indicating whether the contributor is allowed to annotate the image or not. **imageRotation**: An integer value representing the angle of rotation of the image in degrees. **groups**: An array containing the groups associated with the annotation. * **id**: A unique identifier for the group. * **class**: The class of the annotation group. * **shapes**: an array of shape IDs that belong to the group. * **number**: the number of shapes in the group. **annotation**: It contains information about the annotations made on the image in an array. **pages**: An object that represents each page of the image, if it has multiple pages and contains information about the shapes and their instances. In this example, there is only one page. **shapes**: an object that represents each shape on the image, where each shape has a unique ID and the following attributes: * **id**: a unique identifier for the shape. * **number**: the number of the shape. * **color**: the color of the shape. * **pageNumber**: the page number where the shape is located. * **type**: the type of shape (in this case, a "box"). * **class**: the class of the shape (in this case, "Car"). **shapesInstances**: Within the "pages" object, there is an object shape that represents instances of each shape on the image, where each shape instance has a unique ID and the following attributes: * **visible**: a boolean value that indicates whether the shape is visible or not * **x**: the x-coordinate of the shape instance * **y**: the y-coordinate of the shape instance * **height**: the height of the shape, for instance * **width**: the width of the shape, for instance * **angle**: the rotation angle of the shape for instance * **coordinates**: an object that represents the coordinates of the shape, for instance, where "x" and "y" are the top-left corner coordinates and "w" and "h" are the width and height of the shape for instance, respectively. * **metadata**: an object that contains additional metadata about the shape instance, such as transcription information. In this example, there is a "shape Transcription" attribute that contains information about the text transcribed by a machine. ### Using Multipage PDF When a multipage PDF is used as the source data, the tool will automatically transition to multipage mode, which will affect the output. The output format is as follows: { ableToAnnotate: true | false, annotation: { shapes: { [shapeId]: { category, report_value, type } // Static properties of shapes }, groups: { // Groups related properties } pages: { [pageId]: { shapesInstances: { [shapeId: { x.y.w.h. annotated_by, metadata } // Dynamic properties of shapes } } }, } } It should be noted that `output-format=url` is required to allow multipage PDFs. *** ** * ** *** ## Using Bulk Select This feature improves the work efficiency by allowing the user to select and edit (moving or deleting) many shapes at once. It can be done by holding SHIFT and drawing or clicking. Then, they all can be deleted or moved. ![image-20260626-054919.png](https://success.appen.com/__attachments/a_f885aa7b1bb50fdc4e12234791ff76244a7ec03f9c11dc34acf0ba70c679d987/image-20260626-054919.png?cb=74b136a818167cfaec5eefe811d5de69) Fig.:*Selecting many shapes by using bulk select* ## Groups ### Annotating with Groups With the groups feature, the user can create group shapes by using classes already defined for each job. #### Defining the grouping classes On the Design / Ontology Manager page, the job designer can create a class for groups by selecting GROUPING in the toggle. When creating a grouping class, it is needed to define a title and a color. The class description and report value are optional fields. ![image-20260626-055033.png](https://success.appen.com/__attachments/a_3d2030979568b99119f80a2e23bc9a06261d538655faad67edc2f01ab68c9524/image-20260626-055033.png?cb=000845e5393b024a672bad3be61d305b) #### The groups tab When the job is configured with grouping classes, a tab named "GROUPS" will appear along with the "SHAPES" tab. When the latter is selected, the annotation shapes mode is enabled, and the tool behaves as usual. ![image-20260626-055101.png](https://success.appen.com/__attachments/a_a2a2b368006d912504ffcdac2d98973c17f9997355c33693d3d2ba5ee309078f/image-20260626-055101.png?cb=04c7f32a9aa449a22e4608841ec004c4) Fig*.: Image Transcription Tool sidebar with SHAPES tab toggled* By toggling the "GROUPS" tab, the grouping mode is enabled. The sidebar will show the group classes available for the job. ![image-20260626-055251.png](https://success.appen.com/__attachments/a_f6c67a4efa9b76c80e3177c5f201df359f723073c312a6c090bbc1b24d409724/image-20260626-055251.png?cb=dfb4de789d5ca37fd2f51d7240d1a787) Fig.I*mage Transcription Tool sidebar with GROUPS tab toggled* ### Creating a group With one of the group classes selected, the user can bulk select shapes (by holding SHIFT and selecting multiple shapes) and click on the grouping icon (cmd/ctrl + G) to create a group for the selected class. ![image-20260626-055338.png](https://success.appen.com/__attachments/a_ed71ebb3898c3acbd5e6d08818b743b55f0a0eff79b6446771ddca7a5ac82a14/image-20260626-055338.png?cb=824a0b9e7fc0fa70e44fc09a71b96b7b) *Fig.: Selecting many shapes by using bulk select* ### Editing a group To edit a group, the user needs to click on the edit group icon in the toolbar. Once enabled, the group editing mode allows the user to add/remove shapes from the group by holding SHIFT and selecting/unselecting shapes. ![image-20260626-055435.png](https://success.appen.com/__attachments/a_8cfe9d77b35783cbfceb2a1b69d6dc250a362159ca2d76e9bd53c31bb9a8cc6d/image-20260626-055435.png?cb=cee9594d107488c015cbf7ddacf35bd7) *Fig.: Editing group by clicking on the edit group icon* To remove a shape from a group, the user can also click on the ᐅ icon on the group class to view the group instances of that class. Then, by hovering/selecting the shape on the sidebar, click on the minus symbol to remove it from the group. ![image-20260626-055509.png](https://success.appen.com/__attachments/a_61abb9a294594333c55a0a5da29aca4a15d33b6b9c61ce2512fc858d1db9417c/image-20260626-055509.png?cb=da7723588f0ab6149606cab89137f0fd) *Fig.: Removing a shape from the group by clicking on the minus iconFig* If a group has less than two shapes, the group is deleted. ### Deleting a group There are two ways to delete a group: 1. By selecting a group and clicking on the trash icon on the toolbar. 2. Or by clicking on the trash icon on the group accordion on the sidebar. ![image-20260626-055843.png](https://success.appen.com/__attachments/a_92cfdc638a9af5fe4312d03384b637c08827cb8211bb4c1f932afb6d9dc15425/image-20260626-055843.png?cb=fe8441149cb13b05553b01d1ef2ef3b6) *Fig.: Deleting a group* *** ** * ** *** ## Pre-Labelling Pre-labelling dramatically improves the ability to obtain high-quality image transcriptions efficiently. When the user draws a box around a line of text, the Pre-Labelling model will automatically assign individual boxes to each word in that line. Then, the OCR model will predict and transcribe each word. Currently supported in thirty-one languages. ![image-20260626-055803.png](https://success.appen.com/__attachments/a_8f318f3ef1459e2d1732ce18b003a253be58ad341a339f7e03eda4f4888f70b6/image-20260626-055803.png?cb=c31abf063f47f23effd03b989a193dce) *Fig.: Example of the pre-labelling feature* [Next Page](https://success.appen.com/llms-full.txt/1)