Guide to: Running a Text Relationship Job


The Text Relationship tool allows users to create a job that annotates relationships between spans of text with a custom ontology. 


Note: As this tool is currently in its Beta phase, please contact your Customer Success Manager to gain early access.

Note: For Dedicated customers, this feature is currently not available On-Premises and only available via the cloud multi-tenant Appen Data Annotation Platform.

Figure 1: Text Relationship Tool via Preview Page


    • Span - a string of text with an assigned class label; the output of a model or contributor judgment.
    • Relationship - consists of two spans (from a span and to a span) and the relation between them.
    • Relation - the name/type of the relation between two spans as defined in the job's ontology.
    • From span - the starting span in a relationship
    • To span - the ending span in a relationship

Upload Data

The source data of a Text Relationship job can come from two different sources:

  • The output of a previous text annotation job on the Appen platform
    • The output of a previously ran text annotation job on the Appen platform can be uploaded to a text relationship job directly without any modification.
  • Data created externally
    • A source file containing data created externally can be uploaded to a text relationship job, but an identical JSON format (hosted in a URL and a CORS configured bucket) as the output of a text annotation job on the Appen platform will be required.

Note: There is an example file attached below on how to format source data.

Build a Job


Below are the parameters available for the text relationship tool. Some are required in the element, some can be left out.

  • source-data (required)
    • The name of the source data column containing the data to be annotated
  • name (required)
    • The result header where the result links will be stored
  • context-column(optional)
    • The name of the source data column containing the context of each data row
  • review-data (optional)
    • The name of the source data column containing pre-annotated text relationships
  • task-type (optional)
    • Please set task-type=”qa” when designing a review or QA job. This parameter needs to be used in conjunction with review-data . See this article for more details.


  • The Ontology Manager allows job owners to create and edit the ontology within a Text Relationship job. Text Relationship Jobs require an ontology to launch.
  • When the CML for a text relationship job is saved, the Ontology Manager link will appear at the top right of the Design page.
  • The ontology of Text Relationship jobs allows you to create relationship restrictions to each span class.
  • The ontology of a Text Relationships job can be copied from a Text Annotation job or another Text Relationships job, via download and upload.


Figure 2: Ontology Manager for Text Relationship 

Ontology Manager Best Practices 

  • The limit of ontology is 1,000 classes, however, as best practice, we recommend not exceeding 16 classes in a job to ensure contributors can understand and process the different classes. 
  • Choose from 16 colors pre-selected or upload custom colors as hex code via the CSV ontology upload.

Important Note: If there is no relationship restriction defined to a class, the class will not able to relate to any other classes in the job.


  • Results are links to a JSON file that contains a list of relationships.
  • The links are found in the Full or Aggregated report under the column header that was specified as the value for the name attribute.
  • Result links will expire 15 days after generation; to access results after the links have expired, you will need to re-generate the result report.
  • Each relationship instance is an array of five attributes:
    • id: the unique ID of each relationship instance
    • name: the class name of the relation
    • from_span: contains the details of from_span
    • to_span: contains the details of to_span
    • annotated_by: indicates whether a relationship instance is pre-loaded into the job or manually added by an annotator
  • Below is an example of result JSON


"id": "ec9a7581-970b-42b3-a0c9-45fac8dfeb2e",

"name": "sells",

"from_span": {

"id": "f048c11b-4f7d-40d9-984d-89c1f5f6a1fa",

"classnames": ["Company"],

"tokens": [


"id": "8f3afc82-6c20-4a33-87ed-ad4298c1fb73",

"text": "Nike",

"startIdx": 0,

"endIdx": 3



"annotated_by": "human"


"to_span": {

"id": "7b860a72-4991-419e-a46e-c722bb85d6d6",

"classnames": ["Product"],

"tokens": [


"id": "f94e3afd-1528-4422-9535-a488d4e71a6d",

"text": "Air",

"startIdx": 5,

"endIdx": 7



"id": "2c58bf8e-f2aa-402e-b95a-b4827217b37c",

"text": "Max",

"startIdx": 9,

"endIdx": 11



"annotated_by": "human"




Was this article helpful?
0 out of 0 found this helpful

Have more questions? Submit a request
Powered by Zendesk