Guide to: Quality Flow Project Set-up – Appen Success Center

This article is intended as a starter guide for users who want to learn how to set up or configure projects within the Quality Flow Projects framework, which is different from ADAP jobs and workflows. Please talk to your Appen contact if you'd like to know more.

Example Quality Flow Project (JOBS canvas view)

Create a Project

Quality Flow can be enabled as a flag on your team, which you can request from your Appen contact. Once enabled, look for the “Projects” icon in the lefthand sidebar.

When you click the icon, you’ll be brought to an empty project page, or a list of existing projects if someone else on your team has already created any, along with some high level information about the Project, such as the creation date, number of units and a raw progress bar.

To create a project, click “Create Project”, in the top right of the Projects list page (alternatively you can copy an existing project).

The first pop-up you’ll see will look as follows. You will need to name your project and optionally give it a description. Once you click "Confirm" you will be taken back to the Projects list page.

Once back in the Project list page, click on your new project; you will be taken to the DATASET page.

Quality Flow Navigation

On opening your new Project you will see four or five tabs (depending on your Team's settings) across the top of the page and a space to upload your data. These tabs appear at the top of almost every page when you are working within a project.

DATASET — this is your landing page where you upload and download data and perform actions on data such routing to jobs
JOBS — this is a canvas where you create Jobs and link them via flows
CURATED CONTRIBUTORS & CURATED CONTRIBUTORS (MERCURY) — this is where the Appen Managed Services team will manage any curated contributors
DASHBOARDS — this is where you monitor your throughput and quality, as well as download project-level reports

Note: once you have created a job and you are working inside that job these high level tabs are no longer visible. To navigate back to the project-level from within a job there are a couple of options:

there is a "Return to Jobs" link in the top left of each page, this link will take you back to the project-level JOBS canvas where the other project-level tabs are accessible.
additionally, on the DATA tab within each job there is also a button "Go to Project Dataset" which will take you back to the project-level DATASET tab where the other project-level tabs are accessible.

Upload data to your Project

On the DATASET page you'll be prompted to add data. You can skip ahead and create Jobs without adding data, but it is recommended to add data first as this will help to ensure your Job design and data structure are aligned from the start.

Quality Flow Projects expects a .csv or .tsv like other ADAP jobs
If you are using image, video or audio annotation tools, Quality Flow Projects expects input data to be an object such as a CDS link
If you wish to use the Unit Groups feature, please make sure to read this article on formatting your upload data first

After you have uploaded some data, various areas of the page will become populated:

A: the status banner across the top of the page will be populated with the number of units you have uploaded, this banner will provide high-level progress information as the project continues.
B: the ALL DATA table will become populated. Additional columns will appear, such as the following, which are grouped into sections including “unit” and “source”.
- - Unit ID — this is an automatically generated ID specific to Quality Flow
  - Job title — this will be blank until the unit is sent to a Job, after that it will show the Job that the unit is currently in.
  - Job ID — this is the Job ID associated with the Job title.
  - Status — status of the unit. See Statuses.
  - Created At — this will show the date/time of the record.
C: Add more data to the project at any time by clicking "Add More Data"

Columns & Filters

To see all available columns, or to select which columns you want to view click on Columns.
Columns are filterable and sortable, but note that while your column selections will persist until you change them again, your filters will only persist while you do not navigate away from the page.
Used saved filters to manage your common filters
Notice the "Source" columns are deselected by default, select these columns to view your source data and check it has uploaded as expected.

Create a Job

To create a job in your project click on the JOBS tab. If you have already uploaded data, you’ll see a box on the canvas that says “DATASET” which will show the number of units you’ve uploaded. At the bottom of that box, click the small + to add a job.

If you have not already loaded data you will prompted to do so; it is recommended to add data before designing your job(s).

When you click the + at the bottom of the ALL DATA box, you’ll be prompted to give your new job a title.

Notice this job is called a "leading job"; Jobs that are created from the DATASET box are called "leading jobs". For information on use cases & settings where there are multiple leading jobs, see this article.

Once you name your job and click “Confirm”, you’ll be brought to the template screen.

Notice that there are two boxes at the top: “Work Job” and “Quality Assurance (QA) Job”. You must select one of these. In most cases, you will start by creating a leading Work Job. After you have chosen "Work" choose "Start From Scratch" or else a job template. See this article for information on usecases & settings for leading QA jobs.

Design your Job

From here, you will be taken to the DESIGN tab for the new job and you can proceed with designing your Job, as usual (refer to these articles).

Configuring Leading Job Settings

If you are not using Test Questions, and not ready to set up any manual review jobs, skip the QUALITY tab and go straight to JOB SETTINGS, where you will see three settings areas: Test Question Settings, Rows Settings & Contributor Settings.

Note the "Save settings" and "Start data routing" buttons at the top right of the page. During set-up you can save your settings and your configurations will continue to be editable, however once you have clicked "Start data routing", certain settings such as number of judgments to collect will no longer be editable (see Job States in Quality Flow Projects).

If you are using Test Questions, refer to this article, for Test Question set-up as well as information on Test Question specific Job Settings and Job Running instructions. If you are not using Test Questions move on to the Rows Settings and Contributor Settings, described below.

Rows Settings

This is where you set your rows per page, lease expiry time and max number of units.

Note

If you are using Unit Groups there will be additional settings and considerations, see Unit Groups for details.
For information on Dynamic Judgments settings, see this article.

A: Judgments per row can be set from 1-200. Please see this article for important information about collecting multiple judgments in Quality Flow -- also note, the number of judgments cannot be changed once you have started data routing.
B: Adjust the number of units you would like displayed per page.
C: Adjust the lease expiry in line with how much time you think a contributor will need to complete units.
D: Set the time limit alert threshold. For example if the contributor has 10 minutes to complete the page they are working on, and the alert threshold is set to 90%, the contributor will be alerted when they have 1 minute left to complete the page, and the timer will flash for the remaining time
E: If you want to limit the number of units a contributor can complete in the Job, ensure you select Enable max Judgments per Contributor and then specify the max number of units (this can be useful in early testing and callibration stages for example, and it can be adjusted after a job has been running). Otherwise, keep this toggle to OFF.
F: If you want to allow contributors to abandon (or skip over) units, you can select Allow Abandon Units. Abandoned units are either automatically sent back to the dataset pool or you can manage them directly, depending on the additional settings outlined here.

Contributor Settings

External

This is where your Managed Services team will manage any curated contributors. It is not yet possible for the general contributor population to work on jobs in Quality Flow Projects.

Internal

Here is also where you can assign contributors from your Project Resources (see this article), or copy the internal link to share with your in-house teams and start annotating.

Note that the Public link is disabled by default, you will need to enable it before sharing. You can change this setting after launch as long as the job is Paused.

Once you are happy with your selections, at the top of the page click on the button “Start Data Routing”. This will move the job from DRAFT to PAUSED state and make it available to have units routed to it. You can only route data into Jobs that are PAUSED or RUNNING see (see Job States).

Sending data to a Job

Once you’ve instantiated your job, you’ll want to send data to it. After you click "Start data routing" you will be brought to the job's dataset page. Click "Go to Project Dataset" to be taken back to the DATASET tab and start selecting the data you'd like to route to this job. See this article for data management options and suggestions). See this article if you are using Test Questions.

On the DATASET tab, once you have some units selected, click on the Actions dropdown, and choose Send Selected Units To A Job. Start typing the name of your target job in the field and select it when it pops up.

Note: Jobs in DRAFT state will not be available to send units to (see Job States).

From this pop-up you can also set your sample rate for routing:

if you want all of the units you’ve selected to go into this job, leave the sample rate at 100.
You can also choose the number of units you wish to send from this selection, or a percentage. This sample is random.
If you are sending units to a job to be worked on for the first time, you can ignore the options to carry over and overwrite judgments section, these options become important if you are sending units to be worked on again for rework or to collect additional judgments or annotations (see this article).
If you are using Unit Groups you will have some additional options to select (see this article)

Refresh the page and the Data table will be updated. The units will now have been updated with their current Job Title and Job ID as well as a Status. You will also see the DATASET summary banner at the top of the page has become updated (see Unit Statuses for details).

Navigate back to the the Job, by clicking the Job's name in the DATASET table, or else via the JOBS canvas. Navigate to the job-level DATA tab, to view and manage units at the job-level.

Actions you can perform from the Job’s DATA page are Assign/Unassign to Contributor. These actions are useful if someone has had units checked out and you want to give them to someone else to finish, or you want to assign particular units to particular people, in a practice job, for example.

Run a Job

When you are ready to fully launch the Job to contributors, click on Run Job at the top right of the page, from any tab within the job. If you have navigated away from the job, click back into it from the link that will be visible in the DATASET table, or from the JOBS canvas (see Quality Flow Navigation, above).

Once your job is running, this button will now display "Pause Job", see below.

Your Job will now show as RUNNING on the Canvas.

If you ever want to make the job unavailable to contributors, click on Pause Job at the top right of the Job's page (see Job States). You can restart at any time by Clicking on Run Job again later.

Leading Jobs that have never been used can be deleted, under certain conditions:

there are no units in the job
there is no submission history in the job

Any following jobs will automatically be deleted as well. To delete a job click on the Trash icon in the top right on any tab in the Job.

Create a Quality Assurance Job

Next - Guide to: Quality Assurance (QA) in Quality Flow Projects