Guide to: Quality Flow Project Basics – Appen Success Center

This article is intended as a starter guide for users who want to learn how to set up or configure projects within the Quality Flow Projects framework, which is different from ADAP jobs and workflows. Please talk to your CSM if you'd like to know more.

Project set-up

Quality Flow can be enabled as a flag on your team, which you can request from your CSM. Once enabled, look for the “Projects” icon in the lefthand sidebar.

When you click the icon, you’ll be brought to an empty project page, or a list of existing projects if someone else on your team has already created any, along with some high level information about the Project, such as the creation date, number of units and a raw progress bar.

Create a Project

To create a project, click “Create Project”, in the top right of the Projects list page. The first pop-up you’ll see will look as follows. You will need to name your project and optionally give it a description. Once you click "Confirm" you will be taken back to the Projects list page.

Once back in the Project list page, click on your new project; you will be taken to the DATASET page.

You will see four tabs across the top of the page.

Dataset — this is your landing page where you upload and download data and perform actions on data such routing to jobs.
Jobs — this is a canvas where you create Jobs and link them via flows.
Curated Contributors — this is where your Appen Managed Services team will manage any curated contributors.
Dashboards — this is where you monitor your throughput and quality, as well as download your results as reports.

Add data to your Project

On the DATASET page you'll be prompted to add data. You can move on and create Jobs without adding data, but it is recommended to add data first as this will help to ensure your Job design and data structure are aligned from the start.

Quality Flow Projects expects a .csv or .tsv like other ADAP jobs
If you are using image, video or audio annotation tools, Quality Flow Projects expects input data to be an object such as a CDS link.
If you wish to use the Unit Groups feature, please make sure to read this article on formatting your upload data first.

You can add more data at any time by clicking on "Add More Data".

After you have uploaded some data, the table on the DATASET page will become populated. Additional columns will appear in the data table, such as the following, which are grouped into sections including “unit” and “source”.

- Unit ID — this is an automatically generated ID specific to Quality Flow
- Job title — this will be blank until the unit is sent to a Job, after that it will show the Job that the unit is currently in.
- Job ID — this is the Job ID associated with the Job title.
- Status — status of the unit. See Statuses.
- Created At — this will show the date/time of the record.

To see all available columns, or to select which columns you want to view click on Columns.

Columns are filterable and sortable, but note that while your column selections will persist until you change them again, your filters will only persist while you do not navigate away from the page.

Notice the "Source" columns are deselected by default, select these columns to view your source data and check it has uploaded as expected.

Create a Job

To create a job in your project click on the JOBS tab. If you have already uploaded data, you’ll see a box on the canvas that says “ALL DATA” which will show the number of units you’ve uploaded. At the bottom of that box, click the small + to add a job.

If you have not already loaded data you will prompted to do so. Click on "Add data to Project" and you will be taken back to the DATASET page (see Add data to your Project, above).

When you click the + at the bottom of the ALL DATA box, you’ll be prompted to give your new job a title.

Notice this job is called a "leading job", all Jobs that flow directly off the ALL DATA box are "leading jobs".

Once you name your job and click “Confirm”, you’ll be brought to the template screen.

Notice that there are two boxes at the top: “Work Job” and “Quality Assurance (QA) Job”. You must select one of these first. In most cases, you will start by creating a leading Work Job. Leading QA Jobs are typically used later in a project for targeted revisions of previous work (however, see this article for more information about QA Jobs). When you are ready to run Quality Assurance on your existing work jobs, you will use a "following QA job" (see this article).

After you have chosen "Work" choose "Start From Scratch" or else a job template. From here, you will be taken to the DESIGN tab for the new job and you can proceed with designing your Job, as usual (see this article).

Note that you can create multiple leading jobs. Routing units between leading jobs is either can be done via filtering and Actions > Send Selected Data to a Job on the DATASET table. You can do this manually at any time, and you can also set a sampling frequency, see Data Management options, below, and this article.

Configuring Leading Job Settings

Unlike in ADAP, when you first create a Job in Quality Flow Projects it will be in DRAFT state. You will need to go through the Settings to get the Job to “Preview Mode” before you can send data to the Job (see below and Job States).

In order to configure settings for your Work job, skip the QUALITY tab and go straight to SETTINGS.

If you click into the QUALITY tabs, for Work Jobs you’ll see that there’s nothing to do here — Quality Flow Projects does not support test questions yet, and quality configurations are configured in QA Jobs instead, see this article.

Contributor Settings

This is where your Managed Services team will manage any curated contributors. It is not yet possible for the general contributor population to work on jobs in Quality Flow Projects. Here is also where you can find the internal link to share with your in-house teams.

Prices & Rows Settings

This is where you set your rows per page, lease expiry time and max number of units.

Judgments per row can be set from 1-20. Please see this article for important information about collection multiple judgments in Quality Flow -- the number of judgments cannot be changed once a job is running.
If you are using Unit Groups you will be asked to retain or ignore groupings, see Unit Groups for details.
Adjust the number of units you would like displayed per page.
Adjust the lease expiry in line with how much time you think a contributor will need to complete units. (If you are using Unit Groups, set the Assignment Lease Expiry time to allow the contributor to complete the entire unit group.)
If you want to limit the number of units a contributor can complete in the Job, ensure you select Enable max Judgments per Contributor and specify the max number of units (this can be useful in early testing and callibration stages for example, and it can be adjusted after a job has been running). Otherwise, keep this toggle to OFF.
If you want to allow contributors to abandon (or skip over) units, you can select Allow Abandon Units (Abandoned units will go back to the contributor pool to be worked on by someone else, they will not go back to the original contributor). Otherwise keep this toggle OFF.

Save your settings.

Review

On the next page click on the button “Switch to Preview Mode”. This will put the job into a PAUSED state.

You will be taken to the Job’s data page and then redirected back to the DATASET page to add data to the Job (see Sending data to a Job below). You can add data to Jobs that are PAUSED or RUNNING see (see Job States).

Once you have added data to a PAUSED job, you will be able to Preview how your Job will look with the chosen data in it by clicking on a unit id link in the data table.

Sending data to a Job

Once you’ve instantiated your job, you’ll want to send data to it. Return to the DATASET tab and select some data that you want to be processed in this job (see this article for data management options and suggestions).

Once you have some units selected, click on the Actions dropdown, and choose Send Selected Units To A Job. Start typing the name of your target job in the field and select it when it pops up.

Note: Jobs in DRAFT state will not be available to send units to (see Job States).

From this pop-up you can also set your sample rate for routing:

if you want all of the units you’ve selected to go into this job, leave the sample rate at 100.
Or you can also choose the number of units you wish to send from this selection, or a percentage. This sample is random.
If you are sending units to a job to be worked on for the first time, you can ignore the options to carry over and overwrite judgments section, these options become important if you are sending units to be worked on again for rework or to collect additional judgments or annotations (see this article).

Refresh the page and the Data table will be updated. The units will now have been updated with their current Job Title and Job ID as well as a Status. You will also see the DATASET summary banner at the top of the page has become populated (see Unit Statuses for details).

If you go back to the job now (click either the Job Title or Job ID links in the data table), on the DATA tab there will be units listed in the job’s data table.

Note the only Actions you can perform from the Job’s DATA page are Assign/Unassign to Contributor. These actions are useful if someone has had units checked out and you want to give them to someone else to finish, or you want to assign particular units to particular people, in a practice job, for example.

Run a Job

When you are ready to fully launch the Job to contributors, click on Run Job at the top right of the page.

Your Job will now show as RUNNING on the Canvas.

Pause a Job

If you ever want to make the job unavailable to contributors, click on Pause Job at the top right of the Data page (see Job States). You can restart at any time by Clicking on Run Job again later.

Delete a Job

Leading Jobs that have never been used can be deleted, under certain conditions:

there are no units in the job
there is no submission history in the job

Any following jobs will automatically be deleted as well. To delete a job click on the Trash icon in the top right on any tab in the Job.

Downloading your data

Your data can be downloaded from the DATASET tab at any time, and it will always output the latest result in the "latest..." columns (there is no need to regenerate). First, make the selection of the rows and columns of data you wish to download, or Select All. Click the button in the top row of the data table to select all units in the page.

In the coloured panel above the table, you will also be prompted whether you would like to Select All units matching your filter if you have one, or all units in the Project in the case of no filters.

Click on Actions > Download

A pop-up will appear at the bottom of the screen.

Click on the "Check progress and download" button to be taken to a table where you can view progress and download the results.

Note, after your download process has finished, you can click Download under the Actions column to download the data you selected on the DATASET page.

You can use this table to view progress on any Action you are performing. It can be accessed by clicking on the "Processes Running In Background" Icon at the top right of the DATASET page.