Follow

Guide to: Fair Pay

Overview

Our Fair Pay initiative aims to ensure that each contributor working on the Appen platform is paid fairly for their work, which is based on the contributors' geographical location. Users set their target hourly rate, and only contributors residing in countries eligible for that rate will have access to the job. 

Important Note: Fair Pay is not enabled by default for all customers. For more information regarding this feature, please reach out to your Customer Success Manager or the Appen Platform Support team via chat or email (help@appen.com).

How it Works

Before a job is launched, we use a machine learning model to predict how long it will take contributors to submit a judgment; from there, the price per judgment is automatically set based on the target hourly rate set by the user. 

Once the job has collected 300 judgments, we run a second, statistical model to calculate the 75th percentile time per judgment from actual submissions. This new number informs what the price per judgment should be to reach the target hourly rate set by the user. 

The Fair Pay launch flow is centered around two main pages: The Fair Pay job settings and the Launch page. 

Fair Pay Settings

The Fair Pay Settings page allows you to determine what happens once the second model is run to get the actual time per judgment (TPJ). 

blobid0.png

Figure 1: Fair Pay Settings Page

Actual Time per Judgment Decreases

When the actual TPJ is lower than the predicted TPJ, this means contributors are submitting work faster than predicted, and therefore the cost per judgment needed to reach your target hourly rate is less than initially determined.

Your options are the following:

  1. Pause the job and ask me about price changes
    • The job will automatically pause when model 2 runs. 
    • The price per judgment (PPJ) will be decreased automatically to maintain the target hourly pay, but you will need to manually resume the job.
  2. Decrease my PPJ to maintain the initial hourly pay
    • The job will not pause when model 2 runs. 
    • The PPJ will be decreased automatically to maintain the target hourly pay and no action is needed on your part. 
  3. Keep my PPJ constant and increase the reach of my job
    • The job will not pause when model 2 runs. 
    • The PPJ will remain the same as pre-launch, but the hourly pay will be increased, as contributors are working faster. This means your job is open to more contributors. 

Actual Time per Judgment Increases up to X%

When the actual TPJ is higher than the predicted TPJ, this means contributors are submitting work more slowly than predicted, and therefore the cost per judgment needed to reach your target hourly rate is higher than initially determined.  

You will set the percentage price increase that you want to be actionable. For example, if you want to take action at a 10% increase in the PPJ, you will input that value. If the actual TPJ is up to 10% higher than the predicted TPJ, the option you choose will be executed. 

Your options are the following:

  1. Pause the job and ask me about price changes
    • The job will automatically pause when model 2 runs. 
    • The price per judgment (PPJ) will be increased automatically to maintain the target hourly pay, but you will need to manually resume the job. 
  2. Increase my PPJ to maintain the initial hourly pay
    • The job will not pause when model 2 runs. 
    • The PPJ will be increased automatically to maintain the target hourly pay and no action is needed on your part. 
  3. Keep my PPJ constant and decrease the reach of my job
    • The job will not pause when model 2 runs. 
    • The PPJ will remain the same as pre-launch, but the hourly pay will be decreasedas contributors are working slower. This means your job is open to fewer contributors. 

Actual Time per Judgment Increases beyond X%

The value you input for the actionable threshold is applied to both scenarios.

For example, if your PPJ increases more than 10% in this case, you may choose to handle that differently. Of course, you can choose the same option for this setting as the setting outlined above as the options are the same. This exists to give you more flexibility and control over the job’s behavior. 

Launch Page

The Launch Page contains settings that will impact the cost of your job:

Crowd Channels

Select which crowd(s) you want to target:

  • External
    • General
    • Custom Channel
  • Internal
    • Fair Pay has no impact on this channel selection as payment is not used through the internal channel.
  • NDA (if enabled for your team)

Please see the Contributor Channels article for more information on channel settings.

blobid1.png

Figure 2: Crowd Channel selection via Launch Page

Crowd Settings

  • Custom Channel selection and creation (if selected in Crowd Channel section)
    • Please see this article for more information on Custom Channels.
  • Maximum Hourly Pay
    • This is the target maximum hourly pay that you set.
    • Contributors will be paid based on their location, and your job will be targeted to all countries at or below your Maximum Hourly Pay. 
      • For example, if you set your Maximum Hourly Pay to $10, your job will be available to countries where the hourly wage is $10 or less, and contributors will be paid the hourly wage corresponding to their location – not all contributors are paid $10/hr.  
    • Contributors are still paid per judgment, which is why this is a target hourly rate 
  • Maximum Price per Judgment (automatically calculated)
    • This is a function of the predicted Time per Judgment (see below) and the Maximum Hourly Pay (above). Again, please note that contributors are still paid per judgment.
  • Time per Judgment (predicted)
    • We use a machine learning model to predict the time it will take a contributor to submit a judgment in your job. This model is trained on thousands of jobs run on the platform and takes into account the complexity of the job design. 
  • Advanced Settings
    • Explicit Content, Contributor Level, and Contributor Channels
    • Geography Based Filters
      • When geo-targeting, you will need to set your hourly pay to the highest of the included countries’ wages to ensure your job is available to all countries you target.
      • The launch page uses a validator to ensure the correct value is set. 
    • Language Based Filters
      • Please see this article for information on geography and language targeting.
    • Internal Channel link (if selected in Crowd Channel section)

blobid2.png

Figure 3: Crowd Settings via Launch Page

Job Launch Settings

  • Judgments per Row
    • This is the number of unique, trusted judgments that the system will collect for each row of source data. The number of judgments per row determines the number of unique contributors you would like to view and judge each row of source data.
    • Please note: Test Question rows are not affected by the number of judgments per row.  
  • Rows per Page
    • Contributors see one to several rows in a single page. A page is a collection of rows that a contributor must complete before submitting their answers. This setting determines how many rows are on each page of contributor work.

blobid3.png

Figure 4: Job Launch Settings via Launch Page

Rows to Order

  • You can select how many rows you would like to order for each run of the job.
  • The default is 100 rows for any new job created. It is recommended that 100 rows be run first to determine if test questions, job design and instructions are clear. 

Estimated Cost

Because payment is dynamic based on contributor location, we’re unable to provide one defined job cost estimate. As such, we provide a range: 

  • The lower end is the case where all contributors are working from the lowest-wage country in our system 
  • The upper end is the case where all contributors are working from the highest-wage country based on your hourly rate, minus 30% 
    • Analysis shows that jobs rarely reach the upper end of the cost estimate (where all contributors come from the highest-wage country), so we deduct 30% to provide a more realistic estimate.

Allocate Funds

  • Because the job cost estimate can be a wide range, we allow you to determine how much money you wish to allocate to the job. We provide a recommended value typically closer to the lower range, but you may allocate anywhere within the range.  
    • Please note, your job can still run out of funds if it collects too many untrusted judgments, at which point you will be prompted to allocate more funds. 
  • The value must be between $10 and the upper range of the estimate 
    • If your cost estimate is below $10 even at the upper range, we will default to a $10 order, which is our required minimum. 

blobid4.png

Figure 5: Rows to Order, Estimated Cost, and Allocate Funds via Launch Page

Updated Price Prediction

  • Once your job has collected 300 judgments and the second model has been run, a notification of an updated price per judgment notification will appear on the Launch Page.
  • If you chose to pause the job in your Fair Pay Settings, an email alerting you of the pending changes will also be sent. 
  • If your Fair Pay settings dictate that your job should be paused, you need to review these changes and manually resume the job from the launch page or API. You can either choose to accept the changes or change your target hourly pay. 

blobid5.png

Figure 6: Updated Price Prediction Notification on Launch Page

Out of Allocated Funds

  • Jobs may run out of funds if too many contributors are failing out. This could be due to difficult test questions, unclear instructions, or overall a challenging job design. In the case that your job runs out of funds, you will receive an email alerting you, along with a notification on the Launch Page of the job. 
  • We provide the estimated cost range for the ordered rows that have not yet finalized. You will need to allocate more funds and resume the job. 

blobid6.png

Figure 7: Out of Allocated Funds Notification on Launch Page

Using the API

In order to launch a Fair Pay job via the API, you’ll need to know the new and/or updated endpoints. 

There are four fields needed to set your Fair Pay settings: 

  • decrease_behavior
    • This corresponds to the setting labeled “When the price per judgment falls (PPJ) below current PPJ:” 
    • The options are:
      • pause 
        • This will pause the job upon judgment time recalculation
      • adjust_pay
        • This will adjust the price per judgment to maintain the initial hourly pay 
      • adjust_reach
        • This will adjust the hourly pay to maintain the initial price per judgment
  • small_increase_behavior
    • This corresponds to the setting labeled “When the price per judgment increases up to:” 
    • The options are the same as above  
  • large_increase_behavior
    • This corresponds to the setting labeled “When the price per judgment increases beyond:” 
    • The options are the same as above 
  • large_increase_threshold
    • This is the threshold you set for both the small_increase_behavior and the large_increase_behavior; they cannot be unique values 
    • This accepts a float between 1 and 100 

See example below:

curl -X PUT https://api.appen.com/v1/jobs/{job_id}/pay_settings --data-urlencode "decrease_behavior=adjust_pay" --data-urlencode "small_increase_behavior=adjust_reach" --data-urlencode "large_increase_behavior=pause" --data-urlencode "large_increase_threshold=15.0" -d "key={api_key}" 

 

To set the hourly pay, we’ve added the hourly_pay field to the job update API: 

curl -X PUT --data-urlencode "job[hourly_pay]={n}" 'https://api.appen.com/v1/jobs/{job_id}.json?key={api_key} 

 

If you’re geotargeting, you can execute the following command to see what value you need to set for the hourly pay in order to reach your targeted countries: 

curl --location --request GET 'https://api.appen.com/v1/jobs/{job_id}/max_pay_for_crowd?key={api_key} 

 

To see the estimated cost range of the job, execute the following command: 

curl -L 'https://api.sandbox.cf3.us/v1/jobs/{job_id}/allocation_range?key={api_key} 

 

To allocate funds to a job, pass the allocated_funds parameter with your order endpoint. This is the amount in cents you wish to allocate to the job; the example below is allocating $10: 

curl -X POST https://api.appen.com/v1/jobs/{job_id}/orders.json --data-urlencode "channels[0]=on_demand--data-urlencode "debit[units_count]=100" --data-urlencode "debit[allocated_funds]=1000-d "key={api_key}" 


Was this article helpful?
0 out of 0 found this helpful


Have more questions? Submit a request
Powered by Zendesk