AI Detector gathers behavioral signals from your contributors and computes an AI Detection score. The AI Detector works on jobs containing one smart text box (and no other questions). We do not recommend to use AI Detector when you are collecting texts of fewer than 150 words.
To enable, toggle AI Detector to Enabled on the job's design page:
or switch to the code editor and add <cml:ai_detector/>, to your job's cml.
How it works
When you enable the AI Detector, behavioral data is continuously gathered from your contributors as they complete their task. Our proprietary model identifies units that have a 92% chance of being AI-generated and once three such units are found submitted by the same contributor we know with 99.9% accuracy that one of these three units is AI generated. The contributor is considered positive to AI Detector and suspicious units are available to be downloaded in a report, see below.
If a contributor only shows two units that might have used AI, they will not be considered as positive to AI Detector. If they submit more units and within this new batch of units AI Detector spots a third unit that may have used AI, then the contributor will hit the AI threshold and be considered positive to AI Detector.
Scores are computed every two hours (UTC), on the even hour, after the job is launched.
Reports
Two reports can be found in the RESULTS tab at the job level:
1. AI Detector Report - Contributor Level: will list all the contributors that have been considered positive to AI Dectector, along with their number of units submitted and flagged by AI Detector
2. AI Detector Report - Unit Level: will list all the units considered positive to AI Detector
Note:
As signals are continually processed, and computed every two hours, your job's AI Detection scores will change while the job is running.
Report details
AI Detector Report - Contributor Level
This report is a breakdown of the job's submissions by contributor. For each contributor (Column B), the report list the total units worked (Column D), the number of units which our model predicts may have used AI (Column C) and finally, the number of units we predict as AI generated with 99.9% accuracy (Column E).
AI Detector Report - Unit Level
This report is a breakdown of all units submitted in the job. When the value in predicted LLM (Column E) is 1, then the probability is that this unit has been generated using AI is at least 92%. When the value is 0, then the probability is not high enough to infer that the contributor has used AI on this unit.
The PROMPT_ID (Column D) corresponds to the UnitID column in your full DATASET report and can be further mapped to the UnitDisplayID in the Quality Flow DATASET table. You can review the data in your report, or search for the PROMPT_ID in the dataset page and click on the unit view.
Limitations
When you enable the AI Detector in your Smart Text job in Quality Flow, make sure to:
- use AI Detector with text inputs of at least 150 words.
- configure only one row per page in the job settings, you will also be prompted to adhere to this when saving your job design
- use only one tool per page in the job design, combining an additional tool to your smart_text box will create noise in the signal collection, leading to decreased accuracy in the AI Detector
- do not disable copy/paste, as this would also impact the accuracy of the detection
- enable AI Detector prior to running the job, it is not possible to enable AI Detector after a job has been running.