Data quality
Checks that run before field, not after it
Fraud caught after delivery has already cost you money and distorted a result someone may have acted on. Most of our checks therefore run before an invitation is sent, where a removal costs nothing and changes no counts. This page names each check and says plainly what it catches and what it misses.
The checks, one at a time
- Device clustering
- Accounts sharing a device fingerprint are grouped and treated as one participant for invitation purposes. Catches: one person running several accounts. Misses: a household sharing a laptop, which is why a cluster reduces invitations rather than banning accounts outright.
- Profile contradiction tracking
- The same underlying attribute is asked in different forms at different times, and disagreements are recorded. Catches: members reshaping a profile to reach higher-paying work. Misses: genuine life changes, which is why a single contradiction lowers a score rather than closing an account.
- Pre-study edit flag
- A profile answer changed within minutes of a study becoming available is treated as a signal. Catches: targeted misrepresentation. Misses: nothing much — this one is narrow and precise, which is why it is weighted heavily.
- Speeding, measured per study
- Completion below a share of that study’s own median rather than a fixed number of seconds. Catches: people clicking through. Misses: genuinely fast readers on short questionnaires, which is why the threshold moves with the study.
- Straight-lining
- Identical responses down a matrix grid. Catches: disengagement. Misses: respondents who legitimately hold the same view across a battery, so it is combined with timing rather than used alone.
- Open-end repetition
- Free-text answers reused across sessions or across accounts. Catches: copy-paste farming. Misses: short generic answers that are indistinguishable from honest brevity.
- Callback signature verification
- Every completion callback is signed and compared in constant time. Catches: anyone trying to credit themselves a complete by guessing a session id. Misses: nothing in this class — an unsigned callback is simply not settled.
Why every entry above has a “misses” line
Because a check with no false negatives would also have to be so strict that it removed honest people, and removing honest people is its own kind of data damage. Panel members who are rejected without cause stop taking the work seriously, and a demoralised panel produces worse data than a slightly leaky one.
That is also why removals are appealable. A member can see which rule fired and ask for it to be reviewed, and a successful appeal restores both the payment and the score. Providers who treat participants as suspects end up with a panel of people who are only there for the money, which is precisely the population you were trying to screen out.
Rate, not perfection
The measure worth publishing is not “we catch all fraud” but the share of completes removed before delivery, reported every quarter on the same basis. A rate that never moves is a rate nobody is really measuring.