Fraud and inattention are not the same problem

Every panel eventually builds a list of things that get a response thrown out: finished too fast, failed the attention check, straight-lined the grid, came in through a data centre, gave an open-end that reads like nothing. Then the list gets applied as one rule with one outcome, and the panel starts losing people it wanted to keep.

The Insights Association's fraud taxonomy makes a distinction that most internal quality rules do not: some of these signals indicate somebody trying to extract money without doing the work, and some indicate a real person doing the work badly. Those are different problems. They have different frequencies, different remedies, and — this is the part that costs money — different base rates among your most valuable respondents.

The two classes, and why the line matters

Fraud is intent. A bot farm, a shared account being run by four people in rotation, somebody using a VPN to claim a market they do not live in, an answer pattern generated rather than given. The defining property is that the person on the other end is not the person the buyer paid to reach, and no amount of better survey design fixes that.

Inattention is not intent. A senior manager answering on a phone between meetings goes faster than the same person at a desk. Somebody in their fourth grid of twelve identical scale items starts straight-lining because the instrument is exhausting, not because they are cheating. An open-end of four words from a respondent who wrote eighty words on the previous study is a signal about the question, not about the person.

Both produce data you should not ship. Only one is a person you should remove.

Where the cost lands

Here is the failure that follows from a single bucket. Speeding is the most common flag in almost every panel, and speed correlates with two things at once: with fraud, and with seniority. The respondent who is hardest to recruit — the VP with purchase authority who costs four times a consumer complete — is also the one most likely to be answering in a hurry on a phone.

Apply one rule with one outcome, set it at a threshold strict enough to catch fraud, and the people it removes first are disproportionately the expensive ones. The panel gets cleaner by every metric you are measuring and worse at the job you actually need it for. Nobody notices, because the measurement that would show it — incidence among verified senior respondents, six months out — is not on the dashboard.

How the two classes are scored differently here

Quality score starts at 80 and moves on evidence. The two classes carry deliberately different weights, and the gap is the whole point:

Signal	Class	Score effect	Reasoning
Completed under one third of median time	Inattention	−8	Recoverable. Frequent enough among real respondents that a heavy penalty would remove ordinary people at scale.
Straight-lined a full grid	Inattention	−8	Often a statement about instrument length. The response is dropped; the person is not.
Open-end unrelated to the question	Inattention	−8	Reviewed before it counts. Automated text scoring is unreliable enough that acting on it alone removes people for writing tersely.
Geography contradicts declared market	Fraud	−25	Intent. The buyer paid for a market and did not get it.
One device, several accounts, overlapping sessions	Fraud	−25	Intent, and it scales — one operator can run dozens.
Profile answers rewritten immediately before a high-value screener	Fraud	−25	Answer history exists for exactly this. A correction and a fabrication look identical without it.

Three inattention events and the account is under review rather than removed. One fraud event and it is held. The asymmetry is not softness; it is a bet about base rates. Inattention is common and mostly innocent. Fraud is rarer and never is.

The measurement that keeps this honest

A quality regime that only ever reports how much it caught cannot be evaluated. The number that matters is what share of held accounts were held wrongly, and the only way to produce it is to let people contest a hold and then count how often they turn out to be right.

So every hold is appealable, appeals are read by a person, and reversals are recorded as reversals rather than quietly deleted. The quality score is recomputed from the full event stream rather than incremented, which means a reversed event genuinely disappears from the total instead of leaving a residue that nobody can see.

If the reversal rate is near zero, the thresholds are too loose and real fraud is passing. If it climbs, the thresholds are removing ordinary people. Neither of those is visible from a catch count alone, and a supplier who cannot tell you their reversal rate is telling you they have never measured whether their screening works.

What we deliberately do not do

We do not use a third-party risk score as a removal decision on its own. Those scores are useful as one input and dangerous as a verdict: they are trained on traffic patterns, they are opaque by design, and the person they remove has no way to contest something nobody will explain. A panel that removes people on an unexplainable number is not running quality control, it is outsourcing a judgement it is not willing to defend.

We also do not remove for a single soft signal, ever. One fast completion is not evidence of anything. The pattern across a respondent's history is.