Reading a sample supplier’s quality report

The ESOMAR 37 questions are good questions. The answers are usually written by someone whose job is to make the document pass, and a document can pass while describing a process nobody runs.

The difference is almost always visible, because a supplier who measures something answers with a number and a supplier who does not answers with a process description. Below are the questions that separate them most reliably, and what each kind of answer sounds like.

Question by question

“Where do your respondents come from?”

Evasive: "A diverse mix of proprietary and partner sources, carefully vetted." This describes every supplier that has ever existed, including the ones you would not use.

Useful: A split, with percentages, for work resembling yours. Plus the recruitment channels behind the proprietary share — organic search, referral, paid acquisition and affiliate traffic attract measurably different people, and affiliate traffic is where fraud concentrates. A supplier who will not break that out either does not know or would rather you did not.

“How do you prevent duplicate respondents?”

Evasive: "Digital fingerprinting and industry-standard deduplication technology."

Useful: Within their own membership, the specific identifier. Across partners, an honest admission that it mostly does not work, followed by what they do instead. The honest admission is the signal — cross-panel deduplication is genuinely hard, and a supplier claiming it works cleanly across a blended chain is describing something that does not exist.

“What share of completes do you reject?”

Evasive: A number with no denominator, or worse, a range like "typically 3–5%", which is a way of saying nobody pulled the actual figure.

Useful: One number, one period, one definition, and a split between automated removal and human review. A rejection rate near zero means nothing is being caught. A rate above roughly 15% usually means the screening is removing ordinary people, and both are worth asking about.

“Do you pay for screen-outs?”

This question is not on most standard lists, and it predicts data quality better than several that are. An unpaid screen-out teaches a respondent that the screener is an obstacle rather than a question, and a panel that has taught its members this spends the rest of its life catching behaviour it created. Ask it.

“What is your median time to field completion?”

Evasive: An average. Field time distributions have a long right tail, so the mean is always prettier than the experience.

Useful: A median and a 90th percentile. The gap between them is what your project timeline actually needs to absorb.

The four numbers

Underneath every question above sits the same test: has this supplier measured anything. Four figures answer it, and a supplier who has all four has a functioning quality operation almost by definition:

Number	What it proves exists	What a bad answer sounds like
Rejection rate, with period and denominator	Removal decisions are recorded rather than made ad hoc	A range, or a figure with no period attached
Reversal rate on contested removals	Removals can be contested, and somebody counts the mistakes	"We rarely have appeals" — which usually means there is no way to make one
Own-panel share on comparable work	Supply chain depth is tracked per project, not asserted in general	"The majority are ours"
Median and 90th percentile field time	Delivery is measured as a distribution rather than a promise	An average, or a target presented as a result

The second row is the one almost nobody can answer. A supplier who removes respondents and never hears from them again has no evidence their screening works — only evidence that it fires. Catch counts prove activity. Reversal rates prove accuracy, and they are the only thing that does.

What to do with a bad answer

Not necessarily walk away. Vague answers often mean the person writing the document is three departments from the people who have the data, which is a procurement problem rather than a quality problem. Ask once more, specifically, and ask for the number rather than the explanation.

What should end a conversation is a confident answer that is not true — a claimed cross-panel deduplication that turns out to be within-panel only, a rejection rate that nobody can reproduce, an own-panel share that changes when you ask about a specific project. Vagueness is a document problem. A confident wrong answer is a culture problem, and it will be in your data.