OpinionReady

Buying sample

Three hops and the loss of provenance

Sample rarely travels in a straight line. Each time it changes hands something specific is lost, and by the third hand nobody can tell you where a respondent came from.

A whiteboard sketch of sample routing between several supply partners

You buy 500 completes from a supplier. The supplier fills 340 from their own panel and routes 160 to an exchange. The exchange fills 90 from members and routes 70 to two other suppliers, one of which routes part of it onward again. Your dataset has four sources in it and your contract names one.

This is ordinary. Blended supply is how the industry meets volume, and there is nothing wrong with a respondent arriving through three intermediaries as long as somebody can still say who they were and how they were recruited. The problem is that the ability to say it degrades at every hop, and it degrades in a specific order.

What is lost, hop by hop

Hop Usually still known Usually already gone
First — your named supplier Recruitment source, profile history, prior participation, reward paid, quality history Nothing
Second — their exchange or partner Market, broad demographics, session integrity signals, a partner identifier Recruitment source, how long the person has been on a panel, what they were paid
Third — that partner's partner Market, a session id, whatever the redirect carried Any profile not re-asked in your own screener, duplicate detection across the chain, quality history of any kind
Fourth and beyond That a session arrived and returned a complete Effectively everything, including whether the same person is already in your dataset

Two of those losses matter more than the rest.

Duplicate detection stops working across hops. Each operator deduplicates within their own membership. Nobody deduplicates across the chain, because the identifiers are deliberately not shared. A person who belongs to four panels can enter your study four times through four routes and be counted as four respondents, and every operator's internal report will correctly show zero duplicates.

Reward information disappears first. By the second hop nobody downstream knows what the respondent was actually paid, and the amount can have been reduced at every step. A respondent receiving a fraction of what you paid per complete is both an ethical problem and a data problem — underpaid respondents speed, and speeding is what your quality controls then spend their time catching.

Two questions that reveal the depth

Suppliers will describe their supply as blended. Almost nobody will volunteer the depth. These two get you an answer:

  1. "For the last study you ran for someone like us, what share of completes came from your own members, and what share from partners?" A supplier who measures this answers with two numbers. A supplier who does not says "the majority are ours", which is a sentence that survives any actual ratio above zero.
  2. "If I find a duplicate in the delivered data, can you tell me which partner it came through?" This is the one that separates a real audit trail from a claimed one. It is answerable if partner identifiers are carried through to delivery and unanswerable if they are not — and it is unanswerable far more often than the first question's answer would suggest.

What we do about it here

The supplier code and their identifier for the study are written onto the attempt row at the moment somebody leaves for a questionnaire, not reconstructed afterwards from logs. That means every settled complete can name the route it came through, and a duplicate found in delivery can be traced to a partner rather than argued about.

Settlement is idempotent on that same row, which matters more than it sounds. Both a server-to-server postback and a browser redirect can report the same completion, and the two arrive in either order or at the same moment. Deciding what to pay by reading the row and then writing it back pays twice under concurrency, silently, and the accounting still balances against itself. So the decision is made by a conditional write and the answer is whether that write changed anything.

We also publish the depth. Our own members are the first source; partner supply is used and named rather than described as a blend. A buyer who asks the two questions above gets two numbers and a traceable identifier, which is the minimum that makes the phrase "blended sample" mean anything at all.

The case for tolerating hops

None of this is an argument for single-source sample. Single-source has its own failure, and it is worse: a panel asked to deliver volume beyond its natural reach starts over-using the members it has, and the same two hundred people answer everything in a category. That produces internally consistent data that is consistently wrong, and unlike a duplicate it leaves no trace at all.

Blended supply with a named chain beats single-source at volume. Blended supply with an anonymous chain is a dataset you cannot defend when somebody asks where it came from — and somebody eventually asks.

Questions people ask

What does blended sample actually mean?
That some completes came from the supplier you contracted with and some came from their partners, possibly several hops away. It is normal and fine, provided somebody can still say which route each respondent arrived through.
Can duplicate respondents be detected across different panels?
Mostly not. Each operator deduplicates within its own membership, and the identifiers are deliberately not shared between them. A supplier claiming clean cross-panel deduplication is describing something that does not exist.

Filed under Buying sample

Who wrote this

OpinionReady Method Desk — Panel operations and data quality

The people who run the exchange day to day — supply operations, quality review and the payout queue. These notes come out of work we had to do anyway, written down because the same questions keep arriving from buyers and from members.

Written from our own operating record: the screening calls, field estimates and payout runs described here are ones we make on this exchange, not secondhand summaries of research done elsewhere.

More on buying sample

Everything on this topic