Writing a screener people cannot game

A screener is read by people who have been screened out before and would rather not be again. Some will answer honestly. Some will infer the target and adapt. Some automated or professional respondents will arrive already trained to recognize the patterns.

If a screener makes the qualifying profile obvious, a high qualify rate can look like a fieldwork success while producing weak research data. Good screener design does not rely on one “gotcha” question. It combines neutral wording, plausible alternatives, behavioral evidence, profile consistency, and proportionate incentives. Industry data-quality work increasingly emphasizes early screening, logic checks, response-pattern monitoring, and documented review rather than a single post-fieldwork removal rule.

The first two questions usually give it away

Many screeners reveal the target before the respondent has answered anything meaningful. A question about job title followed by a question about purchasing authority tells an experienced respondent exactly which answers will open the study.

Consider this version:

Q1. What is your current job title?

IT Director / CIO / VP of IT / Other

Q2. Are you involved in technology purchasing decisions?

Yes / No

A respondent does not need to know the research objective to see the intended route. The screener has effectively announced: we need senior technology decision-makers.

The better approach is to ask a broader question first, use neutral categories, and separate role from actual involvement. For example:

Q1. Which functions are part of your current role?

Finance / Operations / IT / HR / Sales / Product / Other

Q2. In the past 90 days, which of the following activities have you personally completed?

Reviewed supplier proposals / Approved a budget / Requested a product demo / Renewed a software contract / None of these

The second version still identifies relevant participants, but it asks about evidence rather than offering the qualifying identity as an answer choice.

Decoys have to be plausible, not absurd

Decoys help prevent respondents from learning the answer pattern. But an obviously fake option does the opposite. It teaches people that the list contains a trap and encourages them to search for the “safe” brand, category, or behavior.

A plausible decoy belongs in the respondent’s real decision environment. If a study concerns business software, the list should include adjacent tools or credible alternatives that someone in the target market might genuinely use, evaluate, or confuse with the target category.

For a consumer study, do not include a fictional brand with a name that sounds invented. For a B2B study, do not include a product category that no serious buyer would recognize. The respondent should not be able to identify the correct answer by spotting the least believable option.

Decoys are not a substitute for verification. They are one layer in a broader screener design: neutral framing, behavioral follow-ups, consistency checks, device and identity controls, and post-entry quality review.

Ask about behaviour, not self-description

Self-description is easy to optimize. “I am a decision-maker” is a label. “I approved a supplier renewal in the last 90 days” is a claim that can be tested.

Where possible, screen on recent, concrete behavior. Use a defined timeframe. Ask what action was taken, what stage of the process the person reached, and what happened next. These questions are harder to answer consistently without relevant experience.

For example, instead of asking:

“Are you responsible for selecting cybersecurity vendors?”

Ask:

“Which of the following have you personally done in the past six months?”

Compared vendor proposals / Joined a security-vendor briefing / Recommended a vendor for approval / Signed a contract / None of these

Then ask a follow-up that checks the answer without repeating the same wording:

“What was the primary reason your organization reviewed that category?”

The aim is not to create an adversarial interview. It is to make qualification depend on credible, internally consistent evidence. Survey fraud and overclaiming are better detected through layered checks than through one qualification question.

Cross-check against something asked months ago

A panel profile is useful only if it is treated as a living record rather than a permanent truth.

If a member reported working in operations six months ago and now qualifies as a senior IT buyer, that may be entirely legitimate. People change roles. But it is worth checking whether the change is plausible, whether the account has a history of frequent profile shifts, and whether the new screener responses align with other known facts.

A good cross-check does not automatically reject a participant because an old answer differs. It flags the difference for a proportionate follow-up. For example: “Our records show you previously selected Operations as your primary function. Has your role changed in the last six months?” The participant can correct the profile, and the panel retains an audit trail of the update.

This requires a clear privacy boundary. Participants should be told that profile information may be used to match them to relevant studies, validate eligibility, prevent duplicate participation, and maintain research data quality. Do not use historical profile data for unrelated purposes, and do not retain it longer than the stated policy permits.

Unpaid screen-outs sabotage your own screener

A screener works best when honest answers feel safer than strategic answers.

If a participant repeatedly spends several minutes answering qualification questions and receives nothing after a screen-out, the panel is teaching them that candour has a cost. Over time, some will learn to choose the options that keep them moving. That is not a defense of overclaiming; it is a predictable response to a poorly designed incentive experience.

A documented screen-out policy helps. A small courtesy reward for a verified, good-faith screen-out can protect participant trust without turning screening into a paid qualification game. The policy should be consistent, clearly communicated, and separated from fraud decisions. A genuine screen-out is not the same as a rejected fraudulent session.

What a good qualifying rate looks like

A good qualifying rate is close to the estimate used to plan the fieldwork.

If a study is expected to qualify 20% of invited participants, a final rate of 18% or 22% may be normal. A sudden rate of 62% is not automatically good news. It may indicate that the target was revealed too early, a traffic source learned the qualification pattern, or a screener rule is not operating as intended.

The useful question is not, “Did we qualify enough people?” It is, “Does the qualify rate match what we know about the target population, recruitment source, and screener design?”

Treat large deviations as a finding. Review them by source, device, geography, time period, and response path. Look for abrupt volume spikes, unusually uniform answers, inconsistent profile data, or very fast completions—signals that have been identified as useful early indicators of low-quality or fraudulent participation.

A screener should not be judged by how many people it lets through. It should be judged by whether the people it lets through are the people the study was designed to understand.