Soft launch: what to look at in the first fifty completes

A soft launch is usually read as a fill-rate check. It is a diagnostic, and it has a short shelf life: after about two days the cost of changing anything exceeds the benefit.

Median length against the estimate

The first thing to check, because everything else is priced off it. The point is to establish whether the survey is taking materially longer than the length promised to respondents and priced into the study.

Median length is the right measure, not the mean. A few respondents who leave the survey open while taking a call, feeding a child, or answering on a broken mobile connection can inflate the mean without telling you anything about the ordinary respondent’s experience. If the agreed length was 12 minutes and the first 50 completes show a median of 14 minutes 40 seconds, the study is running about 22% long. That is not a rounding problem. At a cost-per-interview of ¥800, a 22% overrun implies roughly ¥176 of unbudgeted incentive pressure per complete—or, if the incentive is fixed, a meaningful increase in mid-interview abandonment as respondents discover the survey is longer than they were told.

A practical threshold is this: stop and revise if median length exceeds the estimate by more than 15%, or if more than one in five completes exceeds the estimate by 30%. Suppose 50 completes produce a median of 13 minutes against a 12-minute estimate, but 12 respondents took longer than 15 minutes 36 seconds. The median is acceptable, yet the tail is not: one quarter of respondents are experiencing a materially different survey. Before changing wording, look first at whether a particular device type, a video asset, or a free-text question is producing the tail.

Length also has a second-order effect on sample quality. If you promised 12 minutes and deliver 16, you are not merely overpaying; you are selecting for people willing to tolerate a broken promise. By complete 300, that selection can quietly change who is in the dataset. That is why length is the first check: every other diagnostic—drop-off, open-end quality, quota skew—becomes harder to interpret once the survey has become longer than the contract respondents believed they were accepting.

Where people leave

Drop-off concentrated on one screen is a routing fault, not a respondent problem. This section establishes how to distinguish a design defect from ordinary attrition.

Expect some abandonment at the start, especially on mobile: people click, see the topic, and leave. What matters is a spike. If 4 of the first 50 abandons occur at the same question, that is 8% of the sample lost at one point. If the question is a long matrix, a forced-choice screen, or a page where earlier logic unexpectedly filters people into a dead end, treat it as a defect until proven otherwise.

Worked example: imagine a 12-question screener with a page that asks respondents to select the brands they have purchased in the past six months, then immediately asks detailed usage questions for every selected brand. If 6 of 11 people who selected five or more brands abandon there, the problem is likely workload, not disinterest. The fix may be as simple as capping the follow-up to the three most-used brands, or splitting the page. That change can recover several percentage points of completion without weakening the data.

The diagnostic test is to compare exits by path, not just by question number. A screen with a 10% exit rate among respondents routed from Quota A may be fine, while the same screen shows a 35% exit rate among those routed from Quota B. That pattern points to a logic error, contradictory wording, or an incompatible profile being sent into the branch. Do not “fix” it by relaxing the quota or lowering the incentive; fix the route.

Screen-out reasons, not the screen-out rate

Which condition is doing the rejecting tells you whether the brief or the panel is wrong. The point is to establish that a high screen-out rate is not automatically a sample-quality complaint.

A 70% incidence sounds alarming only if you expected 70%. If the brief requires “IT decision-makers at companies with 1,000+ employees who evaluated a security platform in the past quarter,” a 25% incidence may be excellent. The first question is whether the client’s feasibility estimate was based on a comparable definition, or on a broader label such as “IT decision-maker” that quietly included influencers, managers without budget authority, or people whose evaluation happened last year.

Break every screen-out into its reason code. Suppose the first 200 starts produce 50 completes, 90 qualification failures, and 60 other exits. If 55 of the 90 failures fail the same “purchased in past 6 months” item, that is not a bad panel; it is either an overly narrow purchase window or a real-market constraint the feasibility model missed. If, instead, 40 respondents fail because they selected “none of the above” on a brand-awareness list that omits an obvious local competitor, the instrument is likely wrong.

The useful output is a small table: screen-out condition, number failing, share of all failures, and the implied incidence if that condition were relaxed. For example, removing an over-tight tenure requirement might move incidence from 25% to 38%, while removing the purchase-recency requirement might move it from 25% to 60% but destroy the study’s purpose. That comparison turns an argument about “sample quality” into a decision about which eligibility rule the client is actually willing to pay for.

Quota skew in the first fifty is not noise

Easy cells fill first. The point is to establish how to tell the difference between early skew and a real problem.

With only 50 completes, quota cells will not look like the final sample. If the plan is 40% male and 60% female, an opening split of 30/20 is unsurprising: the easier-to-reach cell arrives first. The relevant question is whether the panel can still reach the underfilled cells at the required pace, not whether the first 50 mirror the target exactly.

A useful calculation is remaining demand versus observed arrival rate. Suppose the study needs 100 males and 100 females. After 50 completes, you have 35 males and 15 females. If males are arriving at roughly 7 per 10 completes and females at 3 per 10, the remaining requirement is 65 males and 85 females. At the current pace, you would reach the male target after about 93 more completes, but would still need roughly another 188 completes to satisfy the female target. That is a feasibility warning, not noise.

Distinguish three cases. First, early skew with stable cell-level incidence: monitor, do not intervene. Second, skew where one cell’s incidence is far below its feasibility estimate—for example, a specialist clinician cell promised at 20% but converting at 4%: revisit sourcing or the qualification logic. Third, skew where one cell is filling so quickly that it will exhaust before harder cells can be met: tighten routing or pause the easy cell before it consumes budget. The first fifty tells you which of these situations you are in; it does not yet tell you the final proportions.

Open-end quality as an early instrument signal

Short answers at question five usually mean the question, not the sample. The point is to establish that open-end responses are an instrument check, not merely a content-quality complaint.

If the first 50 respondents produce one- or two-word answers to a question intended to elicit reasoning—“Why do you prefer this option?”—resist the conclusion that the panel is low quality. Ask whether the question gives respondents a reason to write. A prompt such as “What do you like about this concept?” often yields “price,” “design,” or “nothing.” A more specific prompt—“What, if anything, would make you choose this option instead of your current provider? Please mention the specific feature or situation that matters most”—usually produces longer, more usable text.

Use a simple readability measure in the soft launch: median word count, share of responses under five words, and share that are pure repetitions of a closed-ended answer. For example, if 32 of 50 open-ends contain fewer than five words and 18 simply repeat “cheaper,” the problem is likely question design. If median length is 18 words but many answers are generic, the issue may be that respondents have not yet been given enough stimulus to reason about.

Open-ends also reveal comprehension failures earlier than scaled questions do. If several respondents write “I don’t understand what ‘usage occasion’ means,” or repeatedly describe a feature that the concept does not contain, you have found a wording defect while it still costs two days to repair. Fifty responses are enough to see the pattern; they are not enough to code themes reliably, so use them to repair the instrument, not to report findings.

The decision the soft launch exists to make

Go, fix, or stop. Naming the third option is what makes the first two credible. The point is to establish that a soft launch must end in an explicit decision with an owner and a deadline.

Go means the median length is within tolerance, exits are dispersed rather than concentrated, screen-out reasons match the brief, quota cells remain reachable, and open-ends show comprehension. In that case, release the remaining sample and resist the temptation to keep editing. Every change after fieldwork is underway has a cost: new wording can make early and late respondents answer different questions.

Fix means there is a specific, bounded defect: one routing error, one over-long page, one impossible quota, one ambiguous question. Set a rule such as: pause fieldwork, make no more than three changes, retest with 10–20 completes, and restart only when the specific metric moves. For example, if a matrix page is causing a 20% single-screen exit, replace it with two shorter pages and require the replacement to bring that screen’s exit rate below 5% before full launch.

Stop means the study cannot produce the evidence it was commissioned to produce. This is the option teams avoid naming, but it is the one that protects the client. If the target incidence is 2% rather than the 15% sold, if the eligible population cannot meet interlocked quotas, or if the core screening condition eliminates the very behaviour the study is meant to explain, continuing will produce completes rather than insight. A useful rule: if the feasible incidence is less than one third of the feasibility estimate and the sample plan cannot be redesigned within the remaining two-day window, recommend stopping and revising the design rather than burning budget on a study that will fail later.

The soft launch is therefore not a preliminary formality. It is the last moment when the study can be changed cheaply. Its output should be a one-page decision memo: median length versus estimate, top three exit points, top three screen-out reasons, quota arrival rates, open-end comprehension flags, and one of three words—go, fix, or stop.