Sampling bias

The forms it takes

Coverage error

Part of your population had no chance of being sampled at all. An online-only survey about digital exclusion is the textbook case: the people it is about are the people it cannot reach. A staff survey sent to work email addresses misses everyone without one.

Self-selection

Everyone could respond, but only people with a reason to bother did. Open consultation links reliably over-represent the strongly opposed and the strongly organised, because indifference does not fill in forms.

Non-response bias

You invited a proper sample, and the people who declined differ systematically from the people who agreed. This is the one that hides inside a respectable response rate: 40% sounds fine until you notice the missing 60% are concentrated in one age group.

Survivorship

You sampled the people still present. A customer satisfaction survey sent to the current customer list cannot hear from anyone who left, which is exactly the group with the most to say.

Why a bigger sample does not help

This is the point most worth internalising. Sample size controls random error — the wobble you would get from drawing a different handful of the same population — and confidence intervals describe that wobble. Bias is systematic: it pushes every draw in the same direction. Doubling the sample halves nothing and tightens the interval around the wrong number.

A survey of 10,000 self-selected respondents is not more reliable than one of 800 properly sampled ones. It is more confidently wrong, and the large number makes it harder to argue with.

A worked example

A council posts a consultation link on social media about a proposed parking charge. 3,000 people respond and 78% oppose it.

What that number actually measures: opposition among people who follow the council or a local group, saw the post, felt strongly enough to click, and had ten minutes. Supporters of a charge rarely have the same activation energy as people facing a new bill. The finding is real — 2,340 people object, and that is a fact worth knowing — but "78% of residents oppose" is not supported by it, and publishing it that way is the error.

The honest write-up names the method, reports the count, and says who is missing. If you need a population estimate, you need a sample frame and an invitation, not an open link.

What actually reduces it

• Start from a sample frame. A list of the population, and an invitation drawn from it, is the only thing that gives you a real denominator.
• Use more than one channel. Post as well as email, in-person as well as online. Then record which channel each response came through — a hidden field in each link does this — so you can see how differently they behaved.
• Chase non-responders specifically. A reminder to the people who have not answered is worth more than a broadcast to everyone.
• Collect the demographics you would need to weight. You cannot correct for an imbalance you did not measure. Put them at the end.
• Write the limitations section first. If you draft it before fieldwork, you find out what the design cannot tell you while there is still time to change the design.

In NumoForms

No survey tool fixes sampling — it is a property of how you distribute, not of the software. What the product can do is remove the barriers that skew who finishes: conditional logic so people are not asked irrelevant questions, save and continue for anything long, an accessible respondent experience so disabled respondents are not filtered out by the form itself, and hidden fields to tag the channel each response arrived through. See also screener questions, which define the population you are biased within rather than removing the bias.