Online surveys are fast and efficient, but the same speed also lets in answers that do not belong in a report: bots, people who enter the same survey twice, and respondents who go through the questionnaire without reading. One such respondent will not change a result. But a few percent of answers that are not real, inside a small subgroup, can produce a finding that does not exist.
Quality control is therefore not left to the end. It starts with the questionnaire, continues through fieldwork and ends in the data file.
Before fieldwork
- Bot protection. A check that runs in the background and flags suspicious entries without bothering real respondents.
- Blocking repeat entries. By panel ID and by device, even when sampling from several providers at once.
- Attention checks. One or two questions written into the questionnaire in advance, such as "For quality purposes, please select the third answer". Fair and simple, not trick questions.
- Soft launch. A small first sample to check completion times, drop-off and logic before going on.
During fieldwork
Daily monitoring: quota progress, completion rate, completion time by device type and answer distributions by source. A provider that starts showing suspicious patterns is stopped after a day, rather than discovered at the end of fieldwork.
After fieldwork
Checks run on the full file and flag each respondent:
- Speeders. A completion time far below the typical time for the survey, and especially too little time on a page that required reading a text.
- Very slow respondents. Anyone who left the survey open for a long time is flagged, and what to do with them depends on the survey.
- Straight-lining. The same answer to every statement in a grid, even when some are worded in the opposite direction.
- Duplicates. The same ID, the same device or an identical answer pattern, including across different panels.
- Open-ended answers. Meaningless strings of characters, copied text, or an answer unrelated to the question.
- Contradictions. For example an age that does not match the year of birth, or years at work that exceed age.
Every respondent removed from the analysis is flagged in the data file together with the reason for removal.
Transparency, not just cleaning
Instead of deleting, we add flag variables to the data file: who is a duplicate, who was too fast, who failed the attention check, and who is included in the analysis. The client sees exactly what was removed and why, and can run the analysis with or without those groups.
How strict to be
Rules that are too strict also remove real respondents, such as fast readers or older respondents who answer slowly, and so bias the sample themselves. Thresholds are therefore set for the specific survey, written down in advance where possible, and documented in the report.
Quality control is not visible in the report itself, but it is the condition for relying on every number in it.