Say you want to know which message is more persuasive: one that stresses saving money, or one that stresses health. You could show both messages and ask which is more convincing. But the answer to that question is shaped by what sounds reasonable, by what seems right to say, and by the order in which the messages were shown.
An experiment inside a survey works differently. Each respondent sees only one of the messages, at random, and then everyone answers the same questions. The difference between the groups is the effect of the message itself.
Uses
- Testing messages, ads and product names.
- Pricing: willingness to pay, when each group sees a different price.
- Policy: how the wording of a proposal changes support for it.
- Academic research: vignette experiments, where one detail of a description changes and its effect is tested.
What to plan
- Number of versions and group size. Every extra version needs more respondents. Eight versions of 250 respondents each add up to 2,000 respondents, after cleaning.
- Random, balanced assignment. The platform splits respondents evenly between versions and closes a version once it is full.
- Making sure the text was read. A minimum time on the text page, and a question that checks whether the respondent noticed the detail that changed. That question must be neutral, so it does not hint at the answer.
- Balance checks. After fieldwork we check that the groups are similar in age, gender and region. If they are, the difference in answers comes from the version, not from the make-up of the group.
- Cleaning that does not harm the experiment. Cleaning rules apply to all groups in the same way, and each group stays at its planned size.
In an experiment, respondents are not asked to judge what influences them. The effect is measured directly from the difference between the groups.
The analysis
The groups are compared with significance tests, and we also check whether the effect differs between audiences, for example between younger and older respondents. A well-designed experiment gives a clear answer to the question: what happens when one thing changes and everything else stays the same.