How to Test Your Discussion Guide Before Fieldwork (Without Burning Your Recruit Budget)

The most expensive question in a qualitative study is the one nobody tested. You find it on interview three of twelve, when a respondent misreads what you meant, or answers a leading question exactly the way you accidentally pushed them to. By then the recruit is booked and the budget is spent.

The fix has always existed - pilot interviews, cognitive pretesting - and teams skip it for one reason: a real pilot costs almost as much as the study it is meant to protect. So the guide goes to field untested, and you pay for its mistakes at full price.

What actually breaks in an untested guide

• Leading questions that hand the respondent your hypothesis instead of testing it.
• Vocabulary the respondent does not use, so they answer a different question than the one you asked.
• An order that primes later answers, so you learn what your earlier question planted.
• A hypothesis that was never the real barrier, so every question circles the wrong thing.
• Segments that were supposed to differ but answer the same, because the guide never pulls them apart.

None of these show up when you read the guide at your desk. They show up when a person actually sits in front of it - which is exactly why you want that to happen before the expensive people arrive.

Pressure-testing the guide before field, in minutes

Synthetic respondents let you run your real guide - the actual questions, in order, with follow-ups - against a described segment, and read the transcripts back in minutes instead of weeks. You are not testing what people think. You are testing whether your instrument works: whether each question produces a story or a shrug, whether your hypothesis survives contact, whether two segments actually diverge.

It is cheap enough to do it badly, fix it, and do it again. That is the whole point: you iterate the guide as many times as it takes, then send only the version that held up to real fieldwork.

A pre-fieldwork checklist

Run the guide against a synthetic version of your target segment and read for these, in order:

• Does each question produce a story, or a yes/no? Rewrite anything that closes.
• Does your central hypothesis survive, or does the respondent keep pointing somewhere else? If they do, that somewhere else is your real study.
• Do your segments actually answer differently? If not, your segmentation is decoration.
• Where do the follow-ups stall, with nothing more to give? That is a dead branch of the guide.
• Is any word doing work the respondent does not recognise? Swap it for theirs.
• Does an early question colour a later one? Reorder so you measure, not manufacture.

What synthetic pre-testing will not do (the honest part)

It will not tell you your market size, and it will not replace the humans. Synthetic respondents are directional: good at surfacing where a guide leaks and where a hypothesis is soft, and no substitute for hearing a real person contradict themselves in the room. Use them to fix the instrument. Then field it with humans, on solid ground, for the answer you can defend.

Synthetic pre-research does not tell you the answer. It tells you whether you are asking the right question.

Try it on your own guide

Describe your target segment in plain text, drop in the first few questions of your guide, and read where they break. It takes a few minutes and no signup.