Here's an uncomfortable truth about surveys: you can get almost any result you want, and you can do it without lying. You just have to word the question the way you're hoping people will answer. Most of the time nobody's being sneaky — the researcher genuinely believes their question is neutral. It reads fine to them because they already know what they think. The respondent doesn't, and the wording fills in the blank.
The frustrating part is that the number still comes out clean. "68% agreed." It looks like a fact. It gets a bar chart, a slide, a line in the abstract. And it's an artifact of the sentence, not a finding about the world.
Watch how far one question can move just by re-dressing it:
Same policy, four framings, and the "agree" bar swings from 21% to 71%. Nobody changed their mind between those versions — the question changed what counted as agreeing. If you only ran one of them, you'd walk away certain, and wrong.
The three ways a question tilts
Almost every biased question is one of these, and once you can name them you'll start seeing them everywhere — including in your own drafts.
Leading. The question hints at the "right" answer. "Wouldn't it be better if…", "Do you agree that…", "How much did you enjoy…" — each one hands the respondent a default. The fix is boring on purpose: ask for the view, don't propose it. "How do you feel about X?" beats "Don't you think X is a good idea?" every time.
Loaded. The question smuggles in an assumption or an emotionally charged word. "Forced," "without consent," "waste," "reform" — these aren't measuring the thing, they're measuring the reaction to the word. A respondent who's never thought about the policy now has a feeling about "forced," and you've recorded it as data. Strip the adjectives and let the noun stand on its own.
Double-barreled. Two questions glued into one, so a single answer can't mean anything. "Should lectures be recorded and posted publicly on YouTube?" — what does a "no" mean? No to recording? No to YouTube? No to both? Someone who wants recordings for the class but not on the open web has no honest way to answer. If your question has an "and" or an "or" in the middle, it's probably two questions. Split it.
The scale matters as much as the sentence
You can write a perfectly neutral question and still bias it with the options. A few things that quietly do damage:
- Unbalanced choices. If your scale runs Excellent · Very good · Good · Fair, three of the four options are positive. People anchor to the middle of what's offered, and the middle here is "good." Balance it: an equal number of positive and negative points around a genuine neutral.
- No neutral, or no escape. Forcing everyone to pick a side manufactures opinions that don't exist. Offer a real middle, and — where honesty demands it — a "don't know" or "not applicable" so people aren't guessing on your behalf.
- Vague quantifiers. "Often," "sometimes," "regularly" mean wildly different things to different people. If you can attach a number — "more than once a week" — do it.
The test that catches most of it
Before a question goes out, read it and ask: could someone guess what I'm hoping to hear? If yes, it's leading. Then ask: is there a word in here doing emotional work? If yes, it's loaded. Then: is there more than one question hiding in this one? If yes, split it. Three questions, thirty seconds, and you'll catch the majority of the damage before it reaches a single respondent.
Better still, pilot it on five people who aren't in your field and ask them to say out loud what they think each question is asking. You'll be surprised how often "obvious" wording gets read the opposite way.
Where Folio fits
This is exactly the kind of thing Folio Surveyor is built to keep you honest about. When you draft an instrument, it flags the patterns above — leading phrasing, loaded terms, lopsided scales, double-barreled items — the way a careful colleague would, before you distribute. It won't write your study for you, and it shouldn't. But it will stop you from shipping a question that answers itself, which is the difference between a survey that measures something and one that just confirms you.
A good question is quiet. It doesn't argue, it doesn't flatter, it doesn't flinch. It asks, and then it gets out of the way — and whatever comes back is worth believing.