Understanding Polls and Surveys

Polls and surveys are everywhere — election predictions, customer satisfaction scores, health studies, public opinion trackers. They shape headlines, influence decisions, and drive policy. But a poll result like “54% of voters support the measure” is not a single, definitive fact. It is a statistical estimate with built-in uncertainty, and understanding what that means is a critical life skill for anyone who reads the news.

This page breaks down every component of a poll result so you can evaluate whether a headline is telling the full story or oversimplifying.

Anatomy of a Poll Result

A properly reported poll looks something like this:

“54% of voters support the measure, with a margin of error of ±3 percentage points at a 95% confidence level. The survey of 1,200 likely voters was conducted October 10-12.”

There are five key pieces of information packed into that sentence:

1. The result: 54% support the measure
2. Margin of error: ±3 percentage points
3. Confidence level: 95%
4. Sample size: 1,200 likely voters
5. Timing: October 10-12

Each of these matters. Let’s break them down one at a time.

Margin of Error

The margin of error (MOE) is the ± number attached to a poll result. It tells you the range within which the true population value likely falls.

If the poll says 54% with a margin of error of ±3%, the confidence interval is:

$54\% - 3\% = 51\% \quad \text{to} \quad 54\% + 3\% = 57\%$

This means the true level of support is likely somewhere between 51% and 57%.

Where Does the Margin of Error Come From?

For a simple random sample at 95% confidence, the margin of error is approximately:

$\text{MOE} \approx \frac{1}{\sqrt{n}}$

where $n$ is the sample size. This is a simplified formula that works well when the proportion is near 50%. The exact formula uses the standard error of a proportion, but this approximation is close enough for evaluating polls.

Example 1: Sample Size of 1,000

$\text{MOE} \approx \frac{1}{\sqrt{1000}} = \frac{1}{31.62} \approx 0.032 = 3.2\%$

If the poll says 54%, the true value is likely between 50.8% and 57.2%.

Example 2: Sample Size of 400

$\text{MOE} \approx \frac{1}{\sqrt{400}} = \frac{1}{20} = 0.05 = 5\%$

Much less precise. If the poll says 54%, the true value could be anywhere from 49% to 59%.

Notice the pattern: to cut the margin of error in half, you need to quadruple the sample size (because of the square root). Going from $n = 400$ to $n = 1{,}600$ cuts the MOE from 5% to 2.5%.

Sample Size and Margin of Error Reference

Key insight: There are diminishing returns. Going from 100 to 1,000 (10x) gains 6.8 points of precision. Going from 1,000 to 10,000 (another 10x) gains only 2.2 points. This is why most major polls survey 1,000 to 2,000 people — bigger samples cost much more but add little precision.

When Is a Poll Lead “Real”?

This is one of the most misunderstood aspects of polling. When two candidates are close, the headline often says the race is “too close to call” or “within the margin of error.” Here is what that actually means.

Example 3: Overlapping Confidence Intervals

A poll shows: Candidate A at 48%, Candidate B at 45%, with a margin of error of ±3%.

• Candidate A’s range: $48 - 3 = 45\%$ to $48 + 3 = 51\%$
• Candidate B’s range: $45 - 3 = 42\%$ to $45 + 3 = 48\%$

The ranges overlap: A could be as low as 45% and B could be as high as 48%. It is statistically possible that B is actually ahead. This race is within the margin of error — the poll cannot reliably distinguish who is actually leading.

Example 4: Non-Overlapping Intervals

A poll shows: Candidate A at 55%, Candidate B at 42%, with a margin of error of ±3%.

• Candidate A’s range: 52% to 58%
• Candidate B’s range: 39% to 45%

The ranges do not overlap. Even in the worst case for A (52%) and the best case for B (45%), A is still ahead. This is a statistically significant lead — the difference is larger than the margin of error can explain.

Rule of thumb: A lead is “real” (statistically significant) when the gap between the two percentages is roughly twice the margin of error or more. In Example 3, the gap is 3 points and the MOE is 3 points — not enough. In Example 4, the gap is 13 points against a MOE of 3 — very clear.

Confidence Level

The confidence level — almost always 95% in published polls — tells you how reliable the margin of error is.

A 95% confidence level means: if we repeated this exact survey 100 times with different random samples of the same size, about 95 of those surveys would produce a confidence interval that contains the true population value.

It does not mean:

• “There is a 95% chance the true answer is in this range” (the true value either is or is not in the range — it is not random)
• “95% of people agree” (the confidence level is about the method, not the result)
• “We are 95% sure we are right” (it is about long-run reliability of the procedure)

Some polls use 99% confidence, which produces a wider margin of error, or 90% confidence, which produces a narrower one. If the confidence level is not reported, assume 95% — it is the standard convention.

What Polls Can’t Tell You

Even a perfectly designed poll with a large sample and small margin of error has fundamental limitations:

• Future behavior: A poll measures opinions at the time it was taken. People change their minds. A poll showing 54% support in October does not guarantee 54% support on Election Day in November.
• Causation: A poll can tell you that 60% of people who exercise regularly also report being happy. It cannot tell you whether exercise causes happiness, happiness causes exercise, or a third factor (like income or health) drives both.
• Opinions of non-respondents: If only 30% of contacted people agreed to take the survey, the other 70% might have very different views. This is the nonresponse bias problem covered in Bias in Studies.
• Intensity of opinion: A poll might show 52% support, but if the 48% opposed feel extremely strongly while the 52% in favor are lukewarm, the opposition may be more politically relevant.

Red Flags in Poll Reporting

When you encounter a poll in the news, watch for these warning signs:

• No sample size reported: Without knowing $n$, you cannot assess the margin of error or the poll’s reliability.
• No margin of error mentioned: A headline that says “54% support the policy” without mentioning the margin of error is telling you only half the story.
• Online self-selected poll presented as representative: “We asked our website visitors…” is a voluntary response poll, not a random sample. The results tell you about the visitors who chose to respond, not the general population.
• Loaded or leading questions: “Do you support the irresponsible spending bill?” is not a neutral question. The wording biases the response.
• Headline focuses on a subgroup with a tiny sample size: A national poll of 1,200 people might include only 80 respondents from a particular state. The margin of error for that subgroup is $\frac{1}{\sqrt{80}} \approx 11\%$ — nearly useless for drawing conclusions.
• No disclosure of who commissioned the poll: A poll paid for by a political campaign or advocacy group may use question wording designed to produce favorable results.

Interpreting Health Statistics in the News

Health reporting often involves the same statistical concepts as polling, with an additional layer of complexity: relative risk vs. absolute risk.

Example 5: “Drug Reduces Risk by 50%”

A headline reads: “New drug cuts heart attack risk in half!” Here is the underlying data:

• Control group (no drug): 2 out of 100 people had a heart attack (2% risk)
• Treatment group (with drug): 1 out of 100 people had a heart attack (1% risk)

Two ways to describe this:

$\text{Relative risk reduction} = \frac{2\% - 1\%}{2\%} \times 100 = 50\%$

$\text{Absolute risk reduction} = 2\% - 1\% = 1 \text{ percentage point}$

Both statements are mathematically true:

• “The drug reduces heart attack risk by 50%” (relative)
• “The drug reduces heart attack risk by 1 percentage point” (absolute)

The first sounds dramatic. The second sounds modest. Advertisements and press releases almost always use the relative number because it is more impressive. The absolute number gives you a more honest sense of the actual benefit.

Another useful metric: The Number Needed to Treat (NNT) — how many people need to take the drug for one person to benefit:

$\text{NNT} = \frac{1}{\text{Absolute Risk Reduction}} = \frac{1}{0.01} = 100$

One hundred people need to take the drug for one heart attack to be prevented. The other 99 take the drug without benefit (though also potentially without harm). Whether that tradeoff is worthwhile depends on the drug’s side effects, cost, and the individual’s baseline risk.

Real-World Application: Retail — Interpreting Customer Satisfaction Surveys

A retail chain conducts a customer satisfaction survey and reports: “87% customer satisfaction rate, ±2%, based on 2,500 responses.”

Questions a manager should ask:

• Response rate: 2,500 responses out of how many customers contacted? If 25,000 were surveyed and only 2,500 responded (10% response rate), dissatisfied customers may have been less likely to respond — or more likely, depending on the survey context.
• How was “satisfaction” defined? If the scale was 1-5 and anyone who rated 3 or above was counted as “satisfied,” the 87% includes people who were merely neutral. A more meaningful metric might be the percentage who rated 4 or 5.
• Subgroup differences: The overall 87% might mask significant variation — 95% satisfaction in stores with recent renovations and 72% in older locations. The average hides the problem.
• Trend over time: Is 87% better or worse than last quarter? Without historical context, a single number is hard to evaluate.

Practice Problems

Test your understanding with these problems. Click to reveal each answer.

Key Takeaways

• The margin of error creates a confidence interval — the range within which the true value likely falls. Use $\text{MOE} \approx \frac{1}{\sqrt{n}}$ for quick estimates at 95% confidence.
• A poll lead is only meaningful if the gap is larger than the margin of error — otherwise the race is “too close to call.”
• Confidence level (usually 95%) describes the long-run reliability of the method, not the probability that one specific result is correct.
• Watch for red flags: missing sample sizes, no margin of error, self-selected samples, loaded questions, and headlines built on tiny subgroups.
• Relative risk vs. absolute risk matters enormously — “50% reduction” and “1 percentage point reduction” can describe the same result, but they create very different impressions.
• Polls measure opinions at a point in time and cannot predict future behavior or establish causation.
• Being an informed consumer of polls does not require advanced statistics — just the habit of asking the right questions.

Return to Statistics for more topics in this section.