What it says about democracy
One simulated poll works like this. Pick 1,000 of the 1,000,000 voters at random — every voter equally likely, the way names drawn from a hat would be — ask each one which candidate they support, which the computer already knows, and report Harris’s share of the 1,000. Since 49.25% of the country is for Harris, each person drawn has that chance of being a Harris voter, and the share among a thousand of them will land near 49.25 but almost never on it. Poll the built country that way three times:
| Poll | Estimate of Harris’s share (%) | Truth (%) |
|---|---|---|
| 1 | 50.90 | 49.25 |
| 2 | 52.10 | 49.25 |
| 3 | 48.20 | 49.25 |
Three honest polls, three different answers. That wobble is sampling error, the error that comes from asking 1,000 people instead of everyone, and it is the only kind of error the ± describes. Now run a thousand honest polls and pile up their answers in a histogram — the sampling distribution, the spread of results that polls of one size produce from one country — and beside them a thousand polls drawn from a pool where Harris voters are 20% likelier to pick up the phone.
Figure 1. A thousand honest polls, and the same thousand polls with a response gap. Two histograms, because the question is about the spread of many polls, not any single one. Gray: honest polls, centered on the truth at 49.25%. Orange: the same polls when Harris voters are 20% likelier to answer, centered on 53.88%. Both distributions are equally tight. Only one is in the right place, and nothing about the width of either says which.
| Quantity | Value |
|---|---|
| Margin of error claimed at n = 1,000 | ±3.10 points |
| Polls landing inside it | 94.5% |
| Average of the thousand polls | 49.27% |
| The truth | 49.25% |
The advertised promise is kept exactly. About 95% of the honest polls land inside ±3.10, and their average is the truth to two decimal places. Read carefully, the promise says: if my only problem is that I asked 1,000 people instead of all of them, I will usually be within three points. It says nothing whatever about any other problem.
Bigger samples buy more of the same promise:
| People polled | Margin of error (points) |
|---|---|
| 500 | ±4.38 |
| 1,000 | ±3.10 |
| 5,000 | ±1.39 |
| 20,000 | ±0.69 |
| 100,000 | ±0.31 |
Precision improves with the square root of the sample, so a hundredfold increase buys a tenfold improvement. Now grade the biased polls against it:
| People polled | Estimate (%) | Advertised margin | Actually wrong by | As a multiple of its own margin |
|---|---|---|---|---|
| 1,000 | 54.00 | ±3.10 | +4.75 | 1.5× |
| 10,000 | 54.63 | ±0.98 | +5.38 | 5.5× |
| 100,000 | 53.68 | ±0.31 | +4.43 | 14.3× |
Most readers write down something smaller than the ±3 they started with, reasoning that a poll a hundred times larger cannot be worse than a small one. Instead, a poll of 100,000 people reports a margin of ±0.31 points and misses by 4.43 — about 14 times its own stated precision. And it does not drift toward the truth as the sample grows. It converges on 53.80%, which is not the answer.
Gray band: where 95% of honest polls land. Orange band: where 95% of polls land once Harris voters are 20% likelier to answer, and the orange dots are the three polls in the table above. Past about 464 respondents the orange band is narrower than the 4.55-point bias, so from there on the poll’s own interval never contains the truth again, and it gets worse, not better, with every extra respondent.
Figure 2. Where polls land, as the sample grows. The gray band is where 95% of honest polls fall; the orange band is where 95% fall under the response gap, with the three polls from the table as dots. Both bands narrow at the same rate, but only one narrows onto the truth. Past about 464 respondents the orange band is narrower than the 4.55-point bias, so from there on the poll’s own interval never contains the truth again, and it gets worse with every extra respondent.
That convergence point is algebra, not a draw: if Harris voters answer 20% more readily, the responding pool is 53.80% Harris no matter how many of them you talk to. More data makes you more confident about the wrong number. The tight orange cluster is not evidence of accuracy but of agreement, and every poll in it agrees on a figure 4.55 points from the truth. Averaging them changes nothing, because they are all wrong in the same direction. That is exactly what a polling average does, and why its narrow interval does not mean what readers take it to mean.
How big does the gap have to be before it matters? Smaller than intuition says:
| Response gap | Bias (points) | Advertised margin at n = 1,000 | As a multiple of the simulated election margin |
|---|---|---|---|
| 2% | +0.50 | ±3.10 | 0.3 |
| 5% | +1.22 | ±3.10 | 0.8 |
| 10% | +2.38 | ±3.10 | 1.6 |
| 20% | +4.55 | ±3.10 | 3.0 |
| 40% | +8.35 | ±3.10 | 5.6 |
Figure 3. What each response gap costs, against what the poll advertises. One row per gap. The hollow gray dot is the ±3.10 a poll of 1,000 advertises, identical on every row because sample size is the only thing it responds to. The filled orange dot is the bias the gap actually produces. 2 of the 5 gaps put the bias past the advertised margin, and 3 of 5 put it past the 1.50-point margin of the election itself.
A 5% difference in willingness to answer produces 1.22 points of bias. No pollster could detect a gap that size, and it sounds like nothing. Measured against the 1.50-point election this simulation is built around, it is most of the answer. And that, not the advertised margin, is the right yardstick: the bias should be compared to the thing you are trying to detect.
Pollsters know all this, and they correct for it by weighting: counting some answers more than others, to fix a sample that does not look like the country. Build a population where answering the phone is driven by trust in institutions rather than by the vote itself, and try it.
Orange cells are over-represented in the poll, purple under-represented. Nothing in this grid is about the vote directly: the poll over-samples people who trust institutions, and it is only because trust and vote are correlated that Harris comes out at 56.3% of the poll against 49.3% of the country. Weight the two rows back to their true sizes and the error falls to +0.70 points.
Figure 4. Who the poll reached, and who it should have reached. One cell per combination of vote and trust, shaded by the poll’s share of that cell minus the population’s: orange over-represented, purple under. Nothing in the grid is about the vote directly. The poll over-samples people who trust institutions, and only because trust and vote go together does Harris come out at 56.3% of the poll against 49.3% of the country.
| Estimate | Harris share (%) | Error (points) |
|---|---|---|
| The truth | 49.28 | — |
| Raw poll | 56.26 | +6.98 |
| Weighted on trust | 49.98 | +0.70 |
| Weighted on a variable unrelated to response | 56.26 | +6.98 |
Weighting on the thing that actually drives non-response takes the error from +6.98 to +0.70. Weighting on a real, measurable, irrelevant variable moves it essentially not at all, to +6.98. Weighting works exactly and only when you already know what is wrong. After 2016 the industry discovered it had not been weighting by education and added it;3 in 2020 it missed again, in the same direction, and the review by AAPOR, the pollsters’ professional association, could not identify the mechanism.4 The variable you have not thought of is invisible by construction, and no amount of arithmetic finds it.
The record outside the simulation says the mechanism is real. Meng (2018) proves the formal version, the big data paradox: “the more the data, the surer we fool ourselves.”5 He computes that a self-selected sample of 2.3 million voters in 2016 carried the same error as a random sample of about 400. Bradley and colleagues (2021) watched it happen outside politics: a 250,000-response-per-week survey overstated vaccine uptake by up to 17 points while a small random sample was close.6 The margin of error, honest within its terms, describes the smallest problem a poll has — and for the states that decide elections, the problem it describes was never the one going wrong.