Sample Size and Small States in the 2024 CES

Sixty thousand people, and sixty-eight Vermonters: what the largest political survey in America buys, where it runs out, and the difference between being precise and being right

Democracy’s Data

Last updated September 15, 2026 at 4:42p EDT

A survey of sixty thousand people sounds enormous, and it is. In 2024 the Cooperative Election Study interviewed 60,000 Americans, roughly sixty times a standard national poll. The thing usually said about a sample that large is that its margin of error is negligible: how far off the answer could be from the luck of who happened to get drawn.

Nationally, it nearly is. But a national sample is not a national sample everywhere. The same 60,000 respondents are 3,618 Texans and 68 Vermonters, and the second number decides what the survey can say about Vermont. So the question this brief asks of the biggest political survey in America is a small one: is it big enough to say anything about a small state?

The test

The data is the 2024 CES Common Content: 60,000 respondents interviewed around the 2024 election, released by Ansolabehere, Schaffner and Pope on Harvard Dataverse. Every state estimate it yields is then graded against the certified 2024 presidential result in that state — the official count each state signs off on after the election — which this book carries in its historical campaigns chapter.

Why this source, of all surveys? Because it is the largest, and the question is precisely about what largeness buys. It is also unusually honest equipment. The CES publishes the weights it applies to itself — the factors that count some answers more heavily than others, so the sample resembles the country. This book’s weighting chapter reads that file at length.

The benchmarks those weights aim at come from the Census Bureau’s counts; the weighting is only the method that pulls a sample toward them. This brief leans on that chapter for what the CES is, and asks one further question of it.

And why the presidential vote, of all its 694 questions? Because it is the one question with a certified answer. A survey estimate of trust or ideology can only be compared with another survey. An estimate of the two-party vote — the Democratic share of the votes cast for the two major candidates, with third parties left out — can be graded, state by state, against what the state actually did.

The data

Everything below is built into one table, states.csv, with a row per state. It starts from three columns of the big file: inputstate, the state each respondent lives in as a numeric code; CC24_364b, the presidential vote; and commonweight, the survey’s own weight. Of 60,000 respondents, 43,749 named one of the two major candidates; the other 16,251 named somebody else, did not vote, or did not answer.

The file is a free download from Harvard Dataverse for anyone with an account — a download a person has to make, because the archive turns away programs, a wall this book’s survey access brief measures directly. This brief reads the copy the weighting chapter keeps rather than holding a second one.

Two sibling chapters supply what the CES does not. The certified returns come from historical campaigns. The crosswalk, a table that matches one set of codes to another, comes from data sources, and turns the numeric state codes into the abbreviations the returns use. Neither file is typed here. And the vote coding was checked rather than assumed: before any state is graded, the national estimate is set beside the certified national result, and a swapped coding — Democrats counted as Republicans — would miss it by far more than the five points allowed. That check is in the table at the end of this brief.

For each state, the table then records n, the respondents in the state, and est, the weighted two-party Democratic share they imply. moe is that estimate’s margin of error; deff and n_eff are the design effect and effective sample size, explained below. actual is the certified share, error the difference, and covers whether the certified result falls inside the margin.

Before you read on, commit to a view. 60,000 people, spread across 51 jurisdictions. How many of them live in the smallest state cell? Write down a number. Most people put it an order of magnitude too high.

What it says about democracy

Nationally, the machine works. The CES puts the two-party Democratic share at 47.79%; the certified national result was 49.25%. It misses by 1.46 points — a good result for a survey, and a bad result for the usual explanation of survey error. A sample of 43,749 carries a national sampling margin of about half a point. The miss is roughly three times that, so most of it is something other than sampling.

Where it runs out

State Respondents Margin ± CES says Result was Error
Vermont 68 15.3 66.5 66.4 +0.1
Wyoming 77 13.7 24.6 26.5 -1.9
North Dakota 85 13.3 50.7 31.3 +19.4
Alaska 106 15.8 45.6 43.2 +2.4
District of Columbia 110 10.5 79.6 93.3 -13.7
…
Texas 3,618 2.7 42.9 43.1 -0.2
California 3,429 2.8 59.0 60.4 -1.4
Florida 3,106 3.5 46.6 43.4 +3.2

Vermont has 68 respondents. Texas has 3,618, which is 53.2 times as many. The margin of error runs from 2.71 points in the best-sampled state to 15.81 in Alaska, and the median state’s margin is 6.20 points. A survey described in every write-up by one national sample size is, at the level most people want to use it, an ordinary small poll fifty-one times over.

The respondent counts overstate even that. Weighting costs precision, and the cost has a name: the design effect, the factor by which the effective sample — the number of equally weighted respondents that would give the same precision — is smaller than the headcount. The median state’s design effect is 2.78, so a state cell of 300 respondents carries roughly the precision of a simple random sample of about 110 — a sample in which everyone had the same chance of being picked and every answer counts once. Every margin above already accounts for that; the point is that the number in the press release does not.

Here is the arithmetic for the smallest cell, step by step. Vermont has 68 respondents, but their weights are unequal, and a design effect of 1.86 says those 68 carry the precision of 36.5 equally weighted people. The CES puts Vermont’s two-party Democratic share at 66.5%. A margin of error is 1.96 times the square root of p(1 − p) divided by the effective sample, so: 0.665 times 0.335 is 0.223; divided by 36.5 that is 0.00610; its square root is 0.0781; and 1.96 times that, as a percentage, is ± 15.3 points. Texas’s 3,618 respondents go through the same steps and come out at ± 2.7. Nothing about Vermont is being measured badly. There are simply too few Vermonters in the room for the answer to be narrow, and no amount of care in the interviewing changes that.

Precision is not accuracy

Now grade every state against what it actually did.

Figure 1. Every state twice: the margin of error it claims, and the error it actually made against the certified result. A scatter against sample size, because the question is what sample size buys. Hollow points are the claimed margins; filled points are the errors, red where the result falls outside the margin.

Two correlations say the whole thing. A correlation runs from −1 to +1: −1 means one number falls exactly as the other rises, 0 means they have nothing to do with each other. A state’s sample size predicts its margin of error at -0.75. It predicts how wrong the state actually was at -0.25. Sample size buys precision. It barely buys accuracy. The hollow points fall away steeply to the right; the filled ones scatter almost as widely at 3,000 respondents as at 300.

A margin of error is drawn, by convention, so that the true value falls inside it nineteen times in twenty. So if sampling were the only thing going wrong, about 95% of states would have the certified result inside their margin — sampling error is what a margin of error describes, and nothing else. 45 of 51 do, which is 88.2%, against the 48.4 states that 95% coverage predicts. The median state misses by 2.49 points; the worst, North Dakota, misses by 19.41.

That shortfall is everything a margin of error does not promise. It covers who answered and who did not, whether the people in a small state’s cell resemble that state, and whether a person who says they voted for somebody did. For a reader of democracy’s data, the lesson is structural: most public claims about opinion in a small state rest on cells this size, and the printed margin understates the doubt. The weighting chapter takes the machinery apart; the poll weighting brief shows the judgment calls inside it on a single poll.

What this brief cannot tell you

It cannot tell you the CES is bad. It is among the best survey instruments in American political science, and a 1.46-point national miss is a good result. The brief is about what a large sample does and does not buy, and the CES is the right file precisely because it is large and careful.

One question, one year. Presidential vote in 2024, which is the most-asked question in the survey and the easiest to grade. Nothing here says how the CES performs on questions with no certified answer, which is most of them.

The benchmark is not the same population. The certified result counts everyone who voted. The CES estimate counts people who said they voted a particular way, weighted to a model of the electorate. Some of the gap in Figure 1 is that difference rather than survey error.

And the design effect is estimated, not published. The formula used here is a rule of thumb devised by the statistician Leslie Kish: it reads how unequal the weights are and returns the precision they imply, and the more unequal they are, the fewer people the sample is effectively worth.1 A design-based standard error from the survey’s own documentation would differ, probably not by much, and this brief does not have one.

What you should have learned

Extensions

Sources

Data

The data itself

Every figure above is drawn from this table. It is plain CSV, it opens in a spreadsheet, and it is yours to take further.

states.csv

Two more files here are not data about the subject at all – they are how this chapter checked its own work: facts.csv, the individual numbers the text above quotes, and checks.csv, the arithmetic verified before anything was drawn.

Checks

Check Passed
the whole common content arrived yes
every row carries the common weight yes
the sibling chapter’s county file, which carries the state crosswalk, is present yes
every respondent’s state code resolves to a state yes
the sibling chapter’s returns are present yes
the sibling chapter’s national series is present yes
exactly two major-party rows for 2024 yes
the vote coding is not reversed – the estimate is near the result yes
the largest state cell is at least twenty times the smallest yes
weighting costs precision everywhere yes
the margin of error does not account for most of the error yes
sample size buys precision far more than it buys accuracy yes

  1. Leslie Kish, Survey Sampling (New York: John Wiley & Sons, 1965), https://www.wiley.com/en-us/Survey+Sampling-p-9780471109495.↩︎