What a Survey Can Establish That No Record Can

What no sample size will buy

Democracy’s Data

Last updated September 15, 2026 at 4:42p EDT

Most of this book reads records. A record is made because somebody was required to make it: a ballot cast, a district drawn, a contribution filed, a person counted. The record exists whether or not anyone ever studies it, and the questions it can answer were fixed by whoever was obliged to write it down.

This chapter is about the other kind of source. A survey exists because somebody asked. It records not what a person is but what a person says — in the categories the question allowed, to an interviewer, on one day. That difference is not a limitation to be apologised for. It is the reason surveys exist at all, because there are things about a democracy that no record anywhere contains.

The deeper difference is in how the two instruments reach people. A census is an enumeration: it tries to reach everyone, and what it produces is a count. A survey reaches a sample, a few thousand people chosen to stand in for everyone, and what it produces is an estimate — a number that comes with a stated range of doubt attached. Everything in this section of the book follows from choosing the second instrument instead of the first.

The question Census ANES CES
How many people live on this block? Yes, by enumeration No – a sample cannot count a block No – a sample cannot count a block
What share of adults in Wyoming are registered? Yes, by enumeration No – too few respondents per state Yes, at about +/- 6 points
Does this person trust the federal government? No – no record contains an attitude Yes, and since 1958 Yes, but only back to 2006
Has the country’s view of abortion changed since 1980? No – the question is never asked Yes – this is what it is for No – the series is too short
Do Black Republicans differ from white Republicans on immigration? No – neither variable exists No – the subgroup is too small Yes – this is what the sample size is for
What is this person’s race, as they describe it? Yes, in the categories the form allows Yes, in the categories the question allows Yes, in the categories the question allows

Read the third row. No administrative record in the United States contains whether a person trusts the government. There is no register of opinions, there has never been one, and a free society is unlikely to build one. If you want to know whether trust in government fell after 1964, there is exactly one way to find out, and it is to have asked people at the time. Somebody did, which is why the question is answerable now and would be unanswerable otherwise.

Then read the first row. A survey cannot tell you how many people live on a block, and this is not a matter of trying harder. It is arithmetic, and the arithmetic is unforgiving in a way that surprises people. This chapter shows both halves: what asking can reach that counting cannot, and what counting delivers that no amount of asking will buy.

Where the data comes from, and what it is for

Survey data has no single publisher the way census data has a Bureau. This chapter reads two academic studies, the two national election surveys a student of American politics meets first: the American National Election Studies (ANES), run by the University of Michigan and Stanford, which has interviewed a national sample around every presidential election since 1948; and the Cooperative Election Study (CES), run from Harvard, which since 2006 has interviewed a far larger sample online every two years. What it says about them is true, with the details changed, of nearly every survey a newspaper will ever quote.

Who is in it, and how they got there. The people in a survey file are the people who were asked and agreed. The asking is designed: a probability sample gives every kind of person a known chance of being picked, which is the property all the supporting arithmetic depends on. The agreeing is not designed, and it has collapsed. The response rate — the share of the people a survey approaches who actually complete it — was once a majority. Pew Research Center measured its own telephone response rates falling from 36% in 1997 to 6% in 2018, and the slide has continued since.1 The people who still answer are older, more settled, and far more interested in politics than the rest, so every modern survey is built on a small, self-selected sliver of the people it approached.2

Who it was made for. Researchers, in the first instance. ANES exists so that the same question, asked in the same words, can be compared across decades. CES exists so that dozens of university teams can share one very large sample. Neither was built to report the news, and both publish their questions, codes and weights, because their readers are people who check. A campaign poll is made for a client and publishes what the client permits, which is usually a topline — the headline percentages — and nothing underneath it.

What it costs to get. Money to nobody, and effort that varies. The academic files are free downloads, though some sit behind registration walls that a person can pass and a program cannot — the survey-access brief measures exactly that. Government survey files are free and open. Media and campaign polls mostly never release the individual answers at all, so what they cost to get is often: everything, because you cannot.

What has already been done to it. More than most readers suspect. Before a survey number reaches print it has been through weighting: counting some answers more than others, to fix a sample that does not look like the country. The targets the weights aim at are census figures. The questions themselves were written, tested, and frozen by somebody. A survey estimate is a manufactured thing, and the manufacture is where a careful reader looks.

The surveys you will meet

Three families of survey produce nearly every number about American opinion you will encounter, and they publish in three different ways.

The academic time series. ANES has interviewed a national sample around every presidential election since 1948. The General Social Survey, collected by NORC at the University of Chicago, has asked about society and belief since 1972, and CES has fielded roughly sixty thousand respondents every cycle since 2006. Each publishes its individual-level answers, one row per person and one column per question, together with a codebook — the document that says what every numeric code means. The long-running ones also publish a cumulative file stitching every wave — each round of interviewing — into one table, which is the object the ANES chapter opens. These files are free, documented to an almost archival standard, and slow: a wave arrives months or years after the interviews.

The government surveys. The Census Bureau and the Bureau of Labor Statistics run the biggest surveys in the country. The one this book uses is the Current Population Survey (CPS), a monthly survey of about 60,000 households best known as the source of the unemployment rate.3 Every other November it carries a Voting and Registration Supplement, which is the longest-running record of who says they voted — the age-structure brief reads sixty years of it. Government surveys publish on a fixed cadence, as open files and published tables, and their samples dwarf anything academics can afford.

The media and campaign polls. These are the surveys most people meet first: fielded in days, a thousand or so respondents, quoted in headlines. What they publish is usually a topline (the percentages) and a methodological note, not the answers themselves. They also have one property nothing else in this section has: when they measure a vote, an election eventually arrives and grades them in public. Surveys about issues and attitudes never get that grading, which is why everything rests on their design, and why this chapter dwells on the machinery.

What no sample size will buy

Here is the machinery under strain. The most recent American National Election Study is a large, expensive, carefully drawn national survey.

Quantity Value
Respondents in the study 5,521
States and equivalents represented 53
Respondents in the median state 26
Respondents in the smallest state 2
Respondents in the largest state 629
Margin of error, median state, at 50% ± 19.2 points
Margin of error, smallest state, at 50% ± 69.3 points
Margin of error, whole sample, at 50% ± 1.3 points

A margin of error says how far off a sample’s estimate could be, and still be doing its job. 5,521 respondents give a national estimate accurate to about ± 1.3 points, which is excellent. Split those same respondents by state and the median state — the one in the middle, with as many states above it as below — holds 26 people, giving ± 19.2 points. That range is so wide that a result of 50% is consistent with a true value anywhere from a third to two-thirds. The smallest state has 2 respondents and a margin of ± 69.3 points, which is not a measurement at all.

The arithmetic behind those margins is short enough to do by hand. For a result near 50%, the margin is 1.96 times the square root of 0.25 divided by the number of respondents. Take the median state’s 26 people: 0.25 divided by 26 is 0.0096, its square root is 0.098, and 1.96 times that is 0.192 — which, as a percentage, is the ± 19.2 points in the table. Nothing about the state is being measured badly. There are simply too few people in the cell for the answer to be narrow.

Figure 1. Margin of error at 50%, by number of respondents. Each point is a sample size and the range of doubt it buys. The chart is a curve on a logarithmic axis because the lesson is the shape. Precision improves with the square root of the sample size, so the curve falls like a cliff on the left and goes nearly flat on the right; no other chart form shows that flattening. The two marked points are a typical state subsample in ANES and a whole CES.

Because precision follows the square root, a national survey cannot be turned into a state survey by analysing it harder. Cutting a sample into fifty pieces costs a factor of about seven. This is the entire reason two national election surveys exist rather than one.

ANES buys depth and time. The same questions since 1948, long interviews, and a design built so that 1972 and 2024 can be set beside each other. It is the only instrument that can answer has this changed in seventy years.

CES buys breadth. About sixty thousand respondents in one year. Split the same way, its median state would hold about three hundred people and a margin near ± 6 points. That is still not wonderful, but it is the difference between a number you can publish and one you cannot, and it is what makes state-level and small-subgroup work possible.

Neither is better. They are different purchases with the same currency, and a researcher who wants both depth and breadth has to want them in separate projects.

Two more decisions stand between a person’s answer and a published number, and each can move the result more than the thing being measured.

The wording is the measurement

Three questions, exactly as the interviewer reads them. Each one is
chosen because the wording is not a window onto an opinion -- it is
the instrument that produces one.

---- 1. The escape hatch is inside the question --------------------

  We hear a lot of talk these days about liberals and conservatives.
  Here is a 7-point scale on which the political views that people
  might hold are arranged from extremely liberal to extremely
  conservative. Where would you place yourself on this scale,
  OR HAVEN'T YOU THOUGHT MUCH ABOUT THIS?

  The last clause invites a fifth of respondents to decline, and the
  codebook files that answer as VALID. A survey that omitted the
  clause would report a country with no such group in it.

---- 2. One column, three questions --------------------------------

  Generally speaking, do you usually think of yourself as a
  Republican, a Democrat, an Independent, or what?
    ...if Republican or Democrat:
       Would you call yourself a STRONG [R/D] or a NOT VERY
       STRONG [R/D]?
    ...if Independent or other:
       Do you think of yourself as CLOSER to the Republican
       or Democratic party?

  The famous seven-point scale is the record of a branch. Nothing
  in the seven numbers says which branch a respondent came down.

---- 3. The options are the opinion --------------------------------

  1  By law, abortion should never be permitted.
  2  The law should permit abortion only in case of rape, incest,
     or when the woman's life is in danger.
  3  The law should permit abortion for reasons other than rape,
     incest, or danger to the woman's life, but only after the need
     for the abortion has been clearly established.
  4  By law, a woman should always be able to obtain an abortion
     as a matter of personal choice.

  There is no fifth option, and a person whose view is not on the
  card must pick the nearest one. Whatever share option 3 receives
  is a fact about the card as much as about the country.

The temptation is to treat a question as a window — as though the opinion were already there and the question merely lets you see it. These three show why that is wrong. The ideology question carries its own exit, and a fifth of respondents take it; remove the clause and that group vanishes into the answers. The party question is three questions wearing one column. And the abortion card has four options and no fifth, so whatever share the third option receives is a fact about the card as much as about the country.

Each of these has a brief of its own in the Public Opinion cluster; this chapter’s point is the one they share. The instrument is not neutral, and reading the question text is reading the variable definition.

Who answers is not who lives here

People who agree to take surveys differ from people who do not, in ways that correlate with politics. The standard repair is weighting, and the size of that repair is worth seeing.

Variable Category Interviews Raw Weighted Adjustment
votereg Yes 54,913 91.5% 72.5% -19.0
votereg No 4,424 7.4% 23.2% +15.8
pid7 Strong Democrat 16,576 27.6% 21.9% -5.7
newsint Most of the time 29,960 51.2% 46.4% -4.8
educ No HS 2,133 3.6% 7.5% +3.9
ideo5 Not sure 3,603 6.0% 9.2% +3.2
votereg Don’t know 663 1.1% 4.3% +3.2
educ Some college 13,961 23.3% 20.4% -2.9
ideo5 Liberal 11,187 18.6% 15.8% -2.8
gender4 Man 27,454 45.8% 48.4% +2.6

The table is drawn from the 2024 Cooperative Election Study, which publishes each answer’s share twice: once as a plain count of the people interviewed, and once after weighting. In it, 91.5% of the people interviewed said they were registered to vote, and the published estimate is 72.5% — an adjustment of 19.0 points on a single item. That is not evidence of a problem: it is the correction working, and a survey that reported the raw figure would be badly wrong about the country. But it means the weights do a great deal of work before any result appears, and a reader who does not know they exist is reading a number that is substantially a model.

Follow one respondent through it. She was asked, agreed, and told the CES she is not registered to vote. The 4,424 people who said that are 7.4% of everyone interviewed. Nobody set a target for registration; the weights aim at the census’s counts of adults by age, sex, race and education. But the people the raw sample was short of — the young, the less educated, the less attentive to politics — are also the people least likely to be registered, so when their answers are counted up, so is non-registration, and the weighted file says 23.2% of adults are unregistered. On average, then, each unregistered respondent’s answer is counted as about 3.1 answers, and each registered respondent’s as about 0.79 of one. She was asked as one person. In the published estimate she stands in for three, because the sample caught only about a third as many people like her as the country holds. The weighting chapter opens that machinery fully.

What this data cannot tell you

A survey cannot become a count. There is no sample size at which an estimate turns into an enumeration. A survey can say what the country thinks and cannot say how many people live on a block; the census is the mirror image, and the two instruments need each other. Every weighted survey estimate quietly carries census data inside it, as the description of what the country the weights aim at looks like.

The margin of error covers sampling and nothing else. The margins in this chapter are the simple ones — 1.96 times the square root of p(1−p)/n, at p = 0.5, the widest case. Real surveys have design effects — the precision lost by counting some answers more than others, and by interviewing people in clusters rather than one at a time — that make the true interval larger, so every figure here is optimistic. And no margin, however computed, covers a bad question, a sample that draws itself, or refusers who differ from answerers in the very thing being measured. No census table records enthusiasm or trust, so no weight can repair a shortfall in it.

A survey records what people say, not what they did. Whether a stated answer matches a stable underlying attitude, or is partly invented on the spot, is a real and unsettled argument,4 taken up in the ANES chapter. Whether it matches behavior is checkable only by joining a survey to a record of the same people, and that audit waits for Section IV.

The state counts here are one study’s. They come from where respondents were interviewed in the most recent ANES. A different year would give somewhat different per-state counts, but not a different conclusion, because the arithmetic that produces ± 19.2 points from 26 people does not vary.

What you should have learned

Where this data appears in this book

The Political Polls cluster has no chapter of its own, so its two briefs lean on this one. One Poll, Five Answers reweights a single real poll five defensible ways. What a Margin of Error Does and Does Not Cover measures how little the margin actually promises.

Sources

Data

Academic literature

The data itself

Every figure above is drawn from these tables. They are plain CSV, they open in a spreadsheet, and they are yours to take further.

capacity.csv, instruments.csv, moe.csv, weighting.csv

Your turn

None of these is answered above, and every one can be answered from the tables you just opened.

  1. Pew Research Center, “Response rates in telephone surveys have resumed their decline,” 27 February 2019, https://www.pewresearch.org/short-reads/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/.↩︎

  2. Pew compared the people who do answer with census benchmarks and found them more civically and politically engaged than the country as a whole: “What Low Response Rates Mean for Telephone Surveys,” 15 May 2017, https://www.pewresearch.org/methods/2017/05/15/what-low-response-rates-mean-for-telephone-surveys/.↩︎

  3. The Bureau describes the survey and its sample at https://www.census.gov/programs-surveys/cps/about.html; the Bureau of Labor Statistics publishes the unemployment rate from it. The voting supplement’s tables are at https://www.census.gov/topics/public-sector/voting.html.↩︎

  4. The case that answers are assembled on the spot from whatever considerations happen to be in mind is John Zaller and Stanley Feldman, “A Simple Theory of the Survey Response: Answering Questions versus Revealing Preferences,” American Journal of Political Science 36, no. 3 (1992): 579–616, https://doi.org/10.2307/2111583.↩︎