Section II — Survey Data

The one thing no record contains

There is no register of opinions in the United States. Nothing in any government file records whether you trust Congress, which party you think of yourself as belonging to, or whether you approve of how an election was run. There has never been such a file, and a free country is unlikely to build one.

Every other kind of data in this book is a record of something that happened. A person was counted, a ballot was cast, a form was filed, a car was stopped. A survey is the only instrument that can reach what leaves no trace at all: opinion, belief, and how people describe themselves. A voter file can tell you that a person voted. Only a survey can tell you what she says she was voting for.

So if you want to know whether trust in government fell after 1964, there is exactly one way to find out. Somebody had to have asked people at the time, in words that stayed the same afterward. Somebody did, which is the only reason the question is answerable now.² The long-running academic studies in this section exist precisely because somebody kept asking, decade after decade, whether or not anyone was paying attention that year.

Asking a sample instead of counting everyone

A survey replaces enumeration with statistics. Instead of trying to reach every person, it selects a sample: a group chosen so that every kind of person has a known chance of being picked. Then it asks them, and uses arithmetic to say how far the sample’s answer could sit from the country’s.

That arithmetic is the margin of error: how far off the estimate could be, and still be doing its job. You met it in Section I, printed beside every ACS number. What is remarkable is how cheap a good estimate is. A few thousand people, properly chosen, can stand in for a country of hundreds of millions — not exactly, but within a stated few points, and the stating is the point.

The word chosen is doing real work in that sentence. The statistics only hold when the researcher draws the sample; they say nothing about a sample that draws itself. A million volunteers who clicked a link describe the people who click links, and no arithmetic promotes them into the country. The famous polling disasters are mostly this mistake wearing different clothes.³

What sampling cannot do is become a count. Spread even a very large national survey across the country and it places roughly nobody in any particular place. There is no sample size at which a sample becomes a count. A survey can tell you what the country thinks and cannot tell you how many people live on a block. The census is the mirror image, and the two instruments need each other.

That contrast is the real bridge between Section I and this one. An enumeration tries to reach everyone and produce a count. A survey reaches a sample and produces an estimate. Everything else in this section follows from choosing the second instrument instead of the first: the weights, the margins, the mistakes.

Weighting: making the sample representative

A raw sample never quite looks like the country, even when it was drawn honestly. The people who agree to answer are older, more settled, more educated, and far more interested in politics than the people who do not.⁴ Left alone, their answers describe the kind of people who answer surveys, which is a real population but not the one anybody asked about.

The repair is weighting: counting some answers more than others, so that the sample’s shape matches the country’s shape. If the sample came back short of young adults, each young adult in it counts for a little more than one person. If it came back heavy with retirees, each retiree counts for a little less. Suppose young adults are one adult in five nationally but only one in ten of the people who answered: each young answer is then counted as two, and the sample’s age mix now matches the country’s. Every other imbalance is repaired the same way, all at once.

Weighting is not a bridge to anywhere. It is the statistical method that makes a sample representative. But matching the country’s shape requires knowing the country’s shape, and that knowledge is Section I’s. The benchmarks a survey is weighted to come from the Census Bureau: how many adults there are by age, sex, race, education and state. A sample only describes the country after it has been weighted to look like the country, and the description of what the country looks like is census data. Every weighted survey number in this section quietly carries Section I inside it.

That is why weighting is this section’s second cluster rather than a technical appendix. The weights are not a finishing touch applied after the findings. They are part of how the finding was manufactured, and the cluster’s chapter opens a real survey’s weights and shows what turning them does to a published result.

Two kinds of question, and only one gets graded

The surveys in this section split into two families, and the difference between them is a difference in accountability.

Political polls measure vote choice and approval. They are the surveys most people meet first: run fast and cheap, fielded in days, quoted in headlines. And they have one property nothing else in this section has — eventually an election happens, and the poll is graded against a real result. A pollster can be wrong in public, by a measurable amount, and the polling cluster here is largely about what that grading does and does not cover.

Public-opinion surveys measure issues and attitudes: party identification, ideology, abortion, confidence in institutions. The big academic studies have asked some of these questions in the same words for half a century or more, which is what makes a trend line possible at all. But no election ever arrives to grade them. There is no true national figure for trust in government against which a survey can be scored.

That asymmetry should change how you read each kind. A poll’s errors are eventually visible, so the interesting questions are about what happens before election day. An opinion survey’s errors may never be visible, so everything rests on the design: the sampling, the wording, and the weighting. When nothing can grade the answer, the method is the credential, and this section spends most of its pages on exactly that.

The wording carries particular weight in the second family. A survey answer is an answer to particular words, in a particular order, and the long-running studies guard their words the way an archive guards paper, because changing the question quietly changes the trend. One brief here watches one of the most repeated numbers in American politics — the share of Americans who are independents — move dramatically on nothing more than a coding decision about a single follow-up question, the one that asks self-described independents whether they lean toward a party.

Nonresponse, the problem that never goes away

Section I kept returning to the denominator, the bottom number of a fraction and the thing you divide by. This section keeps returning to nonresponse: the people who were asked and did not answer, who today are nearly everyone asked. The response rate — the share of the people a survey approaches who actually complete it — was once a majority and is now in single digits.⁵ When most people answered, a refusal here or there was a nuisance. Now the answerers are a small, self-selected sliver, and every survey estimate rests on how well that sliver can be reshaped into a country.

Weighting can fix what it can name. Short of young people, and the census says how short: weight them up. What weighting cannot fix is a difference it has no benchmark for. Suppose the people who refuse differ from the people who answer in the very thing being measured: trust, enthusiasm, which candidate they favor. No census table records that, and no weight can repair it. The margin of error does not cover it either, because the margin covers sampling and nothing else.

The briefs ahead keep meeting nonresponse in different costumes. One reweights a single real poll five defensible ways and watches the winner change. One measures how little a margin of error actually promises. One lines up sixty years of what people told a survey about voting against how many people could have voted at all. None of them concludes that surveys are worthless. All of them conclude that a survey number is a manufactured thing, and that the manufacture is where to look.

There is one check this section postpones. What people say is not always what they did, and the gap between the two is measurable. But it is measurable only by joining a survey to a record of the same people, and that is a join with problems of its own. That audit waits for Section IV, where surveys meet records on the record’s terms.

What is in this section, and how to use it

The pattern is the same as Section I’s. A chapter is a reading about one kind of data: who makes it, why, and what it structurally cannot say. A brief is a lab: one dataset, one real question, an answer you could defend. The section runs in 4 clusters, each a reading and then the briefs that lean on it. Where the table shows no reading, the cluster is all lab, and its briefs lean on the section’s opening chapter instead.

Cluster	The reading	Briefs
II.1 · What a Survey Can Establish	What a Survey Can Establish That No Record Can	2
II.2 · Weighting	Survey Weighting in the Cooperative Election Study	1
II.3 · Political Polls	—	2
II.4 · Public Opinion	Reading the ANES Cumulative File	8

The order is a teaching order. First what a survey is and what it alone can establish. Then the machinery: weighting, the statistical method that makes a sample representative, with census benchmarks as its targets. That comes second because you should see how the instrument works before trusting anything it produces. Then the two applications in turn: the polls that an election eventually grades, and the opinion studies that nothing ever grades. In full, in reading order:

Cluster	Type	Title	What it is about
II.1	chapter	What a Survey Can Establish That No Record Can	What no sample size will buy
II.1	brief	Free, Public, and Unreachable	Six survey archives, two HTTP clients, and a wall that judges the program rather than the person
II.1	brief	The Age Structure of the Electorate	Who is old enough to vote, who says they voted, and sixty years of the gap between the two
II.2	chapter	Survey Weighting in the Cooperative Election Study	The Cooperative Election Study, the weights it applies to itself, and what a room can measure
II.2	brief	Sixty Thousand People, and Sixty-Eight Vermonters	What the largest political survey in America buys, where it runs out, and the difference between being precise and being right
II.3	brief	One Poll, Five Answers	867 Florida voters, the weights that turned them into a result, and the winner that changes when you turn the dial
II.3	brief	What a Margin of Error Does and Does Not Cover	Everything that goes wrong outside the sampling
II.4	chapter	Reading the ANES Cumulative File	Seventy-six years of survey answers, and three columns that run backwards
II.4	brief	Party Identification and the Leaner Problem	One coding decision, and the most repeated number in American politics moves by a factor of five
II.4	brief	The Gender Gap Is a Subtraction	Fifty-two years of party identification, the one number everybody quotes, and the two numbers underneath it
II.4	brief	Ideological Self-Placement in ANES	A fifth of Americans decline to place themselves, and the survey invited them to
II.4	brief	What a Question About the Economy Measures	One economy, one question, and an answer that changes sign every time the White House changes party
II.4	brief	Abortion Opinion, 1980-2024	The most recent study first, and the fifty years underneath it
II.4	brief	Confidence in Institutions, GSS 1973-2024	Fifty-one years of one question, and the summary that does not survive it
II.4	brief	Nobody Started Liking Their Own Side More	Forty-six years of feeling thermometers, the half of the gap that moved, and the two numbers that are not temperatures
II.4	brief	Everyone Guesses Forty	What Americans think each party is made of, how wrong that is, and the one thing they are most wrong about

That is 3 chapters and 13 briefs.

Where this section sits

Quantity	Docs
Docs in this section	16
Docs elsewhere that read this section’s files	0
Docs here that read another section’s files	2

Look at the middle row, which counts the documents elsewhere in the book that read this section’s files: 0. That is not a disappointment, and it is not an accident. Survey data answers questions about what people think, and those answers do not become inputs to anything else in this book. This section is a destination rather than a supply line.

The bottom row counts the traffic running the other way: the documents here that reach into another section for a file. Those reaches matter more than their number suggests, because they are how a survey gets checked against something that is not another survey. The census benchmarks under every weight are the same kind of reach, made so routinely that nobody thinks to call it borrowing.

What you should have learned

• A survey is the only instrument that reaches an opinion. No government file records whether you trust Congress or which party you think of yourself as belonging to, and no free country is likely to build one. A record can say a person voted; only a survey can say what she says she was voting for, which is why every question about belief in this book ends up in one of these 16 documents.
• An enumeration reaches everyone and produces a count; a survey reaches a sample and produces an estimate. That is the method contrast between Section I and this one, and it decides what each instrument can be asked. There is no sample size at which a sample becomes a count, and no census table at which a count becomes an opinion.
• Weighting is the method that makes a sample representative, and its targets come from the census. Counting some answers more than others is what turns the people who agreed to answer into a description of the country. The benchmarks it matches — adults by age, sex, race, education and state — are Section I’s numbers, so every weighted figure here quietly carries a census product inside it.
• A poll gets graded by an election and an opinion measure never does, so design carries the weight. Vote choice and approval are eventually checked against a real result. Trust in government, ideology and party identification have no result to be checked against, and when nothing can grade the answer the sampling, the wording and the weighting are the whole credential.
• The people who did not answer are the standing problem, and no arithmetic reaches them. Weighting repairs the differences the census can name. It cannot repair a difference in the very thing being measured, and the margin of error does not cover that either, because a margin describes sampling and nothing else.

How to read this section

Start with what a survey can establish, the chapter that opens the first cluster. That is the argument for the whole method. Then read whether you can get one at all, a short brief with an uncomfortable finding about how many survey archives will answer a program rather than a person.

Take the weighting cluster before the applications if you possibly can, because it changes how every later number reads. After that, the polling and opinion clusters can be taken in either order, and the opinion briefs can be read singly, by whichever question you care about — party, ideology, abortion, confidence in institutions. If you are short of time, read the opening chapter and the weighting chapter. Those two are the ones that will change how you read a poll in a newspaper.

Sources

• The corpus itself. The clusters, their order and each document’s title are read from this book’s own section map (_lib/make-index.py) and from each document’s front matter, so this page cannot disagree with the section it opens; reorganise the section and rebuild, and the tables above follow.
• The instruments themselves are introduced, with their documentation and their terms of use, in the chapters that use them. None of them is redistributed here; several are gated downloads, and the chapters say so where that is true.