Survey Weighting in the Cooperative Election Study

Where the data comes from, and what it is for

Nobody is required to run the Cooperative Election Study. It exists because dozens of university research teams each buy a module of it and share a common core of questions, coordinated since 2006 by Stephen Ansolabehere, Brian Schaffner and their collaborators. The release read here is the 2024 Common Content, Data Release 2, published on Harvard Dataverse with a DOI attached.

Who is in it, and how they got there. About sixty thousand American adults — and, crucially, not a random sample of them. The CES is an opt-in online panel: a standing pool of people who signed up with the polling firm YouGov to take surveys online, and are invited back to take more. Nobody drew them at random. What YouGov does instead is draw an imaginary random sample of the country from census-style lists and, for each imaginary person, pick the real panel member who most resembles them in age, sex, race, education and a few other things — a technique its designers call sample matching — and then weight the result so that it resembles the country.¹ Sixty thousand respondents sounds like rigor, but size is not design, and everything this chapter shows follows from who joins a political survey panel voluntarily.

Who it was made for. Political scientists who need subgroups. A national poll of a thousand cannot say anything about one state or one small group; sixty thousand can, which is the whole purchase. The common content asks what political scientists have agreed to ask, in wording they have agreed on, and it would stop the year the money did.

What it costs to get. Nothing in money. The file is a free download from Harvard Dataverse for anyone with an account: about 60,000 rows and 694 variables, 184 MB. The file itself is not reproduced here — it is large, and it is not this book’s to redistribute — so those three figures are quoted from the release rather than computed. What this chapter carries instead is a 39-row summary derived from it — one row per answer category of 7 questions, each carrying a headcount and the same share computed twice. The class file is not published either, by design: it is one room’s real answers, replaced in public copies by a placeholder with the same shape.

What has already been done to it. Almost everything. The CES teams drew the target sample, matched the panel onto it, wrote the wordings, assigned the numeric codes, validated voter registration against state records, and computed the weight, commonweight. The file arrives usable because a team spent years making it so, and their judgments arrive with it, unlabeled and inherited whole. The weighted column is not this chapter’s arithmetic; it is somebody else’s model, shipped as a column of numbers that looks exactly like the column of counts beside it.

How the data is published

A release appears on Dataverse in the months after each November election, and is revised once or twice as validation completes. Alongside it sit the questionnaires and the official Guide, the codebook that says what every numeric code means. Every percentage on this page began life as a column of single digits. Here is the class file, exactly as it is stored — the first line is its header, and the eight after it are eight students answering the same question:

19 students, seven questions, 133 rows of that. There is no word anywhere in the file. 2 is not a quantity, and it does not mean the same thing in the gender4 rows as in the educ rows below them. The file is unreadable without a document that is not in the file, and this is what almost every survey looks like when it arrives. The CES respondent file has the same shape at a different scale: one row per person, roughly 694 columns of digits, of which these are the seven this chapter reads.

The Guide that decodes those digits has to be read carefully, because it once nearly wrecked this chapter. The Guide displays the race categories in a different order than their numeric codes run: it lists Middle Eastern sixth, while the category that actually carries code 6 is “two or more races”. Read the labels off the printed order and you get this:

1,947 people called Middle Eastern and 166 called two or more races — a 12-fold error inside one category, in a table that would have looked entirely normal. The labels here were therefore attached to codes by matching the headcount the Guide publishes for each category, because the count is the only part of that mapping that can be checked against anything.

Two files sit under this chapter, and they store different things.

The benchmark file is already a summary: not 60,000 interviews, but what those interviews produced, with the individual respondents gone. You can read the margins of any question and you can no longer cross one question against another. A summary is a decision about which questions can still be asked later, made before anyone knows what they will want. The class file is individual records, which sounds better and is not, because there are 19 of them.

What the weights do to a published number

A weight is a number attached to each respondent saying how many people like them the sample ought to have contained. If the panel caught half as many people without a high-school diploma as the country has, each one is counted roughly twice. That is not a hypothetical. 2,133 respondents said they had no high-school diploma, 3.6% of everyone interviewed; after weighting, the same people are 7.5% of the sample, because that is the share the targets say the country has. Each of those 2,133 answers is therefore counted, on average, as about 2.1 answers — one person asked, standing in for two. The targets the weights aim at are counts of adults by age, sex, race and education. They come from the census and from federal surveys, which is how census data ends up quietly inside every survey estimate.

The rare thing about the CES is that both columns survive into print. Here is one row of the benchmark file:

Read that row slowly. Two percentages, one set of 54,913 people. Nobody was asked twice and nobody was added. % unweighted is arithmetic on the interviews; % weighted is the sum of commonweight over the same people. The two numbers sit in adjacent columns, formatted identically, and one of them is a model.

Every CES figure anybody has ever seen has been through this, and the unweighted column is the only reason you can see how far it travelled. Here are the eight categories the weights move furthest:

The largest adjustment in the file is voter registration. 91.5% of CES respondents said they were registered to vote. The weights cut that to 72.5% — 19.0 points. Nobody lied. The adjustment is not correcting an answer; it is correcting a sample. People who join an online political panel are far more likely to be registered than people who do not, and the weight is a formal confession of that fact.

The same mechanism runs down the table. Strong Democrats fall 5.7 points, and people who follow the news “most of the time” fall 4.8. People with no high-school diploma, the group least likely to opt into a political panel, rise 3.9. Summed question by question, the weights move 19.0% of the sample on voter registration, more than on any political question. The panel’s problem is engagement, not opinion, and the weights say so.

Now watch that machinery move a headline. Seven-point party identification is the standard survey scale of partisanship, built from two questions: whether you think of yourself as a Democrat, a Republican or an independent, and then either how strongly, or — for independents — which party you feel closer to. The seven answers run from Strong Democrat through Independent to Strong Republican, with the leaners, independents who admit to a side, one step in from each end. The scale collapses naturally into three groups, leaners counted with their party. Do it to both columns.

Figure 1. Seven answer options collapsing into three groups, and the weights changing the headline on the way. Each ribbon is one answer option, entering on the left at the width the researchers collected and leaving on the right at the width they publish; ribbon width is the share of the sample. Ribbons fit this figure because the two columns are the same people counted twice: nothing enters or leaves, it only thickens and thins. A ribbon is the one chart form that shows a reallocation without pretending anything moved in or out. The Democratic block enters at 49.1% and leaves at 41.6%; the Republican block enters at 35.6% and leaves at 39.7%. Hover a ribbon for the answer option behind it.

The sample the CES collected is D+13.5 — Democrats ahead of Republicans by 13.5 points. The sample it publishes is D+1.9. The weights move the headline partisan balance of the largest political survey in America by 11.6 points. That is not a scandal — the weighted figure is very likely the better estimate of the country. It is a demonstration that a published survey number is a construction, built by people who had to decide which variables mattered, and that the deciding happened before you ever saw it.

A room is a census of itself

The class survey has no weights. Nobody built any, and nobody could: a weight requires a population the sample is a sample of, and there is no population of which this room is a sample. The room is a census of itself, so the class column is the equivalent of the unweighted CES column — raw, and describing whoever was there. Here it is against the country, on the question everybody wants:

Reminder: the first column is placeholder data, not this class. The two CES columns are real.

The instinct is to read across the rows and start explaining the differences, and the instinct is wrong for a second reason that has nothing to do with weights.

With 19 respondents, the margin of error on any percentage the class produces is about ±22.5 points — how far off the estimate could be, and still be doing its job. A fifteen-point class-versus-country gap is not a finding; it is inside the noise. Across all 39 class-versus-CES comparisons these seven questions support, only 2 clear that band, and the other 37 sit where the room and the country cannot be told apart. Getting the room to a respectable ±3 points would take about 56 times as many people, because precision improves only with the square root of the sample.

So the two halves of this chapter fail in two different ways, and the reflex to conflate them is the thing worth unlearning.

Neither one fixes the other. A bigger sample makes you more precise about whatever your sample happens to be. Weighting corrects the imbalances somebody thought to measure, and nothing else. The 19.0-point registration adjustment and the ±22.5-point margin are two entirely different kinds of wrongness, and only one of them gets mentioned when a survey reaches a newspaper.

None of that is yet a statement about any particular room, because the class file in use is the placeholder. The arithmetic is correct; the responses it was applied to are invented.

What this data cannot tell you

Whether the weighted column is right. A weight can only correct an imbalance on something that exists in both the panel and the census tables it aims at. Whatever made someone join an opt-in panel is not a column in any census table, so it can be corrected only as far as it travels with the demographics that are. Which variables the weights are still missing is, by construction, not in the file, and never will be. The review of the 2016 polls by AAPOR, the American Association for Public Opinion Research — the pollsters’ professional body — found exactly this failure at scale: state polls over-represented college graduates, mostly did not weight on education, and that single unadjusted variable explains much of the miss.²

Whether opt-in panels beat probability samples. A probability sample is one the researcher draws at random from a list of everyone, so that each person’s chance of being picked is known — the design that all the textbook arithmetic assumes. Ansolabehere and Rivers make the case that a large, matched, transparently weighted panel now beats a probability sample that almost nobody answers; the AAPOR reviews supply the caution. Groves adds that a response rate on its own settles nothing, because nonresponse bias depends on whether answering correlates with the thing measured — which is precisely the shape of the CES’s own registration adjustment.³ Serious people hold both positions, and everything above is compatible with either.

Anything that needs the individual respondents. The benchmark file here is a summary, so there is no cross-tabulation: you cannot ask whether a gap survives holding age constant. Item non-response — people who took the survey but skipped a particular question — is present and invisible throughout: 1,478 people skipped the news-attention question, and no published figure mentions them.

Anything confident about the room. 19 respondents cannot resolve a difference smaller than about 22.5 points, and almost every interesting difference is smaller than that. The mode — the way a survey reaches a person — differs too: the CES is taken alone at home, the class survey in a room where an instructor could see the screen — identical words, a different instrument. And nobody can say what the room would look like weighted, because there is nothing to weight it to.

To which one more must be added here: the class responses in use are placeholder data, invented to give the tables their shape. Every class figure in this document is illustrative only.

What you should have learned

• A weight says how many people like this respondent the sample should have contained. It is the statistical method that makes a sample representative, and census figures are the targets it aims at.
• The CES publishes both columns, so the adjustment is visible: registration moves 19.0 points, and the collapsed partisan margin moves from D+13.5 collected to D+1.9 published.
• The largest adjustments all describe engagement with politics rather than opinion about it, which is what self-selection into a political panel predicts.
• Weighting fixes only what somebody measured; a bigger sample fixes only imprecision. Bias and noise are different failures, and neither remedy touches the other.
• A complete count of a room has no margin of error and no weights, because there is no population it samples — and at 19 people it resolves nothing finer than about ±22.5 points anyway.

Where this data appears in this book

• Sixty Thousand People, and Sixty-Eight Vermonters — what the CES’s size buys state by state, and where even sixty thousand runs out.

The section’s opening chapter, what a survey can establish, reprints this chapter’s weighting benchmarks as its evidence that the repair is large.

Sources

Data

• Cooperative Election Study, 2024 Common Content, Data Release 2 (Ansolabehere, Schaffner and Pope), Harvard Dataverse. https://doi.org/10.7910/DVN/X11EP6. The benchmark file is derived from it; the weight is commonweight, and the Guide is the codebook every numeric label above was checked against.
• Class replication survey, first day of term, recoded to CES codes. Not redistributed; the committed placeholder has the same shape.

The data itself

Every figure above is drawn from these tables. They are plain CSV, they open in a spreadsheet, and they are yours to take further.

ces2024_benchmarks.csv, class_responses_EXAMPLE.csv

Your turn

None of these is answered above, and every one can be answered from the tables you just opened.

• ces2024_benchmarks.csv reports pct_unweighted next to pct_weighted for every category. Sort by the difference. Which groups is the weighting working hardest to fix?
• Take one variable in ces2024_benchmarks.csv and find the category the raw sample is furthest off on. What kind of person is hardest to get to answer a survey?
• class_responses_EXAMPLE.csv is one room’s answers in the same coding. Compare a variable in it against the national benchmark. Where is a classroom least like the country?
• Weighting fixes what it can see. Name something about a respondent that no weight in this file could correct for.

Academic literature

• Ansolabehere, S. and Rivers, D. (2013). “Cooperative Survey Research.” Annual Review of Political Science 16: 307–329. doi:10.1146/annurev-polisci-022811-160625.
• Groves, R. M. (2006). “Nonresponse Rates and Nonresponse Bias in Household Surveys.” Public Opinion Quarterly 70(5): 646–675. doi:10.1093/poq/nfl033.
• Kennedy, C. et al. (2018). “An Evaluation of the 2016 Election Polls in the United States.” Public Opinion Quarterly 82(1): 1–33. doi:10.1093/poq/nfx047. The committee’s report: https://aapor.org/wp-content/uploads/2022/11/AAPOR-2016-Election-Polling-Report.pdf
• YouGov. “Panel methodology.” How the opt-in panel is recruited and matched. https://today.yougov.com/about/panel-methodology

The file behind this chapter comes from a source no script can download
today, though a person with a browser can.

WHY NOT. The chapter is built on the 2024 Cooperative Election Study
Common Content — 60,000 Americans interviewed before and after the
2024 election — held on Harvard Dataverse at
https://doi.org/10.7910/DVN/X11EP6
Dataverse sits behind an AWS web application firewall that answers an
automated request with status 202, "Accepted", and a small challenge
page instead of the data. The code says success; the body is a wall
only a real browser can pass. And at 184 MB the survey file is too
large to travel with this book in any case.

WHAT EXISTS INSTEAD. Open the address above in an ordinary browser
and the download works normally; the data are free. The benchmark
table this chapter actually uses — every category of seven core
questions, with its unweighted share and its share under the study's
own weight — is small and ships alongside, so nothing needs
downloading to follow the argument.

WHAT YOU CAN STILL CHECK. If you do download the survey file, these
numbers say whether you hold the same object:
- 60,000 rows and 694 columns
- 54,913 respondents said they were registered to vote — 91.5% of
  the sample unweighted, and 72.5% once the weight is applied. That
  nineteen-point correction is the chapter's point.
- 16,576 called themselves strong Democrats and 10,826 strong
  Republicans, before any weighting