The first three sections of this book each worked with one kind of data at a time. A count described the population, a survey asked it questions, an administrative record wrote down what a system did. That one-source-at-a-time discipline was a simplification, and this section drops it.
Almost nothing here introduces new data. What is new is the combining. A name and an address joined to a guess about race. A survey’s claim about turnout joined to the counted ballots. A precinct result joined to the demographics of the precinct, and a district map graded against a rule somebody wrote down.
That is where the questions people actually care about live. It is also where nearly all the wrong answers come from, and the two facts are related.
Start with what these 19 docs produce, because it is a genuinely different kind of object from anything so far.
There is no file anywhere that records a voter’s race, if the voter’s state does not ask. There is no file that records how Black voters in a county voted, because ballots are secret. There is no file that records whether a district was drawn to dilute somebody’s vote, or what the House will look like next November. Every one of those numbers is manufactured — computed from other numbers, by somebody, using a method with assumptions in it.
You have already met one manufactured number without being asked to notice. A member of Congress’s ideology score, in the roll-call cluster of Section III, appears in no clerk’s record. It is computed from the votes, by a method, and the method’s choices are inside every polarization chart built on it. This section is where numbers of that kind stop passing unexamined.
Because once such a number is computed, it looks exactly like the measured numbers around it. It has the same number of decimal places. It goes in the same column, and it joins to other files as easily as a count does. The assumptions do not come with it. The point of this section is to keep hold of them anyway.
Here is the mistake this section keeps returning to, stated once so you can watch for it everywhere. A number built from two sources inherits the assumptions of both. Then the join — the step that matches the two files row by row on something they share, a county code, a year, a name — adds one more of its own: an assumption about how the two files line up. The first two you have spent three sections learning to see. The third is new, it belongs to nobody’s file, and no column in either source records it.
It fails in a few repeating ways, and you have met each of them once already, in single-source form.
A code outlives what it names. Two files agree that
a row is 11001, and mean different ground by it — the join
that opens this section, where a county code silently switched from a
city to an eighth of one. Match on a precinct name, a county code or a
district number across years, and part of what you measure is the drift
of the label rather than a change in the world.
A denominator is borrowed across sources. A denominator is the bottom number of a fraction, the thing you divide by. Here the top of the fraction comes from one file and the bottom from another, made by different people, on different dates, with different rules about who counts. Divide counted ballots by a census population, and the fraction quietly assumes the two files describe the same people at the same moment. They rarely do. One doc here shows a famous thirty-year decline in American turnout that lives entirely in the borrowed bottom number, and disappears when the denominator counts only people actually eligible to vote.1
A probable match is treated as a certain one. Most joins in real political data are guesses with a score attached — this voter is probably that record, this surname probably belongs to this group. The arithmetic that follows almost always drops the “probably”. A guess laid over a guess can still print a tight, confident interval — a margin of error that looks precise — and one doc here exists to show exactly that happening.
None of these failures announces itself. That is what makes join error the section’s defining problem rather than its nuisance: the join succeeds, the table is complete, and the number is wrong. Every check a careful person runs on a single file comes back clean, because no single file contains the mistake. The mistake lives between them.
The section runs in 6 clusters. Only 1 doc is a data-type chapter, because only one new kind of raw material is introduced: the join key itself. Everything after it combines files you have already met.
| Cluster | Docs |
|---|---|
| IV.1 Joining Files | 1 |
| IV.2 Survey Meets Record | 1 |
| IV.3 Race Inference | 4 |
| IV.4 Districts | 5 |
| IV.5 Forecasting | 4 |
| IV.6 Proxies and Indices | 4 |
Joining Files opens with the join key: one federal county code, two files four years apart, and a match that succeeds on every row while being wrong in one of them. It is the mechanics of every join after it, worked on the simplest possible case.
Survey Meets Record puts Section II against Section III directly: what people say they did against what the ballots show, over sixty years. People report voting more than the counted ballots can support. The gap is not noise, it runs one way, and it is one of the most reliable findings in American survey research.2 That makes it the cleanest possible demonstration of what a record can do to a survey.
Race Inference builds a number no file contains. A federal surname table turns a name into a probability. A check against a state that records race says how often the guess is right, and for whom it fails. A third doc shows a tight confidence interval sitting on top of a number that describes nobody. The cluster ends with ecological inference, guessing what individuals did from group totals, run on a county where the answer is known and producing an estimate above 100%.
Districts joins census counts, election returns and geography to grade maps. Thousands of computer-drawn plans stand in for the maps nobody drew. The other docs test real maps against them: the same votes poured into redrawn lines, and a dilution claim about a map that does not exist. A third asks how many majority-Black districts a state has, and finds six defensible answers. A bias statistic gets a doc of its own, because what it measures is not intent.
Forecasting joins everything to the calendar: polls, history and judgment compressed into a probability, and a working model of an election that has not happened. A forecast is the join at its most exposed, because the thing it is joined to is the future, and the future supplies no file at all until the night it arrives. Grading one is its own problem. A forecast that gave the loser thirty per cent was not wrong, and finding out whether it was any good takes hundreds of resolved forecasts, which is exactly what the cluster assembles.
Proxies and Indices closes the book with numbers that stand in for other numbers. There are two denominators for turnout, fifty-six election laws compressed into one score, and a famous shorthand about grocery stores rebuilt from scratch to see what it is made of. A proxy is a join whose second file is an idea, and the cluster asks what each one quietly decided on your behalf.
A manufactured number cannot be checked against a file, so it has to be checked another way. Run the method on a case where the true answer is already known, and see whether it gets it.
That is what this section does, over and over. Guess race from names and addresses, then compare against a state that records race. Estimate how a group voted from precinct totals, then compare against ballots that were counted separately. Grade forecasts against the elections they forecast, and search for a fairer map with an algorithm, then check whether the map it could not find has been law for years.
The answers are not reassuring, and that is the point of putting them last. You now know enough about where each file comes from to understand why the methods fail where they do — which is a different and more useful thing than being told not to trust them.
This is the destination of the book. Its docs read files built all over the earlier sections: a census denominator here, a voter file there, a set of returns underneath nearly everything. No doc anywhere reads what this section computes. Everything flows in, and nothing flows out.
That is what a final section should look like. The manufactured numbers here are ends of chains, not links in them. Each doc names the docs it borrowed from, so you can walk any chain back to a source with a signature on it. Walking the chain is worth doing at least once. A racially polarized voting estimate rests on a race guess, which rests on a surname table and a census block count, which rest on a form somebody filled in. Every link was somebody’s decision, and the estimate at the end is only as good as the weakest of them.
Read it in order. This is the one section of the book where that matters. The docs build. The join-key doc supplies the mechanics, the race-inference cluster supplies a method and its error rate, the voting-estimate doc uses that method, and the district docs put the whole apparatus up against real law.
Two docs are fixed to dates rather than to places in the sequence. The Senate-rating brief asks you to commit to a map in August, to a number you cannot revise in November. The election-night brief runs on the night itself, with results that do not exist until then, and grades the predictions made earlier against what the country actually did.
If you only read one doc, read the racially polarized voting brief: a method used in federal voting-rights litigation, run against an answer key, getting an impossible number with an excellent fit statistic on top.
INDEX.md, so
this page cannot disagree with the section it opens; reorganise the
section and re-render, and the table above follows.The decline, and its disappearance, were first shown by Michael P. McDonald and Samuel L. Popkin, “The Myth of the Vanishing Voter,” American Political Science Review 95, no. 4 (2001): 963–974, https://doi.org/10.1017/S0003055400400134.↩︎
The modern statement of it, matching a large survey to the voter rolls person by person, is Stephen Ansolabehere and Eitan Hersh, “Validation: What Big Data Reveal About Survey Misreporting and the Real Electorate,” Political Analysis 20, no. 4 (2012): 437–459, https://doi.org/10.1093/pan/mps023.↩︎