Confidence Intervals and What They Do Not Cover

One surname model scored 49,771 times, and a tight interval around a number that describes nobody

Democracy’s Data

Last updated September 15, 2026 at 4:42p EDT

Every estimate you meet in public life is supposed to come with a margin of error: a number saying how far off the estimate could be and still be doing its job. A narrow margin reads as a settled number. This brief asks what a narrow margin is actually a fact about.

It asks that in the one place in this book where an estimate can be graded against a recorded answer. Georgia writes race on the voter registration form, so for 49,771 registered voters in Houston County there are two things on file. A guess, worked out from the voter’s surname and the census block they live in, the smallest area the census reports and often a single city block. And the answer, which is what the voter wrote down.

The guess is right for 65.53% of them, and its 95% confidence interval, the range that would capture the true share in 95 of every 100 samples drawn this way, runs from 65.11% to 65.94%. That is less than a point wide, which makes it one of the most precisely measured numbers in this book. Whether it describes anybody is a separate question, and the interval has nothing to say about it.

The test

The measurement joins three files. The first is Houston County, Georgia’s voter registration file, carrying the race each voter reported and the census block each one lives in. Georgia sells its voter file to anyone who orders it (https://sos.ga.gov/page/order-voter-registration-lists-and-files); this copy is working data for the county’s 2026 redistricting matter, in which the instructor is retained, and no material from any expert report is used here. The second is the U.S. Census Bureau’s Frequently Occurring Surnames from the 2010 Census, downloadable at https://www2.census.gov/topics/genealogy/2010surnames/names.zip. The third is the Bureau’s 2020 Census count of who lives on each of the county’s blocks, from the redistricting file it publishes for every state (https://www2.census.gov/programs-surveys/decennial/2020/data/01-Redistricting_File--PL_94-171/Georgia/ga2020.pl.zip). The voter file and the block counts are the same ones the BISG check brief uses, and the surname list is the one the surnames brief describes.

These are the right files because grading a guess requires a recorded answer, and almost no file in American politics has one. A handful of states, mostly ones once subject to federal supervision of their election law, put a race question on the registration form.1 That question is why the quantity a surname model is trying to produce is already written down here.

Everything below rests on two joins holding. Join Keys and Crosswalks is the reading on what a join key is and how one fails. It also covers the repair for a key that will not line up: a crosswalk, a table that translates one set of identifiers into another.

The data

Every table under this brief comes from running the surname-and-block method on all 49,771 voters and comparing each answer against the race that voter reported. The same method was scored again inside every group, and again inside every kind of neighbourhood.

The join has two keys and both are strings of text. The surname joins a voter to the Census surname list, and it does not always land. 7.6% of these voters, 3,787 people, carry a name the Bureau does not publish, because fewer than a hundred Americans shared it. Those voters fall back to the national spread of surnames, which is a much weaker starting guess. The fifteen-character census block identifier, GEOID20, joins a voter to the block table. It arrived in the extract already filled in, matched from a street address by somebody else.

One piece of cleaning decides what can be claimed. The source file marks records whose race was supplied by a model rather than by the voter, and every one of those was dropped before any of this ran. Scoring a method against its own output would produce an excellent number that means nothing.

Four tables come out of that run. by_group.csv holds one row per racial group with n_true, n_pred, recall and its interval rec_lo and rec_hi, and precision with pre_lo and pre_hi. by_decile.csv holds the same recall in ten strata of neighbourhood, with block_black_lo, block_black_hi, black_voters, recall, lo and hi. thresh.csv sweeps the cutoff, carrying t, the four counts tp, fp, fn and tn, and the rates tpr, fpr and prec. calib.csv checks the model’s calibration, whether its stated confidence can be taken at its word: it gathers the voters the model gave a similar probability of being Black and compares that probability with the share who actually were. Here it cannot be taken at its word. Among the 2,369 voters the model scored between 25 and 30 percent likely to be Black, 49.2% were.

Four quantities do all the work below, and they answer different questions:

Quantity What it answers Computed from
recall of the people who ARE this race, how many did the model find truth, then prediction
precision of the people the model CALLED this race, how many were prediction, then truth
the interval how far the estimate would move if you drew the sample again the count and the sample size
AUC how well the score separates the two groups, at any threshold the ranking, not the threshold

The intervals are Wilson intervals, a 1927 formula that keeps an interval for a percentage between 0 and 100,2 rather than the textbook estimate plus or minus two standard errors. One group below sits so close to zero that the textbook formula puts the bottom of its interval below zero. An interval containing impossible values is a sign the approximation has been pushed outside its range.

Before you read on, commit to a number. The model is 65.53% accurate, give or take half a point. Write down what you expect its accuracy to be for the smallest group in the file — American Indian and Alaska Native voters, 1,236 of them. A number, and a range around it.

What it says about democracy

Score the same model separately for each group and the pooled figure stops describing any of them.

Figure 1. The same model, scored separately for each group. A forest plot fits this question, because the question is whether one pooled number covers the rows underneath it. The bar is each group’s 95% interval and the diamond is its estimate. The shaded band carries the pooled interval up through the figure, so you can see what falls inside it. Almost nothing does.

The model finds 90.1% of white voters and 39.3% of Black voters. For American Indian and Alaska Native voters it finds 0.32%: 1,236 people, of whom it identifies almost none, with an interval of 0.13% to 0.83% that never comes near the pooled figure. Put that row in people. Of the 1,236 voters who reported American Indian or Alaska Native, the model found 4; that is its recall, 4 divided by 1,236, or 0.32%. It put the label on 159 voters, and those same 4 were the only ones it had right; that is its precision, 4 divided by 159, or 2.5%. So on the rare occasion it does say American Indian, it is almost always wrong.

Compare those rows with the number you wrote down. The pooled 65.53% is not an average of them in any useful sense. It is dominated by the largest group, and a reader given only the pooled figure has no way to discover a group the model cannot see at all.

The obvious objection is that groups differ, so of course their numbers differ. So here is one quantity, how often the model finds a Black voter, computed ten times on ten slices of the same population. Nothing changes except the neighbourhood.

Figure 2. The same estimate, ten times, on ten strata, or slices, of one county. Rows are tenths of the census block’s Black share, from the whitest blocks at the top to the Blackest at the bottom. The band and the dashed line are the pooled figure for Black voters, 39.3%.

Read the diamonds down the page and they run from 3.1% to 91.6%. The model finds nine in ten Black voters who live in a majority-Black block, and one in twenty who live in a mostly white one. Now read the bars. The widest interval anywhere in that figure is 4.0 points. Every one of these estimates is precise, and they disagree with each other by 88.5 points.

That is the thing a confidence interval does not cover. It answers one question: how much would this number move if I drew the sample again? Here the honest answer is hardly at all. It says nothing about whether the number carries to a different neighbourhood, a different county or a different year, which is almost always what is being asked of it. A narrow interval on a pooled estimate is evidence of a large sample, not of a reliable estimate.

Notice what the model is doing to earn its score. Its accuracy tracks the racial composition of the block almost exactly, which means most of its apparent skill is the geography rather than the name. It works because Georgia’s blocks are segregated.

Everything so far used the model’s own top guess. But the model does not produce a guess. It produces a probability, and turning a probability into a decision takes a cut that nobody in the file chose.

Figure 3. The same model at every possible cut. Each point on the curve is one threshold, and the panel beside it reports what that choice does to four counts of real people. The diagonal is what a coin would achieve. The area under the curve is 0.83, a summary of the ranking that depends on no threshold at all, on a scale where 1 is a perfect ranking and 0.5 is the coin.

Move the cut and the four counts trade against each other. At the conventional cut of one-half, calling a voter Black whenever the model’s probability reaches 50%, the model finds 35.7% of Black voters at the cost of calling 5.3% of everyone else Black. No setting is correct, because the two errors are not the same kind of thing. One is a Black voter the analysis loses; the other is a white voter the analysis invents. Which matters more depends entirely on what the number is for.

The area under the curve is a property of the model. Every other number in this brief is a property of a decision somebody made, and none of those decisions is recorded beside the estimate when it travels.

What this brief cannot tell you

One county, one state, one model. Houston County, Georgia has 49,771 scored voters and a particular racial geography. Figure 2 is, if anything, an argument that these numbers do not carry even between neighbourhoods of the same county.

The intervals assume the sample is the only source of error. A Wilson interval describes sampling variation and nothing else. It does not cover the surname list being from a different decade than the voters, or self-reported race being a question people answer inconsistently, or a census block being a crude stand-in for a neighbourhood. Those errors are real, they are larger than the bars in Figure 1, and no interval computed from this data can see them.

Recall is not accuracy, and most of what is shown here is recall. A model that called every voter Black would find every Black voter and be useless. The American Indian and Alaska Native row is the case where both measures fail at once: recall 0.32%, precision 2.5%.

What you should have learned

Extensions

None of these is answered above, and every one can be answered by sorting or grouping the tables the Sources section hands over.

Two stretch ideas point past the spreadsheet. North Carolina also records race on its registration form and publishes its voter file; score the same method there and see which of these numbers travel. And any survey estimate you read this week will carry a margin of error somewhere in its methods note — find one, and work out which of the errors listed above it covers.

Sources

Data

The data itself

Every figure above is drawn from these tables. They are plain CSV, they open in a spreadsheet, and they are yours to take further.

by_decile.csv, by_group.csv, calib.csv, roc.csv, thresh.csv

One more file is not data about the subject – it is how this chapter checked its own numbers: facts.csv, the individual numbers the text above quotes.

  1. The national mail registration form’s state-by-state instructions show which states ask: U.S. Election Assistance Commission, https://www.eac.gov/voters/national-mail-voter-registration-form.↩︎

  2. Edwin B. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference,” Journal of the American Statistical Association 22, no. 158 (1927): 209–212, https://www.jstor.org/stable/2276774.↩︎