Testing BISG Race Imputation Against Known
Race
BISG checked against the truth, on forty-six
thousand Georgia voters
Democracy’s Data
Last updated September 15, 2026 at 4:42p EDT
Courts are told every year how many Black voters live in a district,
and nobody counted them. Voter files do not record race, so the number
is manufactured by a method: Bayesian Improved Surname
Geocoding, or BISG, which multiplies what a surname implies
about a person by what their neighbourhood implies (the surnames brief works the
arithmetic by hand). Elliott and colleagues (2009) built it for health
records;
Imai and Khanna (2016) brought it to voter files. It is the standard
evidence in vote-dilution cases under Section 2 of the Voting Rights
Act, the provision that forbids election rules giving minority voters
less opportunity than others to elect candidates of their choice, and in
federal lending-discrimination work. How good is the guess?
Almost everywhere, that question cannot be asked, because an
inference is an inference when the answer is not written down. Georgia
writes it down. Race appears on the voter registration form, collected,
as the federal registration form puts it, “in order to administer the
Federal Voting Rights Act.” So in one Georgia county there are 45,984
people whose race a widely used method is guessing, and whose race is
already on file.
The question sounds like it has one answer. It has five, and they
disagree with each other by 90 points.
The test
The test joins three files, none of which was built to meet the
others. The first is a Houston County, Georgia voter registration
extract, carrying each voter’s surname, the race they reported and the
census block they live in. It is working data from the county’s 2026
redistricting matter, and it is not published: a Georgia voter file is
ordered from the state (https://sos.ga.gov/page/order-voter-registration-lists-and-files),
cut on one day and never regenerated. The second file is the 2020 Census
count of who lives on each of the county’s blocks. The third is the
Census Bureau’s Frequently Occurring Surnames table from the
2020 Census — every surname held by about a hundred or more people, with
the race shares of its holders — downloadable at https://www2.census.gov/topics/genealogy/2020surnames/,
and the same copy the surnames
brief uses, so the two briefs cannot disagree about it.
These files can answer the question because one of them contains the
answer. A small number of states put a race question on the registration
form, most of them once subject to federal supervision of their election
law. Nobody collected that column to grade a statistical method, which
is what makes it a fair answer key.
It is also the deepest problem in this literature. An
inference can only be graded where the answer is recorded, and the
answer is only recorded where the inference is unnecessary.
Every published accuracy figure for BISG comes from a place that did not
need it.
Two joins have to hold before anything can be scored. Join Keys and
Crosswalks is the reading on what a join key is and how one fails
silently. It also covers the repair when two files carry different keys:
a crosswalk, a table that translates one set of identifiers into
another.
The data
Every number below comes from running BISG here, on every voter, and
comparing each answer with the race that voter reported. The surname’s
race shares, the block’s, the two rules for when one of them fails, and
the scoring are all recomputed on this page; nothing is copied from a
published accuracy figure.
The join runs on two keys and both are strings of text. The
surname joins a voter to the Census surname list. The
fifteen-character census block identifier,
GEOID20, joins a voter to the block table — a GEOID is the
Bureau’s code for any piece of its geography, and a block’s is the
state, county, tract and block numbers run together — and it arrived
already filled in. Somebody matched street addresses to blocks before
this brief existed, and every address that failed was settled by a rule
no column records. Geography is half of what the method uses, so part of
what gets graded below is that match.
Three cleaning decisions got 53 columns down to three, and each one
changes what can be claimed.
Rows whose answer key was written by the method being graded
are dropped first. 11,168 of the 155,756 records carry a flag
saying the voter left the race question blank and somebody filled it in
by running BISG. Scored against those rows the method looks superb, and
nothing in the numbers would give the circularity away.
A further 854 voters reported a race the five-category scheme
has no box for, or none at all. They are dropped too, which is
not a neutral act: the method is now graded only on people who fit the
categories it predicts.
The identifiers go on purpose. The registration
number joins the two state files and is discarded as soon as the join is
done, so the committed table cannot be linked back to a person. It
cannot be audited against its source either.
What survives is 49,771 voters across 2,130 blocks, in three columns:
surname, race as the voter reported it, and
GEOID20. The census side is aggregated to the same 3,269
blocks, one column of residents per group. One more thing happens before
scoring:
| Voters in the file |
49,771 |
| Whose surname appears in the Census surname list |
45,984 (92.4%) |
| Whose surname does not |
3,787 |
| Voters the method is scored on |
45,984 |
3,787 voters, 7.6% of them, hold a surname the Census list does not
carry, because too few Americans shared it in 2020 for the Bureau to
publish it. The method has no opinion about them, so they appear in no
figure below. Who they are turns out to matter.
Here is one case, worked the way the method works it. Take the
commonest surname in the county’s file, Williams, and the census block
where the most Williamses are registered. (The block is not named here;
nothing from the voter file is reproduced except a tally.)
| white |
46.2 |
198 |
30.0 |
53.9 |
0.56 |
25.7 |
20.8 |
| Black |
48.7 |
445 |
67.4 |
35.3 |
1.91 |
92.9 |
75.3 |
| Hispanic |
3.7 |
7 |
1.1 |
7.4 |
0.14 |
0.5 |
0.4 |
| Asian |
0.6 |
3 |
0.5 |
3.1 |
0.14 |
0.1 |
0.1 |
| Am. Indian |
0.8 |
7 |
1.1 |
0.2 |
5.00 |
4.1 |
3.3 |
The surname table says a Williams is 48.7% Black and 46.2% white, so
the name alone settles nothing. The 2020 count found 660 people living
on the block, 445 of them Black and 198 white. That makes the block
67.4% Black against 35.3% for the county as a whole, 1.91 times the
county share, and 30.0% white against 53.9%, 0.56 times. Multiply each
group’s name share by its ratio, add the five products, and each
product’s share of the total is the posterior, the
probability once the block has been taken into account: 75.3% Black,
20.8% white. The method’s guess is Black. On the answer key, 11 of the
11 registered voters named Williams on that block reported Black, and
across the county’s 699 Williamses 72.5% did. That is one case going
right. The rest of this brief counts how often it does.
Before you read on, write down three numbers. BISG
scores 67.3% overall on these voters. First: what share of
white voters do you think it gets right? Second: what share of
Black voters? Third, and this is the one that matters —
what would a method score if it simply guessed “white” for every
voter in the county, with no surname, no geography and no model
at all?
What it says about democracy
Start with the third number, because it is free. Guessing
“white” for everybody scores 51.5%, since 51.5% of these voters
are white. Most people put that floor near a fifth, on the intuition
that a five-category problem gives a blind method one chance in five.
The surname alone scores 61.3%, and BISG with the block scores
67.3%.
The dashed line is free: it is what a method with no method scores. 2 of
the five groups sit below it, and so would the whole electorate if it
were not 51.5% white. Hover a bar for the group size.
Figure 1. Every accuracy in this brief, against the number
available for free. One bar per group fits this question,
because the question is a comparison against a fixed reference. The bars
are how often BISG identifies each group correctly, with the whole
scored electorate on the right, and the dashed line at 51.5% is what a
method with no method collects. 2 of the five groups sit below it, and
so would the overall bar if the county were not 51.5% white.
Read down the column rather than across the bottom:
| white |
23,668 |
94.5% |
90.6% |
-3.9 |
| Black |
16,841 |
16.9% |
40.7% |
+23.8 |
| Hispanic |
2,319 |
71.4% |
65.9% |
-5.5 |
| Asian |
2,038 |
64.2% |
55.3% |
-8.8 |
| Am. Indian |
1,118 |
0.2% |
0.3% |
+0.1 |
The overall figure is not a summary of how the method
performs. It is a summary of who lives in Houston County. On an
unbalanced problem, overall accuracy mostly reports the balance.
Adding the census block is not a free improvement either. It is a
redistribution. Black voters gain +23.8 points, from 16.9% to 40.7%,
because Black residents of this county are geographically concentrated
and the block genuinely carries information about them. Asian voters
lose 8.8 points, Hispanic voters 5.5, white voters 3.9. Those
populations are spread thinly, so no block is distinctively Asian or
Hispanic, and the geographic term drags the estimate toward the local
majority. One group gained and three lost, while the overall figure
moved +6.0 points and reported none of it.
Accuracy asks whether the winning label was right. Ask instead how
much probability the model put on the answer that turned out to be
correct — a question of calibration, whether a model
that says “70%” is right about seven times in ten — and the per-person
picture changes.
Figure 2. How much probability the model placed on the answer
that was correct. One panel per group on a common scale, each
bar a five-point slice of probability, with the vertical rule at that
group’s mean. White voters pile against the right-hand wall and Am.
Indian voters against the left. The Black panel is the one to sit with:
spread through the middle, at neither end.
| white |
0.79 |
90.6% |
| Black |
0.43 |
40.7% |
| Hispanic |
0.60 |
65.9% |
| Asian |
0.54 |
55.3% |
| Am. Indian |
0.01 |
0.3% |
For Black voters the model puts 0.43 on the truth. It is not
confidently wrong; it is genuinely uncertain, and then the decision rule
forces a hard guess anyway. Much of the information loss is in
collapsing a probability into a label rather than in the model. For
American Indian voters it puts 0.01 on the truth, and those 1,118 voters
cannot be found by any threshold at all.
Being wrong is also not one thing. The errors go somewhere, and where
they go is the rest of the table.
Figure 3. Where the errors go. Each row is a group
of voters as they reported themselves, each column is what BISG guessed,
and rows sum to 100%. The outlined diagonal is the accuracy column from
the table above; everything off it is a specific mistake. The largest
off-diagonal cell is 67.0% of Am. Indian voters labeled white, and 57.7%
of Black voters are labeled white.
The errors are not scattered. They run one way,
toward the group that is locally largest, which is exactly what
multiplying by the block’s make-up does.
The same mechanism breaks the method outright for 394 people. 392
live in a block the census records as empty; for 2 more, surname and
block contradict each other so flatly that every category multiplies to
zero. Those are the two rules mentioned above: a voter on an empty block
is scored with the county’s make-up in place of the block’s, and a voter
whose name and block cancel out is scored on the name alone. Without
them all 394 would have been labeled white, and 158 of them are
Black.
Two checks cut the other way. Replace every block with the county’s
make-up and the method scores 58.8% overall, worse than no
geography at all, because one composition applied to everybody
tilts every voter toward white. And on the 8,450 voters whose surname
gives no group above 60%, adding the block is worth +16.1 points, more
than twice what it gains on the full file. The method is strong exactly
where the surname was weak.
The number that should stay with you is that with
everything it has, BISG identifies Black voters correctly 40.7% of the
time. It is wrong about them more often than it is right, in a county
where 37% of these voters are Black, on the question Section 2
litigation turns on. That is not an argument for throwing the method
out, because the alternative is no evidence and no way to prove vote
dilution at all. It is an argument that an error rate this uneven
belongs beside every estimate built on it, and it almost never is.
What this brief cannot tell you
It is one county. Houston County, Georgia is about
51% white and moderately segregated, and nothing here says whether it is
typical. The shape of these findings travels; the figures should not be
quoted anywhere else.
Whoever the surname list missed never entered the
test. The 3,787 unscored voters are 23.3% Hispanic against 6.4%
among all registered voters, over-represented by a factor of about 3.6.
So the Hispanic accuracy above was measured on Hispanic voters with
common Hispanic surnames. Every figure here is conditional on the method
having had an opinion in the first place.
The answer key is itself a measurement. The race on
a registration form is one box, chosen by the voter, flattening the same
people every categorical race question flattens. Any-part Black —
counting as Black anyone who checked Black alone or together with
another race — is the definition used throughout, following Voting
Rights Act practice, and the stricter one would move every
figure above. And these are registered voters rather than residents.
And the validation comes from a biased set of
places. Checking BISG requires a state that records race on
registration, which is a small group concentrated in the South and
overlapping heavily with the states most often in Section 2 litigation.
That is either reassuring, because it is where the method is used, or
circular, for the same reason.
What you should have learned
- On 45,984 people whose race is on record, BISG scores 67.3% overall,
against a free floor of 51.5% for guessing “white” every time.
- That overall figure conceals the groups: 90.6% for white voters,
40.7% for Black, 0.3% for American Indian.
- Adding the census block is a redistribution, not an improvement:
+23.8 points for Black voters, a loss for Asian, Hispanic and white
voters, and +6.0 overall.
- The errors run one way, toward the locally largest group, which is
what multiplying by a geographic prior does: 57.7% of Black voters are
labeled white.
- 7.6% of voters were never scored, their surname being absent from
the Census list, and they are 23.3% Hispanic against 6.4% of all
registered voters.
Extensions
None of these is answered above, and every one can be answered by
sorting or grouping the tables the Sources section hands over.
houston_voters.csv gives every voter a
surname, a self-reported race and a
GEOID20. Group by race and count. Which groups
are large enough here to say anything about?- Take a common surname from
census_surnames.csv and find
its holders in houston_voters.csv. Compare
pctwhite and pctblack against what these
voters reported.
houston_blocks.csv gives each block’s residents by
race. Sort on the five count columns to find the blocks with nobody in
them. What should a method do when it lands on one?- Pick a surname in
census_surnames.csv whose shares are
close to even, and work out the most confident BISG could possibly be
about anyone carrying it.
Two stretch ideas point past the spreadsheet. North Carolina also
records race on its registration form and publishes its file (https://www.ncsbe.gov/results-data/voter-registration-data);
run the same comparison there and see whether the pattern holds. And
find an expert report quoting a BISG accuracy figure, then check whether
it reports by group or only overall.
Sources
Data
Literature and cases
- Elliott, M. N. et al. (2009). “Using the Census Bureau’s Surname
List to Improve Estimates of Race/Ethnicity and Associated Disparities.”
Health Services and Outcomes Research Methodology 9(2): 69–83.
https://doi.org/10.1007/s10742-009-0047-1
- Imai, K. and Khanna, K. (2016). “Improving Ecological Inference by
Predicting Individual Ethnicity from Voter Registration Records.”
Political Analysis 24(2): 263–272. https://doi.org/10.1093/pan/mpw001
- Imai, K., Olivella, S. and Rosenman, E. T. R. (2022). “Addressing
Census Data Problems in Race Imputation via Fully Bayesian Improved
Surname Geocoding and Name Supplements.” Science Advances
8(49): eadc9824. https://pmc.ncbi.nlm.nih.gov/articles/PMC9728965/
- Consumer Financial Protection Bureau (2014). Using Publicly
Available Information to Proxy for Unidentified Race and Ethnicity.
https://files.consumerfinance.gov/f/201409_cfpb_report_proxy-methodology.pdf
- Thornburg v. Gingles, 478 U.S. 30 (1986). https://www.law.cornell.edu/supremecourt/text/478/30
- Georgia v. Ashcroft, 539 U.S. 461 (2003) — the
any-part-Black count. https://www.law.cornell.edu/supremecourt/text/539/461
- Voting Rights Act, Section 2, 52 U.S.C. § 10301. https://www.law.cornell.edu/uscode/text/52/10301
The data itself
Every figure above is drawn from these tables. They are plain CSV,
they open in a spreadsheet, and they are yours to take further.
houston_blocks.csv,
houston_voters.csv,
census_surnames.csv
