Traffic Stops and the Benchmark Problem

The test

The stop records are the ones San Francisco’s police department keeps for each stop, standardized and published by the Stanford Open Policing Project, a research group at Stanford that has collected such records from police departments across the country by public-records request.² Its San Francisco file, released 1 April 2020, covers 2007 to 2016 and can be downloaded at https://openpolicing.stanford.edu/data/. No law requires a national record of traffic stops, and none exists. The project assembled what there is one agency at a time, and then forced dozens of incompatible extracts into a single layout with the same column names and the same coding.

That layout is why this source, of all sources, can answer the question. Each row carries whether a search happened and what the search turned up, on the same record. Many agencies publish one and not the other, and a department’s own summary table publishes neither. The population figures come from a different kind of source entirely. They are U.S. Census Bureau counts for San Francisco County from the American Community Survey, the Bureau’s continuous survey of a sample of households, the one that asks about race, age and how people get to work; its five-year releases pool five years of answers to give a steady figure for a county. The counts here are the average of the two releases that cover the stop years exactly, 2007–2011 and 2012–2016. Every table is free at https://data.census.gov/; the Bureau asks anyone who downloads its tables through its data service, rather than by hand, to register for a free key at https://api.census.gov/data/key_signup.html.

A stop record is administrative. It is paperwork an institution left behind while doing something else, and nobody in it was being measured or asked. The section introduction sets out what that kind of record can bear, and why it nearly always arrives as a top number with nothing underneath it.

The data

Everything below rests on by_race.csv, which has 5 rows and five columns: race, stops, searched, contraband_found and arrests, the project’s one-row-per-stop file added up by the driver’s race. by_year.csv does the same split by year, and acs_denominators.csv holds the population counts, one row per group per denominator, each naming the census table it came from in source.

The one-row-per-stop file carries these fields, and two of them decide everything that follows:

subject_race is the officer’s perception, not the driver’s self-report. It is a box somebody ticked at the roadside. For a question about the officer’s behaviour that is arguably the right variable, because what matters is what the officer believed he was looking at. It is not the census’s variable, and the two must never be quietly mixed.

contraband_found exists only where search_conducted is true. Where nobody searched, the value is blank, and that blank is not missing data. It records that there was nothing to find because nobody looked.

Cleaning here mostly meant throwing things away on purpose. Every column of the stop file was dropped except race, the two search columns, the arrest column and the year inside the date, so no individual-level record is kept here at all. The yes/no columns arrive as text, and only the word TRUE is read as a yes, which makes a blank a no rather than a mystery. Nothing was filled in by guessing and nothing was reweighted. A stop with no race recorded is dropped rather than filed under other, because “the officer did not write it down” and “the officer wrote other” are different facts. That rule costs nothing here: the 5 categories account for all 905,070 stops.

The census side has one flaw worth naming. The residents table, the Bureau’s count of Hispanic origin by race, splits the population cleanly, while the adult and commuter tables, its counts by age and by journey to work, each published separately for each race group, fold Hispanic respondents into the other races. The commuter table also counts workers who live in San Francisco, so it holds a resident driving out to San Mateo and misses one driving in from Oakland. Where no honest match existed, as for the other category past the residents row, the row was left out rather than approximated.

Here is the cleaned table:

Before you read on, commit to two numbers. You have seen what happens to Hispanic drivers when the bottom number of the fraction changes, from 0.86 against residents to 1.01 against drivers. Now abandon population altogether. Of the drivers an officer had already pulled over, what share was searched? Write down the ratio you expect for Black drivers against white ones. Then write down whether searches of Black drivers turned up contraband more often, less often, or about as often as searches of white drivers.

What it says about democracy

To turn stops into a rate you need a denominator, the bottom number of the fraction — the thing you divide by. Here is what you would actually need, and whether it exists:

Follow one group through the arithmetic. The file holds 152,196 stops of Black drivers over ten years, and the census counts 45,552 Black residents of San Francisco, so that is 3.34 stops per resident. White drivers: 372,318 stops over 342,448 residents, 1.09 per resident. Divide the first rate by the second and Black drivers are stopped 3.07 times as often, a disparity ratio, one group’s rate over another’s. Now change only the bottom number. The census also counts who drives to work: 7,656 Black residents and 89,679 white ones. The stops have not moved, but the rates become 19.88 and 4.15 stops per driver, and the ratio 4.79.

Figure 1 does that for every group. It puts the same 905,070 stops over three real census counts and one question needing no population at all.

Log scale, so equal distances are equal ratios.

Figure 1. The same stops, four denominators, and a disparity that changes sign. One line per group, on a log scale, because the whole question is how far a ratio travels when only the number underneath it changes; equal distances on this axis are equal ratios. Against residents Hispanic drivers read 0.86, against adults 0.98, against people who drive to work 1.01. Black drivers move from 3.07 to 4.79. The gray band is the denominator nobody has.

Read the Hispanic line first, because it breaks the argument most people want to make. Three defensible census counts give three different stories, and the sign of the disparity flips between the first and the third. Black drivers move a long way too, in the other direction. Which conclusion you reach was decided by a choice made before any analysis began, and the data cannot settle it.

The column that would settle it is the gray band. You would need to know who was on that road, at that hour, driving in a way that could attract a stop. No agency collects that, and no survey asks it. Most public arguments about policing stall here, with each side asserting a denominator neither can produce.

So change the question. Take the stop as given, and ask what happened next to the people an officer had already pulled over: of each group’s stops, what share led to a search. That is the search rate.

Black drivers were searched in 15.52% of stops and white drivers in 3.14%, a ratio of 4.9 to one. Hispanic drivers, level or under-stopped on every population denominator available, were searched 3.14 times as often as white drivers. No count of who was driving is needed for this comparison, because the officer supplied the comparison group.

There is a serious objection: perhaps the stopped groups genuinely differed. If officers were meeting more grounds for suspicion among the drivers of one group, a higher search rate would follow from correct policing rather than from a lower bar. Search rates cannot answer that, so take the search as given too, and compare hit rates: of each group’s searches, the share that found something, contraband found divided by searches made. In 2001 the economists Knowles, Persico and Todd proposed the move:³

An officer searches when suspicion crosses some internal threshold. If the same threshold is applied to everyone, the searches it authorises should succeed at roughly equal rates across groups. A lower threshold means more searches on weaker evidence, and weaker evidence means a lower success rate. So unequal hit rates imply unequal thresholds.

Contraband was found in 9.2% of searches of Black drivers and 24.2% of searches of white drivers: 2,183 finds in 23,622 searches against 2,834 in 11,707. Searched 4.9 times as often, and found with something 2.6 times less often. That comparison is between searches, not between populations, so no denominator had to be chosen to reach it.

Two groups could be a coincidence. Figure 2 puts all 5 groups on the same plane.

Hover a point for that group’s counts.

Figure 2. The more often a group was searched, the less often the searches found anything. Share of stops leading to a search against share of searches finding contraband, one point per officer-perceived race group. A scatter is the right form because the claim is about two rates moving together, and the dashed line is the overall hit rate, 14.9%, where equal thresholds would put every group.

The exception is the most useful part of the figure, because it runs the other way and still fits. Asian and Pacific Islander drivers were searched least often of all, in 1.81% of stops, and their searches found contraband most often, 36.1% of the time. The ordering runs one way without a single exception across all 5 groups, which is what makes an innocent explanation hard to build: it would have to account for the Black and Hispanic results and this reversal at once.

Nor is the pattern the work of one strange year. Split the same counts by year and searches of white drivers found contraband more often than searches of Black drivers in 10 years out of 10. The levels move a great deal. The ordering does not move at all.

What this brief cannot tell you

It cannot tell you who was driving. The population at risk of being stopped was never counted and cannot be reconstructed, so Figure 1’s first three columns are benchmarks rather than measurements. The hit-rate comparison, what economists call the outcome test, routes around that problem rather than solving it.

It cannot separate the causes. Individual animus, departmental policy, and where officers were deployed all produce the same department-level pattern. A pattern like this one can also be produced by a small number of officers behaving very differently from their colleagues. Nor does the record say why any stop was made: since Whren v. United States (1996) any traffic violation, however minor, justifies a stop whatever else the officer had in mind, so reason_for_stop records a legal basis rather than a motive.⁴

It cannot treat “contraband found” as untouched. What counts as a find, and whether it is written down, are decisions made by the officers being measured.

And the test compares averages, not margins. The theory is about the weakest search an officer chooses to make; the file supports only the average one. If suspicion is spread differently across groups, average hit rates can differ even under one threshold. Simoiu and colleagues named that problem in 2017, and it bounds the finding rather than contradicting it.⁵

What you should have learned

• The same 905,070 stops support opposite conclusions about Hispanic drivers: 0.86 times the white rate against residents, 1.01 against people who drive to work.
• A rate needs a bottom number, and for traffic stops that number does not exist. Saying so plainly is worth more than a confident ratio built on a census count that measures residence rather than driving.
• Taking the stop as given gets rid of the problem, because the officer chose the comparison group: Black drivers were searched in 15.52% of stops against 3.14% for white drivers.
• Those searches succeeded less often, not more: contraband in 9.2% of searches of Black drivers against 24.2% for white drivers, with the ordering holding across all 5 groups and in 10 years out of 10.
• Race in this file is an officer’s perception recorded at the roadside, which is the right column for a question about the officer and the wrong one for a comparison against census categories.

Extensions

None of these is answered above, and every one can be answered from the tables the Sources section hands over.

• by_race.csv carries stops, searched, contraband_found and arrests. Compute an arrest rate per stop for each group and set it beside the search rate. Do the two measures rank the groups the same way?
• acs_denominators.csv has a count for each race at each denominator. Recompute Figure 1 using other where it exists, and say what the missing rows cost.
• by_year.csv has stops, searched and contraband_found by year and race. Compute the search rate per year for each group and find the year in which the gap between Black and white drivers was widest.
• by_year.csv also holds the size of the file itself. Sum stops by year and read the trend. Would a department stopping fewer people change any rate above?
• Stretch. The Open Policing Project publishes dozens of other departments in the same layout. Download one and run the search rate and hit rate again. Does the ordering survive the change of city, even though the levels will not?
• Stretch. The census tables named in source can be pulled for a later five-year release. Rebuild the residents denominator for 2018–2022 and ask how much of a decade-old disparity is really a changing city.

Sources

Data

• Pierson, E. et al. Stanford Open Policing Project, San Francisco, CA, 2020-04-01 release. https://openpolicing.stanford.edu/data/ — one row per stop, reduced here to the by-race and by-year counts every figure above is drawn from.
• U.S. Census Bureau, American Community Survey five-year estimates, San Francisco County — the three population denominators in Figure 1. The exact table behind each row is recorded in acs_denominators.csv beside the count. The tables are free at https://data.census.gov/; the Bureau’s page on the five-year release is https://www.census.gov/data/developers/data-sets/acs-5year.html.
• Bureau of Justice Statistics, Contacts Between Police and the Public, 2018 – Statistical Tables — the share of adults who meet the police in a year. https://bjs.ojp.gov/library/publications/contacts-between-police-and-public-2018-statistical-tables

The data itself

Every figure above is drawn from these tables. They are plain CSV, they open in a spreadsheet, and they are yours to take further.

acs_denominators.csv, by_race.csv, by_year.csv

Academic literature

• Knowles, J., Persico, N. and Todd, P. (2001). “Racial Bias in Motor Vehicle Searches: Theory and Evidence.” Journal of Political Economy 109(1): 203–229. https://www.jstor.org/stable/10.1086/318603
• Pierson, E., Simoiu, C., Overgoor, J. et al. (2020). “A large-scale analysis of racial disparities in police stops across the United States.” Nature Human Behaviour 4: 736–745. https://www.nature.com/articles/s41562-020-0858-1
• Simoiu, C., Corbett-Davies, S. and Goel, S. (2017). “The problem of infra-marginality in outcome tests for discrimination.” Annals of Applied Statistics 11(3): 1193–1216. https://projecteuclid.org/journals/annals-of-applied-statistics/volume-11/issue-3/The-problem-of-infra-marginality-in-outcome-tests-for-discrimination/10.1214/17-AOAS1058.full

Cases

• Whren v. United States, 517 U.S. 806 (1996). https://www.law.cornell.edu/supremecourt/text/517/806

I want to rebuild a public dataset from its original sources, without
any starting files. Please do the following and show your work.

THE FIRST SOURCE. Stanford Open Policing Project, stops by the San
Francisco Police Department, release dated 2020-04-01. One zip, about
25 MB, at
https://stacks.stanford.edu/file/druid:yg821jf8611/yg821jf8611_ca_san_francisco_2020_04_01.csv.zip
No account or key is needed. It unzips to a 177 MB CSV of 905,070
stops, 2007 through 2016 — each row one stop, with the driver's race
as the officer perceived it, whether a search was conducted, whether
contraband was found, and whether an arrest was made.

THE SECOND SOURCE. U.S. Census Bureau, American Community Survey
5-year estimates, through the Census Data API — population
denominators for San Francisco County (state 06, county 075). Two
vintages tile the stop years exactly, 2007-2011 and 2012-2016:
  https://api.census.gov/data/2011/acs/acs5
  https://api.census.gov/data/2016/acs/acs5
This one DOES need a key. It is free and instant from
https://api.census.gov/data/key_signup.html — sign up, then attach
the key to every request. Keep the state and county codes as text:
"06" stops being California the moment it becomes 6.

WHAT TO BUILD.
1. One row per race group: stops, searches, contraband found,
   arrests. Then the two rates the comparison turns on — the share
   of stops that led to a search, and the share of searches that
   found contraband.
2. The same counts split by year, to see the pattern over time.
3. Population counts by race from both ACS vintages, averaged: total
   residents, adults 18 and over, and people who drive to work.

CHECK YOUR WORK. The copies this book used were downloaded in August
2026. If your rebuild matches them, you should find:
- 905,070 stops carrying a race code
- 372,318 stops of white drivers and 152,196 of Black drivers
- 11,707 searches of white drivers found contraband 2,834 times;
  23,622 searches of Black drivers found it 2,183 times
The release date is part of the URL, so this exact file should not
change; if the project posts a newer release, its numbers will differ
because the data were extended or revised, not because you made an
error.