The great academic surveys of American politics are free and public. That is the claim, it is repeated on every archive’s front page, and it is true. The General Social Survey is free. The American National Election Studies are free. The Cooperative Election Study is free and openly licensed. Nobody is refusing anybody anything as a matter of policy.
So can you actually get them? Not “you” the person with a browser and an afternoon. “You” the person who wants a file the way this book gets everything else: by a program that can be run again next year and produce the same table. The distinction matters, so state it plainly once. A person downloads a file by opening a page and clicking a link. A program downloads it by sending the web server the same request with nobody at the keyboard, which is what makes the work repeatable — and an archive can tell the two apart, and may treat them differently. How differently turns out to have a measurable answer, and the answer is the subject of this brief.
This is the fourth time this book has gone to get something, and the fourth different kind of difficulty. The census had too many answers: a dozen published populations for one county, and the work is naming which you mean. Election returns had none, so you assemble fifty-one publications yourself. The voter file is public by law and gated by law. Surveys are none of those: every archive here is free, and four of the six addresses will not answer a program.
The measurement is a scan of six addresses, made on 2026-08-13 and kept frozen at that date. The addresses are the ones this book’s own chapters read: the Census Bureau’s voting table from the Current Population Survey (the monthly household survey best known for the unemployment rate), the General Social Survey’s cumulative file (every year of NORC’s survey of American society in one table), and the download page for the American National Election Studies’ cumulative file. To them add the Cooperative Election Study on Harvard Dataverse (the archive Harvard runs for research data), a second dataset on the same archive, and a study page at ICPSR, the Inter-university Consortium for Political and Social Research at the University of Michigan, which holds most of the older survey files in the social sciences. These are the archives behind everything a survey can establish that no record can, which is why their front doors are worth a chapter of their own. If the section’s data lives behind these six addresses, the six addresses are the section’s infrastructure.
Each address was requested twice, with byte-identical headers, from
two ordinary programs: curl, a small tool that downloads a
web address from the command line, and urllib, the module a
Python program uses to do the same.1 Same User-Agent — the line
in a request that names the program making it — same
Accept, same Accept-Language. That makes 12
requests in all, and what each one answered was written down.
Sending the same request twice sounds like padding. It is the whole design. Two identical requests from two different programs can only differ in the program that made them, and whether that difference matters is the thing being tested.
The scan produced three small tables, and the raw record they are built from is committed beside them exactly as it arrived.
status.csv holds one row per archive.
archive and what name the address and what
sits at it. curl and urllib record the HTTP
status code each program received: the three-digit number every web
server attaches to its answer, where 200 means success.2 wall
names the bot wall in front of the address, if any — a
screen a hosting service puts in front of a page to turn away automated
visitors — and max_bytes counts the largest answer either
program got. open is TRUE only when both programs received
the data itself.
| Archive | What the address is | Curl | Urllib | Wall | Bytes |
|---|---|---|---|---|---|
| CPS (Census Bureau) | the voting supplement table itself | 200 | 200 | 43,643 | |
| GSS (NORC) | the cumulative file itself | 200 | 200 | 47,562,980 | |
| ANES | the cumulative file’s download page | 403 | 403 | Cloudflare | 5,965 |
| CES (Harvard Dataverse) | the dataset’s metadata API | 202 | 202 | AWS WAF | 2,071 |
| Harvard Dataverse, another dataset | a second dataset’s metadata API | 202 | 202 | AWS WAF | 2,071 |
| ICPSR | a study’s catalogue page | 403 | 200 | Cloudflare | 74,193 |
refusal.csv groups the walled archives by how they say
no: the status code, the wall that sends it, and
what_a_program_sees when it arrives.
client_dependent.csv holds the one archive whose two
answers disagree, which is the row this brief turns on. Nothing was
cleaned, because nothing needed cleaning. The decisions were all in the
design, and the one that matters, asking twice and identically, has
already been stated.
Before you read on, commit to a view. Four of the six addresses are behind a bot wall. What should a refusal look like? Write down the HTTP status code you would expect a program to receive when a wall turns it away — and then ask what your own code would do if it received that.
The CPS table and the GSS cumulative file answer any program, with no gate of any kind. The GSS hands over 47,562,980 bytes, forty-seven megabytes, to anybody who asks. No registration, no agreement, no wall. One is a federal statistical agency and the other is a university survey center that decided to publish. Neither is under more obligation than the other four; they simply made a different choice about the front door.
| HTTP | Wall | Archives | What a program sees |
|---|---|---|---|
| 403 | Cloudflare | ANES; ICPSR | an error. The request failed and the code says so. |
| 202 | AWS WAF | CES (Harvard Dataverse); Harvard Dataverse, another dataset | a success. 202 is Accepted, and the body is a challenge page. |
The two walls named in the table are commercial services. Cloudflare is a company that many websites put in front of themselves to absorb attacks and screen visitors; AWS WAF is Amazon’s equivalent, and Harvard Dataverse uses it.3
2 of the walled archives answer 403. That is a refusal, and it behaves like one: the request failed, the status says so, and any script that checks its return value stops.
2 answer 202. HTTP 202 is Accepted. It is a success code.4 The body is 2,071 bytes of challenge page — the screen that asks a visitor to prove it is a browser, which a program cannot do — where the data should be.
This book has met that pattern before. The bellwether chapter hit it on a Dataverse archive and had to fetch the file by hand. The media ideology chapter hit the same wall and had to narrow what it claims. Here it is again, measured directly rather than worked around.
The recurring hazard is not the refusal. It is the refusal shaped like a success. A status code in the 2xx range is what a careless script records as “fine”. The zero rows that follow look like a finding about the world rather than a fact about the network.
Here is why every address was tried twice.
| Client | HTTP | Bytes returned |
|---|---|---|
| curl | 403 | 5,790 |
| python-urllib | 200 | 74,193 |
ICPSR told curl 403 and told Python 200
— and handed Python 74,193 bytes of the page curl was not allowed to
see. The headers were identical. The URL was identical. They ran seconds
apart, and it reproduces.
So the thing being judged is not the request. It is the program that made it: how it negotiates the connection — the technical handshake every program performs with a server before any page is sent, which differs from one program to the next — a fingerprint below the level anything in your code can set. You cannot fix this by changing the User-Agent, because the User-Agent was never what was read.
Two consequences follow, and the second is the uncomfortable one. A wall like this does not distinguish a researcher from a scraper; it is not equipped to know who you are or what you intend. And “can this be fetched?” has no single answer, because it depends on the library you happened to import. A chapter reporting “ICPSR blocks automated access” would be treating a property of the pairing as a property of the archive.
2 Harvard Dataverse datasets were probed, and 2 are behind the wall — including the one the bellwether chapter has been treating as an awkward property of its own source. It is not that dataset’s setting. It is Dataverse, and every archive hosted there behaves the same way. That is the more useful fact, because it predicts the next dataset instead of explaining the last one.
Meanwhile, none of the four walled archives is withholding anything. ANES asks people to take the file from ANES, and the file is free. The CES is openly licensed on a public archive. ICPSR’s holdings are free to member institutions. The permission and the retrieval are separate systems, and they do not know about each other. The voter file was the opposite case: North Carolina, under the strictest kind of legal regime, serves its file to any client that asks.
That separation is what this measurement says about the infrastructure of public knowledge. Whether a democracy’s data is reachable is decided less by law or licence than by a security setting at a hosting platform, tuned by nobody in particular, applied to everyone alike. Six chapters of this book read the ANES cumulative file, and every one requires a person with a browser to download it first. The reason is a setting at Cloudflare — not a fee, and not anybody’s decision about who should have the data.
It does not say what a browser gets. Every measurement here is what a script — a program running unattended — meets. A person with a browser can reach all six archives, and in four cases that is the intended route. The scan measures the distance between “public” and “programmatically available”, which is a narrower thing than access.
Two clients are two clients. curl and
Python’s urllib disagreed once. A third client might
disagree somewhere else, and the right reading of the ICPSR row is not
“Python works” but “this is unstable in a way nothing in the request
controls.”
It is one day, and bot walls are tuned continuously. The results are frozen at 2026-08-13 for that reason. Re-running the scan and finding different numbers would not mean this brief was wrong; it would mean the measurement is of something that moves, which is itself the point.
And nothing here is an argument against bot mitigation. These archives are small operations serving large files, and the walls exist because automated traffic is a real cost. What is measured is a consequence, not a verdict.
status.csv records max_bytes, the largest
answer each archive gave. Sort by it. How large is a challenge page next
to a data file, and could you tell them apart from size alone?refusal.csv pairs each wall with
what_a_program_sees. Group the archives by wall technology.
Does each wall refuse the same way everywhere it appears?client_dependent.csv holds the archive whose
curl and urllib columns disagree. Write the
sentence a careful chapter could publish about that archive’s
accessibility, and the sentence it could not.open in
status.csv and work out, step by step, what a person must
do to get its data legitimately. Count the steps.status.csv. Every difference is a
fact about how the walls moved since 2026-08-13.Data
cf-mitigated and
x-amzn-waf-action headers quoted throughout were recorded
on 2026-08-13 and are kept alongside this brief exactly as they
arrived.curl, https://curl.se/, and Python’s urllib, https://docs.python.org/3/library/urllib.html.The data itself
Every figure above is drawn from these tables. They are plain CSV, they open in a spreadsheet, and they are yours to take further.
client_dependent.csv,
refusal.csv, status.csv
Two more files here are not data about the subject at all – they are
how this chapter checked its own work: facts.csv, the individual
numbers the text above quotes, and checks.csv, the
arithmetic verified before anything was drawn.
The dataset behind this chapter cannot be rebuilt, because it is a measurement of one particular day. WHY NOT. The data record what happened on 13 August 2026 when six survey archives were each asked, by machine, for the thing this book wants from them: the Census Bureau's CPS voting table, the GSS cumulative file, the ANES download page, two datasets on Harvard Dataverse, and an ICPSR study page. Every address was requested twice, with byte-identical headers, from two different programs. Bot walls are infrastructure, and infrastructure changes without notice: send the same requests next month and some door now closed will be open, or the reverse. Repeating the scan does not rebuild this dataset; it builds a new one, about a new day. WHAT EXISTS INSTEAD. Make the same twelve requests yourself, today, and compare. Ask each archive for the data file or its catalogue entry — not the home page — twice, from two different programs sending identical headers. The question being tested is whether the wall reads the request or the client. WHAT YOU CAN STILL CHECK. These numbers identify the scan of 13 August 2026 that this book quotes: - six archives, twelve requests; two archives handed over the data and four put a bot wall in front of it - the GSS handed both programs its full 47,562,980-byte file, with no gate at all - Harvard Dataverse answered with status 202 — "Accepted" — and a 2,071-byte challenge page instead of the data - ICPSR told one program 403 and the other 200, with identical headers: that wall judges the client, not the request
Both are free and documented: https://curl.se/ and https://docs.python.org/3/library/urllib.html.↩︎
The codes and their meanings are catalogued at https://developer.mozilla.org/en-US/docs/Web/HTTP/Status.↩︎
https://www.cloudflare.com/ and https://aws.amazon.com/waf/.↩︎
Defined in RFC 9110, the current specification of HTTP, section 15.3.3: https://www.rfc-editor.org/rfc/rfc9110.html.↩︎