Two pollsters holding the same interviews can publish different
winners. That sounds like an accusation, and it is not one; it is a
documented event, and the document is public. On 20 September 2016
The New York Times published a poll of Florida and then did
something almost no pollster does: it gave the raw file away.
The Times and Siena College had interviewed 867 Florida voters and
published a result, Clinton by one point. Alongside it, the Upshot, the
Times’s data-journalism desk, handed the respondent-level data — one row
per person interviewed, with every answer — to four outside pollsters
and asked each to produce the estimate themselves. The four came back
with four different answers, spread across five points, and they did not
agree on who was ahead.
Nobody cheated and nobody made an arithmetic error. The 867 people
were the same 867 people in every case. What differed was the
weighting: counting some answers more than others, so
that a sample which does not look like the electorate can be made to.
How does the same file honestly yield different winners — and which
decisions inside the weighting actually move the answer? The file that
lets anyone check is still public, so this brief turns the dials and
watches.
The test
The data is the 2016
Times/Siena respondent-level poll file, released by the Times on
GitHub: 4,879 respondents across six separate state polls, of which the
September Florida poll is the 867 this brief reads. Florida’s certified
November result, the answer sheet, comes from this book’s historical
campaigns chapter.
Why this source, of all polls? Because it is very nearly the only one
of its kind. A respondent-level poll file is normally the last thing a
pollster releases, since it is the only thing that lets somebody else
disagree with them in public. The Upshot released one on purpose, so
that four rivals could contradict it in the same week. It carries the
interviews and the two weights the Times applied, so the
published answer can be checked rather than taken on trust — and every
rival scheme can be rebuilt on the same people.
The mechanics of weighting are this section’s subject, and its weighting chapter walks
through them on the largest survey that publishes its own. The short
version to carry in: weighting is only the statistical step that makes a
sample resemble a chosen picture of the electorate. The picture itself,
how many college graduates and how many seniors, is benchmarked from
outside. For a national poll that usually means the Census Bureau’s own
surveys of who lives in the country; for a poll like this one, drawn from a
state’s voter file — the state’s list of everyone registered to vote,
with each person’s party registration and voting history — the file
itself supplies much of the picture. This brief is about choosing the
picture.
The data
Everything below is built from upshot-siena-polls.csv,
the poll file the New York Times and Siena College published on GitHub,
downloadable by anyone: no key, no account, no guestbook. This is a
newspaper’s file on a code-hosting site, not an archive; no statute
obliges anyone to keep it there, and the version you download tomorrow
is whatever the repository holds tomorrow. The small tables in this
brief’s folder are kept for exactly that reason.
This brief keeps the 867 September Florida respondents, in three
tables. fl_respondents.csv holds one row per person:
vote; the Times’ two weights, rvweight and
lvweight — a registered-voter weight,
which makes the sample resemble everyone registered to vote in Florida,
and a likely-voter weight, which makes it resemble the
smaller group the Times expected to actually turn out; and seven columns
describing who the person is — educ4, race4,
race_said, party3, age4,
turnout3, region5. Two of those deserve a
flag. race_said is the race the respondent gave the
interviewer; race4 is the race Florida’s voter file records
for the same person, and the two will matter separately below.
party3 and turnout3 also come from the voter
file: registration and vote history are records, not answers.
schemes.csv holds one row per weighting scheme — a
margin and a leader for each way of treating
the same people. And targets.csv holds what each scheme
assumes the electorate looks like, variable by variable. Its Times
columns are not copied from anywhere: they are recovered from the
published weights, by a trick worth knowing because it applies to any
survey that ships them. A weighted tabulation is a target.
Weight the sample by the Times’ likely-voter weight and tabulate
education, and what comes back is the education profile the Times
assumed Florida’s likely voters had, because the weights were built to
hit it.
Of the 867 respondents, 720 named one of the two major candidates;
the other 147 named somebody else, refused, or said they would not vote.
Every margin below is computed over the 720.
Before you read on, commit to a view. The published
poll said Clinton +1. What was Clinton’s margin
among the respondents before anybody weighted anything? Not the
published number — the raw count of who said what. Write it down; the
gap between your guess and the answer is the brief.
What it says about democracy
The ladder
Three numbers, from the same 720 people.
| As collected |
that the people who answered are the electorate |
+9.2 |
Clinton |
| Registered voters (Times) |
that the electorate looks like Florida’s registered
voters |
+4.0 |
Clinton |
| Likely voters (Times, published) |
that it looks like the Times’ model of who turns
out |
+0.6 |
Clinton |
As collected, Clinton led by 9.2 points. Published,
she led by 0.6. The same people, 8.5 points apart. If you guessed two or
three points above the published figure, you were short by a factor of
three: the gap is more than eight. Weighting is not a rounding step
applied to a result that already exists — it is most of how the result
came to exist at all.
Figure 1. The same 720 respondents under three
weights, drawn as bars from zero because the question is
distance from a tie. The red rule is Florida’s actual two-party margin,
-1.2.
Here is what a weight is, on one person. Of the 720 respondents who
named a candidate, 393 said Clinton and 327 said Trump. 393 minus 327,
divided by 720, is Clinton +9.2: that is the raw count, and it treats
every answer as one answer. A weight is the number of answers the Times
decided a person’s one answer should stand for. The 867 people who
picked up the phone were more Democratic-registered, younger and less
reliable past voters than the electorate the Times expected in November,
so a respondent from a group the sample holds too many of counts for
less than one, and a respondent from a group it holds too few of counts
for more. The most heavily weighted respondent over 65 is a White
registered Republican with a high turnout history from the Central
region: the Times’ likely-voter weight for that person is 1.63, so the
answer counts 1.63 times. The most lightly weighted registered Democrat
sits at the other end of those dials — aged 18-34, Hispanic in the voter
file, with a low turnout history — and carries a weight of 0.06, so that
answer counts for roughly one part in 16 of one.
Now redo the subtraction with the weights. Under the likely-voter
weight, Clinton’s 393 answers add up to 305.2 and Trump’s 327 add up to
301.3: 305.2 minus 301.3, divided by 606.6, is Clinton +0.6. Nobody’s
answer changed. What changed is how many answers each one was worth.
What the four pollsters were asked to do
The Upshot’s exercise was not a test of arithmetic. Each pollster
received the same respondents and made their own decisions about two
separate things.
First, which people count as the electorate. Every
pollster applies some rule — a likely-voter screen —
about who will actually turn out: a stated intention, a record of having
voted before, a modeled score. The rules disagree. Second, what
the electorate is assumed to look like: how many college
graduates, how many registered Democrats, how many over 65. Then each
scaled the respondents until the sample matched their picture.
Their published answers, as the article reported them:
| Omero, Green and Rosenblatt |
Clinton +4 |
| Charles Franklin |
Clinton +3 |
| The Times and Siena, as published |
Clinton +1 |
| Patrick Ruffini |
Clinton +1 |
| Gelman, Rothschild and Corbett-Davies |
Trump +1 |
These five figures are quoted from the
article, not computed here. The pollsters described their methods in
prose rather than publishing code, and two of them used targets that are
not in the released file. Nothing in this brief claims to have
reproduced them.
That table is the reason this brief exists, and the one thing in it
this brief cannot rebuild. What can be rebuilt is the mechanism the five
answers came out of, on the file they all worked from.
The electorate the Times assumed, recovered from its own
weights
Read the Times’ recovered targets against the standard story about
2016 and one of them runs the wrong way.
| Education |
Bachelor’s |
24.0 |
24.1 |
24.6 |
| Education |
HS or less |
19.7 |
19.3 |
17.3 |
| Education |
Postgraduate |
19.6 |
19.2 |
20.6 |
| Education |
Refused |
1.4 |
1.3 |
1.3 |
| Education |
Some college |
35.3 |
36.1 |
36.2 |
| Party registration |
Democratic |
41.0 |
38.1 |
38.3 |
| Party registration |
Other |
27.1 |
25.8 |
22.2 |
| Party registration |
Republican |
31.9 |
36.1 |
39.5 |
The standard account of 2016 is that state polls held too many
college graduates, so weighting should have pushed that share
down. It pushed it up. 43.6% of the respondents held at
least a bachelor’s degree, and the electorate the Times weighted to was
45.2% — 1.6 points more educated, not less. The reason is not
an error. Likely voters really are better educated than registered
voters, and the target was a likely-voter electorate, so the correction
for one thing moved the other the wrong way.
Turn the dial yourself
Here is the whole exercise, on the 867 respondents. Two sets of
controls, because a pollster makes two separate decisions and they do
not have the same consequences. Which variables get
corrected at all, on the left. What the electorate is assumed to
look like on each of them, on the right. The margin recomputes
as you change either.
The numbers on the right are shares of that assumed electorate, not
shares of the 867 people who answered. The levels of a variable
therefore always add to 100%, and raising one lowers the others. Each
slider is marked where the Times’ own target sat. The readout under the
bar also reports an effective sample size: how many
unweighted answers the weighted ones are worth. A respondent counted at
a tenth is contributing almost nothing, so a sample whose weights vary
widely is telling you less than its headcount suggests.
Figure 2. In the browser, two sets of controls: which
variables are corrected, and what the electorate is assumed to look like
on each. In print, eight fixed schemes instead, every one of
them reachable with those controls.
Which dial actually matters
Turn only the education slider and very little happens. Turn on party
registration and the winner changes.
| As collected |
+9.2 |
Clinton |
| Education only |
+9.3 |
Clinton |
| Party registration only |
-0.2 |
Trump |
| Education and race |
+4.6 |
Clinton |
| Everything at once |
-0.4 |
Trump |
Correcting the education skew moves the margin by less than a
fifth of a point. Correcting the party mix moves it by more
than nine, and across zero. The two corrections are the same operation
on the same people. Only the choice of column is different.
Here is that operation for one dial at a time, because with one
variable it is arithmetic anyone can check. 19.7% of the respondents
have a high-school education or less, and the electorate the Times
assumed has 17.3%, so every such respondent is given the weight 17.3 ÷
19.7 = 0.88. Postgraduates are 19.6% of the sample and 20.6% of the
target, so each of them counts 1.05. No education weight strays more
than 0.12 from one, because the sample’s education mix was already close
to the target, and the margin barely moves: that is the Education
only row. Party registration is a different dial. Registered
Democrats are 41.0% of the sample against 38.3% of the target, so each
counts 0.94; registered Republicans are 31.9% against 39.5%, so each
counts 1.24. And 86.3% of the registered Democrats named Clinton while
9.5% of the registered Republicans did, so shrinking the first group and
enlarging the second moves the answer a long way: to the Party
registration only row, and across zero.
That is the answer to the question the controls are really asking.
The target you set matters, and it matters less than which variable you
decided was worth setting a target for. A pollster who never thought
about party registration never sees the number that flips the state, and
nothing in the output announces the absence.
Two files disagree about who these people are
One more thing sits inside the file, and it is the reason race is a
harder dial than it looks. Every respondent has a race twice: the race
they gave the interviewer, and the race the voter file records.
They disagree about 203 of the people who answered the
question, which is 24.7% of them.
Neither column is the truth. A pollster weighting to census
categories uses the first, a pollster weighting to registration data
uses the second, and the two are choosing between different people
before either one sets a target. The BISG
chapter grades a method that guesses race from a name and a
neighbourhood against a state’s own record; this is the same problem
arriving one step earlier, in a poll.
Put the pieces together and the finding is this. A published
poll is an estimate plus a set of judgments, and honest judgments
disagree. Both the Times’ answer and its opposite are reachable
from the same interviews by decisions that are defensible one at a time.
Reading polls well does not mean finding the pollster who cheated —
nobody here cheated. It means knowing which dial moved, and noticing
that the printed margin of error, how far off sampling luck alone could
put the result, says nothing about any of the dials.
What this brief cannot tell you
It cannot reproduce the four pollsters’ answers.
Their methods were described in prose, not published as code. Two of the
four used targets that are not in the released file, including a modeled
estimate of the 2016 electorate. The five figures in the table above are
quoted from the article.
It cannot tell you which weighting was right.
Florida went -1.2, so the published poll missed by 1.9 points on the
two-party margin and named the wrong winner. That does not make any of
the rival schemes correct. One election is one draw, a scheme can be
right for the wrong reason, and a poll taken in September is not a
forecast of November.
It cannot separate weighting error from everything
else. A poll can miss because the weights were wrong. It can
miss because the people who answered differ from the people who did not,
in ways no column captures. It can miss because voters changed their
minds in the seven weeks that followed. Nothing here distinguishes
those.
The dials are coarser than a pollster’s. Six
variables, collapsed to a few levels each, on one state in one month of
one year. A real weighting scheme runs on more variables and finer
categories, and the sliders cannot express a likely-voter screen, which
is a rule about who counts rather than about what they look
like.
And 867 people is a small number. Before any of
this, a sample that size carries a margin of error of roughly three
points on a two-party split. Several of the gaps in Figure 2 are smaller
than that.
Sources
Data
- Poll data. New York Times Upshot / Siena College,
2016 voter-file polls.
upshot-siena-polls.csv, 4,879
respondents, 34 columns. https://github.com/TheUpshot/2016-upshot-siena-polls
- The article. Nate Cohn, “We Gave Four Good
Pollsters the Same Raw Data. They Had Four Different Results,” The
Upshot, 20 September 2016. The five quoted results come from it.
The Times’s own address is
www.nytimes.com/interactive/2016/09/20/upshot/the-error-the-polling-world-rarely-talks-about.html;
an archived copy is at http://web.archive.org/web/20161002005137/http://www.nytimes.com/interactive/2016/09/20/upshot/the-error-the-polling-world-rarely-talks-about.html.
- Florida’s actual 2016 result, read from this book’s
historical
campaigns chapter rather than retyped, so the two cannot
disagree.
- The tables here. The 867 September Florida
respondents, cut from that file, with the Times’ targets recovered from
its own published weights. Every number above is computed from them at
the moment this document was rendered.
Also cited
The data itself
Every figure above is drawn from these tables. They are plain CSV,
they open in a spreadsheet, and they are yours to take further.
fl_respondents.csv,
schemes.csv, targets.csv
Two more files here are not data about the subject at all – they are
how this chapter checked its own work: facts.csv, the individual
numbers the text above quotes, and checks.csv, the
arithmetic verified before anything was drawn.
Checks
| the wave is the 867 the article is about |
yes |
| both of the Times’ own weights are present on every
row |
yes |
| educ4 has no missing values to weight around |
yes |
| race4 has no missing values to weight around |
yes |
| party3 has no missing values to weight around |
yes |
| age4 has no missing values to weight around |
yes |
| turnout3 has no missing values to weight around |
yes |
| region5 has no missing values to weight around |
yes |
| the published Clinton +1 reproduces from the released
weights |
yes |
| weighting moves the margin by more than five
points |
yes |
| the education profile is not left where it was
found |
yes |
| the rebuilt weight lands within two points of the
published margin |
yes |
| the sibling chapter’s state returns are where this
expects them |
yes |
| exactly one Florida 2016 row |
yes |
| the poll and the result disagree about who won
Florida |
yes |
| the committed respondent file is the whole wave, not a
sample |
yes |
| it carries nothing that could identify anybody |
yes |
Rebuild this data yourself, with an AI assistant
I want to rebuild a public dataset from its original source, without any
starting files. Please do the following and show your work.
THE SOURCE. The New York Times' Upshot, respondent-level data from the
2016 New York Times/Siena College polls, released on GitHub in the
repository TheUpshot/2016-upshot-siena-polls. One CSV, about 2.3 MB,
downloaded raw from
https://raw.githubusercontent.com/TheUpshot/2016-upshot-siena-polls/master/upshot-siena-polls.csv
No account or key is needed. Each row is one person interviewed —
4,879 in all, across six state polls. The columns hold each person's
answers plus two weights the Times computed: rvweight, for registered
voters, and lvweight, for likely voters.
WHAT TO BUILD.
1. Keep only the September Florida poll. Among its respondents who
named one of the two major candidates, compute Clinton's two-party
margin over Trump three ways: with no weights at all, under
rvweight, and under lvweight. The last is the figure the Times
published as "Clinton +1".
2. A count of the Florida respondents who named one of the two major
candidates, and of those who named somebody else, refused, or said
they would not vote.
CHECK YOUR WORK. The copy this book used was downloaded in August 2026.
If your rebuild matches it, you should find:
- 4,879 rows in the file, 867 of them in the September Florida poll
- 720 of the 867 named Clinton or Trump; 147 did not
- the margin is Clinton +9.17 with no weights, +4.03 under rvweight,
and +0.64 under lvweight — which rounds to the published +1
This is a finished release about one election and should never grow;
if your numbers differ, you are holding a changed copy of the file,
not making an arithmetic error.
