Grading Probabilistic Election Forecasts

The test

Grading a probability takes many probabilities, each carrying a number recorded before the answer and an answer that arrived later. This brief uses 746 resolved prediction markets from Polymarket (https://polymarket.com/), with closing dates running from 2024-01-15 to 2027-01-01. A prediction market is a forecast with money behind it. Traders buy contracts that pay one dollar if a thing happens and nothing if it does not, so a contract trading at 70 cents is the crowd betting that the chance is about 70%, and when the market settles everyone finds out. The record is Polymarket’s own: it publishes a list of every market that has closed and how it settled, downloadable at https://gamma-api.polymarket.com/markets?closed=true&order=volumeNum&ascending=false, and a day-by-day price history for each contract, described at https://docs.polymarket.com/.

That venue answers the question because almost nothing else in public life leaves a record of this shape. Forecasting models publish pictures rather than files, so the comparison everyone wants, market against model, cannot be run at all. The betting site publishes its trading record because its traders want it, not for anyone’s research, and it can withdraw the record without notice. Economists have argued since the early 2000s that prices like these pool scattered information well;¹ this brief asks the narrower question of whether the prices this venue actually produced were honest.

Those two records are joined here. The list of closed markets carries each contract’s identifier, a long code, and the price history is looked up by that same code. Every join takes on the assumptions of both files plus one more about how they line up, which is the subject of Join Keys and Crosswalks; that chapter also explains the crosswalk, a table that translates one set of identifiers into another. A key is a column two files share, and nothing in either file promises that the same code means the same thing on both sides.

The data

One row is one resolved market, and this is the largest of them.

That is the whole design: a claim, a number attached to it before the answer was known, and the answer. question holds the contract’s wording. p7 and p30 hold the prices seven and thirty days before it closed. outcome is 1 if the thing happened and 0 if it did not, volume is the dollars traded, and end_date is the day it closed.

The prices had to be dug out of a trading record. A market issues no forecast. It trades continuously and leaves a price for every day it was open, so p7 and p30 were picked by counting back from the closing date. Those are two moments chosen out of hundreds, after the fact, and the second is carried beside the first so the choice can be tested rather than trusted.

Three cleaning decisions do real work. Only markets that settled cleanly to yes or no can be scored, so contracts that voided, that settled to a number, or that never settled were dropped. A market younger than thirty days has no thirty-day price, and 69 of 746 rows are left blank rather than filled in. And the file is ordered by money rather than drawn at random: it holds the resolved markets with the most dollars traded, taken in that order.

The last two decide how far anything here travels. This is not a sample of prediction markets and cannot be read as one. The markets that vanished are mostly the ambiguous ones, whose wording did not survive contact with events. A score computed on what is left is a score on the questions that turned out to be answerable.

Three facts in that table shape everything below. Most of these questions are about things that did not happen: 70.6% were priced under 0.10 and only 19.4% resolved yes. Most are not political either, because the highest-volume contracts include sport, cryptocurrency and Federal Reserve decisions. And volume runs from $10m to $1,531m, which matters more than any other column.

Before you read on, commit to a number. Take every market that traded between 0.70 and 0.80 seven days before it closed. What share of them actually happened? Write the number down. Then write down how far off the prices would have to be before you would call the venue unreliable.

What it says about democracy

Start with what cannot be done. Suppose the 70% chance came in; a defender says the forecast was good. Suppose it did not; a defender says three-in-ten events occur three times in ten.

This is a property of probability rather than a weakness of forecasters, and the only escape from it is volume.² Gather many forecasts, group them by what each one claimed, and check the groups against what happened.

746 dots, one per market, area proportional to traded volume: the largest market traded $1,531m and the median $22m. The lower strip is crowded against the left wall, 71% of every question asked here was priced under 0.10, and 81% of the file resolved no. Hover any dot for the question behind it.

Figure 1. Every resolved market in the file. One dot per market, placed at the price it carried seven days before it closed, with dot area proportional to traded volume. A dot sits on the upper strip if the thing happened and on the lower strip if it did not. Two strips and one axis is the right form because the whole record is a price and a yes-or-no, and calibration asks how those two line up. 527 of the 746 markets sit in the leftmost tenth of the axis.

Now sort every market into a band by its seven-day price and ask what share of each band happened. If the prices mean anything, the two middle columns should match.

Circle area is the number of markets in the band. Points on the dashed line are perfectly calibrated. Hover for detail.

Figure 2. What each price band claimed, against what happened to it. One circle per band, placed at the band’s average price and the share of it that occurred, with circle area the number of markets in the band. The dashed diagonal is perfect calibration, and the large circle at the bottom left is the 527 markets priced under 0.1.

Read the top of the table first. Markets priced 0.9 to 1.0 happened 97.8% of the time, those priced 0.8 to 0.9 happened 84.6% of the time, and those priced 0.7 to 0.8 happened 76.9% of the time. A 70% chance, from this venue, has meant roughly a 70% chance. One number compresses the picture: the Brier score, the average squared distance between the probability and the outcome, where zero is perfect.³ Take the largest market in the file, the one in the table above. It was priced at 0.6585 seven days out and then happened, so its outcome is 1. Its distance is 1 − 0.6585 = 0.3415, and squared that is 0.1166. Had it gone the other way the score would have been 0.4336. A forecast of exactly 0.5 scores 0.25 whichever way things go; a confident forecast scores near zero when right and near one when wrong. The venue’s Brier score is that calculation averaged over all 746 markets.

The market scores 0.0644, against 0.2500 for a coin flip. Notice the last row, though. Saying “nothing will happen” to every question scores 0.1944, which is worse but not absurdly so, because most of these questions really are about things that did not happen. A summary number is flattered by the mix of questions asked, which is why the calibration table sits beside it rather than behind it.

What this brief cannot tell you

Whether markets beat models. Model outputs are not published as data anybody can score, so one side of the comparison is a picture.

Anything about prediction markets as a class. This is one venue, and the file holds its highest-volume resolved contracts rather than a random draw. Markets that voided or settled ambiguously were dropped, and there is no reason to think they were a random subset.

Anything specific to elections. Most of these questions are about sport, cryptocurrency and interest rates. The calibration establishes a property of the mechanism at this depth of trading, not a fact about the American electorate.

Whether a well-calibrated forecaster can be beaten. It can. Calibration says the labels are honest, not that they are as sharp as the available information allows. A manipulated market can also be well calibrated on average.

What you should have learned

• A single probability cannot be refuted by a single outcome, so grading a forecaster takes many claims with the answers attached — here 746 of them.
• Calibration is the test that volume makes possible. Markets priced 0.7 to 0.8 happened 76.9% of the time and those priced above 0.9 happened 97.8% of the time.
• One summary number hides the question mix. The Brier score here is 0.0644 against 0.2500 for a coin flip, but answering “no” to everything scores 0.1944, because only 19.4% resolved yes.
• Ten bands were tested and 1 cleared p < 0.05, which is what ten tests produce by accident.
• The apparent longshot bias is depth rather than price. In the mid-range, markets above the median volume miss by 0.3 points and those below it by 8.7.

Extensions

None of these is answered above, and every one can be answered from the table the Sources section hands over.

• polymarket_resolved.csv has p7 and outcome. Group the rows into price bands of your own choosing, five or twenty, and see whether the low-band anomaly survives a different cut.
• Sort the same file by volume and split it into thirds rather than halves. Does calibration improve smoothly with money, or only at the top?
• Compare p30 against p7 on the rows carrying both. Do the prices get more accurate as the closing date approaches, or only more confident?
• Sort by the distance between p7 and outcome, then read the question column for the worst twenty. Do the misses have anything in common?
• Stretch. Polymarket’s two records are still being published today. Draw a fresh sample of resolved markets and run the calibration table on it.

Sources

Data

• Polymarket’s record of every resolved market and how it settled, published at https://gamma-api.polymarket.com/markets?closed=true&order=volumeNum&ascending=false, and the price history it publishes for each contract, documented at https://docs.polymarket.com/. 746 markets survived the filters. https://polymarket.com/

The data itself

Every figure above is drawn from this table. It is plain CSV and it opens in a spreadsheet.

polymarket_resolved.csv

Also cited

• Brier, G. W. (1950). “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review 78(1): 1–3. https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2 The scoring rule used above.
• Wolfers, J. and Zitzewitz, E. (2004). “Prediction Markets.” Journal of Economic Perspectives 18(2): 107–126. https://www.aeaweb.org/articles?id=10.1257/0895330041371321 The case for prices as an aggregator, and its limits.
• Berg, J. E., Nelson, F. D. and Rietz, T. A. (2008). “Prediction market accuracy in the long run.” International Journal of Forecasting 24(2): 285–300. https://doi.org/10.1016/j.ijforecast.2008.03.007
• Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press. https://press.princeton.edu/books/paperback/9780691175973/expert-political-judgment Tens of thousands of scored expert forecasts, none of them refutable one at a time.
• Griffith, R. M. (1949). “Odds Adjustments by American Horse-Race Bettors.” American Journal of Psychology 62(2): 290–294. https://www.jstor.org/stable/1418469 The first measurement of the longshot bias.
• Thaler, R. H. and Ziemba, W. T. (1988). “Anomalies: Parimutuel Betting Markets: Racetracks and Lotteries.” Journal of Economic Perspectives 2(2): 161–174. https://www.aeaweb.org/articles?id=10.1257/jep.2.2.161

The dataset behind this chapter cannot be rebuilt, even though its
source is open, keyless, and answering right now.

WHY NOT. The data are a snapshot of Polymarket, a real-money
prediction market: the highest-volume markets that had closed and
resolved to a definite yes or no, each with the price it traded at
7 and 30 days before it closed. The snapshot was drawn in August
2026 from Polymarket's public interfaces, starting at
https://gamma-api.polymarket.com/markets?closed=true&order=volumeNum
and following each market into the matching price-history service.
Ask the same question today and you get a different answer,
honestly: markets resolve every day, so the top of the volume
ranking is no longer the same list. A snapshot of a moving thing
cannot be drawn twice.

WHAT EXISTS INSTEAD. The same interfaces, today. They need no
account and no key — only a browser-style User-Agent header, since
bare requests are refused. A sample drawn now is just as valid as
this one; it is simply a different sample, and every calibration
question asked here can be asked of it.

WHAT YOU CAN STILL CHECK. These numbers identify the August 2026
snapshot this book uses:
- 746 resolved markets, each priced 7 and 30 days out
- the largest, "Will Donald Trump win the 2024 US Presidential
  Election?", with 1,531,479,285 dollars traded
- a Brier score of 0.064 for the 7-day prices — against 0.25 for
  always answering fifty-fifty