How we score an election forecast — and what the score does not measure

1. Pre-registration

Every answer was collected before the vote (26 August – 30 October 2026) and is published verbatim. The date attached to each model is the date its forecast entered the benchmark.

2. The judge

Each answer is rated by a fixed judge — DeepSeek V4 Pro at temperature 0 — on three axes, each scored 0–100: specificity (a named candidate or party and a stated probability), grounding (polls, incumbency, district and electoral structure), and calibration (a probability that matches the real uncertainty). The category score is the mean of its questions; the overall score is the mean of the two categories.

3. What the score does not measure

It does not measure whether the forecast is right. A model can be wrong and score well; a vague answer always scores badly. It also does not measure the quality of the underlying polls — only how the model uses them.

4. After the vote

Once results are certified we add a second layer: winner accuracy, calibration curves (did 70% calls happen about 70% of the time?) and the Brier score for the probabilistic calls. Both layers will be published side by side, so quality and accuracy are never conflated.

5. Limits

Prompts are in French, which disadvantages models with weaker French. Some models answer in a single sentence; others write four paragraphs. The judge controls for substance, not length, but style still matters at the margin. Finally, several newer models were judged by a model from the same family; we disclose it rather than hide it.

6. Cite this

LFORLA (2026). Election Predictions — US 2026 & France 2027 (v1.0) [Data set]. https://elections.lforla.org/data.html