Calibrate your confidence
Answer trivia, rate your certainty, and compare your confidence with your accuracy.
Ten questions. How sure are you?
- Choose an answer without looking it up.
- Pick how sure you are, from 50% to 100%.
- Check your curve, then try a fresh round.
About 3 minutes 5 fresh rounds
What does 80% sure mean?
Across ten answers you rate 80%, you should expect about eight to be correct.
After each round, your calibration curve compares how sure you felt with how often you were right. Repeat with fresh questions to see the pattern more clearly.
How calibration and scoring work
Calibration: among answers you rate 80%, about 80% should be correct over many attempts. Accuracy alone measures how much trivia you know. Calibration measures whether your certainty fits that knowledge.
Brier score: for each answer, we square the difference between your confidence (as a decimal) and the outcome (1 for correct, 0 for incorrect), then average. An 80% answer scores 0.04 if correct and 0.64 if wrong. Scores range from 0 to 1; a 50% guess always scores 0.25. This measures overall probability accuracy, not calibration alone.
Keep perspective: confidence groups can contain only one or two answers in a round. A small gap between averages is not proof of good calibration. Use the curve and sample sizes across rounds. This trivia exercise does not establish how well you forecast real-world events.
Practice and feedback: Tetlock and colleagues’ work on forecasting tournaments provides broader context for practicing explicit probabilities and reviewing outcomes.
Inspired by Julia Galef’s The Scout Mindset. All 50 questions are original to Cognitive Bias Lab.
Progress saved in this browser. No account needed.