You saw in the Sampling Machine that pure chance makes results wobble around.
So when something looks impressive — a "lucky" coin, a sales bump, a drug that "worked" —
the real question is: could plain luck have produced something this extreme?
Pick a case, then let chance answer for you.
Step 1
Pick a case
Guess: real signal, or just noise?
Step 2
Let chance run
Build the "if it were random" world
Step 3
Read the verdict
How often does luck do this?
Three suspicious cases
Before running anything — which do you think is a real effect, and which is just luck?
🪙
The "lucky" coin
A friend claims their coin is biased toward heads.
They flipped 100 times → 58 heads.
📈
The sales campaign
Marketing says a new ad lifted average daily sales.
30 days after → mean up 1.6 units.
💊
The "miracle" supplement
A vendor says their pill improves a test score.
40 people → mean 4.1 points higher.
The "if it were just luck" world
Start by assuming the boring explanation: the coin is fair (50/50) and 58 heads was just luck.
We'll flip a fair coin 100 times, over and over, and see how often luck alone reaches the observed result.
outcomes luck produced as extreme as what we observed observed result
Random trials run0
…that hit the observed result or beyond0
Share that did—
That share you just measured is the p-value.
It's the probability of getting a result this extreme or more, if the boring "just luck" explanation were true.
Small p → luck rarely does this → the effect looks real. Large p → luck does this all the time → not convincing.
Careful what it is NOT: it's not the probability the claim is true, and not the probability "it was due to chance."
It's strictly: how surprising the data is, assuming nothing special is going on.
✦ One question
You run a test and get p = 0.03. What does that actually mean?
AThere's a 3% chance the claim is true.
BThere's a 97% chance the effect is real.
CIf nothing special were going on, a result this extreme would happen only about 3% of the time.
Is it Real or Random? · built for teaching · Jan Erik Meidell