Pitfalls: peeking, multiple testing, novelty effect

Stopping a test as soon as it first reaches p < 0.05, then declaring victory, keeps your actual false-positive rate at 5%.

You test 5 independent guardrail metrics, each at α = 0.05, and none of them truly changed. What is the probability that at least one shows a false positive, as a percentage?

Using the Bonferroni correction for 5 comparisons at a family-wise α = 0.05, what per-test significance threshold should you use?

What's the safest way to handle wanting to check results before your pre-computed sample size is reached?

Match each pitfall to its description.

Which of these are classic causes of a false 'winning' variant? Select all that apply.

CatChow's dashboard shows the banner test 'already significant' after 2 days, but the pre-registered sample size needs 2 weeks. In 2-3 sentences, explain to Sofia's teammate why they shouldn't ship yet.

Write a one-paragraph recommendation for the team.