Cohen's d

A standardised effect size: difference in means divided by the pooled standard deviation.

Cohen's d expresses the gap between two group means in units of standard deviation:

$d = \frac{\bar x_1 - \bar x_2}{s_p}, \qquad s_p = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}$

Here $s_p$ is the pooled standard deviation, a weighted blend of both groups' spread. When the groups are the same size it simplifies to $s_p = \sqrt{(s_1^2 + s_2^2)/2}$ . A d of 0.5 means the two means sit half a typical spread apart.

Because d has no units, you can compare effects on very different metrics (order value in dollars, session time in minutes, a 1–10 satisfaction score) and feed it into power calculations. For a two-sample t-test at α = 0.05 with 80% power you need roughly $n \approx 16 / d^2$ users per group, so halving d quadruples the sample.

Cohen offered rough labels: 0.2 small, 0.5 medium, 0.8 large. He meant them as a fallback when nothing better is known, not as a verdict on value. In product analytics a d of 0.05 on revenue per user can be worth millions, while a d of 1 on a vanity metric may be worth nothing, so always translate back into natural units too.

d is closely tied to Student's t-statistic. With equal groups of size $n$ , $t = d \times \sqrt{n/2}$ : t grows with the sample, d does not. That's exactly why t (and the p-value) answers "is it real?" while d answers "how big is it?". For conversion rates, report the lift in percentage points instead (or use Cohen's h, the proportion counterpart).

Example

CatChow's upsell-widget test: 10 orders per group.

Control: mean $42.50, $s_1 \approx \$ 3.03 $(variance$ \approx 9.17$)
Widget: mean $52.50, $s_2 \approx \$ 3.37 $(variance$ \approx 11.39$)

Pooled SD (equal groups): $s_p = \sqrt{\frac{9.17 + 11.39}{2}} = \sqrt{10.28} \approx \$ 3.21$.
Cohen's d: $d = \frac{52.50 - 42.50}{3.21} = \frac{10.00}{3.21} \approx 3.12$ .
Cross-check with the t-test: $t = 3.12 \times \sqrt{10/2} = 3.12 \times 2.236 \approx 6.97$ , the same t as in the course's t-test lesson.

A d of 3.1 is enormous, far beyond the "large" mark of 0.8. It's large because order values in this small synthetic sample barely vary, so a $10 gap is more than three typical spreads.

Now imagine real CatChow orders, where the standard deviation of order value is closer to $25. The same $10 gap gives $d = 10 / 25 = 0.4$ , between small and medium. Planning a test to detect it: $n \approx 16 / 0.4^2 = 16 / 0.16 = 100$ orders per group.

Common mistakes

Treating 0.2 / 0.5 / 0.8 as a law. They're rough conventions from behavioural research, not a measure of business value.
Dividing by the standard error instead of the standard deviation. SE shrinks with sample size, so you get an inflated "d" that is really just a t-statistic in disguise.
Comparing d across very different populations. A narrow segment (say, only subscription customers) has a smaller SD, which makes the same dollar gap look like a bigger d.
Ignoring skew and outliers. A few very large orders inflate the SD and shrink d; consider a log scale, a trimmed mean or a rank-based measure for heavy-tailed revenue data.
Using d for conversion rates. For yes/no outcomes, report the lift in percentage points (or Cohen's h), not d.