Statistics glossary

Plain-language definitions of the statistics terms product managers and marketers meet in dashboards, A/B tests and research.

A

A/B test: A randomised controlled experiment that splits users between a control (A) and one or more variants (B) and compares a metric.
alternative hypothesis (H1): The claim accepted when H0 is rejected: there is a difference or effect.
analysis of variance (ANOVA): Tests whether the means of three or more groups differ by comparing between-group and within-group variance.
average order value (AOV): Revenue divided by the number of orders; usually right-skewed, so outliers matter.

B

bar chart: A chart comparing counts or values across categories with separate bars.
baseline rate: The current value of the metric in control; an input to sample-size calculations.
Bayesian statistics: An approach that combines prior beliefs with new data (Bayes' theorem) to get updated, posterior probabilities.
between-group variance: The part of total variability explained by differences between group means.
Bonferroni correction: Divide alpha by the number of comparisons, e.g. 0.05 / 3 = 0.017 for three tests.
box plot (box-and-whisker plot): A chart showing the median, the IQR as a box, whiskers for the bulk of the data and dots for outliers.
bubble chart: A scatter plot where point size encodes a third variable.

C

categorical variable: A variable whose values are labels or groups, e.g. traffic channel, device type or plan tier.
causation (correlation is not causation): A cause-effect link; a correlation alone can also come from reverse causation, a confounder or chance, so causality needs an experiment.
central limit theorem: For large enough samples the sample mean is approximately normally distributed, even if the data are not.
centroid: The centre of a cluster: the mean of its members on every variable.
Chebyshev distance: The largest absolute difference on any single variable.
classification: Assigning observations to known classes, e.g. predicting whether a user will churn.
classification threshold: The predicted probability above which an observation is labelled positive, often 0.5 but tuned to business costs.
click-through rate (CTR): Clicks divided by impressions for an ad, email or link.
cluster analysis (clustering): Methods that find groups of similar observations without predefined labels, e.g. customer segments.
coefficient of determination (R²): The share of the outcome's variance explained by the model, from 0 to 1.
coefficient of variation: Standard deviation divided by the mean; lets you compare spread of metrics on different scales.
Cohen's d: A standardised effect size: difference in means divided by the pooled standard deviation.
cohort: A group of users who share a starting event in the same period, e.g. everyone who signed up in March.
confidence interval (CI): A range of plausible values for a parameter; a 95% CI is built by a method that captures the true value in 95% of repeated samples.
confidence level: The long-run share of intervals that contain the true value, typically 95%; equals 1 - alpha.
confounding variable (confounder): A third variable that affects both the supposed cause and the outcome, creating a spurious correlation.
confusion matrix: A 2x2 table of true/false positives and negatives for a classifier.
contingency table: A table of counts for combinations of two categorical variables, e.g. variant x converted / not converted.
control group: The group that does not get the change and serves as the baseline.
controlled experiment: A study where the researcher assigns the treatment, so a difference between groups can be attributed to it.
conversion rate: The share of users who complete a target action: conversions / users. A proportion between 0 and 1.
correlation: A statistical association between two variables: when one goes up the other tends to go up (positive) or down (negative).
correlation matrix: A symmetric table of correlation coefficients for every pair of variables, with 1 on the diagonal.
covariance: The average product of the deviations of two variables from their means; its sign shows the direction of the relationship.
critical value: The value of a test statistic beyond which H0 is rejected at the chosen alpha.

D

degrees of freedom (df): The number of values free to vary once the estimated quantities are fixed, e.g. n - 1 for one sample, n1 + n2 - 2 for a two-sample t-test.
dendrogram: A tree diagram of hierarchical clustering; cutting it at a chosen distance gives the clusters.
dependent variable (outcome): The variable a model explains or predicts, e.g. revenue per user.
descriptive statistics: Numbers and charts that summarise a dataset (typical value, spread, shape) without generalising beyond it.
deviation: The difference between a value and the mean; deviations always sum to zero.
dimensionality reduction: Replacing many correlated variables with a few summary variables (factors or components).
discriminant analysis: Finds boundaries (discriminant functions) that best separate known groups using predictors; groups minus one functions.
distance metric: A rule for measuring dissimilarity between observations: Euclidean, Manhattan or Chebyshev.
distribution: How often each value (or range of values) of a variable occurs.

E

effect size: How big a difference or relationship is, independent of sample size, e.g. absolute lift, relative lift or Cohen's d.
eigenvalue: How much of the total variance a factor explains (the sum of its squared loadings).
error bars: Lines around a point estimate showing its SD, standard error or confidence interval; always state which one.
Euclidean distance: The straight-line distance: square root of the sum of squared differences across variables.
expected frequency: The count a cell would have if the variables were unrelated: row total x column total / grand total.

F

F-statistic (F-ratio): In ANOVA, between-group mean square divided by within-group mean square; large values mean groups differ.
F-test for equality of variances: The ratio of the larger to the smaller sample variance; tests whether two groups differ in spread.
factor (in ANOVA): A categorical explanatory variable whose levels define the groups, e.g. landing page version or traffic channel.
factor analysis: Explains correlations among many observed variables by a few hidden factors; used for surveys and to reduce dimensions.
factor loading: The correlation between an observed variable and a factor.
factor rotation (orthogonal, oblique): Rotating factor axes so each variable loads strongly on one factor; orthogonal keeps factors uncorrelated, oblique allows correlation.
false discovery rate (FDR): The expected share of false positives among significant results; controlled by the Benjamini-Hochberg procedure.
forecast: A model-based estimate of a future value, ideally with an uncertainty range.
frequency (absolute and relative): The count of observations with a given value (absolute) or its share of the total (relative, in %).
frequency polygon: A line chart that joins the frequencies of consecutive values; useful when there are many values.
frequency table: A table listing each value or bin and how many observations fall into it.
Friedman test: The non-parametric alternative to repeated-measures ANOVA: ranks each subject's measurements and compares rank sums.
functional vs structural model: A functional model links inputs to an outcome (e.g. a regression); a structural model describes the components of the outcome.

G

guardrail metric: A metric that must not get worse (e.g. refunds, page speed) even if the primary metric improves.

H

hierarchical clustering: Repeatedly merges the most similar observations or clusters until one cluster remains; the number of groups need not be known.
histogram: A chart of a quantitative variable's distribution: adjacent bars show how many values fall into each bin.
hypothesis testing: A procedure that decides whether data give enough evidence against a null hypothesis.

I

independent samples: Groups made of different, unrelated observations, e.g. users in variant A vs users in variant B.
independent variable (predictor): A variable used to explain or predict the outcome, e.g. ad spend or number of sessions.
information criterion (AIC, BIC): A model-comparison score that rewards fit and penalises the number of parameters; lower is better.
interaction effect: When the effect of one factor depends on the level of another, e.g. a discount works on mobile but not on desktop; shows as non-parallel lines.
intercept: b0: the predicted outcome when all predictors equal zero.
interquartile range (IQR): Q3 minus Q1: the spread of the middle 50% of the data; robust to outliers.

K

k-means clustering: Splits data into k clusters by alternately assigning points to the nearest centroid and moving centroids to cluster means.
Kaiser criterion: Keep only factors with an eigenvalue of at least 1.
Kruskal-Wallis H test: The non-parametric alternative to one-way ANOVA, based on rank sums of three or more groups.

L

latent factor: An unobserved trait inferred from correlated indicators, e.g. 'price sensitivity' behind several survey answers.
least squares method: Fits a regression line by choosing coefficients that minimise the sum of squared residuals.
lift (absolute and relative): The change of a metric in the variant vs control: absolute lift in percentage points, relative lift in % of control.
Likert scale: A survey rating scale such as 1 = strongly disagree to 5 = strongly agree; an ordinal variable.
linear regression: A model that predicts an outcome as a straight-line function of a predictor: y = b0 + b1 x.
log-odds (logit): The natural logarithm of the odds; positive when p > 0.5, negative when p < 0.5.
logistic regression: A model for a yes/no outcome that predicts the log-odds as a linear function of predictors and returns a probability.

M

main effect: The average effect of one factor across all levels of the other factors.
Manhattan distance: The sum of absolute differences across variables, like walking along a street grid; less sensitive to outliers.
Mann-Whitney U test: A non-parametric test comparing two independent groups by pooling, ranking and comparing rank sums.
margin of error: Half the width of a confidence interval, e.g. about 1.96 x SE for 95%.
mathematical model: A simplified formal description of a system that lets you explore scenarios without real experiments.
mean (arithmetic mean): The sum of all values divided by their count; sensitive to outliers.
measure of central tendency: A single number describing the typical value of a variable: the mean, median or mode.
measure of variability (spread): A number describing how much values differ from each other: range, IQR, variance or standard deviation.
median: The middle value of the sorted data (the average of the two middle values when n is even); robust to outliers.
metric: A quantitative measure tracked for a product or campaign, such as conversion rate, AOV, retention or CTR.
minimum detectable effect (MDE): The smallest effect a test is designed to detect with the planned power; smaller MDE needs a larger sample.
mode: The most frequent value in the data; the only measure of central tendency that works for categories.
multicollinearity: Strong correlation between predictors that makes regression coefficients unstable and hard to interpret.
multiple comparisons problem: Running many tests (metrics, segments, variants) raises the chance that at least one is a false positive.
multiple regression: Regression with several predictors; each coefficient is the effect of its predictor holding the others constant.

N

Net Promoter Score (NPS): Share of promoters (9-10) minus share of detractors (0-6) on a 0-10 'would you recommend' question; ranges from -100 to 100.
non-parametric test: A test that makes no normality assumption, usually by working with ranks; robust to outliers and skew.
normal distribution: The symmetric bell-shaped distribution; about 68% of values lie within 1 SD of the mean and about 95% within 2 SD.
novelty effect: A temporary change in behaviour because something is new; it fades, so short tests can overstate the effect.
null hypothesis (H0): The default claim of no difference or no effect, e.g. 'the new checkout does not change conversion'.

O

observation: One measured unit in a dataset (one user, order or session); usually one row of a table.
odds: Probability of an event divided by the probability of it not happening: p / (1 - p).
odds ratio: How many times the odds change per unit of a predictor; exp of a logistic regression coefficient.
one-tailed vs two-tailed test: A two-tailed test detects a difference in either direction; a one-tailed test only in a direction fixed in advance.
ordinal variable: A categorical variable whose values have a natural order but uneven gaps, e.g. a 1-5 satisfaction rating.
outlier: An observation far from the rest of the data (a common rule: beyond 1.5 IQR from the quartiles); can distort the mean and SD.
overfitting: A model that fits noise in its sample and predicts poorly on new data; checked with holdout data, limited with AIC/BIC.

P

p-value: The probability of getting a result at least as extreme as the observed one if H0 were true. It is not the probability that H0 is true.
paired (related) samples: Measurements that come in matched pairs, typically the same users before and after a change.
paired t-test: Tests whether the mean of before-after differences is zero: t = mean difference / (SD of differences / sqrt(n)).
parameter: A true (usually unknown) property of the population, such as the population mean.
parametric test: A test that assumes a distribution (usually normal) and works with means and variances, e.g. the t-test or ANOVA.
Pearson correlation coefficient (r): Measures the strength of a linear relationship from -1 to +1; sensitive to outliers.
Pearson's chi-square test: Tests association between categorical variables by summing (observed - expected)^2 / expected over all cells.
peeking (optional stopping): Checking results repeatedly and stopping as soon as p < 0.05; inflates the false-positive rate far above alpha.
percentage point (pp): The unit of an absolute difference between two percentages: 4% to 5% is +1 pp but +25% relative.
percentile: The value below which a given percentage of observations falls, e.g. the 90th percentile of page load time.
pie chart: A circle split into sectors proportional to shares of the whole; readable only with a few categories.
placebo: A dummy treatment that looks real; separates the effect of the change from the effect of expecting a change.
polynomial (non-linear) regression: Regression that includes squared or higher-power terms to fit curved relationships, e.g. diminishing returns on ad spend.
population: The whole group we want conclusions about, e.g. all current and future visitors of the shop.
post hoc test: A pairwise comparison run after a significant ANOVA with control of multiple comparisons (Tukey, Bonferroni, Games-Howell).
practical significance: Whether an effect is large enough to matter for the business, judged by effect size and cost, not by p-value.
precision and recall: Precision: share of predicted positives that are correct. Recall: share of actual positives that were found.
primary metric (OEC): The single metric, chosen before launch, that decides whether a test wins.
principal component analysis (PCA): Rotates the data to new uncorrelated axes ordered by explained variance; a common way to reduce dimensions.
prior and posterior probability: The prior is what we believe before seeing the data; the posterior is the belief after updating on the data.
proportion: The share of observations with a yes/no property; its standard error is sqrt(p(1 - p) / n).

Q

quantitative (continuous) variable: A variable measured on a numeric scale where arithmetic makes sense, e.g. revenue or time on page.
quartile: One of three cut points (Q1, Q2 = median, Q3) that split sorted data into four equal parts.

R

randomisation: Assigning units to groups by chance, so groups are comparable and confounders are balanced on average.
randomisation unit: The entity assigned to a variant (user, session, device, shop); analysis should be done at the same level.
range: The difference between the largest and the smallest value; very sensitive to outliers.
rank: The position of a value in the sorted data; rank-based tests compare positions instead of raw values.
repeated-measures ANOVA: ANOVA for three or more measurements on the same subjects; removes between-subject variability from the error.
representative sample: A sample whose composition mirrors the population, so results generalise; random sampling helps achieve it.
residual: Observed value minus the value predicted by the model.
retention rate: The share of users who are still active after a given period, e.g. day-30 retention.

S

sample: The subset of the population that we actually observe or measure.
sample ratio mismatch (SRM): When the observed split between groups differs significantly from the planned one (e.g. 50/50); signals a broken experiment.
sample size: The number of observations (per group in a test); planned in advance from alpha, power, baseline and MDE.
sampling bias: A systematic difference between sample and population caused by how observations were selected.
sampling distribution: The distribution of a statistic (e.g. the mean) across many hypothetical samples of the same size.
scatter plot: A chart with one point per observation, placed by the values of two quantitative variables.
scree plot: A plot of eigenvalues by factor number; factors after the 'elbow' are dropped (Cattell's criterion).
seasonality: A repeating pattern tied to the calendar, such as weekday effects or holiday peaks.
segmentation: Splitting users or customers into groups that behave differently, so each can be targeted differently.
sequential testing: Test designs with adjusted thresholds that allow valid interim looks at the data.
sign test: Compares the number of positive and negative before-after changes, ignoring their size.
significance level (alpha): The threshold for rejecting H0, fixed before the test, usually 0.05; equals the accepted false-positive rate.
Simpson's paradox: A trend that appears in every subgroup reverses when the groups are combined, because group sizes differ.
skewness (skewed distribution): Lack of symmetry in a distribution; with a long right tail (typical for revenue) the mean is above the median.
slope (regression coefficient): b1: how much the predicted outcome changes when the predictor increases by one unit.
Spearman rank correlation (rho): Pearson's r computed on ranks; captures monotonic relationships and is robust to outliers.
standard deviation (SD): The square root of the variance; spread expressed in the same units as the data.
standard error (SE): The standard deviation of a statistic's sampling distribution; for a mean it is SD / sqrt(n).
standardisation (scaling): Converting variables to z-scores so that variables on large scales do not dominate distances.
static vs dynamic model: A static model describes a state at one moment; a dynamic model describes change over time.
statistic (estimate): A number computed from a sample that estimates a population parameter, e.g. the sample mean.
statistical power: The probability that a test detects a real effect of a given size; 1 - beta, usually planned at 0.8.
statistical significance: A result is significant when p < alpha; it says the effect is unlikely to be pure noise, not that it is large or important.
stepwise regression: Builds a model by adding or removing predictors one at a time based on their contribution.
Student's t-test (independent samples): Compares the means of two independent groups: t = (mean1 - mean2) / SE of the difference.

T

test assumptions: Conditions a test needs to be valid, e.g. independent observations, approximate normality, equal variances, no extreme outliers.
test statistic: A number computed from the data (t, U, F, chi-square...) that is compared with its distribution under H0.
time series: Observations of a metric ordered in time, e.g. daily orders; often has trend and seasonality.
train/test split (holdout): Fitting a model on one part of the data and evaluating it on unseen data to estimate real performance.
treatment (experimental) group: The group that gets the change being tested (the variant).
trimmed mean: The mean computed after dropping a fixed share (e.g. 5-10%) of the smallest and largest values.
truncated axis: A value axis that does not start at zero; it exaggerates small differences in bar charts.
Tukey's HSD test: A common post hoc test comparing all pairs of groups when variances are equal.
two-proportion z-test: Compares two conversion rates using the normal approximation: z = (p1 - p2) / SE of the difference.
two-way (multifactor) ANOVA: ANOVA with two or more factors that tests each main effect and their interaction.
type I error (false positive): Rejecting a true H0: declaring an effect that does not exist. Its probability is alpha.
type II error (false negative): Failing to reject a false H0: missing a real effect. Its probability is beta.

V

variable: A measurable property of each observation, such as order value, session length or whether a user converted.
variance: The average squared deviation from the mean; the sample version divides by n - 1. Measured in squared units.

W

weighted mean: A mean in which each value counts in proportion to its weight, e.g. average conversion across segments weighted by traffic.
Welch's t-test: A version of the two-sample t-test that does not assume equal variances; a safe default.
Wilcoxon signed-rank test: The non-parametric alternative to the paired t-test: ranks absolute differences and compares sums for positive and negative shifts.
within-group variance: The variability of observations around their own group mean; the 'noise' in ANOVA.

Z

z-score (standard score): How many standard deviations a value is from the mean: (x - mean) / SD.