Choosing a test is a matter of answering four questions about your design. Answer them in order and the test is usually determined before you open any software.
- What type is my outcome variable — continuous, ordinal, binary or count?
- How many groups or predictors am I comparing?
- Are the observations independent, or repeated on the same people?
- Am I comparing groups, testing a relationship, or predicting an outcome?
Comparing groups
| Groups | Independent | Repeated / paired | Non-parametric alternative |
|---|---|---|---|
| Two | Independent t-test | Paired t-test | Mann-Whitney U / Wilcoxon signed-rank |
| Three or more | One-way ANOVA | Repeated-measures ANOVA | Kruskal-Wallis / Friedman |
| Two factors | Two-way ANOVA | Mixed ANOVA | Aligned rank transform |
| With a covariate | ANCOVA | Mixed model | — |
| Several outcomes | MANOVA | Repeated MANOVA | — |
Testing relationships
- Two continuous variables, linear relationship — Pearson correlation.
- Ordinal data, or a non-linear monotonic relationship — Spearman rho.
- Two categorical variables — chi-square test of independence, or Fisher's exact test when expected counts are small.
- Agreement between two raters — Cohen's kappa, not correlation.
- Internal consistency of a scale — Cronbach's alpha or McDonald's omega.
Predicting an outcome
| Outcome variable | Model |
|---|---|
| Continuous | Multiple linear regression |
| Binary (yes/no) | Binary logistic regression |
| Three or more unordered categories | Multinomial logistic regression |
| Ordered categories | Ordinal logistic regression |
| Counts | Poisson or negative binomial regression |
| Time until an event | Cox proportional hazards |
| Nested or clustered data | Multilevel / mixed-effects model |
When assumptions fail
A violated assumption is a fork in the road, not a dead end. There are four normal responses, and all of them are reportable.
- Use a robust variant — Welch's t-test and Welch's ANOVA do not assume equal variances and cost you very little when variances are equal anyway.
- Bootstrap — resampling gives confidence intervals without distributional assumptions, and most software offers it as a checkbox.
- Transform — log or square-root transformation can fix skew, but interpretation is then on the transformed scale, which you must state.
- Switch to a non-parametric test — reliable, though usually with some loss of power, and the hypothesis being tested subtly changes.
Whichever you choose, report the violation and the response. A reader who sees that you noticed and handled it trusts the rest of the analysis more, not less.
Three errors that recur constantly
- Running many t-tests instead of an ANOVA. Six pairwise comparisons at α = .05 gives roughly a 26% chance of at least one false positive.
- Treating a Likert item as continuous. A single item is ordinal. A summed scale of several items is conventionally treated as continuous, and that distinction matters.
- Testing significance instead of the hypothesis. Decide what result would support or contradict your hypothesis before you run anything.
Questions this raises
A single Likert item is ordinal and is best analysed as such. A scale summing several items measuring one construct is conventionally treated as continuous, and that is widely accepted.
Unequal group sizes are fine for most tests, but they make the homogeneity-of-variance assumption more consequential. Welch's correction handles it well.
Still stuck after reading this? That is usually the point at which it is worth asking someone. Describe your project or ask on WhatsApp.
