A student compares exam scores across three teaching methods: lecture, flipped classroom, and hybrid. Instead of running one analysis, they run three separate t-tests: lecture vs. flipped, lecture vs. hybrid, flipped vs. hybrid. Each test uses a significance level of 0.05. The problem? The probability of at least one false positive across those three tests is not 5%. It is about 14%. The student thinks they found a real difference, but the statistics say they might be chasing noise. Both t-tests and ANOVA compare group means, but they are not interchangeable. Choosing the wrong one can inflate your error rate, and reviewers will notice.
What Does Each Test Actually Do?
The t-test compares the means of exactly two groups. It answers one question: is the difference between these two means larger than what you would expect by chance? There are two main versions. The independent-samples t-test compares two separate groups of people, such as a treatment group and a control group. The paired-samples t-test compares the same group of people measured at two time points, such as before and after an intervention.
ANOVA stands for Analysis of Variance, which is a misleading name because it also compares means. The difference is scope. ANOVA handles two or more groups in a single test. When you apply it to exactly two groups, the result is identical to a t-test: the F-statistic equals t-squared, and the p-value is the same. ANOVA becomes necessary when your design includes three or more groups.
Consider a psychology student who measures stress levels (on a validated 0-to-40 scale) in 90 participants randomly assigned to meditation, exercise, or a control condition. That is three groups, one continuous outcome variable. The correct test here is a one-way ANOVA, not three separate t-tests. The reason comes down to error control.
Why Not Just Run Multiple T-Tests?
Every t-test carries a 5% risk of a Type I error, which means concluding that a difference exists when it does not. One test, one 5% risk. That is manageable. But run multiple tests on the same data, and the risks compound. The probability of at least one false positive across k independent tests is 1 - (1 - α)k. For three tests at α = 0.05, this is 1 - 0.953 = 0.143. Your actual false-positive rate has nearly tripled.
Scale this up and the problem gets worse. Five groups require 10 pairwise comparisons. The cumulative error rate reaches 1 - 0.9510 = 0.40, or about 40%. At that point, you are almost as likely to find a false positive as not.
ANOVA solves this by testing all groups in one shot. The F-test asks whether any of the group means differ from one another, and it does so while keeping the overall alpha at 0.05. One test, one controlled error rate, regardless of how many groups you have.
This is not a minor statistical technicality. A reviewer who sees three pairwise t-tests where a one-way ANOVA was appropriate will question whether you understand the analysis. We have seen this mistake in dozens of thesis drafts. It is one of the fastest ways to get a revision request.
Assumptions
Both tests rest on the same foundation. The dependent variable must be continuous (interval or ratio scale). Observations must be independent, meaning one participant's score does not influence another's. The data within each group should be approximately normally distributed. And the variances across groups should be roughly equal, a property called homogeneity of variance.
Test normality with Shapiro-Wilk for samples under about 50 per group, or inspect Q-Q plots visually for larger samples. For homogeneity of variance, Levene's test gives you a direct answer: a significant result (p < 0.05) means the variances differ enough to cause concern.
When normality fails, you switch to nonparametric alternatives. The Mann-Whitney U test replaces the independent-samples t-test. The Kruskal-Wallis test replaces the one-way ANOVA. Both rank the data instead of working with raw means, so they do not require a normal distribution.
When the variance assumption fails but normality holds, use Welch's correction. Welch's t-test adjusts the degrees of freedom to account for unequal variances, and Welch's ANOVA does the same for multiple groups. Both are available in SPSS (under Options in the One-Way ANOVA dialog) and through the oneway.test() function in R.
What Happens After a Significant ANOVA?
A significant F-test tells you that at least one group mean differs from the others. It does not tell you which one. If you have three groups and get F(2, 87) = 6.42, p = .003, you know something is going on, but not where. That is where post-hoc tests come in.
Tukey's HSD (Honestly Significant Difference) is the most common choice. It compares every possible pair of groups while controlling the family-wise error rate. Bonferroni correction is more conservative: it divides your alpha level by the number of comparisons, which keeps false positives very low but reduces statistical power. Games-Howell is the option when your group variances are unequal, since it does not assume homogeneity.
Returning to the stress study: suppose Tukey's post-hoc reveals that the meditation group scored significantly lower on stress than the control group (mean difference = 4.2, p = .002), but the exercise group did not differ significantly from either the meditation group (p = .11) or the control group (p = .34). Now you know exactly where the effect lies. The ANOVA opened the door, and the post-hoc test walked through it.
For guidance on reporting these p-values correctly, see our article on how to interpret p-values.
Effect Sizes
A significant p-value tells you an effect exists. It says nothing about how large that effect is. A study with 10,000 participants can produce a statistically significant result for a difference so small it has no practical meaning. That is why every analysis should include an effect size measure.
For t-tests, the standard measure is Cohen's d. It expresses the difference between group means in standard deviation units. The conventional benchmarks: 0.2 is small, 0.5 is medium, 0.8 is large. A Cohen's d of 0.5 means the two groups are separated by half a standard deviation.
For ANOVA, the standard measure is partial eta-squared (η²). It tells you what proportion of the variance in the dependent variable is accounted for by the grouping variable. The benchmarks: 0.01 is small, 0.06 is medium, 0.14 is large.
In the stress study, η² = .13 means the group variable (meditation, exercise, or control) accounts for about 13% of the variance in stress scores. That falls just below the large-effect threshold and is a meaningful finding. Most journals now require effect sizes alongside p-values, and the APA Publication Manual has mandated them since its sixth edition.
Quick Reference Table
| Feature | T-Test | ANOVA |
|---|---|---|
| Number of groups | Exactly 2 | 2 or more |
| Test statistic | t | F |
| Relationship | F = t² (for 2 groups) | — |
| Post-hoc needed? | No | Yes, if significant |
| Effect size | Cohen's d | Partial eta-squared (η²) |
| Nonparametric alternative | Mann-Whitney U / Wilcoxon | Kruskal-Wallis / Friedman |
The decision is straightforward. Two groups: use a t-test. Three or more groups: use ANOVA. If your ANOVA returns a significant F-test, follow up with post-hoc comparisons to identify which specific groups differ. Always report an effect size alongside your p-value, and always check your assumptions before running either test.
If you are unsure which test fits your particular research design, our guide to choosing a statistical test covers the full decision framework. And if you want to skip the manual work, Academic Stats Agent handles the test selection automatically: describe your variables, upload your data, and it runs the appropriate analysis with assumption checks and effect sizes included.
Key Takeaway: Use a t-test for two groups and ANOVA for three or more. Running multiple t-tests instead of ANOVA inflates your false-positive rate, and any reviewer will catch it.
Frequently Asked Questions
Can I use a t-test to compare 3 groups?
No. Running multiple t-tests on three groups inflates the false-positive rate from 5% to roughly 14%. Use a one-way ANOVA instead, followed by post hoc tests such as Tukey HSD to identify which specific groups differ.
When should I use Welch's t-test?
Use Welch's t-test when Levene's test shows the two groups have unequal variances, or when group sizes differ substantially. It adjusts the degrees of freedom and does not assume equal variances, so many statisticians recommend it as the default.
What do I do after a significant ANOVA result?
A significant ANOVA only tells you that at least one group mean differs. Run post hoc comparisons (Tukey HSD with equal variances, Games-Howell with unequal variances) to identify which pairs of groups differ, and report an effect size such as eta squared.