Introduction to Hypothesis Testing

Hypothesis testing is how we use data to make decisions. Is a new drug more effective than the current treatment? Is a factory’s defect rate higher than the acceptable threshold? Is there a real difference between two teaching methods, or is the observed difference just due to random chance? These questions all follow the same logical framework — and that framework is hypothesis testing. In this lesson, you will learn the core concepts: null and alternative hypotheses, test statistics, p-values, significance levels, and the types of errors that can occur.

The Logic of Hypothesis Testing

The reasoning behind a hypothesis test follows a structured sequence:

1. Start with a claim — assume some default position (the null hypothesis) is true
2. Collect data — draw a random sample and compute a relevant statistic
3. Ask a key question — if the null hypothesis were true, how surprising is the data we observed?
4. Make a decision — if the data would be very surprising under the null hypothesis, reject it. If the data is consistent with the null, fail to reject it.

A helpful analogy is the legal standard of “innocent until proven guilty.” In a trial, the default assumption (null hypothesis) is that the defendant is innocent. The prosecution must present evidence strong enough to reject that assumption beyond a reasonable doubt. If the evidence is not strong enough, the defendant is found “not guilty” — which is not the same as saying “innocent.” Similarly, failing to reject the null hypothesis does not prove it is true; it only means the data did not provide strong enough evidence against it.

Null and Alternative Hypotheses

Every hypothesis test begins with two competing statements about the population:

• $H_0$ (the null hypothesis): the default claim — typically “no effect,” “no difference,” or “the parameter equals a specific value”
• $H_a$ (the alternative hypothesis): what you suspect or want to demonstrate — typically “there is an effect,” “there is a difference,” or “the parameter differs from the null value”

The null hypothesis always contains an equals sign. The alternative hypothesis specifies the direction of the difference (or lack thereof).

Examples of Hypothesis Pairs

Drug effectiveness:

• $H_0: p = 0.50$ (the drug is no better than a placebo — 50% recovery rate)
• $H_a: p > 0.50$ (the drug produces a higher recovery rate)

Manufacturing quality:

• $H_0: \mu = 500$ g (the machine fills packages to the target weight)
• $H_a: \mu \neq 500$ g (the machine is off target — too high or too low)

Comparing two groups:

• $H_0: \mu_1 - \mu_2 = 0$ (no difference between the two groups)
• $H_a: \mu_1 - \mu_2 \neq 0$ (the groups differ)

One-Sided vs Two-Sided Tests

• A one-sided (one-tailed) test has an alternative hypothesis with a direction: $H_a: p > 0.50$ or $H_a: \mu < 500$. Use this when you only care about a deviation in one direction.
• A two-sided (two-tailed) test has an alternative hypothesis without a specific direction: $H_a: p \neq 0.50$ or $H_a: \mu \neq 500$. Use this when a deviation in either direction would be important.

When in doubt, use a two-sided test — it is more conservative and does not assume the direction of the effect in advance.

Test Statistics

A test statistic is a single number that measures how far the sample result is from what the null hypothesis predicts. The general form is:

$\text{test statistic} = \frac{\text{sample statistic} - \text{null value}}{\text{standard error}}$

This is essentially a z-score or t-score: it tells you how many standard errors the observed result is from the hypothesized value. A test statistic of 0 means the data matches the null hypothesis perfectly. A large positive or negative test statistic means the data is far from what the null hypothesis would predict.

For a proportion: $z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}}$

Note that in hypothesis testing for proportions, the standard error uses the null value $p_0$ (not the sample proportion $\hat{p}$) because we are asking: “If $p_0$ were the true proportion, how unlikely is our sample?”

For a mean (when $\sigma$ is unknown): $t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}$

P-Values — What They Mean

The p-value is the probability of observing a result as extreme as (or more extreme than) the sample data, assuming the null hypothesis is true. It answers the question: “If $H_0$ is true, how likely are we to see data this extreme just by random chance?”

• A small p-value (e.g., 0.003) means the observed data would be very unlikely under $H_0$. This is strong evidence against the null hypothesis.
• A large p-value (e.g., 0.42) means the observed data is consistent with $H_0$. There is no strong evidence against the null hypothesis.

What the P-Value is NOT

The p-value is one of the most misinterpreted concepts in statistics. Here are common errors:

• The p-value is NOT the probability that $H_0$ is true. It is the probability of the data (or more extreme data) given that $H_0$ is true — a conditional probability, not a direct statement about $H_0$.
• The p-value is NOT the probability that the result is due to chance. It is calculated under the assumption that chance alone is operating.
• A large p-value does NOT prove $H_0$ is true. It only means the data is consistent with $H_0$ — the data might also be consistent with other hypotheses.

Example 1: Testing a Coin for Fairness

You flip a coin 100 times and get 62 heads. Is the coin fair?

Step 1: State the hypotheses.

$H_0: p = 0.50 \quad \text{(the coin is fair)}$

$H_a: p \neq 0.50 \quad \text{(the coin is not fair — two-sided test)}$

Step 2: Calculate the test statistic. Under $H_0$, the standard error uses $p_0 = 0.50$:

$SE = \sqrt{\frac{0.50 \times 0.50}{100}} = \sqrt{\frac{0.25}{100}} = \sqrt{0.0025} = 0.05$

$z = \frac{0.62 - 0.50}{0.05} = \frac{0.12}{0.05} = 2.4$

Step 3: Find the p-value. Since this is a two-sided test:

$\text{p-value} = 2 \times P(Z > 2.4) = 2 \times 0.0082 = 0.0164$

Interpretation: If the coin were truly fair, there is only a 1.64% chance of getting a result as extreme as 62 (or more) heads out of 100 flips. This is fairly strong evidence against the coin being fair.

Significance Level ($\alpha$)

The significance level $\alpha$ is the threshold you set before collecting data that determines how small the p-value must be to reject $H_0$. It represents the maximum probability of making a Type I error (rejecting a true null hypothesis) that you are willing to tolerate.

Common significance levels:

Decision rule:

• If p-value $\leq \alpha$: reject $H_0$. The result is statistically significant.
• If p-value $> \alpha$: fail to reject $H_0$. The result is not statistically significant.

Example 1 continued: With $\alpha = 0.05$, the p-value of 0.0164 is less than 0.05. We reject $H_0$ and conclude there is statistically significant evidence that the coin is not fair.

Note the careful language: we say “fail to reject $H_0$,” not “accept $H_0$.” Absence of evidence against the null is not evidence that the null is true.

Type I and Type II Errors

Because hypothesis tests are based on sample data, there is always a chance of making the wrong decision. There are exactly two types of errors:

• Type I Error (False Positive): Rejecting $H_0$ when it is actually true. You conclude there is an effect when there really is not one. The probability of this error equals $\alpha$.
• Type II Error (False Negative): Failing to reject $H_0$ when it is actually false. You miss a real effect. The probability of this error is denoted $\beta$.

The following table summarizes all possible outcomes:

Real-world consequences:

• Type I Error in medicine: Approving a drug that does not actually work. Patients receive an ineffective treatment, and resources are wasted.
• Type II Error in medicine: Failing to approve a drug that does work. Patients miss out on an effective treatment.

There is a trade-off between the two errors: lowering $\alpha$ (making it harder to reject $H_0$) reduces the risk of Type I errors but increases the risk of Type II errors. The only way to reduce both simultaneously is to increase the sample size.

Statistical Power

Power is the probability of correctly rejecting a false null hypothesis:

$\text{Power} = 1 - \beta$

A test with high power is good at detecting real effects. A test with low power may miss real effects, leading to inconclusive results.

Three main factors affect power:

1. Sample size — larger samples provide more information and increase power. This is the factor researchers have the most control over.
2. Effect size — a larger true difference from the null value is easier to detect. You cannot control this, but you can design studies around the smallest effect you consider practically meaningful.
3. Significance level ($\alpha$) — a larger $\alpha$ (say 0.10 instead of 0.05) makes it easier to reject $H_0$, increasing power. However, this also increases the Type I error rate.

A commonly cited target for power is 0.80 (80%), meaning the test has an 80% chance of detecting a real effect if one exists. Power analysis — calculating the sample size needed to achieve a desired power — is a critical step in study planning.

Statistical Significance vs Practical Significance

A result can be statistically significant (small p-value) but practically meaningless. This happens when the sample size is very large: with enough data, even a tiny, unimportant difference can produce a p-value below 0.05.

For example, suppose a study of 100,000 patients finds that a new blood pressure medication lowers systolic pressure by 0.5 mmHg compared to the standard treatment, with a p-value of 0.001. The result is statistically significant — but a 0.5 mmHg reduction has virtually no clinical impact. No doctor would change their prescribing based on such a small difference.

Always consider the effect size alongside the p-value. Ask: “Is the observed difference large enough to matter in practice?” Report confidence intervals whenever possible — they convey both the direction and the magnitude of the effect, which a p-value alone cannot do.

The Steps of a Hypothesis Test

Every hypothesis test follows the same seven-step framework:

1. State $H_0$ and $H_a$ — define the null and alternative hypotheses in terms of a population parameter
2. Choose the significance level $\alpha$ — typically 0.05 unless there is a reason to use a different threshold
3. Check conditions — verify that the sample is random, observations are independent, and the appropriate distributional assumptions are met
4. Calculate the test statistic — measure how far the sample result is from the null value, in standard error units
5. Find the p-value — determine the probability of observing a result this extreme (or more extreme) if $H_0$ is true
6. Make a decision — compare the p-value to $\alpha$ and reject or fail to reject $H_0$
7. State the conclusion in context — translate the statistical decision into a plain-language statement about the original research question

Following this framework ensures that every test is conducted systematically and transparently.

Real-World Application: Nursing — Evaluating ER Wait Times

A hospital administration claims that the average emergency room wait time is 30 minutes. A nursing researcher suspects the actual wait time is longer. She collects data from a random sample of $n = 50$ patients and finds $\bar{x} = 34$ minutes and $s = 10$ minutes. Is there evidence that the average wait time exceeds 30 minutes? Use $\alpha = 0.05$.

Step 1: State the hypotheses.

$H_0: \mu = 30 \quad \text{(wait time is 30 minutes as claimed)}$

$H_a: \mu > 30 \quad \text{(wait time exceeds 30 minutes — one-sided)}$

Step 2: Significance level: $\alpha = 0.05$.

Step 3: Check conditions. Random sample ✓. Independence (50 patients is less than 10% of all ER patients) ✓. Sample size $n = 50 \geq 30$, so the CLT applies ✓.

Step 4: Calculate the test statistic.

$SE = \frac{s}{\sqrt{n}} = \frac{10}{\sqrt{50}} = \frac{10}{7.071} = 1.414$

$t = \frac{\bar{x} - \mu_0}{SE} = \frac{34 - 30}{1.414} = \frac{4}{1.414} = 2.83$

Step 5: Find the p-value. With $df = 49$ and $t = 2.83$ (one-sided, upper tail):

$\text{p-value} \approx 0.003$

Step 6: Decision. Since $0.003 \leq 0.05$, we reject $H_0$.

Step 7: Conclusion in context. There is statistically significant evidence at the $\alpha = 0.05$ level that the average ER wait time exceeds 30 minutes. The sample data suggests the true mean is around 34 minutes. The hospital should investigate causes of the longer wait times and consider operational changes.

Clinical note: This finding is also practically significant — a 4-minute increase over the claimed wait time represents a meaningful difference in patient experience and potential triage delays, especially during peak hours.

Practice Problems

Test your understanding with these problems. Click to reveal each answer.

Key Takeaways

• Hypothesis testing is a structured framework for using sample data to make decisions about population parameters
• The null hypothesis ($H_0$) represents the status quo (no effect, no difference); the alternative hypothesis ($H_a$) represents what you want to demonstrate
• The test statistic measures how far the sample result is from the null value in standard error units
• The p-value is the probability of observing data as extreme as yours if $H_0$ were true — a small p-value provides evidence against $H_0$
• Compare the p-value to your significance level $\alpha$: if p-value $\leq \alpha$, reject $H_0$; otherwise, fail to reject
• Type I Error (false positive, probability $\alpha$) means rejecting a true $H_0$; Type II Error (false negative, probability $\beta$) means failing to reject a false $H_0$
• Power ($1 - \beta$) is the ability to detect a real effect — it increases with larger sample sizes, larger effect sizes, and larger $\alpha$
• Statistical significance does not imply practical significance — always consider the effect size and real-world context alongside the p-value
• In healthcare, hypothesis tests help evaluate new treatments, monitor quality metrics, and make evidence-based decisions that directly affect patient care

Return to Statistics for more topics in this section.