Linear Regression

Linear regression finds the straight line that best fits a set of data points. While correlation tells you how strong the linear relationship is, regression gives you the actual equation of the line — which lets you describe the relationship mathematically and make predictions. If you know a student studied 4.5 hours, regression tells you the predicted test score. This is one of the most widely used tools in all of statistics, from predicting sales revenue to estimating patient recovery times.

The Least-Squares Regression Line

The least-squares regression line (also called the line of best fit) is the line that minimizes the sum of the squared vertical distances from each data point to the line. In other words, it is the line where the total squared prediction error is as small as possible.

The equation of the regression line is:

$\hat{y} = a + bx$

Where:

• $\hat{y}$ (“y-hat”) is the predicted value of $y$ for a given $x$
• $b$ is the slope — the predicted change in $y$ for each one-unit increase in $x$
• $a$ is the y-intercept — the predicted value of $y$ when $x = 0$

Formulas for Slope and Intercept

The slope $b$ is calculated from the data:

$b = \frac{n\sum xy - \sum x \sum y}{n\sum x^2 - \left(\sum x\right)^2}$

The intercept $a$ is calculated using the slope and the means of $x$ and $y$:

$a = \bar{y} - b\bar{x}$

Notice that the numerator of the slope formula is the same as the numerator of the correlation coefficient formula. The difference is in the denominator: the slope uses only the $x$-part of the denominator from the $r$ formula.

Calculating the Regression Line

Example 1: Study Hours vs Test Score

Using the same dataset from the correlation page — six students who reported their study hours and test scores:

From the correlation calculation, we already know: $n = 6$, $\sum x = 21$, $\sum y = 433$, $\sum xy = 1608$, $\sum x^2 = 91$.

Step 1: Calculate the slope.

$b = \frac{6(1608) - 21(433)}{6(91) - 21^2} = \frac{9648 - 9093}{546 - 441} = \frac{555}{105} \approx 5.286$

Step 2: Calculate the means.

$\bar{x} = \frac{21}{6} = 3.5 \qquad \bar{y} = \frac{433}{6} \approx 72.167$

Step 3: Calculate the intercept.

$a = 72.167 - 5.286(3.5) = 72.167 - 18.501 = 53.666$

Step 4: Write the regression equation.

$\hat{y} = 53.67 + 5.29x$

This equation describes the best-fitting line through the six data points.

Interpreting Slope and Intercept

Always interpret the slope and intercept in context — using the actual variable names and units from the problem.

Slope Interpretation

The slope $b = 5.29$ means: for each additional hour of study, the predicted test score increases by 5.29 points.

Notice the careful wording: “predicted” score, not “actual” score. The regression line gives predictions — individual students may score higher or lower than the prediction.

Intercept Interpretation

The intercept $a = 53.67$ means: a student who studies 0 hours is predicted to score 53.67 on the test.

However, be cautious about interpreting the intercept if $x = 0$ falls outside the range of your data. In this dataset, the lowest study time is 1 hour. The intercept is a mathematical necessity for defining the line, but it may not represent a meaningful real-world scenario.

Making Predictions

Once you have the regression equation, you can predict $y$ for any value of $x$ by substituting into the equation.

Example 2: Predict the Score for 4.5 Hours of Study

$\hat{y} = 53.67 + 5.29(4.5) = 53.67 + 23.805 = 77.475 \approx 77.48$

A student who studies 4.5 hours is predicted to score approximately 77.5 on the test.

Interpolation vs Extrapolation

• Interpolation: Predicting within the range of the data (here, $x$ between 1 and 6). These predictions are generally reliable.
• Extrapolation: Predicting outside the range of the data (e.g., predicting the score for $x = 15$ hours). Extrapolation is risky because the linear pattern may not continue beyond the observed data. A student studying 15 hours might experience diminishing returns from fatigue — the linear model cannot capture that.

Rule of thumb: Only use the regression line for predictions within (or very close to) the range of $x$-values in your dataset.

Residuals

A residual is the difference between what actually happened and what the regression line predicted:

$\text{Residual} = y - \hat{y} = \text{observed} - \text{predicted}$

• A positive residual means the actual value was higher than predicted (the point lies above the line)
• A negative residual means the actual value was lower than predicted (the point lies below the line)
• A residual of zero means the prediction was perfect (the point lies exactly on the line)

Example 3: Residual Table

Using $\hat{y} = 53.67 + 5.29x$, compute the predicted score and residual for each student:

Notice that the residuals are a mix of positive and negative values, and they are all quite small — which makes sense given the very strong correlation ($r = 0.998$) for this dataset.

Key property: The sum of all residuals for a least-squares regression line always equals zero (or very close to zero due to rounding): $0.75 + 0.46 + (-0.12) + 0.17 + (-0.41) + (-0.96) \approx -0.11$ (the small deviation is from rounding the slope and intercept).

Visualizing the Regression Line

The scatter plot below shows the six data points with the least-squares regression line drawn through them. Each teal dot is an observed data point, and the blue line is $\hat{y} = 53.67 + 5.29x$.

The teal dots represent observed scores, and the blue line is the least-squares fit. Notice how closely the points hug the line — reflecting the very high correlation ($r = 0.998$).

R-Squared: How Good Is the Fit?

The coefficient of determination $R^2$ tells you how well the regression line fits the data. It represents the proportion of the total variation in $y$ that is explained by the linear relationship with $x$.

$R^2 = r^2$

For our study hours example:

$R^2 = 0.998^2 = 0.996$

This means 99.6% of the variation in test scores is explained by the linear relationship with study hours. Only 0.4% of the variation remains unexplained.

Interpreting R-squared Values

A low $R^2$ does not necessarily mean the model is useless — it means other variables (not included in the model) also affect $y$. A high $R^2$ does not prove causation; it only confirms that the linear equation describes the pattern well.

Residual Plots: Checking Model Assumptions

A residual plot graphs the residuals (vertical axis) against the $x$-values or predicted values (horizontal axis). It is the most important diagnostic tool for evaluating whether a linear model is appropriate.

What to look for:

• Good pattern (linear model is appropriate): The residuals scatter randomly above and below zero with no obvious pattern, and the spread stays roughly constant across all $x$-values.
• Bad pattern — curved trend: If the residuals show a U-shape or inverted-U shape, the true relationship is probably curved, not linear. A linear model is not appropriate.
• Bad pattern — fan shape: If the residuals spread out (or narrow) as $x$ increases, the equal-variance assumption is violated. This is called heteroscedasticity.

Key principle: Even if $R^2$ is high, always check the residual plot. A high $R^2$ does not guarantee that a linear model is the right choice — it is possible to get a deceptively high $R^2$ when the true relationship is curved.

Real-World Application: Nursing — Predicting Patient Recovery Time from Age

A hospital collects data on 8 patients who underwent the same knee replacement surgery, recording each patient’s age and recovery time (days until discharge):

A nurse researcher runs a regression and obtains $\hat{y} = -2.85 + 0.178x$ (with $R^2 = 0.97$).

Interpreting the slope: For each additional year of age, the predicted recovery time increases by 0.178 days. Equivalently, for every 5 to 6 additional years of age, recovery is predicted to increase by about 1 day.

Prediction: A 50-year-old patient would be predicted to need $-2.85 + 0.178(50) = -2.85 + 8.90 = 6.05$ days, or roughly 6 days for recovery.

Clinical use: This regression helps nurses plan staffing and discharge logistics. However, individual patients vary based on fitness, complications, and other factors that the model does not capture (the 3% unexplained variation, plus unmeasured variables).

Caution about extrapolation: Predicting recovery time for a 20-year-old ($\hat{y} = -2.85 + 3.56 = 0.71$ days) or a 90-year-old ($\hat{y} = -2.85 + 16.02 = 13.17$ days) may be unreliable because those ages fall outside the observed data range.

Practice Problems

Test your understanding with these problems. Click to reveal each answer.

Key Takeaways

• The least-squares regression line $\hat{y} = a + bx$ minimizes the sum of squared residuals and provides the best linear fit to the data
• The slope $b$ describes the predicted change in $y$ for each one-unit increase in $x$ — always interpret it in context using the actual variable names
• The intercept $a$ is the predicted $y$ when $x = 0$ — it may or may not have a meaningful real-world interpretation
• Residuals ($y - \hat{y}$) measure how far each observed value falls from the regression line; they always sum to approximately zero
• Interpolation (predicting within the data range) is generally reliable; extrapolation (predicting outside the data range) is risky and should be avoided
• $R^2$ tells you the proportion of variation in $y$ explained by the linear model — but a high $R^2$ does not prove causation or guarantee the model is appropriate
• Always examine a residual plot to check whether the linear model is a good fit — look for random scatter with constant spread
• In healthcare and other applied fields, regression helps make predictions, but individual outcomes vary based on factors not captured by the model

Return to Statistics for more topics in this section.