Multiple Regression — Predicting with Several Variables
Predict with study hours alone, then add sleep hours. Even with the same data, the coefficient for study hours can change.
Multiple Regression — Predict with multiple variables
One predictor gives a line; two give a plane in 3D. Start by comparing the coefficients in the panel below. This example uses synthetic data; whether a coefficient represents a causal effect is a separate question.
- Compare the two β₁ estimates at a correlation setting of −0.5.
- Try zero correlation. The gap tends to shrink, but sampling variation remains.
- Try +0.5. Does the direction of the gap change? Regenerate the data and compare.
- Compare R² too. Adding a predictor cannot worsen the fit to the same data, but it need not improve predictions on unseen data.
▶ Simple vs. Multiple Regression — Watch β₁ Shift
- Step 1: Drag the 3D plot to rotate. The translucent surface is the regression plane — data points align along it.
- Step 2: Move the study slider → the prediction dot slides along the x₁ direction. The tilt = β₁.
- Step 3: Move sleep too → it moves along x₂. The tilt = β₂. Each variable's contribution is visible.
- Step 4: Hit Resample a few times → β₁, β₂, R² shift slightly each time. Estimates have variability too.
// Why does adding a variable change the coefficient?
The panel uses synthetic data with correlated study and sleep hours. Scores are generated as 40 + 3 × study hours + 2 × sleep hours + noise.
With study hours alone, the slope also reflects the sleep hours that vary alongside them. With negative correlation, the simple-regression coefficient tends to be smaller than the value of 3 used to generate the data.
A partial regression coefficient describes the change in the prediction when one predictor changes and the others stay fixed. Adding sleep hours tends to bring the study coefficient closer to 3, though sampling and noise still affect the estimate.
Adding predictors does not, by itself, establish causation. Interpreting a coefficient as a causal effect requires separate assumptions about confounders, measurement, and the model.
With the correlation setting at zero, the two coefficients tend to be closer. In a finite sample, they need not be identical.
// Formula used here
Each part
• β₀ (intercept): baseline prediction when all variables = 0
• β₁x₁: "holding x₂ fixed, how much does y change per unit of x₁" × the value of x₁
• β₂x₂: same idea — the isolated contribution of x₂
How it differs from simple regression
• Simple regression β₁: the slope when predicting with x₁ alone
• Multiple regression β₁: the slope for x₁ with x₂ held fixed in the model
• At a correlation setting of zero, the estimates tend to be closer, but sampling differences remain
Geometric picture
• With 2 predictors, the fit is a plane in 3D. Least squares picks the tilt that minimizes the sum of squared vertical residuals from points to the plane
// Worked example — try it by hand
A worked example with 30 fictional students. These illustrative values do not track the panel above.
① Check the averages
Mean study = 5.0h, mean sleep = 7.0h, mean score = 65 pts
Correlation between study and sleep: r = −0.45 (heavy studiers sleep less)
② Simple regression y ~ x₁ (study hours only)
β₁ = +2.4 pts/h, R² = 0.32
→ One more study hour corresponds to a prediction 2.4 points higher
③ Multiple regression y ~ x₁ + x₂ (add sleep hours)
β₁ = +3.1 pts/h, β₂ = +2.0 pts/h, R² = 0.57
→ With sleep held fixed in the model, the study coefficient is +3.1 pts/h
④ Why the change?
Study and sleep hours are negatively correlated in this example.
Including sleep changes the study coefficient.
That difference alone does not establish a causal effect of study or sleep.
⑤ Make a prediction
β₀ = ȳ − β₁x̄₁ − β₂x̄₂ = 65 − 3.1×5 − 2.0×7 = 35.5 (OLS always passes through the means).
Student with 6h study, 7h sleep → ŷ = 35.5 + 3.1×6 + 2.0×7 = 35.5 + 18.6 + 14.0 = 68.1 pts
β₀ is the "score at 0h study, 0h sleep" — a hypothetical baseline with no intuitive meaning (don't extrapolate).
// Common misconceptions
"Study hours (0–10)" and "sleep hours (4–10)" are on different scales. Comparing raw coefficients is meaningless. Use standardized coefficients (both variables rescaled to SD = 1) to compare importance.
R² always increases with more variables, but the adjusted R² can decrease. Irrelevant variables add noise and destabilize estimates. In the simulator above, a coefficient near zero hints that the variable may be unnecessary.
When two predictors are highly correlated — like "study hours" and "library hours" — individual coefficients become erratic (signs can even flip). Watch for VIF > 10 as a warning sign.
// Shapes you'll meet again
Around multiple regression, the same coefficient-interpretation and test-comparison shapes keep returning.
- The partial-coefficient picture: with β₁ = 3, the reading "holding x₂ constant, raising x₁ by 1 lifts y by 3 on average" appears. "Hold the others fixed" is the conditional clause that travels with this picture
- When β₁ shifts between simple and multiple regression: with correlated predictors, adding a variable can change the slope. At a correlation setting of zero, the two estimates tend to be closer
- R² vs. adjusted R² as a pair: adding a variable always raises R², but adjusted R² can fall. The shape "if adjusted drops, the new variable wasn't worth its cost" lives here
- How F-tests and t-tests divide the labor: the F-test takes "all partial coefficients = 0" at once; the t-test asks "is this one coefficient = 0?" — the model-wide and per-coefficient questions split along these two shapes
- The multicollinearity signal: VIF > 10 appears as the rule-of-thumb threshold, foreshadowing scenes where coefficients become erratic