Skip to content
StatPlayMathPlayPartial derivatives
The surface state is also available through each step's numbers and sliders.
    // MATHPLAY 01 · PARTIAL DERIVATIVES

    Toward “set the partial derivatives to zero”How many slopes are there here?

    The diagram shows a surface whose height f(x, y) depends on two inputs, x and y. At point P, there is no single slope: the slope changes with the direction of travel.

    Start with a tangent to one curve, then keep moving the same diagram until it becomes the least-squares valley.

    1. 01One slope
    2. 03Measure a slice
    3. 06Tangent plane
    4. 07Gradient
    5. 08Valley floor
    6. 09Least squares

    Scroll to change the diagram. Drag point P to move it.

    01

    Along one path, there is one slope

    Begin with one input. The only choices are to move right or left.

    Take a second point Δx to the right of P and join the two points with anorange line. Its slope Δf/Δx is the average slope over that interval. Shrink Δx and the line approaches thetangent line at P, whose slope is the derivative f′(x).

    f′(x) = limΔx→0 f(x + Δx) − f(x)Δx

    Watch this As Δx gets smaller, Δf/Δx in the upper left approaches the tangent slope +1.00 .

    02

    On a surface, slope depends on direction

    The previous curve was a cross-section of this surface cut at y = −1.00 . With two inputs, the graph becomes a surface like terrain.

    Move from P in direction θ. The slope may be uphill, downhill, or flat. Rotate θ through a full turn; the angle is measured counterclockwise from the x-axis.

    There are infinitely many directions. Which slopes are enough to determine the rest?

    03

    Hold y fixed to recover one curve

    To return to the single slope in step 01, hold y fixed and move only x. The available path is one curve where the cyan plane cuts through the surface.

    Watch this Changing the fixed value of y changes the slice, but each slice remains a single curve.

    04

    The tangent slope of this slice is ∂f/∂x

    Seen from the side, the slice is a curve like the one in step 01. Write its tangent slope at P as ∂f/∂x. This is the partial derivative with respect to x. A textbook may also write fx.

    ∂f/∂x = limΔx→0 f(x + Δx, y) − f(x, y)Δx

    Compared with step 01, the key difference is that y has the same value in both terms. Treat y as a constant during the calculation.

    Functionf(x, y) = x² + xy + y²
    Set y to a numbery = −1.00 then
    f = x² + (−1.00)x + (−1.00)²
    Differentiate with respect to x2x + (−1.00) ※ (−1.00)x differentiates to (−1.00); the constant term differentiates to 0
    Write y again∂f/∂x = 2x + y
    At P2 × (1.00) + (−1.00) = +1.00

    Watch this The term y remains in 2x + y because this function contains xy. Changing the fixed y shifts the slice and changes its slope. For f = x² + y², by contrast, ∂f/∂x = 2x regardless of the fixed y.

    05

    Hold x fixed to obtain ∂f/∂y

    Now hold x fixed and move only y. The view rotates 90° and shows themagenta slice from the side. Its tangent slope is ∂f/∂y .

    Set x to a numberx = 1.00 then
    f = (1.00)² + (1.00)y + y²
    Differentiate with respect to y(1.00) + 2y
    Write x again∂f/∂y = x + 2y
    At P(1.00) + 2 × (−1.00) = −1.00

    Watch this At the same point P, the x-slope is +1.00, while the y-slope is −1.00. A partial derivative measures change in one input while the others remain fixed.

    06

    Two tangent lines determine a plane

    The plane containing both tangents through P is the tangent plane. Very close to P, this smooth surface lies close to its tangent plane.

    Approximate the height change Δf after moving Δx along x and Δy along y by adding the two slope contributions. The steps in the diagram add thex-directionandy-direction changes in sequence.

    Δf ≈ (∂f/∂x) Δx + (∂f/∂y) Δy

    Current step: Δx = 0.40, Δy = −0.20

    Tangent-plane change
    +0.600
    Actual change Δf
    +0.720
    Gap
    +0.120

    Watch this Halving the step halves the tangent-plane prediction but quarters the approximation error for this function. The error shrinks faster than the step, so the tangent-plane approximation improves near P. The total differential df = (∂f/∂x)dx + (∂f/∂y)dy describes change on that plane and is therefore written with an equals sign rather than ≈.

    07

    Two partial derivatives give slopes in every direction

    This is a contour map viewed from above, with a valley at the center. A unit step in direction θ moves cos θ along x and sin θ along y. On the tangent plane, the resulting height change is the directional derivative below.

    (∂f/∂x) cos θ + (∂f/∂y) sin θ

    Step 02 showed infinitely many directions. Combine the x- and y-partials to calculate the slope in any of them.

    Solid curve: slope by direction Cyan dashed curve: (∂f/∂x) cos θ Magenta dashed curve: (∂f/∂y) sin θ Green point: steepest uphill slope

    Put the two partial derivatives together: ∇f = (∂f/∂x, ∂f/∂y) (∇ is pronounced “nabla”). This is the gradient. It points in the direction of steepest ascent, whose slope is |∇f| = 1.41.

    Watch this The green arrow crosses the contour through P (thedotted line) at a right angle. Along the contour the slope is zero; the opposite of the gradient is the steepest descent.

    08

    Where both slopes become zero

    Where both ∂f/∂x and ∂f/∂y are zero, the formula from step 07 gives zero slope in every direction, and the tangent plane is horizontal.

    x-direction∂f/∂x = 2x + y = 0
    y-direction∂f/∂y = x + 2y = 0
    Intersection(x, y) = (0, 0)

    The cyan and magenta dashed lines mark where each partial derivative is zero. Both vanish at their intersection.

    Watch this Small steps opposite the gradient move toward the intersection: the idea behind gradient descent. A point where both partials vanish can also be a peak or a saddle on another surface. Here it is the valley floor.

    09

    Into statistics: fitting a line also makes a valley

    Fit a line y = a + bx to five points. The vertical difference between a point and the line is its residual ei = yi − (a + bxi). Square and sum these residuals to obtain S.

    S(a, b) = Σ ei² = Σ ( yi − a − bxi )²

    The inputs here are not x, but the parameters of the line: a (intercept) and b (slope). Put a and b along the horizontal and depth axes, and S becomes a valley.

    Watch this As the line approaches the points, S decreases and (a, b) moves down the valley. Least squares finds the valley floor.

    10

    Hold b fixed and move a: sum of residuals

    As in steps 03–04, hold b fixed and move only a. The slope of the cyan cross-section is ∂S/∂a.

    Differentiate with respect to a∂S/∂a = Σ 2( yi − a − bxi ) × (−1)
    = −2 Σ ei
    Current valueΣ ei = +2.00
    ∂S/∂a = −4.00

    Inside the parentheses, ( yi − a − bxi ) falls by one when a rises by one. Therefore the derivative of the square, 2( … ), is multiplied by −1: an application of the chain rule.

    The condition ∂S/∂a = 0 says that the residuals sum to zero. The line balances the points above and below it, and passes through the mean point (x, y) = (2, 3).

    11

    Hold a fixed and move b: residuals and x

    Now hold a fixed and move only b. The slope of themagenta slice is ∂S/∂b.

    Differentiate with respect to b∂S/∂b = Σ 2( yi − a − bxi ) × (−xi)
    = −2 Σ xiei
    Current valueΣ xiei = +10.40
    ∂S/∂b = −20.80

    The condition ∂S/∂b = 0 says that the x-weighted residuals sum to zero. Together with the condition in step 10, the covariance between x and the residuals is zero: no linear pattern with x remains in the residuals.

    Watch this Increase b too far and the line overshoots the points on the right; ∂S/∂b becomes positive.

    12

    Both derivatives vanish at the regression line

    As in step 08, both partial derivatives vanish at the valley floor. The two simultaneous conditions are the normal equations.

    ∂S/∂a = 0Σ( yi − a − bxi ) = 0
    → Σyi − n·a − b Σxi = 0
    → n·a + b Σxi = Σyi
    ∂S/∂b = 0Σxi( yi − a − bxi ) = 0
    → a Σxi + b Σxi² = Σxiyi
    For these five points5a + 10b = 15
    10a + 30b = 39.4
    Solutionb = 0.94, a = 1.12

    S(a, b) is quadratic in a and b. Unless all xi values are identical, it forms a bowl: no peak or saddle, and one point where both derivatives vanish—the minimum.

    Solving the equations gives b = Σ(xi − x)(yi − y) ⁄ Σ(xi − x)², a = y − b x, the familiar regression-coefficient formulas. For these points, b = 9.4 ⁄ 10 = 0.94.

    The textbook instruction “differentiate S with respect to a and b and set the results to zero” states the condition for this valley floor.

    Watch this The cyan and magenta dashed lines show where each partial derivative is zero. Their intersection marks the pair (a, b) for the least-squares regression line.

    // BACK TO THE TEXTBOOK

    Before returning to the textbook

    Here is how the objects in the diagram correspond to the symbols in a textbook.

    SymbolNameWhat the diagram showed
    ∂f/∂xPartial derivative with respect to xTangent slope of the slice with y fixed (04)
    ∂f/∂yPartial derivative with respect to yTangent slope of the slice with x fixed (05)
    df = fxdx + fydyTotal differentialTangent-plane prediction of the height change (06)
    (∂f/∂x) cos θ + (∂f/∂y) sin θDirectional derivativeSlope in direction θ (07)
    ∇fGradientDirection and size of the steepest uphill slope (07)
    ∂S/∂a = ∂S/∂b = 0Normal equationsLeast-squares minimum, corresponding to the regression line (12)

    A partial regression coefficient is the change in the predicted outcome when one predictor increases by one unit while the others are fixed. It is the partial derivative of the prediction equation with respect to that predictor. In steps 09–12, the derivatives were instead taken with respect to a and b. Reading a coefficient as a causal effect requires separate scrutiny of confounder selection and measurement, as well as model assumptions.