Gradient Descent: Learning by Following the Slope

📍 Where we are: Part III · Learning from Scratch — Chapter 9. The previous chapter said learning is fitting a function to data by minimizing a loss. This chapter is the engine that does the minimizing.

In What Is Learning we reframed learning as a search: pick a family of functions, measure how badly a candidate fits the data with a loss, and then hunt for the settings that make the loss small. That last step, the hunt, is where almost all of modern machine learning actually spends its time, and almost all of it uses one idea. The idea is gradient descent: measure which way the loss is increasing, take a small step the other way, and repeat until you cannot go down any further.

This chapter builds that idea from nothing. We will not treat "minimize the loss" as a black box. We will define the loss as a surface over the space of a model's numbers, define exactly what "slope" means in one dimension and then in many, prove that the negative gradient is the steepest way down, derive the update rule and the precise range of step sizes that make it converge, and then run it by hand on two real models from the companion code. By the end you will be able to read the training loop of essentially any neural system and see gradient descent staring back.

The simple version

Imagine you are standing somewhere on a foggy hillside, blindfolded, and you want to reach the lowest point in the valley. You cannot see the valley. But you can feel, through your feet, which way the ground tilts. So you shuffle one small step directly downhill, feel the tilt again, step again, and keep going. If your steps are too big you overshoot the bottom and stagger back and forth; too small and you crawl. Gradient descent is exactly this, except the "hillside" is the loss of a model plotted against its numbers, and "feeling the tilt" is computing a derivative.

What this chapter covers

• The loss surface: how a model's parameters become coordinates and its loss becomes a height, turning learning into "walk downhill."
• The derivative in one dimension: the slope, its sign, the update rule, and a full convergence analysis on a quadratic that tells us exactly which learning rates work.
• The gradient in many dimensions: partial derivatives, the gradient vector, and a proof (directional derivative plus the Cauchy–Schwarz inequality) that the negative gradient is the direction of steepest descent.
• Two full worked derivations: fitting a line by least squares, and training a single-neuron classifier by cross-entropy, each matched line by line to committed code in examples/logic/learning.py, with real numbers from a real run.
• Batch, stochastic, and mini-batch gradient descent, and the trade-off between an exact step and a cheap one.
• The learning rate in a real landscape: conditioning, zig-zagging ravines, momentum, and step-size schedules.
• The unsolved part: why a local slope gives no guarantee of a global minimum once the surface stops being convex.

The loss surface: learning as descent

Fix a model whose behavior is controlled by a list of numbers. Collect those numbers into a parameter vector $\theta = (\theta_1, \theta_2, \ldots, \theta_d)$. For a straight line $y = wx + b$ the parameters are $\theta = (w, b)$, so $d = 2$. For the tiny neural network of the next chapter there are a couple dozen. For a modern language model there are hundreds of billions. In every case, training means choosing the numbers in $\theta$.

The data and the loss turn that choice into a landscape. Recall the recipe: for each choice of $\theta$ the model makes predictions on the training set, and the loss function $L(\theta)$ scores how wrong those predictions are with a single non-negative number, zero meaning perfect. Because $L$ assigns one height to every point $\theta$ in parameter space, it defines a surface. With two parameters you can picture it literally: a landscape of hills and valleys sitting above the $(w, b)$ plane, its height at each point equal to the loss there. Training is the search for a lowest point of that surface, a minimum of $L$.

This is the standard picture for a single neuron, where the space of parameters is called the weight space and the function the network computes is plotted at each point of it [1]. It is worth holding onto, because it makes the whole algorithm visual. We are not manipulating symbols for their own sake. We are trying to walk to the bottom of a valley we can feel but cannot see.

Two questions decide whether we can walk there. First, from where we stand, which way is downhill? Second, how big a step do we dare take? The derivative answers the first. A short piece of analysis answers the second. We take them in order, in one dimension first, because everything in many dimensions is built from the one-dimensional case one coordinate at a time.

One dimension: the derivative is the slope

Suppose for a moment the model has a single parameter $w$, so the loss $L(w)$ is an ordinary function of one real variable, a curve. The derivative $L'(w)$, also written $\frac{dL}{dw}$, is defined as the limit of the rise over the run as the run shrinks to nothing:

$$L'(w) \;=\; \lim_{h \to 0} \frac{L(w + h) - L(w)}{h}.$$

Read the fraction before the limit as "if I nudge $w$ by a small amount $h$, how much does $L$ change, per unit of nudge." The limit makes the nudge infinitesimally small, so $L'(w)$ is the exact slope of the curve at the point $w$: the rate at which the loss changes as $w$ increases. Its sign is all we need to take a step.

• If $L'(w) \gt 0$, the loss is rising as $w$ increases, so to make the loss smaller we must move $w$ in the negative direction, decreasing it.
• If $L'(w) \lt 0$, the loss is falling as $w$ increases, so to make the loss smaller we must move $w$ in the positive direction, increasing it.
• If $L'(w) = 0$, the slope is flat and, at least locally, there is nowhere lower to step. This is a stationary point.

Both non-flat cases are captured by a single rule: step in the direction opposite to the sign of the derivative. Scale the step by a small positive number $\eta$ (the Greek letter eta), called the learning rate, which sets how far we move:

$$w \;\leftarrow\; w - \eta\, L'(w).$$

The arrow means "replace the old $w$ by the value on the right." When $L' \gt 0$ we subtract a positive quantity and $w$ goes down; when $L' \lt 0$ we subtract a negative quantity and $w$ goes up. Either way the loss decreases, provided the step is not too large. The rest of this section is about that proviso, because "not too large" turns out to have an exact meaning.

A quadratic bowl, solved exactly

To see precisely how the step size behaves, take the simplest interesting loss, an upward parabola with its minimum at the origin:

$$L(w) \;=\; \tfrac{1}{2}\, a\, w^2, \qquad a \gt 0.$$

Its derivative is $L'(w) = a w$ (the $\tfrac12$ is there precisely so this comes out clean). The minimum is at $w = 0$, where $L' = 0$. Now watch what gradient descent does. Writing $w_t$ for the value after $t$ steps and substituting $L'(w_t) = a w_t$ into the update rule:

$$w_{t+1} \;=\; w_t - \eta\, a\, w_t \;=\; (1 - \eta a)\, w_t.$$

Every step multiplies the current value by the same constant $r = 1 - \eta a$. That is a geometric sequence, and we can solve it in closed form. Starting from $w_0$,

$$w_t \;=\; (1 - \eta a)^t\, w_0.$$

Now the entire behavior of the algorithm on this bowl is visible in one factor, $r = 1 - \eta a$. The iterates go to the minimum $w = 0$ if and only if the powers $r^t$ shrink to zero, which happens exactly when the size of $r$ is below one, $\lvert 1 - \eta a\rvert \lt 1$. Solving that inequality for $\eta$:

$$-1 \;\lt\; 1 - \eta a \;\lt\; 1 \quad\Longleftrightarrow\quad 0 \;\lt\; \eta a \;\lt\; 2 \quad\Longleftrightarrow\quad 0 \;\lt\; \eta \;\lt\; \frac{2}{a}.$$

That is the exact rule. There is a hard ceiling $\eta = 2/a$; step at or beyond it and the method never settles. Between zero and that ceiling, the sign of $r$ splits the safe zone into two qualitatively different regimes, and the special value $\eta = 1/a$ sits exactly between them where $r = 0$ and one step lands on the minimum. The four cases below are not a story we are telling; they are what the recurrence $w_{t+1} = (1-\eta a) w_t$ literally produces with $a = 1$ and $w_0 = 1$, computed by running the update:

learning rate $\eta$	factor $r = 1 - \eta a$	$w_0, w_1, w_2, w_3, w_4, w_5, w_6$	behavior
$0.1$ (so $0 \lt \eta \lt 1/a$)	$+0.9$	$1.000,\, 0.900,\, 0.810,\, 0.729,\, 0.656,\, 0.590,\, 0.531$	monotone decay toward $0$
$1.0$ (so $\eta = 1/a$)	$0.0$	$1.000,\, 0.000,\, 0.000,\, 0.000,\, 0.000,\, 0.000,\, 0.000$	one step to the exact minimum
$1.5$ (so $1/a \lt \eta \lt 2/a$)	$-0.5$	$1.000,\, {-0.500},\, 0.250,\, {-0.125},\, 0.063,\, {-0.031},\, 0.016$	overshoots, oscillates, still converges
$2.1$ (so $\eta \gt 2/a$)	$-1.1$	$1.000,\, {-1.100},\, 1.210,\, {-1.331},\, 1.464,\, {-1.610},\, 1.772$	oscillates and diverges

The lesson generalizes far beyond this toy bowl. A learning rate that is too small wastes time (the top row crawls); one that is well chosen lands quickly (rows two and three); one that is too large is not merely slow but catastrophic, throwing the parameters further from the minimum every step until they overflow to infinity (the bottom row). The number $a$ is the curvature of the bowl, how sharply it bends. Sharper bowls (larger $a$) demand smaller steps, because the ceiling $2/a$ is lower. This single fact, that the safe step size is set by the curvature, is the seed of every practical difficulty with learning rates that we will meet in real networks.

Many dimensions: the gradient and steepest descent

Real models have many parameters, so the loss is a surface over a high-dimensional space, not a curve. The generalization of the slope to that setting is the gradient, and it is built from one-dimensional slopes taken one coordinate at a time.

The partial derivative of $L$ with respect to the $j$-th parameter, written $\frac{\partial L}{\partial \theta_j}$, is just the ordinary derivative you get by freezing every other parameter at its current value and wiggling only $\theta_j$. It answers "if I nudge only $\theta_j$ and hold the rest fixed, how fast does the loss change." There are $d$ of these, one per parameter, and stacking them into a vector gives the gradient:

$$\nabla_\theta L \;=\; \left( \frac{\partial L}{\partial \theta_1},\; \frac{\partial L}{\partial \theta_2},\; \ldots,\; \frac{\partial L}{\partial \theta_d} \right).$$

The upside-down triangle $\nabla$ is read "del" or "grad." The gradient is a vector in the same space as $\theta$: it has one component for each parameter, and it lives at the point $\theta$ where you evaluated it, pointing in some direction through parameter space. The update rule is the one-dimensional rule with the gradient in place of the derivative, applied to all coordinates at once:

$$\boxed{\;\theta \;\leftarrow\; \theta - \eta\, \nabla_\theta L\;}$$

This says: move every parameter a little, each in proportion to how much it was responsible for the loss, and all in the direction that reduces $L$ fastest. The claim that this direction is the fastest way down is not obvious, and most texts simply assert it [1]. We will prove it, because the proof is short and it is the reason the algorithm is called gradient descent rather than something vaguer.

Why the negative gradient is steepest descent

Stand at a point $\theta$ and consider stepping a tiny distance in some direction. Represent the direction by a unit vector $\mathbf{u}$, meaning $\lVert \mathbf{u} \rVert = 1$ (the double bars $\lVert\cdot\rVert$ denote a vector's length, also called its norm, computed as the square root of the sum of its squared components, so a unit vector is one of length exactly one), so that "direction" and "distance" stay separate. How fast does the loss change as we move along $\mathbf{u}$? That rate is the directional derivative, and multivariable calculus gives it as the dot product of the gradient with the direction:

$$D_{\mathbf{u}} L \;=\; \nabla_\theta L \cdot \mathbf{u} \;=\; \sum_{j=1}^{d} \frac{\partial L}{\partial \theta_j}\, u_j.$$

(This is itself the chain rule: moving along $\mathbf{u}$ changes each $\theta_j$ at rate $u_j$, each of which changes $L$ at rate $\partial L / \partial \theta_j$, and the total is the sum.) We want the direction of steepest descent, the unit vector $\mathbf{u}$ that makes $D_{\mathbf{u}} L$ as negative as possible. Here the Cauchy–Schwarz inequality does the work. It states that for any two vectors,

$$\lvert \nabla_\theta L \cdot \mathbf{u} \rvert \;\le\; \lVert \nabla_\theta L \rVert \; \lVert \mathbf{u} \rVert \;=\; \lVert \nabla_\theta L \rVert,$$

using $\lVert \mathbf{u} \rVert = 1$ at the end. Therefore the directional derivative is boxed into the interval

$$-\lVert \nabla_\theta L \rVert \;\le\; D_{\mathbf{u}} L \;\le\; \lVert \nabla_\theta L \rVert.$$

The most negative value it can possibly take is $-\lVert \nabla_\theta L \rVert$, the left end. Cauchy–Schwarz achieves equality exactly when $\mathbf{u}$ is parallel to $\nabla_\theta L$; to hit the negative end we need $\mathbf{u}$ pointing exactly opposite to the gradient:

$$\mathbf{u}^{\star} \;=\; -\frac{\nabla_\theta L}{\lVert \nabla_\theta L \rVert}.$$

So among all directions you could step, the one that decreases the loss fastest is the direction directly against the gradient. That is precisely the direction the update rule $\theta \leftarrow \theta - \eta \nabla_\theta L$ moves in. The gradient earns its central place not by convention but by this inequality: it is the unique steepest direction, and its negative is the unique steepest way down.

One geometric picture cements it. The set of points at a fixed height forms a contour line (or contour surface) of the loss, like the isolines on a topographic map. Moving along a contour keeps the loss constant, so the directional derivative along it is zero, so by $D_{\mathbf{u}} L = \nabla L \cdot \mathbf{u} = 0$ the gradient is perpendicular to every contour. Steepest descent therefore always heads straight across the contour lines, at right angles, the same way water runs directly down a hillside rather than along it.

The loss $L(\theta)$ drawn as a landscape over a two-parameter weight space. At the current point the gradient $\nabla L$ points uphill, perpendicular to the contour of constant loss; the update steps along $-\eta\nabla L$, straight across the contours toward the basin. The step length is set by $\eta$ and by the steepness $\lVert\nabla L\rVert$.

Original diagram by the authors, created with AI assistance.

Worked example 1: fitting a line by least squares

The linear-regression fitter in examples/logic/learning.py, the function fit_line, is gradient descent in its purest form, and its two parameters let us watch the whole surface. It fits $y = wx + b$ to points $(x_i, y_i)$ by minimizing the mean squared error, exactly the loss defined in mse inside that function:

$$L(w, b) \;=\; \frac{1}{n} \sum_{i=1}^{n} \big(w x_i + b - y_i\big)^2.$$

Because the model is linear in its parameters and the error is squared, this surface is a smooth upward bowl (a paraboloid) with a single lowest point, the setting we solved exactly in one dimension, now in two. To descend it we need the two partial derivatives. Introduce the residual $r_i = w x_i + b - y_i$, the signed gap between prediction and truth on example $i$. Then $L = \frac{1}{n}\sum_i r_i^2$, and the chain rule gives, step by step,

$$\frac{\partial L}{\partial w} \;=\; \frac{1}{n}\sum_{i=1}^{n} \frac{\partial (r_i^2)}{\partial w} \;=\; \frac{1}{n}\sum_{i=1}^{n} 2 r_i \frac{\partial r_i}{\partial w} \;=\; \frac{1}{n}\sum_{i=1}^{n} 2\big(w x_i + b - y_i\big)\, x_i,$$

using $\partial r_i / \partial w = x_i$ because $r_i$ depends on $w$ only through the term $w x_i$. Identically, with $\partial r_i / \partial b = 1$,

$$\frac{\partial L}{\partial b} \;=\; \frac{1}{n}\sum_{i=1}^{n} 2\big(w x_i + b - y_i\big).$$

These two formulas are the code. Line by line, the training loop in fit_line reads:

for _ in range(epochs):
    dw = sum(2 * (w * x + b - y) * x for x, y in zip(xs, ys)) / n
    db = sum(2 * (w * x + b - y) for x, y in zip(xs, ys)) / n
    w -= lr * dw
    b -= lr * db

dw is $\partial L / \partial w$ verbatim, db is $\partial L / \partial b$, and the last two lines are the update rule $\theta \leftarrow \theta - \eta \nabla_\theta L$ split across the two coordinates, with lr playing the role of $\eta$. Running it on the five points in the file, with $\eta = 0.02$ for $2000$ steps, drives the loss from $44.182$ down to $0.0214$ and recovers the line

$$y \;=\; 1.990\, x + 0.050,$$

which is what the function prints. The data were generated near $y = 2x$, and gradient descent found it by nothing more than repeatedly rolling downhill on the squared-error bowl.

Worked example 2: a single-neuron classifier by cross-entropy

The line above regressed a number. The more consequential example, and the classic one for a single neuron [1], is classification: a single neuron that outputs a probability and is trained by cross-entropy. This is the function fit_logistic in the same file, and it is where the mathematics rewards us with a small miracle of cancellation.

The neuron takes a feature vector $\mathbf{x}$, forms the weighted sum $z = \mathbf{w}\cdot\mathbf{x} + b = \sum_j w_j x_j + b$, and squashes it through the sigmoid into a number between zero and one, read as the probability that the label is $1$:

$$p \;=\; \sigma(z) \;=\; \frac{1}{1 + e^{-z}}.$$

Here $e \approx 2.718$ is Euler's number and $e^{-z}$ is the exponential function ($e$ raised to the power $-z$); because $e^{-z}$ is always positive, the denominator $1 + e^{-z}$ exceeds one and the fraction stays strictly between $0$ and $1$, which is what lets $p$ be read as a probability.

For a true label $y \in \{0, 1\}$ (the symbol $\in$ reads "is an element of," so $y$ is either $0$ or $1$), the binary cross-entropy loss on one example is

$$\ell(\mathbf{w}, b) \;=\; -\big[\, y \ln p + (1 - y)\ln(1 - p)\,\big].$$

Here $\ln$ is the natural logarithm (the logarithm to base $e$), the standard inverse of the exponential. Read the loss as a penalty for being confidently wrong: if $y = 1$ it reduces to $-\ln p$, which is zero when $p = 1$ and blows up as $p \to 0$; if $y = 0$ it reduces to $-\ln(1 - p)$, symmetric. Each term is the information content of the observed outcome under the neuron's predicted distribution, and the total is minimized, reaching its floor of zero, exactly when $p = y$ on every example [1]. We want the gradient of $\ell$ with respect to the weights. It comes through a three-link chain: $\ell$ depends on $p$, $p$ depends on $z$, and $z$ depends on $w_j$.

Link one, the loss through the prediction. Differentiating $\ell$ with respect to $p$:

$$\frac{\partial \ell}{\partial p} \;=\; -\left[\frac{y}{p} - \frac{1 - y}{1 - p}\right] \;=\; \frac{-y(1-p) + (1-y)p}{p(1-p)} \;=\; \frac{p - y}{p(1 - p)}.$$

Link two, the prediction through the pre-activation. We need the derivative of the sigmoid, which has a famously tidy form. Writing $\sigma(z) = (1 + e^{-z})^{-1}$ and using the chain rule on the power and the exponential,

$$\sigma'(z) \;=\; -(1 + e^{-z})^{-2}\cdot(-e^{-z}) \;=\; \frac{e^{-z}}{(1 + e^{-z})^{2}} \;=\; \frac{1}{1+e^{-z}}\cdot\frac{e^{-z}}{1+e^{-z}}.$$

The first factor is $\sigma(z)$. The second is $1 - \sigma(z)$, because $\frac{e^{-z}}{1+e^{-z}} = 1 - \frac{1}{1+e^{-z}}$. So the sigmoid satisfies the identity

$$\sigma'(z) \;=\; \sigma(z)\big(1 - \sigma(z)\big) \;=\; p(1 - p).$$

Link three, the pre-activation through the weight. Since $z = \sum_j w_j x_j + b$, nudging $w_j$ moves $z$ at rate $x_j$, and nudging $b$ moves $z$ at rate $1$:

$$\frac{\partial z}{\partial w_j} \;=\; x_j, \qquad \frac{\partial z}{\partial b} \;=\; 1.$$

Now multiply the three links. Watch the $p(1-p)$ terms meet and annihilate:

$$\frac{\partial \ell}{\partial z} \;=\; \frac{\partial \ell}{\partial p}\cdot\frac{\partial p}{\partial z} \;=\; \frac{p - y}{p(1 - p)}\cdot p(1 - p) \;=\; p - y.$$

That cancellation is the miracle, and it is no accident. The cross-entropy loss is chosen to pair with the sigmoid precisely so the awkward $p(1-p)$ denominator from the loss is cancelled by the $p(1-p)$ from the sigmoid's own derivative, leaving the clean, robust error signal $e = p - y$. Had we used squared error on the sigmoid instead, the $p(1-p)$ would survive and shrink toward zero whenever the neuron saturates near $0$ or $1$, stalling learning exactly when a confident mistake most needs correcting. Cross-entropy is engineered to avoid that. Finishing the chain to the parameters:

$$\boxed{\;\frac{\partial \ell}{\partial w_j} \;=\; (p - y)\, x_j, \qquad \frac{\partial \ell}{\partial b} \;=\; (p - y).\;}$$

This is the standard result, the gradient $g_j = -(t - y)x_j$ with the sign flipped because the classic derivation writes the error as $t - y$ (target minus output) and we write $p - y$ (output minus target); the two differ only in which way you then step [1]. And it is the code, exactly. The inner loop of fit_logistic reads:

for xi, yi in zip(X, y):
    p = sigmoid(sum(wj * xj for wj, xj in zip(w, xi)) + b)
    err = p - yi
    for j in range(dim):
        gw[j] += err * xi[j]
    gb += err
w = [wj - lr * gj / n for wj, gj in zip(w, gw)]
b -= lr * gb / n

err is $p - y$, gw[j] += err * xi[j] accumulates $\sum_i (p_i - y_i) x_{ij}$, which is the batch gradient $\partial L / \partial w_j$, and the final two lines are the averaged update $\theta \leftarrow \theta - \eta \nabla_\theta L$. No calculus is hidden; the loop is the boxed formula.

One step, by hand, on the running example

The classifier in learning.py is grounded in this series' running academic world. Its two features are read straight off the advises relation: for each person, how many people they advise (out-degree) and how many advise them (in-degree). Professors are labeled $1$, students $0$. Here is the actual feature table the code builds:

person	out-degree $x_1$	in-degree $x_2$	label $y$
alice	$1$	$0$	prof $(1)$
bob	$2$	$1$	prof $(1)$
carol	$1$	$1$	student $(0)$
dave	$0$	$1$	student $(0)$
erin	$0$	$1$	student $(0)$

Start, as the code does, from $\mathbf{w} = (0, 0)$ and $b = 0$. Then every $z = 0$, so every prediction is $p = \sigma(0) = 0.5$: the untrained neuron is maximally unsure about everyone. The error on each example is $e = p - y = 0.5 - y$, which is $-0.5$ for the two professors and $+0.5$ for the three students. Accumulating the gradient over the batch, coordinate by coordinate:

$$\frac{\partial L}{\partial w_1} = \sum_i e_i x_{i1} = (-0.5)(1) + (-0.5)(2) + (0.5)(1) + (0.5)(0) + (0.5)(0) = -1.0,$$ $$\frac{\partial L}{\partial w_2} = \sum_i e_i x_{i2} = (-0.5)(0) + (-0.5)(1) + (0.5)(1) + (0.5)(1) + (0.5)(1) = +1.0,$$ $$\frac{\partial L}{\partial b} = \sum_i e_i = -0.5 - 0.5 + 0.5 + 0.5 + 0.5 = +0.5.$$

Averaging over the $n = 5$ examples gives the gradient $(-0.2,\, 0.2)$ for the weights and $0.1$ for the bias. One step with learning rate $\eta = 0.3$ moves the parameters to

$$\mathbf{w} \leftarrow (0,0) - 0.3\,(-0.2,\, 0.2) = (0.06,\, -0.06), \qquad b \leftarrow 0 - 0.3\,(0.1) = -0.03.$$

Read the direction of that first step and it already makes sense: the out-degree weight $w_1$ went up (advising many people is evidence of being a professor) and the in-degree weight $w_2$ went down (being advised is evidence of being a student). Gradient descent extracted the right qualitative rule from a single step, purely from the sign of the errors. Let the loop run its $3000$ epochs and the parameters settle at $\mathbf{w} \approx (7.75,\, -8.59)$, $b \approx -2.76$, a cross-entropy loss of $0.010$, and all five people classified correctly, which is exactly what the file prints. Neither feature alone separates professors from students; the neuron had to lean on both, and gradient descent found the combination.

Batch, stochastic, and mini-batch gradient descent

Both worked examples above summed the gradient over the entire training set before taking a single step. That is batch gradient descent (also called "batch learning"): each step uses the exact gradient of the true loss, so it moves in the genuinely steepest direction, but it must touch every example to move even once [1]. On five people that is nothing. On five billion web pages it is ruinous.

The opposite extreme is stochastic gradient descent (SGD), also called "on-line learning": update the parameters after each individual example, using that one example's gradient as a cheap, noisy estimate of the true gradient [1]. For the classifier this is literally the boxed per-example rule $w_j \leftarrow w_j - \eta (p - y) x_j$ applied one point at a time. Each step is $n$ times cheaper, and a single sweep through the data (an epoch) takes $n$ steps instead of one. The price is noise: any single example pulls the parameters in a direction that helps it, not the average, so the path to the minimum jitters rather than glides. Remarkably, that noise is often a feature. It lets SGD rattle out of the shallow dips and narrow ridges that would trap an exact, deterministic step, and it acts as a mild regularizer. The idea traces to the Robbins–Monro stochastic approximation method of 1951, which also tells us the noise must be cooled over time, by shrinking $\eta$ on a schedule, for the jitter to settle onto the true minimum rather than bounce around it forever [2].

Real systems live between the extremes, at mini-batch gradient descent: average the gradient over a small handful of examples (a few dozen to a few thousand), step, and move to the next handful. The batch is large enough that the gradient estimate is fairly stable and the arithmetic maps efficiently onto vector hardware, yet small enough that each step is cheap and a little helpful noise remains [3]. Batch, stochastic, and mini-batch are the same algorithm we derived; they differ only in how many examples go into each estimate of $\nabla_\theta L$ before a step is taken.

The learning rate in a real landscape

The one-dimensional bowl told us the safe learning rate is $0 \lt \eta \lt 2/a$, set by the curvature $a$. In many dimensions the curvature is not a single number but a whole matrix of second derivatives, the Hessian, and along its different eigen-directions the surface bends by different amounts, the eigenvalues $\lambda_1, \ldots, \lambda_d$. The one-dimensional analysis then applies direction by direction, and the stability ceiling is set by the sharpest direction: gradient descent converges only when $0 \lt \eta \lt 2/\lambda_{\max}$ [3]. This is the source of a very practical headache. If the surface is a long, narrow valley, steep across and gently sloping along, then $\lambda_{\max}$ is large and $\lambda_{\min}$ is small. You are forced to pick a small $\eta$ to stay under the ceiling in the steep cross-direction, but that same small $\eta$ makes agonizingly slow progress down the gentle length of the valley. The result is the classic zig-zag: fast bouncing between the steep walls, crawling toward the actual minimum. The ratio $\kappa = \lambda_{\max}/\lambda_{\min}$, the condition number, measures how badly this bites.

The standard first fix is momentum. Instead of stepping purely by the current gradient, keep a running velocity $\mathbf{v}$ that accumulates past gradients, and step by the velocity:

$$\mathbf{v} \;\leftarrow\; \beta\, \mathbf{v} - \eta\, \nabla_\theta L, \qquad \theta \;\leftarrow\; \theta + \mathbf{v},$$

with a momentum coefficient $\beta$ near $0.9$. Along the gentle length of the valley the gradient keeps pointing the same way, so the velocity builds up and the method accelerates, like a ball gathering speed downhill. Across the narrow width the gradient keeps flipping sign, so successive contributions to the velocity cancel and the oscillation is damped [3]. It is worth an honest caveat: momentum is a patch, and the more principled cure for ill-conditioning is a smarter optimizer such as conjugate gradients, which most serious work reaches for [1]. Alongside momentum, practitioners schedule the learning rate, starting larger to cover ground and decaying it over time (a common simple form is $\eta_0/\tau$), so the steps shrink as the parameters approach the basin.

The unsolved part

Every guarantee in this chapter rested quietly on one property: that the loss surface is a single bowl. The quadratic converged because it had one minimum. Linear regression and the single-neuron classifier converge for the same reason: their losses are convex, bowl-shaped everywhere, so any point where the gradient vanishes is the global minimum and downhill always eventually reaches it. This is a real theorem, and it is why these classical models are trustworthy [3].

The moment we stack neurons into the network of the next chapter, that property is gone. A deep network's loss is non-convex: a rugged range of many valleys, ridges, and passes. Gradient descent still does the only thing it knows, follow the local slope, but the local slope no longer knows where the lowest valley is. Three failures become possible and, in high dimensions, common. A local minimum is a valley floor that is lowest in its neighborhood but not overall; descent settles there and stops, blind to a deeper valley elsewhere. A saddle point is a pass where the gradient is zero yet the surface goes up in some directions and down in others; it is not a minimum at all, but it can stall progress for a long time, and saddles vastly outnumber true minima as the dimension grows. A plateau is a near-flat expanse where the gradient is tiny, so the steps are tiny, and the method barely moves. Gradient descent guarantees only that it will reach a stationary point, not the best one, and only if the step size respects the local curvature.

There is a deeper limitation still, true even on a friendly convex bowl: the gradient is a purely local instrument. It reports the slope directly under your feet and nothing else. It cannot see the shape of the valley beyond your immediate step, cannot tell a shallow dip from the entrance to the deepest gorge, and cannot look across a ridge. Everything gradient descent achieves, it achieves by feeling the ground one step at a time. That this simple, blind, local rule nonetheless trains models with hundreds of billions of parameters is one of the genuine surprises of the field, and it is not fully understood why it works as well as it does.

Why it matters

Gradient descent is the load-bearing algorithm of modern machine learning. Every neural network you will meet in the rest of this series, and essentially every one in current research, is trained by some refinement of the update rule $\theta \leftarrow \theta - \eta \nabla_\theta L$ derived here. Understanding it is not optional background; it is the mechanism.

For the neuro-symbolic project specifically, gradient descent is the exact hinge on which integration turns. A symbolic construct, a logical rule, a constraint, a proof score, can be trained jointly with a perception network if and only if it can be written as a differentiable loss whose gradient we can follow. That single condition, "make it differentiable so gradient descent can reach it," is the recurring engineering demand behind Volume 4's differentiable logic, behind semantic loss, behind the whole effort to make reasoning learnable. Everything in this book that learns learns by walking downhill on a loss, and this chapter is the description of the walk.

Key terms

• Loss surface (weight space): the graph of the loss $L(\theta)$ over the space of a model's parameters; training is the search for a low point on it.
• Derivative / partial derivative: the slope of the loss as one parameter changes with the others held fixed; its sign says which way is uphill.
• Gradient $\nabla_\theta L$: the vector of all partial derivatives; it points in the direction of steepest increase, so $-\nabla_\theta L$ is steepest descent.
• Learning rate $\eta$: the step-size multiplier; convergence on a bowl of curvature $a$ requires $0 \lt \eta \lt 2/a$, and the sharpest direction sets the ceiling.
• Update rule: $\theta \leftarrow \theta - \eta\, \nabla_\theta L$, applied repeatedly.
• Cross-entropy and the error signal: the loss $-[y\ln p + (1-y)\ln(1-p)]$, whose gradient through the sigmoid collapses to the clean error $e = p - y$.
• Batch / stochastic / mini-batch: whether each step's gradient is estimated from all examples, one example, or a small group.
• Momentum: a velocity that accumulates past gradients to accelerate along consistent directions and damp oscillation across narrow valleys.
• Convex vs non-convex: a single-bowl loss where descent reaches the global minimum, versus a rugged loss where it may reach only a local minimum, a saddle, or a plateau.

Where this leads

We now have the engine, but only for models whose gradient is easy to write out, a line and a single neuron. The pressing question is how to compute $\nabla_\theta L$ when the model is a deep stack of composed functions, where a single weight influences the output through many intermediate layers. The answer is backpropagation: the chain rule of calculus, organized to reuse its own partial results, run backward from the output to every weight. That is the subject of the next chapter, Neural Networks, where the gradient we have learned to follow here is finally something we can efficiently compute for a network of any depth.

---

Companion code: examples/logic/learning.py implements both worked examples, fit_line (regression, mean squared error) and fit_logistic (classification, cross-entropy), in pure Python with the gradients written out by hand exactly as derived above. Run python3 examples/logic/learning.py to reproduce every number in this chapter; the acceptance harness examples/logic/validate.py checks that both fits reach their target losses and that the classifier labels all five people correctly.