Neural Networks: Differentiable Function Approximators

📍 Where we are: Part III · Learning from Scratch, Chapter 10. What Is Learning tuned numbers until a straight-line boundary matched the data; but some patterns no straight line can ever match, and this chapter builds the network that can, then derives the mathematics that makes it work.

Two earlier chapters set this up. What Is Learning framed learning as the mirror image of deriving: instead of firing rules forward, a learner starts from examples and nudges numbers until its predictions match; Gradient Descent then made the nudging precise. Both of its workers, a line fit to points and a logistic classifier that told professors from students, drew the same shape of decision, a single straight boundary. That shape is also a ceiling. A straight line is all a linear model can ever express, and there are patterns no straight line can cut. This chapter meets the smallest such pattern, proves that the linear model cannot solve it, and then builds and fully derives the thing that can: a network with one hidden layer trained by backpropagation.

The simple version

Imagine you must separate red beads from blue beads scattered on a table by laying down one straight stick, every red on one side, every blue on the other. For some arrangements this is easy. For others it is impossible: the reds sit at two opposite corners and the blues at the other two, so no straight stick can split them. Now imagine you are first allowed to crumple the tablecloth, to lift it and fold it, before laying the stick down. Suddenly the same beads line up and one stick divides them. A neural network's hidden layer is that crumpling. It bends the space until a straight cut works, and, remarkably, it learns exactly how to fold from the examples alone.

What this chapter covers

• A proof, not a picture that the exclusive-or pattern is not linearly separable, and the linear baseline's two-out-of-four score on it.
• The neuron and the sigmoid: an affine pre-activation passed through a nonlinearity, with the sigmoid's derivative derived from scratch, and an argument for why the nonlinearity is not optional.
• The forward pass of a 2-4-1 network, every symbol mapped to a line of the companion neural.py.
• The loss and gradient descent: squared error, its gradient, and the update rule with its learning rate.
• Backpropagation derived by the chain rule, each gradient matched to the exact code line that computes it, with the numbers worked through one step.
• The payoff and the honest limits: why the hidden layer conquers XOR, universal approximation stated precisely, why differentiability is the bridge to neuro-symbolic AI, and the two problems this machinery does not solve.

Why a linear model fails on XOR: a proof, not a picture

A linear classifier scores an input by a single weighted sum followed by a threshold. Write each input as a vector $\mathbf{x} \in \mathbb{R}^n$, which reads "$\mathbf{x}$ is an ordered list of $n$ real numbers $(x_1, \ldots, x_n)$." The integer $n$ is the input dimension: the number of measured features that describe one example (for the professor-versus-student classifier of the previous chapter, $n = 2$, the out-degree and the in-degree). Pair the input with a weight vector $\mathbf{w} = (w_1, \ldots, w_n)$, one weight per feature, and a single number $b$ called the bias. The classifier's prediction is then

$$\hat y = \mathbf{1}\!\left[\, \mathbf{w}^\top \mathbf{x} + b > 0 \,\right],$$

Two pieces of notation need decoding. The expression $\mathbf{w}^\top\mathbf{x}$ is the dot product of the weight and input vectors, the weighted sum $\sum_{i=1}^{n} w_i x_i = w_1 x_1 + \cdots + w_n x_n$; the raised $\top$ (read "transpose") is the bookkeeping symbol that turns the column of weights on its side so it multiplies $\mathbf{x}$ entry by entry and adds the products. The indicator $\mathbf{1}[\cdot]$ returns $1$ when the condition inside the brackets holds and $0$ otherwise. So the neuron outputs $1$ exactly when the weighted sum of the features plus the bias is greater than zero.

Geometrically, the equation $\mathbf{w}^\top\mathbf{x} + b = 0$ marks the boundary where that score is exactly zero, the dividing surface between the two classes. A flat boundary of this kind in $n$-dimensional space is called a hyperplane. When there are just two features, so $n = 2$ and each example is a point $(x_1, x_2)$ in the ordinary plane, the hyperplane is simply a straight line; with $n = 3$ it is a flat plane, and for larger $n$ it is the higher-dimensional analogue that we still call a hyperplane. The classifier labels everything on one side of this boundary $1$ and everything on the other side $0$. A dataset it can split perfectly with a single such boundary is called linearly separable.

The XOR function (short for exclusive-or, "true when exactly one input is true") is the canonical pattern that is not. The companion neural.py writes it out as four points with their labels:

XOR_X = [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]]
XOR_Y = [0.0, 1.0, 1.0, 0.0]

Read as a table of targets: $(0,0)\mapsto 0$, $(1,1)\mapsto 0$, $(0,1)\mapsto 1$, and $(1,0)\mapsto 1$. The two class-$0$ points sit at opposite corners of the unit square, and so do the two class-$1$ points. That no line separates them is not a matter of looking at a picture and being convinced; it is a short proof by contradiction.

Suppose a separating $(\mathbf{w}, b)$ with $\mathbf{w} = (w_1, w_2)$ existed, classifying all four points correctly. The two positive points must satisfy $\mathbf{w}^\top\mathbf{x} + b > 0$:

$$(0,1):\ w_2 + b > 0, \qquad (1,0):\ w_1 + b > 0 .$$

Adding these two strict inequalities gives

$$w_1 + w_2 + 2b > 0. \tag{A}$$

The two negative points must satisfy $\mathbf{w}^\top\mathbf{x} + b \le 0$:

$$(0,0):\ b \le 0, \qquad (1,1):\ w_1 + w_2 + b \le 0 .$$

Now regroup the sum $w_1 + w_2 + 2b$ using these two facts:

$$w_1 + w_2 + 2b = \underbrace{(w_1 + w_2 + b)}_{\le\, 0} + \underbrace{b}_{\le\, 0} \le 0. \tag{B}$$

Statements (A) and (B) directly contradict each other: the same quantity cannot be both strictly positive and less than or equal to zero. Therefore no linear separator exists. The best any single line can do on these four points is three of four correct, because getting all four is exactly what we just ruled out.

Turn the exact logistic classifier from those earlier chapters loose on this data and it does even worse than the best line. The module runs fit_logistic on XOR and prints:

linear classifier on XOR: [1, 1, 1, 1] target [0, 1, 1, 0] -> correct: 2 / 4

Two of four. Unable to bend, gradient descent on the linear model settles into calling everything $1$, no better than a coin flip. The proof told us the ceiling is three of four; this seeded run lands on the floor of two. Either way the linear family is the wrong family, and the fix is not a better line but a richer function.

The neuron and the sigmoid

The atom of a network is the neuron. It computes an affine pre-activation, a weighted sum of its inputs plus a bias,

$$z = \mathbf{w}^\top \mathbf{x} + b = \sum_k w_k x_k + b,$$

and then passes that scalar through a nonlinear activation function to produce its activation $a = \sigma(z)$. The activation used throughout neural.py is the logistic sigmoid,

$$\sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \sigma : \mathbb{R} \to (0, 1),$$

an S-shaped curve that squashes any real number into the open interval $(0,1)$: large negative inputs map near $0$, large positive inputs near $1$, and $\sigma(0) = \tfrac12$. In code:

def sigmoid(z: float) -> float:
    if z < -60:
        return 0.0
    if z > 60:
        return 1.0
    return 1.0 / (1.0 + math.exp(-z))

The two early returns only guard math.exp against overflow when $z$ is extreme; the mathematics is the last line, 1.0 / (1.0 + math.exp(-z)), which computes $\sigma(z)$ exactly.

What makes the sigmoid so convenient for training is that its derivative has a clean closed form in terms of the sigmoid itself. Write $\sigma(z) = (1 + e^{-z})^{-1}$ and differentiate by the chain rule:

$$\sigma'(z) = -(1 + e^{-z})^{-2}\cdot \frac{d}{dz}\!\left(1 + e^{-z}\right) = -(1 + e^{-z})^{-2}\cdot(-e^{-z}) = \frac{e^{-z}}{(1 + e^{-z})^2}.$$

Split that fraction into two factors and recognize each one:

$$\sigma'(z) = \frac{1}{1 + e^{-z}}\cdot\frac{e^{-z}}{1 + e^{-z}} = \sigma(z)\cdot\frac{e^{-z}}{1 + e^{-z}}.$$

The second factor simplifies because $\dfrac{e^{-z}}{1 + e^{-z}} = \dfrac{(1 + e^{-z}) - 1}{1 + e^{-z}} = 1 - \dfrac{1}{1 + e^{-z}} = 1 - \sigma(z)$. Hence

$$\boxed{\,\sigma'(z) = \sigma(z)\,\bigl(1 - \sigma(z)\bigr).\,}$$

This identity is why the code never computes an exponential during the backward pass: once it holds an activation value $a = \sigma(z)$, the local slope is just $a(1-a)$. Numerically the slope peaks at $z = 0$, where $\sigma(0) = 0.5$ and $\sigma'(0) = 0.5 \times 0.5 = 0.25$, and it decays toward zero as $z$ moves away from the origin: at $z = 2$, $\sigma(2)\approx 0.881$ and $\sigma'(2)\approx 0.881 \times 0.119 \approx 0.105$.

Why does the nonlinearity matter at all? Because without it, depth is an illusion. An affine map is a function of the form $\mathbf{x}\mapsto W\mathbf{x} + \mathbf{b}$. Compose two of them and the result is still affine:

$$W_2\bigl(W_1\mathbf{x} + \mathbf{b}_1\bigr) + \mathbf{b}_2 = (W_2 W_1)\,\mathbf{x} + (W_2\mathbf{b}_1 + \mathbf{b}_2),$$

which is just $W\mathbf{x} + \mathbf{b}$ with $W = W_2 W_1$ and $\mathbf{b} = W_2\mathbf{b}_1 + \mathbf{b}_2$. Stacking a hundred linear layers with nothing between them collapses to a single linear layer, so it can never draw anything a single line could not. Slipping a nonlinear squash between the layers breaks this collapse: the composition can now curve, and that curvature is exactly what a hard pattern like XOR demands [1].

The 2-4-1 network: the forward pass

The network in neural.py has the shape 2-4-1: two inputs (the coordinates of an XOR point), a hidden layer of four neurons, and one output neuron. Its parameters live in four blocks, initialized to small uniform random values so the four hidden units start out different from one another:

W1 = [[rng.uniform(-1, 1) for _ in range(din)] for _ in range(hidden)]
b1 = [rng.uniform(-1, 1) for _ in range(hidden)]
W2 = [rng.uniform(-1, 1) for _ in range(hidden)]
b2 = rng.uniform(-1, 1)

Read the names against the math. $W^{(1)}_{jk}$, stored as W1[j][k], is the weight from input $k$ into hidden unit $j$, so $W^{(1)}$ is a $4\times 2$ matrix; $b^{(1)}_j$ is hidden unit $j$'s bias b1[j]; $W^{(2)}_j$ is W2[j], the weight from hidden unit $j$ into the single output; and $b^{(2)}$ is the output bias b2. That is $8 + 4 + 4 + 1 = 17$ trainable numbers in all. Together they form a multilayer perceptron (MLP), the plainest feedforward neural network there is.

The forward pass computes the network's output for one input, left to right. For each hidden unit $j$,

$$z^{(1)}_j = \sum_{k} W^{(1)}_{jk}\, x_k + b^{(1)}_j, \qquad h_j = \sigma\!\bigl(z^{(1)}_j\bigr),$$

which in vector form is $\mathbf{h} = \sigma\!\bigl(W^{(1)}\mathbf{x} + \mathbf{b}^{(1)}\bigr)$, the sigmoid applied elementwise. The output neuron reads the hidden activations:

$$z^{(2)} = \sum_j W^{(2)}_j\, h_j + b^{(2)}, \qquad o = \sigma\!\bigl(z^{(2)}\bigr).$$

Every symbol appears verbatim in the code:

def forward(x):
    h_pre = [sum(W1[j][k] * x[k] for k in range(din)) + b1[j] for j in range(hidden)]
    h = [sigmoid(z) for z in h_pre]
    o = sigmoid(sum(W2[j] * h[j] for j in range(hidden)) + b2)
    return h, o

The list h_pre[j] is the pre-activation $z^{(1)}_j$; h[j] is the activation $h_j = \sigma(z^{(1)}_j)$, the point's new coordinate along hidden axis $j$; and o is the output activation $o = \sigma(z^{(2)})$, a number in $(0,1)$. A discrete prediction is read off by thresholding at one half: the companion's predict_mlp returns $1$ when o >= 0.5 and $0$ otherwise.

The loss and gradient descent

Training needs a single number that measures how wrong an output is, so that "less wrong" has a direction. For one example with output $o$ and target $y$ we use the squared-error loss

$$L = \tfrac12\,(o - y)^2 .$$

The factor of $\tfrac12$ is a convenience: differentiating $\tfrac12(o-y)^2$ with respect to $o$ gives $\partial L/\partial o = (o - y)$ with no leftover constant. The code measures the same quantity without the half, (o - target)**2; that only scales every gradient by a constant factor of $2$, which is indistinguishable from doubling the learning rate, so it changes nothing essential. The reported figure at the end of training is the mean squared error (MSE), the average of $(o-y)^2$ over the four examples.

To shrink the loss we use gradient descent. Collect all seventeen parameters into a single vector $\theta$. The gradient $\nabla_\theta L$ is the vector of partial derivatives $\partial L/\partial\theta_i$, and it points in the direction of steepest increase of $L$. To go downhill we step in the opposite direction:

$$\theta \leftarrow \theta - \eta\,\nabla_\theta L ,$$

where the learning rate $\eta > 0$ sets the step size. In neural.py, $\eta$ is the argument lr=0.5. A step that is too small crawls; one that is too large can overshoot the valley and diverge. The whole art of the update is captured in that minus sign: subtracting the gradient moves each parameter in the direction that most rapidly reduces the loss, for a small enough step.

Backpropagation, derived by the chain rule

The one remaining question is how to compute $\nabla_\theta L$ for a network of composed functions. The answer is backpropagation: the chain rule of calculus applied layer by layer, from the output back toward the inputs, reusing partial results as it goes [2]. The entire backward pass and update in neural.py is remarkably short:

for x, target in zip(X, y):
    h, o = forward(x)
    # output layer gradient (MSE through sigmoid)
    do = (o - target) * o * (1 - o)
    # hidden layer gradients
    dh = [do * W2[j] * h[j] * (1 - h[j]) for j in range(hidden)]
    # updates
    for j in range(hidden):
        W2[j] -= lr * do * h[j]
    b2 -= lr * do
    for j in range(hidden):
        for k in range(din):
            W1[j][k] -= lr * dh[j] * x[k]
        b1[j] -= lr * dh[j]

Derive each quantity in turn.

Output error. The convenient intermediate is the derivative of the loss with respect to the output pre-activation $z^{(2)}$, called the error signal or delta at the output, $\delta^{(2)}$. The chain rule threads through the output activation:

$$\delta^{(2)} = \frac{\partial L}{\partial z^{(2)}} = \frac{\partial L}{\partial o}\cdot\frac{\partial o}{\partial z^{(2)}} = (o - y)\cdot\sigma'\!\bigl(z^{(2)}\bigr) = (o - y)\,o\,(1 - o),$$

using $\partial L/\partial o = (o-y)$ from the loss and $\sigma'(z^{(2)}) = o(1-o)$ from the sigmoid identity, since $o = \sigma(z^{(2)})$. This line computes $\delta^{(2)}$:

do = (o - target) * o * (1 - o)

Output-layer weights and bias. Because $z^{(2)} = \sum_j W^{(2)}_j h_j + b^{(2)}$, the pre-activation depends linearly on each output weight and on the bias, giving $\partial z^{(2)}/\partial W^{(2)}_j = h_j$ and $\partial z^{(2)}/\partial b^{(2)} = 1$. Multiplying by $\delta^{(2)}$:

$$\frac{\partial L}{\partial W^{(2)}_j} = \delta^{(2)}\, h_j , \qquad \frac{\partial L}{\partial b^{(2)}} = \delta^{(2)} .$$

These are exactly the update terms W2[j] -= lr * do * h[j] and b2 -= lr * do: step each output weight opposite $\eta\,\delta^{(2)} h_j$ and the bias opposite $\eta\,\delta^{(2)}$.

Backprop into the hidden layer. The output pre-activation also depends on each hidden activation $h_j$, with $\partial z^{(2)}/\partial h_j = W^{(2)}_j$, so the loss's sensitivity to a hidden activation is

$$\frac{\partial L}{\partial h_j} = \delta^{(2)}\,W^{(2)}_j .$$

To turn that into the hidden error, the delta at hidden unit $j$, push it one more step back through that unit's own sigmoid, whose slope is $\sigma'(z^{(1)}_j) = h_j(1-h_j)$:

$$\delta^{(1)}_j = \frac{\partial L}{\partial z^{(1)}_j} = \frac{\partial L}{\partial h_j}\cdot\frac{\partial h_j}{\partial z^{(1)}_j} = \delta^{(2)}\,W^{(2)}_j\,\sigma'\!\bigl(z^{(1)}_j\bigr) = \delta^{(2)}\,W^{(2)}_j\,h_j\,(1 - h_j).$$

That is precisely the list comprehension dh[j] = do * W2[j] * h[j] * (1 - h[j]): three local slopes multiplied together, the output error do, how strongly unit $j$ feeds the output W2[j], and how responsive unit $j$'s sigmoid is h[j] * (1 - h[j]).

Hidden-layer weights and bias. Finally, $z^{(1)}_j = \sum_k W^{(1)}_{jk} x_k + b^{(1)}_j$ depends linearly on its weights and bias, so $\partial z^{(1)}_j/\partial W^{(1)}_{jk} = x_k$ and $\partial z^{(1)}_j/\partial b^{(1)}_j = 1$, giving

$$\frac{\partial L}{\partial W^{(1)}_{jk}} = \delta^{(1)}_j\, x_k , \qquad \frac{\partial L}{\partial b^{(1)}_j} = \delta^{(1)}_j ,$$

which the code applies as W1[j][k] -= lr * dh[j] * x[k] and b1[j] -= lr * dh[j].

The whole method is nothing more than the chain rule applied layer by layer. The single fact that makes it efficient is the reuse of $\delta^{(2)}$: it is computed once and then multiplied into every output-weight gradient and threaded into every hidden delta, rather than recomputed for each of the seventeen parameters. One forward pass and one backward pass produce every gradient, at a cost proportional to the number of weights, not to the number of weights squared.

Working one step by hand. Suppose for the input $\mathbf{x} = (0,1)$ with target $y = 1$ the network currently outputs $o = 0.7$, and one hidden activation feeding the output is $h_j = 0.6$. Then

$$\delta^{(2)} = (o - y)\,o\,(1-o) = (0.7 - 1)(0.7)(0.3) = (-0.3)(0.21) = -0.063 .$$

The update to that output weight is $-\eta\,\delta^{(2)}h_j = -(0.5)(-0.063)(0.6) = +0.0189$. The sign is exactly right: the output was too low (0.7 against a target of 1), the error signal came out negative, and subtracting a negative quantity raises the weight, which pushes the next output higher. Every one of the seventeen parameters is nudged by this same logic, and one nudge barely moves the loss. The loop runs all four examples for epochs=8000 passes, and over those tens of thousands of tiny corrections the folds settle into place.

Why the hidden layer conquers XOR

Geometrically, each hidden unit draws its own straight boundary in the input square and reports, softly through the sigmoid, which side a point falls on. With four hidden units, every input point is rewritten as four such soft answers, a new position in a four-dimensional hidden space. The learned weights arrange these boundaries so that in that reshaped space the two class-$1$ points land on one side of a plane and the two class-$0$ points on the other. The output neuron then does the very thing the linear classifier could not do in the original square: it draws one flat separating boundary. The network did not discover a cleverer straight line in the input picture, which the proof above forbids; it learned a new picture in which a straight line suffices.

Run the trained 2-4-1 network on the same four points that defeated the linear model, and the module prints:

2-4-1 MLP on XOR:      [0, 1, 1, 0] target [0, 1, 1, 0] -> correct: 4 / 4  (loss 0.0003)

Four of four, at a mean squared error of about $0.0003$, effectively zero, against the linear model's two of four. The exact pattern no straight line could cut, a single hidden layer of four sigmoid neurons learned from scratch.

XOR defeats a straight line but not a bent space: a single hidden layer of sigmoid neurons reshapes the four points until one plane separates them, backpropagation learns the bend, and the trained network scores four out of four at a loss of 0.0003. Original diagram by the authors, created with AI assistance.

Universal approximation

XOR is not a lucky special case. The universal approximation theorem states the general fact precisely: a feedforward network with a single hidden layer of sigmoidal units can approximate any continuous function on a compact (closed and bounded) subset of $\mathbb{R}^n$ to any desired accuracy, provided the hidden layer has enough units. Formally, for any continuous $f$ on a compact set $K$ and any tolerance $\varepsilon > 0$, there exist a width $m$ and parameters such that the network $g$ satisfies $\sup_{\mathbf{x}\in K}\lvert g(\mathbf{x}) - f(\mathbf{x})\rvert < \varepsilon$ (first established in 1989 [3]). In words: $\sup_{\mathbf{x}\in K}$ is the supremum (the least upper bound, met in the order chapter) taken over every point $\mathbf{x}$ in the region $K$, and $\lvert g(\mathbf{x}) - f(\mathbf{x})\rvert$ is the absolute size of the gap between the network and the target at that point, so the inequality says the worst gap anywhere in $K$ can be forced below any error tolerance $\varepsilon$ you name. This is the theoretical license for the whole enterprise: a network is not limited to lines or to any fixed family of shapes, but is a universal function approximator you fit to data.

The theorem is worth stating carefully because of what it does not promise. It is an existence result and is entirely non-constructive: it guarantees that suitable weights exist, but gives no recipe for finding them. It says nothing about how many hidden units the target function needs, and for some functions that number is astronomically large. And crucially, it is silent about training: it never claims that gradient descent, or any other algorithm, will actually locate the weights it promises. Universality is a statement about what networks can represent, not about what we can reliably learn.

Differentiable: the load-bearing word

Notice what every step of the pipeline had in common. The affine pre-activation, the sigmoid, and the squared-error loss are each differentiable: they have a well-defined slope everywhere. That single property is what let the chain rule assign blame backward and gradient descent improve the parameters. Remove differentiability at any step, replace the sigmoid with a hard $0/1$ threshold, say, and the gradient is zero almost everywhere and undefined at the jump, so backpropagation has nothing to grip and learning stalls. Differentiability is not a stylistic choice; it is the mechanism.

This is the exact hinge on which the rest of the series turns. The same descent that trained a net on XOR can train a net on anything you can express as a differentiable loss. Volume 4 makes logical constraints themselves differentiable: a hard rule such as "every professor is a researcher" is relaxed into a smooth truth value that carries a gradient, a penalty that is zero when the rule holds and grows as it is violated. Because that penalty is differentiable, a symbolic requirement can be trained into a network by this very backpropagation, sitting alongside the data loss in the same sum. The bend-and-descend engine derived in this chapter is, quite literally, the machinery the neuro-symbolic program will run on: make the thing you care about differentiable, then let the gradient find it.

The unsolved part

Two honest limits sit inside everything above, and neither is a detail.

The first is non-convexity. Gradient descent follows the local downhill direction, so it converges to a local minimum of $L$, a point where $\nabla_\theta L = 0$ and no small step improves the loss, but not necessarily the global minimum. For a linear model with squared error the loss surface is a single bowl (convex), so the local minimum is the global one, and the earlier linear and logistic fits were guaranteed to find the best answer. A network's loss surface is not a bowl. It is a high-dimensional landscape riddled with many local minima, saddle points, and flat plateaus, and which one descent lands in depends on the random initialization and the learning rate. Our XOR run reached a loss near $0.0003$, but a different seed can stall in a poorer basin. We can find a good solution; we cannot in general certify it is the best.

The second is opacity. Ask the trained network why it outputs $1$ for $(0,1)$, and the only faithful answer is a list of seventeen real numbers, $W^{(1)}, \mathbf{b}^{(1)}, W^{(2)}, b^{(2)}$. Nowhere among them is the sentence "output $1$ when exactly one input is on." The rule is real, but it is smeared across the weights rather than stated anywhere, so a network that is right still offers no auditable reason. Contrast the Horn rules of Part II: researcher ← professor wears its justification on the outside, and you can point to the line. This is the interpretability gap, and Volume 5 confronts it head-on, including the sharp worry of faithfulness, that any human-readable explanation we extract from a network may not match what the network actually computes.

Why it matters

This chapter turned a slogan, "neural networks learn," into mechanism. A single line cannot cut XOR, and we proved it rather than sketched it. A hidden layer of sigmoid neurons can, because a nonlinearity between affine maps lets the composition curve. The chain rule, applied layer by layer and reusing its error signals, computes every gradient in one backward pass, and gradient descent walks the parameters downhill until four of four points are classified at a loss near zero. Every equation here corresponds to a specific line of neural.py, so the theory is not decoration; it is the program. Above all, the property that makes the whole thing run, differentiability, is the same property the later volumes exploit to fuse logic and learning. Master this tiny network and the frontier chapters read as variations on one theme.

Key terms

• Affine pre-activation: the weighted sum plus bias $z = \mathbf{w}^\top\mathbf{x} + b$ a neuron forms before applying its activation.
• Activation: the output $a = \sigma(z)$ of a neuron after its nonlinearity; also the name of the function $\sigma$ itself.
• Sigmoid and its derivative: the logistic squash $\sigma(z) = 1/(1+e^{-z})$ mapping $\mathbb{R}$ into $(0,1)$, with the closed-form slope $\sigma'(z) = \sigma(z)(1-\sigma(z))$.
• Forward pass: computing the network's output from an input by evaluating each layer left to right.
• Loss: the scalar measuring how wrong an output is; here the squared error $L = \tfrac12(o-y)^2$, averaged into the mean squared error over the dataset.
• Gradient: the vector $\nabla_\theta L$ of partial derivatives of the loss with respect to the parameters, pointing in the direction of steepest increase.
• Learning rate: the step size $\eta$ in the update $\theta \leftarrow \theta - \eta\,\nabla_\theta L$; lr=0.5 in the code.
• Backpropagation: the chain rule applied layer by layer, from output to input, to compute every gradient in one backward pass.
• Delta / error signal: the derivative of the loss with respect to a pre-activation, $\delta^{(2)} = (o-y)\,o(1-o)$ at the output and $\delta^{(1)}_j = \delta^{(2)}W^{(2)}_j h_j(1-h_j)$ at the hidden layer; the reused quantity that makes backprop efficient.
• Universal approximation: the theorem that one hidden layer of enough sigmoidal units can approximate any continuous function on a compact set to arbitrary accuracy; an existence result, not a training guarantee.
• Non-convexity: the property that a network's loss surface has many local minima, so gradient descent finds a local, not necessarily global, optimum.

Where this leads

The hidden layer did something we should not gloss over: it turned two raw coordinates into four learned numbers, a new coordinate system in which the task came apart cleanly. That learned space is not a side effect; it is a representation worth studying in its own right. The next chapter, Embeddings: Meaning as Geometry, does exactly that. It takes the objects of the running example, the people and papers and topics of the academic world, and places them as points in a vector space where distance and direction encode meaning. Where this chapter bent space to make one pattern separable, the next asks what it means for meaning itself to live in geometry.