What Is Learning: Fitting Functions to Data

📍 Where we are: Part III · Learning from Scratch — Chapter 8. Fixpoints closed our rules into a 47-atom model by running them until nothing new appeared; now we put the rulebook down and learn the mapping straight from examples instead.

Every chapter so far earned its answers the same way: write down facts and rules, then let an engine grind out everything that logically follows. Chapter 7 pushed that idea to its limit — a fixpoint, the state where running the rules once more changes nothing, closed our 23 base facts into a 47-atom model. That is one whole way to be intelligent: derive the answer. This chapter introduces the other way. Instead of rules that spell out what follows, we are handed examples — inputs paired with the answers we want — and we tune a pile of numbers until the machine's guesses match. No rule for "professor" is ever written down; the machine reads it off the data.

The simple version

Imagine tuning an old radio. You do not know the station's frequency; you just turn the dial and listen. When the music gets clearer you keep turning that way; when it gets worse you turn back. You are not deriving the frequency from a formula — you are adjusting until the signal is best. Machine learning is that dial, scaled up: the dials are numbers inside a formula, "best" is measured by how wrong the guesses are, and the turning is done automatically, one small nudge at a time, always in the direction that cuts the noise.

What this chapter covers

• The shift from deriving to fitting — logic hands you rules and computes the consequences; supervised learning hands you examples of features to a label and asks you to recover the rule.
• Model, loss, and minimization — a model is a parametric function, a loss measures how wrong it is on the examples, and learning is nothing more than making that loss small, every symbol decoded before it is used.
• Gradient descent, derived on the line — the mean-squared-error gradient worked out step by step with the chain rule, matched line for line to the companion code, then checked by hand: the first step lands exactly where the program says it does.
• Two full numeric traces — the line fit falling from a loss of 44.182 to 0.0214 epoch by epoch, and the logistic classifier's loss dropping from ln 2 to 0.010, with a per-person score-and-probability table.
• Logistic regression on the academic world — a classifier that separates professors from students using two honest numbers read off the advises relation, getting all five people right, and why its decision boundary must be slanted.
• Generalization versus memorization, the honest catch of being right for a shallow reason, and the three families of learning.

From deriving to fitting

In logic the direction of travel is fixed: rules plus facts produce answers. Give the engine advises(alice, bob) and the rule that a professor advises a student, and it derives what follows — it never questions the rule. Learning runs the arrow backwards. We are given only the answers — five people, each already tagged professor or student in kb.py (lines 40–44) — and a handful of measurable features about them, and we must recover the mapping from features to label that a rule would have encoded. This is supervised learning: every training example is an input paired with its correct output, and the goal is a function that reproduces those outputs and, we hope, extends to inputs it has never seen. The word "supervised" is literal — a teacher has already marked every example with the right answer, and the learner's only job is to imitate that teacher well enough to stand in for it later [1].

The companion module examples/logic/learning.py makes the contrast explicit in its own docstring (lines 3–7): Volume 1's logic modules derive conclusions from rules, while this one "starts from examples and tunes numbers until predictions match." Same academic world, opposite engine. The 23 base facts and 47-atom closure from the fixpoint chapter are still the ground truth; here we ask a pile of numbers to reconstruct one slice of that world (who is a professor) from clues, rather than deducing it.

A model, a loss, and the thing we minimize

Three ideas do all the work, so meet them plainly before any symbols.

A model is a parametric function: a fixed shape with adjustable knobs. The simplest is a straight line. Writing $x$ for the input (a single real number), $y$ for the output the model produces, and calling the two knobs $w$ (the weight, or slope) and $b$ (the bias, or intercept), the model is

$$\hat{y} \;=\; w\,x + b .$$

The hat on $\hat{y}$ marks it as the model's guess, to be kept separate from the true answer $y$ we are trying to match. The shape — a line — is chosen by us; the knobs $w$ and $b$ are the parameters, and they are what learning adjusts. Different settings of $(w, b)$ give different lines; learning is the search for the pair that fits best. It is convenient to collect all the tunable numbers of any model into one list, the parameter vector $\theta = (\theta_1, \ldots, \theta_d)$ (the Greek letter theta), where $d$ counts the parameters; for the line $\theta = (w, b)$ and $d = 2$.

"Best" needs a number, and that number is the loss: a single non-negative value that grows when the model is wrong and shrinks when it is right, with zero meaning perfect. For fitting a line the natural choice is mean squared error (MSE). Suppose we have $n$ training examples, indexed by $i$ running from $1$ to $n$; example $i$ is the pair $(x_i, y_i)$, an input and its true answer. Then

$$L(w, b) \;=\; \frac{1}{n} \sum_{i=1}^{n} \big(w\,x_i + b - y_i\big)^2 .$$

Decode the symbols one at a time. The subscript $i$ picks out one example, so $x_i$ is the $i$-th input and $y_i$ its truth. The capital sigma $\sum_{i=1}^{n}$ means "add up the following expression for every $i$ from $1$ to $n$." The quantity inside, $w x_i + b - y_i$, is the model's guess minus the truth on example $i$: its error there, which we call the residual $r_i$. Squaring it, $r_i^2$, makes overshooting and undershooting both count as positive error and penalizes big misses far more than small ones. Dividing by $n$ averages over the examples, so the loss does not simply grow with the size of the dataset. A loss of zero means the line passes exactly through every point.

With those two pieces, learning is just minimization: find the parameters that make the loss as small as possible. That is the entire game — pick a model, define a loss, and descend [2]. The rest of this chapter is what "descend" means, and what it feels like when it works and when it is quietly cheating.

Learning in one picture: define how wrong you are (left, the loss valley), and step the parameters downhill until a line fits its points (center) or a boundary separates two classes (right). Original diagram by the authors, created with AI assistance.

Walking downhill: gradient descent

How do we minimize the loss without knowing the answer in advance? We use the loss's own slope. For a function of one variable the slope at a point is its derivative; for a function of several parameters the slope in the direction of one particular parameter, holding the others fixed, is a partial derivative, written with the rounded $\partial$ instead of the straight $d$. Collecting the partial derivatives with respect to every parameter into one vector gives the gradient, written $\nabla_\theta L$ (the upside-down triangle $\nabla$ is read "del" or "grad"):

$$\nabla_\theta L \;=\; \left(\frac{\partial L}{\partial \theta_1},\; \ldots,\; \frac{\partial L}{\partial \theta_d}\right) .$$

The gradient points in the direction that makes the loss grow fastest, so stepping the opposite way makes it shrink fastest. Gradient descent is that step, taken over and over: measure the gradient, nudge every parameter a little way against it, repeat. The update rule, with the arrow $\leftarrow$ meaning "replace the left side by the right," is

$$\theta \;\leftarrow\; \theta - \eta\,\nabla_\theta L ,$$

where $\eta$ (the Greek letter eta) is the learning rate, a small positive number setting how far we move on each step, and one full pass over the data is an epoch. Why the negative gradient is the fastest way down, and how large $\eta$ may be before the steps overshoot and diverge, are the exact questions the next chapter opens; here we take the direction on faith and watch it work, deriving only the one gradient we need to read the code.

For the line, that gradient is two partial derivatives, and the chain rule delivers them cleanly. Write the residual $r_i = w x_i + b - y_i$, so that $L = \frac{1}{n}\sum_i r_i^2$. The chain rule says the derivative of a composed expression is the derivative of the outside times the derivative of the inside. The outside is the square, whose derivative is $2 r_i$; the inside is $r_i$, whose derivative with respect to $w$ is $x_i$ (because $r_i$ contains $w$ only through the term $w x_i$) and with respect to $b$ is $1$. Putting the pieces together, term by term:

$$\frac{\partial L}{\partial w} \;=\; \frac{1}{n}\sum_{i=1}^{n} \frac{\partial (r_i^2)}{\partial w} \;=\; \frac{1}{n}\sum_{i=1}^{n} 2\,r_i\,\frac{\partial r_i}{\partial w} \;=\; \frac{1}{n}\sum_{i=1}^{n} 2\big(w x_i + b - y_i\big)\,x_i ,$$ $$\frac{\partial L}{\partial b} \;=\; \frac{1}{n}\sum_{i=1}^{n} 2\,r_i\,\frac{\partial r_i}{\partial b} \;=\; \frac{1}{n}\sum_{i=1}^{n} 2\big(w x_i + b - y_i\big) .$$

These two formulas are the code. The whole linear learner, copied from fit_line in examples/logic/learning.py (lines 22–37), keeps the mechanics in pure Python with nothing hidden inside a library:

def fit_line(xs, ys, lr: float = 0.02, epochs: int = 2000):
    """Return (w, b, loss_start, loss_end) for the least-squares line through
    the data, found by gradient descent on the mean squared error."""
    w, b = 0.0, 0.0
    n = len(xs)

    def mse(w, b):
        return sum((w * x + b - y) ** 2 for x, y in zip(xs, ys)) / n

    loss_start = mse(w, b)
    for _ in range(epochs):
        dw = sum(2 * (w * x + b - y) * x for x, y in zip(xs, ys)) / n
        db = sum(2 * (w * x + b - y) for x, y in zip(xs, ys)) / n
        w -= lr * dw
        b -= lr * db
    return w, b, loss_start, mse(w, b)

Line 33, dw = sum(2 * (w * x + b - y) * x for ...) / n, is $\partial L / \partial w$ transcribed symbol for symbol; line 34 is $\partial L / \partial b$; and lines 35–36, w -= lr * dw and b -= lr * db, are the update rule $\theta \leftarrow \theta - \eta\,\nabla_\theta L$ split across the two coordinates, with lr playing the role of $\eta$. No calculus is hidden in a library; the loop is the boxed formulas.

The first step, by hand

Watch the very first step, starting where the code starts, at $w = 0,\ b = 0$. Then every residual is $r_i = 0 + 0 - y_i = -y_i$, so the starting loss is just the average of the squared answers:

$$L(0,0) = \frac{1}{5}\sum_{i=1}^{5} y_i^2 = \frac{4.41 + 15.21 + 38.44 + 60.84 + 102.01}{5} = \frac{220.91}{5} = 44.182 ,$$

matching loss_start exactly. The five points in the file are $\mathbf{x} = (1,2,3,4,5)$ and $\mathbf{y} = (2.1, 3.9, 6.2, 7.8, 10.1)$, which hug the line $y = 2x$. The two gradient pieces at the origin are

$$\frac{\partial L}{\partial w} = \frac{2}{5}\sum_i (-y_i)\,x_i = -\frac{2}{5}(2.1\!\cdot\!1 + 3.9\!\cdot\!2 + 6.2\!\cdot\!3 + 7.8\!\cdot\!4 + 10.1\!\cdot\!5) = -\frac{2}{5}(110.2) = -44.08 ,$$ $$\frac{\partial L}{\partial b} = \frac{2}{5}\sum_i (-y_i) = -\frac{2}{5}(30.1) = -12.04 .$$

Both are negative, meaning the loss falls as $w$ and $b$ rise, so gradient descent will raise them. With $\eta = 0.02$ the first update is

$$w \leftarrow 0 - 0.02\,(-44.08) = 0.8816 , \qquad b \leftarrow 0 - 0.02\,(-12.04) = 0.2408 ,$$

which is exactly the $(w, b)$ the program holds after one epoch. One step has already pulled the slope most of the way from $0$ to its target near $2$. The full run continues, and the loss falls monotonically:

epoch	$w$	$b$	loss $L$
0	0.00000	0.00000	44.18200
1	0.88160	0.24080	12.30296
2	1.34640	0.36618	3.45685
5	1.78989	0.47905	0.13083
10	1.86646	0.48410	0.05595
100	1.92458	0.28618	0.03155
500	1.98566	0.06565	0.02144
2000	1.99000	0.05000	0.02140

By epoch 2000 the program prints [2]:

line: y = 1.990 x + 0.050   (loss 44.182 -> 0.0214)

The loss fell from 44.182 to 0.0214, and the recovered slope is 1.990 with intercept 0.050: starting from $w = 0, b = 0$ and seeing nothing but five noisy points, the learner rediscovered the slope of roughly two that generated them. That is fitting a function to data in miniature: no formula for the line was supplied, only the data and the instruction "be less wrong." Notice too that the loss does not reach zero. It settles at $0.0214$ because the points are noisy and no straight line threads all five exactly; gradient descent finds the best line, not a perfect one, and the residual loss is the noise the line cannot explain.

Classifying professors from students

Predicting a continuous number is regression. Predicting a discrete label — professor or student — is classification, and its workhorse is logistic regression. The trick is to keep the linear part but pass it through a squashing function so the output reads as a probability. Now $\mathbf{w}$ and $\mathbf{x}$ are no longer single numbers but short lists — one weight per feature, one feature value per person — and the centered dot in $\mathbf{w}\cdot\mathbf{x}$ is the dot product: multiply each weight by its matching feature and add the results, so with two features $\mathbf{w}\cdot\mathbf{x} = w_1 x_1 + w_2 x_2$. The linear score, or logit, is $z = \mathbf{w}\cdot\mathbf{x} + b$. The squashing function is the sigmoid $\sigma$ (the Greek letter sigma), which bends any real number into the open interval between $0$ and $1$:

$$\sigma(z) \;=\; \frac{1}{1 + e^{-z}} .$$

Here $e \approx 2.718$ is Euler's number and $e^{-z}$ is the exponential function, $e$ raised to the power $-z$; the wavy $\approx$ reads "approximately equal to." Because $e^{-z}$ is always positive, the denominator $1 + e^{-z}$ always exceeds $1$, so the fraction stays strictly between $0$ and $1$, which is exactly why its output can be read as a probability: the model's estimated chance that the label is $1$ (professor). A big positive $z$ drives $e^{-z}$ toward $0$ and the output toward $1$ ("almost certainly a professor"); a big negative $z$ drives it toward $0$ ("almost certainly a student"); and $z = 0$ gives $\sigma(0) = 1/2$ exactly, the fence between the two. The code, sigmoid in learning.py (lines 41–46), is the formula plus a numerical guard:

def sigmoid(z: float) -> float:
    if z < -60:
        return 0.0
    if z > 60:
        return 1.0
    return 1.0 / (1.0 + math.exp(-z))

The two early returns are a safety net: once $z$ runs past roughly $\pm 60$ the sigmoid is already indistinguishable from $0$ or $1$, and evaluating math.exp(-z) there would overflow the floating-point range, so we hand back the limiting value directly.

For a true label $y$, which is either $0$ or $1$ (written $y \in \{0, 1\}$, where the symbol $\in$ reads "is an element of"), the loss is no longer squared error but binary cross-entropy (BCE):

$$\ell \;=\; -\big[\,y\ln p + (1 - y)\ln(1 - p)\,\big], \qquad p = \sigma(z) .$$

Here $\ln$ is the natural logarithm, the inverse of the exponential: $\ln(e^k) = k$, and $\ln$ of a number between $0$ and $1$ is negative, growing without bound toward $-\infty$ as the number approaches $0$. Read the loss as a penalty on the probability assigned to the correct class. If $y = 1$ only the first term survives and $\ell = -\ln p$, which is $0$ when $p = 1$ and blows up as $p \to 0$; if $y = 0$ only the second survives and $\ell = -\ln(1 - p)$, the mirror image. That formula gives the loss $\ell_i$ on a single example $i$; the number the training loop tracks is their mean over all $n$ examples, $L = \frac{1}{n}\sum_{i=1}^{n} \ell_i$, exactly as MSE averaged the squared residuals, and it is this mean $L$ that fills the loss column of the tables below. So a confident-but-wrong guess (probability near $0$ on the true class) makes the logarithm blow up into a huge loss, while a confident-right one drives it toward $0$. That asymmetry — punishing a confident wrong answer far more harshly than a hesitant one — is the right shape when the target is a yes-or-no label, and it is why cross-entropy rather than squared error is the standard companion of the sigmoid [2].

We can pin the starting loss exactly. The classifier begins at $\mathbf{w} = (0,0),\ b = 0$, so every logit is $z = 0$ and every prediction is $p = \sigma(0) = 1/2$: the untrained neuron is maximally unsure about everyone. Each example then contributes $-[\,y\ln\tfrac12 + (1-y)\ln\tfrac12\,] = -\ln\tfrac12 = \ln 2$, so the average loss is $L = \ln 2 \approx 0.69315$, which is exactly what _bce returns at $\mathbf{w} = (0,0),\ b = 0$: the epoch-0 loss the training loop starts from. The training loop, fit_logistic (lines 67–85), accumulates the gradient over all five people and steps once per epoch:

for xi, yi in zip(X, y):
    p = sigmoid(sum(wj * xj for wj, xj in zip(w, xi)) + b)
    err = p - yi
    for j in range(dim):
        gw[j] += err * xi[j]
    gb += err
w = [wj - lr * gj / n for wj, gj in zip(w, gw)]
b -= lr * gb / n

The one line to hold onto is err = p - yi (line 79): the gradient of cross-entropy through the sigmoid collapses to the clean error signal $p - y$, the predicted probability minus the truth, and the per-parameter gradient is simply that error times the feature, $\partial \ell / \partial w_j = (p - y)\,x_j$. Line 81, gw[j] += err * xi[j], is that product summed over the batch, and line 82, gb += err, is the bias gradient. Why the messy derivative of $\ln \sigma(z)$ collapses to something so tidy is the small miracle the next chapter derives in full; for now we take the result and check it moves the way it should.

Reading the features off the advises relation

Now the honest part: what does the classifier actually look at? Two numbers per person, read straight off the advises facts of the running example by advise_features (lines 49–64), which counts an out-degree on line 59 and an in-degree on line 60:

def advise_features():
    """Two features per person, straight from the advises relation:
    [out-degree = how many you advise, in-degree = how many advise you].
    Label 1 = professor, 0 = student. Neither feature alone separates the two
    groups, so the classifier leans on both."""
    people = ["alice", "bob", "carol", "dave", "erin"]
    out_deg = {p: 0 for p in people}
    in_deg = {p: 0 for p in people}
    for f in FACTS:
        if f[0] == "advises":
            out_deg[f[1]] += 1
            in_deg[f[2]] += 1
    profs = {"alice", "bob"}
    X = [[float(out_deg[p]), float(in_deg[p])] for p in people]
    y = [1 if p in profs else 0 for p in people]
    return people, X, y

The out-degree $x_1$ counts how many people you advise; the in-degree $x_2$ counts how many advise you. When we need to name a single feature value we write $x_{ij}$ for feature $j$ of example $i$, so $x_{i1}$ is person $i$'s out-degree and $x_{i2}$ their in-degree. There are exactly four advises facts in kb.py (lines 47–50): alice→bob, bob→carol, bob→dave, carol→erin. Tallying them by hand, each fact adds one to the adviser's out-degree and one to the advisee's in-degree:

person	out-degree $x_1$	in-degree $x_2$	label $y$
alice	1 (advises bob)	0 (advised by none)	prof (1)
bob	2 (advises carol, dave)	1 (advised by alice)	prof (1)
carol	1 (advises erin)	1 (advised by bob)	student (0)
dave	0	1 (advised by bob)	student (0)
erin	0	1 (advised by carol)	student (0)

This is exactly the table the code builds. Each person is now a point in a two-number feature space, and the labels come from the profs set on line 61 (alice and bob), matching the base professor facts in kb.py. Notice already that no single feature sorts everyone: alice and carol share $x_1 = 1$ but have opposite labels, and bob and carol share $x_2 = 1$ but have opposite labels. Any correct rule has to combine both numbers.

The fit, and the decision boundary

Running fit_logistic with its defaults ($\eta = 0.3$, $3000$ epochs), the loss falls from $\ln 2$ toward zero:

epoch	$w_1$	$w_2$	$b$	loss $L$
0	0.0000	0.0000	0.0000	0.69315
1	0.0600	−0.0600	−0.0300	0.66662
10	0.5464	−0.5043	−0.2257	0.49320
100	2.4575	−2.4546	−0.8210	0.15900
1000	5.7214	−6.3606	−1.9076	0.02774
3000	7.7549	−8.5903	−2.7587	0.00987

The epoch-1 row is worth checking by hand, and it confirms the error-signal rule. At the origin every $p = 0.5$, so each person's error is $e = p - y = 0.5 - y$: that is $-0.5$ for the two professors and $+0.5$ for the three students. Summing the weight gradient over the batch, $\sum_i e_i x_{i1} = (-0.5)(1) + (-0.5)(2) + (0.5)(1) + (0.5)(0) + (0.5)(0) = -1.0$ and $\sum_i e_i x_{i2} = (-0.5)(0) + (-0.5)(1) + (0.5)(1) + (0.5)(1) + (0.5)(1) = +1.0$, with bias sum $\sum_i e_i = +0.5$. Averaging over $n = 5$ and stepping by $\eta = 0.3$ gives $w_1 \leftarrow 0 - 0.3(-0.2) = 0.06$, $w_2 \leftarrow 0 - 0.3(0.2) = -0.06$, $b \leftarrow 0 - 0.3(0.1) = -0.03$ — exactly the epoch-1 row. Already the direction is right: the out-degree weight rose (advising many is evidence of professor) and the in-degree weight fell (being advised is evidence of student).

At epoch 3000 the program prints:

logistic: weights=[7.75, -8.59] bias=-2.76 loss=0.010
  alice  label=prof    pred=prof
  bob    label=prof    pred=prof
  carol  label=student pred=student
  dave   label=student pred=student
  erin   label=student pred=student

All five classified correctly, loss driven to 0.010. The learned rule is readable in the weights: out-degree earns a strong positive weight (+7.75, advising people points toward professor), in-degree a strong negative one (−8.59, being advised points hard toward student). The model discovered a rule it was never told, expressed not in logic but in two numbers and a bias.

To see why it works, feed each person's features through the trained logit $z = 7.7549\,x_1 - 8.5903\,x_2 - 2.7587$ and squash with the sigmoid. The decision boundary is the set of points where $z = 0$, i.e. $p = 0.5$; the classifier predicts professor when $z > 0$ and student otherwise. The code's own predict_logistic (lines 98–99) computes exactly these:

person	$x_1$	$x_2$	logit $z$	probability $p = \sigma(z)$	prediction
alice	1	0	+4.996	0.9933	prof
bob	2	1	+4.161	0.9846	prof
carol	1	1	−3.594	0.0268	student
dave	0	1	−11.349	0.000012	student
erin	0	1	−11.349	0.000012	student

The two professors land above the boundary with probabilities near $1$; the three students land far below it, dave and erin most emphatically because they advise no one and are advised. The boundary line $7.7549\,x_1 - 8.5903\,x_2 - 2.7587 = 0$ is slanted in the feature plane precisely because both weights are nonzero: it tilts to keep alice and bob on one side and carol, dave, erin on the other, which no horizontal or vertical line (one feature alone) could do.

Generalization versus memorization

A loss of $0.010$ on the training examples is not the goal. The goal is a model that is right on people it has never seen. The gap between those two is the difference between generalization — capturing the real pattern — and memorization — merely recording the answers it was shown. A lookup table has zero training loss and learns nothing; it fails on the first new example.

The standard referee is the train/test split: hold back some examples, train only on the rest, and grade the model on the held-out ones it never touched. A model that memorized will ace the training set and stumble on the test set; a model that generalized scores well on both. This one habit is the backbone of honest machine learning, and later chapters lean on it constantly [1]. Our academic world has only five people, so there is nothing left over to hold out — which is exactly why "all five correct" here is a demonstration of the mechanics, not a claim about skill on the wider world.

The honest catch: right for a shallow reason

Look again at what separates carol from alice. Both advise exactly one person ($x_1 = 1$), so on that feature they are identical; the model tells them apart entirely on the other feature — carol is advised ($x_2 = 1$), alice is not ($x_2 = 0$) — leaning on that large negative weight. In-degree is the classifier's favorite handle: at −8.59 it is the biggest-magnitude weight in the model, and it reads "someone advises you" as strong evidence for student.

But no single feature separates all five people, and being advised is not a clean proxy for "student." bob is advised too ($x_2 = 1$) and is nonetheless a professor; a rule keyed on in-degree alone would misclassify him — four right out of five, not five. What rescues bob is the other feature: he advises two people ($x_1 = 2$), and out-degree's positive weight is large enough to swamp his in-degree penalty and push him back onto the professor side. His logit works out to $z = 7.7549(2) - 8.5903(1) - 2.7587 \approx +4.16$ (about $+4.15$ with the printed two-decimal weights), comfortably positive, so $p = 0.985$. carol makes the mirror-image point — she advises someone ($x_1 = 1$) yet is a student, and her logit $z = 7.7549 - 8.5903 - 2.7587 \approx -3.59$ lands her safely on the student side. This is exactly why the boundary is slanted rather than flat: both axes carry weight, and the perfect five-out-of-five comes from combining them, not from any single shortcut.

Even so, the model leans hardest on that in-degree proxy, and "someone advises you" is a shallow stand-in for "student," not the meaning of it. A newly hired professor whom no one advises, or a senior student advising juniors, would break the pattern instantly — the model would be confidently wrong, and wrong for a reason it never understood. This is the recurring hazard of pure fitting: a learner can be right for the wrong reason, keying on a surface correlation that holds in the data but not in the world. Volume 5 gives this failure a name and a whole treatment — reasoning shortcuts, where a model reaches the correct answer through a concept that is not the one we meant — and it is one of the sharpest arguments for wiring learning back together with the logic we set aside in this chapter.

The three families of learning

What we did — learning from labeled examples — is supervised learning, and it is one of three classical families. In unsupervised learning there are no labels at all: the learner is handed inputs and must find structure on its own, for instance clustering our five people by shared institution (the affiliated facts group them into mit and cmu) without ever being told the groups. In reinforcement learning there are no fixed answers either, only rewards: an agent takes actions, receives feedback, and adjusts to maximize reward over time — the frame behind game-playing and robot control. All three share the DNA of this chapter — a model, an objective to optimize, and a search over parameters — but they differ in what the teacher provides: the right answer, no answer, or a score [3].

The unsolved part

The uncomfortable question learning cannot answer from the training data alone is: has it actually learned, or merely memorized? With a low loss and no held-out examples — our exact situation with five people — the two are indistinguishable. Worse, the difficulty is not just a shortage of data; it is a matter of principle. Fitting the training examples perfectly guarantees nothing about unseen ones unless you assume something about how the world is shaped — that nearby inputs deserve nearby answers, that the true pattern is simpler than the noise. Change that assumption and a different model fits the same points equally well while predicting the opposite on new ones. There is no assumption-free way to prefer one; the honesty is in stating the assumption, testing on held-out data, and staying suspicious of a model that is right for a reason you cannot articulate [1]. Our classifier passes every training example and still leans hardest on the shallow "being advised" signal — a live reminder that a low loss is a necessary sign of learning, never a sufficient one.

Why it matters

This chapter installs the second of the two engines the rest of the series fuses. Logic derives; learning fits — and neuro-symbolic AI is the discipline of making them work as one system. Nearly every neural model you will meet, however large, is built from the two pieces assembled here: a parametric function and a loss driven down by gradient descent, so logistic regression on five people is, in miniature, what a network with billions of parameters does at scale. And the honest catch is why the "symbolic" half never goes away: a pure learner can ace its examples by latching onto a surface signal, so the structure and guarantees of logic are exactly what fitting cannot supply for itself. Knowing where fitting succeeds — and where it is silently right for the wrong reason — is what separates a model that reasons from one that only pattern-matches.

Key terms

• Supervised learning — fitting a function from labeled examples, each an input paired with its correct output.
• Feature — a measurable input to the model (here, out-degree and in-degree read off the advises relation).
• Model / parametric function — a fixed functional shape with adjustable parameters $\theta$; the line $\hat{y} = wx + b$ is the simplest.
• Weight and bias — the parameters $w$ (slope) and $b$ (intercept) that learning adjusts.
• Loss function — a single non-negative number measuring how wrong the model is; mean squared error (MSE) for regression, binary cross-entropy (BCE) for classification.
• Residual — the signed gap $r_i = wx_i + b - y_i$ between a prediction and its truth on example $i$; MSE averages its square.
• Gradient descent — the core loop: read the loss's slope, the gradient $\nabla_\theta L$, and step every parameter against it by a learning rate $\eta$, once per epoch, via $\theta \leftarrow \theta - \eta\,\nabla_\theta L$.
• Linear regression — fitting a straight line to predict a continuous number.
• Logistic regression — classification by passing the logit $z = \mathbf{w}\cdot\mathbf{x} + b$ through the sigmoid $\sigma(z)$ to produce a probability, then thresholding it at $0.5$.
• Error signal — the quantity $p - y$ (prediction minus truth) that cross-entropy's gradient collapses to; it drives each weight update.
• Decision boundary — the surface $z = 0$ in feature space where the classifier flips label; slanted here because both features carry weight.
• Generalization versus memorization — capturing the real pattern (works on unseen inputs) versus recording the answers (fails on them); refereed by a train/test split.
• Reasoning shortcut — a model reaching the right answer through a shallow proxy rather than the intended concept; the "being advised" signal is a small example.
• Unsupervised / reinforcement learning — learning structure from unlabeled data, and learning from rewards rather than labeled answers.

Where this leads

We defined learning as descent and derived the one gradient the line fit needs, but we took two things on faith: that stepping against the gradient is the fastest way down, and that the learning rate $\eta$ can be trusted not to overshoot. We also stated, without proof, that cross-entropy's gradient through the sigmoid collapses to the clean error signal $p - y$. The next chapter, Gradient Descent, opens that engine in full: it defines the loss as a surface over weight space, proves with the Cauchy–Schwarz inequality that the negative gradient is the unique steepest descent, derives the exact interval of safe learning rates from a one-dimensional recurrence, and works the cross-entropy cancellation that makes $p - y$ appear — the same two fits from learning.py, now taken apart to the last derivative.

---

Companion code: examples/logic/learning.py implements both worked examples, fit_line (regression, mean squared error) and fit_logistic (classification, cross-entropy), in pure Python with the gradients written out by hand exactly as derived above. Run python3 examples/logic/learning.py to reproduce every number in this chapter; the acceptance harness examples/logic/validate.py checks that both fits reach their target losses and that the classifier labels all five people correctly.