Embeddings: Meaning as Geometry

📍 Where we are: Part III · Learning from Scratch — Chapter 11. The previous chapter, Neural Networks, built a differentiable function approximator that turns vectors into vectors and learns by gradient descent. But our academic world is made of symbols — alice, advises, p1 — not numbers. This chapter builds the missing on-ramp: a way to turn those symbols into vectors a network can compute with.

A neural network only knows how to add, multiply, and slide numbers downhill. It has no idea what alice is. So before any of the machinery from the last chapter can touch our knowledge base, we need a translator, something that hands each symbol a vector of real numbers and arranges those vectors so that their positions mean something. That translator is an embedding, and the trick that makes it work is to let geometry carry the meaning: things that behave alike end up near each other, and a relationship becomes a fixed motion through space. This chapter derives that idea from the ground up, ties every equation to the exact line of companion code that runs it, and then walks the mechanism by hand on the running example, from a single random initialization through one gradient step to the trained map.

The simple version

Imagine a map of a country you have never visited. You do not know the language, but you can still read a great deal from the map alone: cities that trade with each other sit close together, a river connects towns in a chain, and "one hour north" is the same arrow whether you start in the capital or a village. An embedding is a map of meaning drawn the same way. Every person, paper, and topic in our world becomes a dot on the map. "Who is similar to whom" becomes "which dots are close." And a relationship like advises becomes a single arrow you can lay down anywhere: put its tail on a professor and its head lands on their student.

What this chapter covers

• Symbols to vectors: what a distributed representation is, what a vector and its coordinates are, and why turning meaning into distance lets a network reason about a knowledge base it could never read as text.
• TransE, a relation as a translation: the model $\mathbf{h} + \mathbf{r} \approx \mathbf{t}$, and the score function derived as a Euclidean distance, decoded operator by operator and matched to the committed code.
• The margin-ranking loss, derived: the hinge objective against corrupted triples, its gradient worked out link by link with the chain rule, and each partial derivative matched to the exact update line.
• One gradient step, by hand: a full numeric trace from the real random initialization, showing a single true-versus-corrupt pair's objective fall from $7.34$ to $4.76$ in one step.
• The run: the loss curve, the sorted score table of all 18 true triples averaging exactly the reported $1.609$, and a ranked query that guesses who alice advises.
• What geometry buys and costs: open-world generalization set against the price of a scorer that is never silent but never certain, made concrete with the numbers.
• The unsolved part: why a flat point-and-arrow geometry cannot faithfully carry a logical hierarchy, shown numerically on bob's two students.

From symbols to vectors

Every earlier chapter in this volume treated alice as an opaque token: a thing you could match, chain rules over, or look up, but never measure. The forward-chaining engine of Part II ground our 23 base facts up to a derived model of 47 atoms and then stopped, because logic only ever states what follows; it has no notion that carol is in some way like dave. An embedding takes the opposite stance. It assigns every symbol a short list of real numbers, its coordinates, and insists that those coordinates, not any stored fact, are where the meaning lives.

A word on the objects themselves before we use them. A vector here is just an ordered list of $d$ real numbers, written $\mathbf{v} = (v_1, v_2, \ldots, v_d)$; the bold face marks it as a vector rather than a single number, and the count $d$ is the dimension. Each $v_i$ is a coordinate, and the subscript $i$ is an index that names which coordinate we mean, running from $1$ to $d$. In the companion code $d = 2$, so every symbol is a point $(v_1, v_2)$ on an ordinary flat plane, which is exactly why we can draw the whole model on paper. When we write $i = 1, 2$ we are just pointing at the first and second coordinate in turn.

This scheme, meaning-as-coordinates, is called a distributed representation: the meaning of a symbol is not one flag in one slot but a pattern spread across every coordinate, so that nearby patterns mean similar things [1]. The payoff is that a fuzzy human question, "who resembles whom?", becomes a crisp geometric one: similarity is distance. Two entities whose vectors sit close together are treated as similar; two that sit far apart are not.

Our companion file embeddings.py starts by gathering exactly the symbols it needs to place. It reads the knowledge base and keeps only the binary facts, the ones that relate two things, like advises or cites, because a relationship needs both a head (where it starts) and a tail (where it ends). This is _binary_triples, lines 23 to 29:

def _binary_triples():
    """Every binary fact (advises, cites, affiliated, about) as an (h, r, t)
    triple, plus the sorted entity and relation vocabularies."""
    triples = [(f[1], f[0], f[2]) for f in FACTS if len(f) == 3]
    ents = sorted({h for h, _, _ in triples} | {t for _, _, t in triples})
    rels = sorted({r for _, r, _ in triples})
    return triples, ents, rels

The filter if len(f) == 3 keeps only facts of arity two (a predicate plus two arguments), and the remap (f[1], f[0], f[2]) reorders each stored fact (predicate, arg1, arg2) into the triple order (head, relation, tail). The two set-builders then collect the sorted vocabulary of entities and of relations, where the symbol | is set union and {... for ...} builds a set (so duplicates vanish). On the academic world this yields exactly 18 triples over 13 entities and 5 relations. The 13 entities are the five people (alice, bob, carol, dave, erin), three papers (p1, p2, p3), two institutions (mit, cmu), and three topics (logic, ml, nesy); the 5 relations, sorted, are about, advises, affiliated, authored, cites. Here is the full triple list the code builds, in order:

#	triple $(h, r, t)$	reads as
0–3	advises chain	alice→bob, bob→carol, bob→dave, carol→erin
4–7	authored	alice→p1, bob→p1, carol→p2, dave→p3
8–9	cites	p2→p1, p3→p2
10–14	affiliated	alice→mit, bob→mit, carol→cmu, dave→cmu, erin→cmu
15–17	about	p1→logic, p2→nesy, p3→ml

Each triple is written $(h, r, t)$: read it as "head $h$ stands in relation $r$ to tail $t$," for example $(\texttt{alice}, \texttt{advises}, \texttt{bob})$. The symbol $\in$, which we will use shortly, reads "is an element of," so "$(h, r, t) \in T$" means the triple belongs to the set $T$ of known true facts. Our job is to place all 13 entities and all 5 relations as points, then judge every triple by geometry alone.

TransE: a relation is a translation

Which geometry? The simplest one that could possibly work, and the one that launched this whole field: TransE, short for translating embeddings [2]. Its single idea is that a relation is a translation, a fixed arrow you add to a point to move somewhere else.

Here is the notation, in plain words first. Write the head entity's vector as $\mathbf{h}$, the relation's vector as $\mathbf{r}$, and the tail entity's vector as $\mathbf{t}$. All three live in the same $d$-dimensional space. TransE demands that for a true triple,

$$\mathbf{h} + \mathbf{r} \;\approx\; \mathbf{t}.$$

The plus sign is ordinary vector addition: adding two vectors means adding them coordinate by coordinate, $(\mathbf{h} + \mathbf{r})_i = h_i + r_i$, which geometrically slides the point $\mathbf{h}$ along the arrow $\mathbf{r}$. The symbol $\approx$ reads "is approximately equal to," here meaning "should land almost exactly on." So if $(\texttt{alice}, \texttt{advises}, \texttt{bob})$ is true, then starting at alice's point and adding the advises arrow should drop you right next to bob's point. The same advises arrow, added to bob, should land near carol; added to carol, near erin. One relation, one arrow, reused everywhere.

The score, derived as a distance

The relation $\approx$ is an aspiration, not a measurement. To train and to test we need a number that says how nearly $\mathbf{h} + \mathbf{r}$ lands on $\mathbf{t}$. That number is the score, and it is the length of the leftover gap vector $\mathbf{h} + \mathbf{r} - \mathbf{t}$:

$$s(h, r, t) \;=\; \big\lVert\, \mathbf{h} + \mathbf{r} - \mathbf{t} \,\big\rVert \;=\; \sqrt{\, \sum_{i=1}^{d} \big(h_i + r_i - t_i\big)^2 \,}.$$

Three symbols to decode. The double bars $\lVert \cdot \rVert$ denote the Euclidean norm, the ordinary straight-line length of a vector, which by the Pythagorean theorem is the square root of the sum of its squared coordinates. The capital sigma $\sum_{i=1}^{d}$ is a summation: it says "add up the following expression as the index $i$ runs from $1$ to $d$," so $\sum_{i=1}^{2} c_i = c_1 + c_2$ in our two-dimensional case. The radical $\sqrt{\phantom{x}}$ is the square root. Put together: form the gap vector, square each of its coordinates, add the squares, take the root. A small score means the translation fit well (plausible); a large score means it did not (implausible). Lower is better, because a score is a penalty, not a reward.

Those two operations are exactly two functions in the file. The squared distance is _dist2 (lines 32 to 33), and score wraps it in a square root (lines 101 to 104):

def _dist2(a, b):
    return sum((ai - bi) ** 2 for ai, bi in zip(a, b))


def score(E, R, h, r, t) -> float:
    """Plausibility as distance: lower means the translation fits better."""
    dim = len(E[h])
    return math.sqrt(_dist2([E[h][i] + R[r][i] for i in range(dim)], E[t]))

E is the dictionary of entity vectors and R the dictionary of relation vectors. The comprehension [E[h][i] + R[r][i] for i in range(dim)] is the vector sum $\mathbf{h} + \mathbf{r}$ built coordinate by coordinate, _dist2 computes $\sum_i (h_i + r_i - t_i)^2$, and math.sqrt takes the root. Nothing here knows what advises means, only where the arrows point. One detail matters for the derivation to come: the training loop below will minimize the squared distance $\sum_i(\cdot)^2$ directly, skipping the square root, while score reports the rooted Euclidean distance. Because the square root is a strictly increasing function, it never changes the order of two scores, so ranking a true triple below a false one is the same problem whether we use the squared distance or its root. We drop the root during training only because it is cheaper and its derivative is simpler.

TransE draws every relation as one reusable arrow: the head's point plus the relation's arrow should land on the tail's point, so a short leftover distance means a plausible fact and a long one means an implausible fact. Original diagram by the authors, created with AI assistance.

Learning by ranking true triples above corrupted ones

We start with random arrows, so at first the geometry is nonsense. The file draws each starting coordinate uniformly from the interval $[-0.6, 0.6]$ and seeds the generator so the run is reproducible. Training fixes the arrows, but there is a subtlety. Our knowledge base lists only true facts; it never says $(\texttt{alice}, \texttt{advises}, \texttt{erin})$ is false. With no negative examples, a lazy model could shrink every vector to the origin, score everything zero, and declare all triples equally plausible. TransE avoids that collapse in two ways at once.

First, it manufactures negatives. For each true triple it builds a corrupted triple by swapping the tail for a random wrong entity $t_n$, then insists the true one score lower than the corrupt one by at least a fixed margin $\gamma$ (the Greek letter gamma), set to $1$ in the code. Second, it forbids the vectors from cheating by inflating to infinity: at the start of every epoch each entity vector is renormalized to unit length, projected onto the unit circle $\lVert \mathbf{e} \rVert = 1$, exactly as the original TransE prescribed [2]. That renormalization is normalize, lines 53 to 55, applied to every entity each epoch:

def normalize(v):
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]

It divides a vector by its own length $n$, so the result has length one. (The or 1.0 guards the impossible case of a zero-length vector, which cannot be normalized.) Only entities are renormalized; the relation arrows are left free to grow to whatever length best fits the translations. As a concrete instance, the first raw draw for alice is $\mathbf{a} = (0.4133, 0.3095)$, whose length is

$$n = \sqrt{0.4133^2 + 0.3095^2} = \sqrt{0.1708 + 0.0958} = \sqrt{0.2666} = 0.5163,$$

so normalization sends alice to $(0.4133/0.5163,\; 0.3095/0.5163) = (0.8004, 0.5995)$, a point on the unit circle since $0.8004^2 + 0.5995^2 = 1.0000 \approx 1$.

The margin-ranking objective and its gradient

Now the mathematics that drives the whole run. Collect the known true triples into a set $T$. For each true triple $(h, r, t) \in T$ we draw a corrupted tail $t_n$ and define two squared distances,

$$d^{+} \;=\; \big\lVert \mathbf{h} + \mathbf{r} - \mathbf{t} \big\rVert^2, \qquad d^{-} \;=\; \big\lVert \mathbf{h} + \mathbf{r} - \mathbf{t}_n \big\rVert^2,$$

read "$d$-plus" for the true triple and "$d$-minus" for the corruption. The margin-ranking loss, summed over the dataset, is

$$L \;=\; \sum_{(h,r,t)\in T} \max\!\big(0,\; \gamma + d^{+} - d^{-}\big).$$

The operation $\max(0, \cdot)$ is a hinge: it returns its argument when that argument is positive and returns zero otherwise. Read the whole term as a demand with an escape hatch. If the true triple already beats the fake by more than the margin, meaning $d^{-} - d^{+} \gt \gamma$, then $\gamma + d^{+} - d^{-} \lt 0$, the hinge clamps it to zero, and this triple contributes nothing to the loss: there is nothing to fix. Only triples that are not yet ranked correctly, or not by enough, incur a penalty and get a nudge. That single line of logic is the continue in the training loop, lines 79 to 88:

        for h, r, t in triples:
            # corrupt the tail with a random wrong entity
            tn = rng.choice(ents)
            while tn == t:
                tn = rng.choice(ents)
            hv, rv, tv, tnv = E[h], R[r], E[t], E[tn]
            d_pos = _dist2([hv[i] + rv[i] for i in range(dim)], tv)
            d_neg = _dist2([hv[i] + rv[i] for i in range(dim)], tnv)
            if margin + d_pos - d_neg <= 0:
                continue  # already ranked correctly by the margin

When the hinge is active, that is when $\gamma + d^{+} - d^{-} \gt 0$, the loss for this triple is $\ell = \gamma + d^{+} - d^{-}$, and we descend it by gradient descent, exactly the update rule $\theta \leftarrow \theta - \eta\, \nabla_\theta \ell$ from the Gradient Descent chapter (recall $\nabla$ is the gradient, the vector of partial derivatives, and $\theta$ collects the numbers being trained), with $\eta$ the learning rate (here lr = 0.05). The gradient is short to derive. Introduce the two leftover vectors coordinate by coordinate,

$$u_i \;=\; h_i + r_i - t_i \quad(\text{the } i\text{-th coordinate of }\mathbf{h}+\mathbf{r}-\mathbf{t}), \qquad v_i \;=\; h_i + r_i - t_{n,i}.$$

Then $d^{+} = \sum_i u_i^2$ and $d^{-} = \sum_i v_i^2$. Each squared distance is a sum of squares, so its partial derivative is found by the power rule and the chain rule, one coordinate at a time. Since $u_i$ increases at rate $1$ when either $h_i$ or $r_i$ increases and at rate $-1$ when $t_i$ increases,

$$\frac{\partial d^{+}}{\partial h_i} = 2 u_i, \qquad \frac{\partial d^{+}}{\partial r_i} = 2 u_i, \qquad \frac{\partial d^{+}}{\partial t_i} = -2 u_i,$$

and identically for the corruption, where $v_i$ depends on the corrupted tail $t_{n,i}$ at rate $-1$,

$$\frac{\partial d^{-}}{\partial h_i} = 2 v_i, \qquad \frac{\partial d^{-}}{\partial r_i} = 2 v_i, \qquad \frac{\partial d^{-}}{\partial t_{n,i}} = -2 v_i.$$

Now assemble $\ell = \gamma + d^{+} - d^{-}$. The margin $\gamma$ is a constant and drops out of every derivative. Collecting terms coordinate by coordinate:

$$\boxed{\; \frac{\partial \ell}{\partial h_i} = 2u_i - 2v_i, \quad \frac{\partial \ell}{\partial r_i} = 2u_i - 2v_i, \quad \frac{\partial \ell}{\partial t_i} = -2u_i, \quad \frac{\partial \ell}{\partial t_{n,i}} = +2v_i. \;}$$

Two things are worth pausing on. First, the head and the relation receive the identical gradient $2u_i - 2v_i$, because they enter the score only through their sum $\mathbf{h} + \mathbf{r}$, so nothing in the loss can tell them apart. Second, read the directions. Descending $-\eta\,\partial\ell/\partial t_i = +2\eta u_i$ moves the true tail $\mathbf{t}$ toward $\mathbf{h} + \mathbf{r}$ (pulling $d^{+}$ down), while $-\eta\,\partial\ell/\partial t_{n,i} = -2\eta v_i$ moves the fake tail away from $\mathbf{h} + \mathbf{r}$ (pushing $d^{-}$ up). This is precisely the code's comment "push d_pos down, d_neg up," and the four update lines 89 to 97 are the boxed formula transcribed verbatim:

            for i in range(dim):
                diff_pos = hv[i] + rv[i] - tv[i]
                diff_neg = hv[i] + rv[i] - tnv[i]
                # push d_pos down, d_neg up
                gh = 2 * diff_pos - 2 * diff_neg
                E[h][i] -= lr * gh
                R[r][i] -= lr * (2 * diff_pos - 2 * diff_neg)
                E[t][i] -= lr * (-2 * diff_pos)
                E[tn][i] -= lr * (2 * diff_neg)

Here diff_pos is $u_i$, diff_neg is $v_i$, gh is $\partial \ell / \partial h_i$, and each -= lr * (...) is one coordinate of $\theta \leftarrow \theta - \eta\,\partial\ell/\partial\theta$. The escape-hatch continue above implements the outer $\max(0,\cdot)$: when the pair is already well ranked, the gradient is zero and nothing moves. This is stochastic gradient descent (SGD), the same downhill-by-small-steps optimizer from Neural Networks, here applied to points on a map, one triple (and one freshly drawn corruption) at a time.

The full routine repeats this for thousands of passes, renormalizing every entity each round. Its signature (lines 36 to 38) fixes the hyperparameters:

def train_transe(dim: int = 2, lr: float = 0.05, epochs: int = 4000,
                 margin: float = 1.0, seed: int = 0, eval_seed: int = 7,
                 loss_curve=None):

Two dimensions (dim=2), 4000 epochs, a learning rate of 0.05, a margin $\gamma = 1.0$, and a fixed seed so the run is reproducible. The last two arguments serve the loss trace below: eval_seed fixes a single set of evaluation corruptions once, and passing a loss_curve dictionary records the per-epoch loss against that fixed set, using its own separate random generator so the measurement never disturbs the training itself.

One gradient step, by hand

Watch the mechanism move once. At the start of the first epoch, after renormalization, the four vectors involved in the very first triple $(\texttt{alice}, \texttt{advises}, \texttt{bob})$ are these real numbers from the seeded run, and the random corruption the generator draws for the tail is p2:

role	symbol	vector after epoch-0 normalization
head	alice = $\mathbf{h}$	$(0.8004,\; 0.5995)$
relation	advises = $\mathbf{r}$	$(0.4956,\; 0.5599)$
true tail	bob = $\mathbf{t}$	$(-0.3129,\; -0.9498)$
corrupt tail	p2 = $\mathbf{t}_n$	$(0.9080,\; 0.4189)$

First the translated head: $\mathbf{h} + \mathbf{r} = (0.8004 + 0.4956,\; 0.5995 + 0.5599) = (1.2960,\; 1.1594)$. Now the two squared distances, coordinate by coordinate:

$$d^{+} = (1.2960 + 0.3129)^2 + (1.1594 + 0.9498)^2 = 1.6089^2 + 2.1092^2 = 2.5887 + 4.4486 = 7.0373,$$ $$d^{-} = (1.2960 - 0.9080)^2 + (1.1594 - 0.4189)^2 = 0.3880^2 + 0.7405^2 = 0.1505 + 0.5484 = 0.6989.$$

Notice the embarrassment the random start hands us: the false tail p2 sits at squared distance $0.70$ from $\mathbf{h}+\mathbf{r}$, while the true tail bob sits far off at $7.04$. The margin test is $\gamma + d^{+} - d^{-} = 1 + 7.0373 - 0.6989 = 7.3384 \gt 0$, so the hinge is active and we take a step. Using $u = \mathbf{h}+\mathbf{r}-\mathbf{t} = (1.6089, 2.1092)$ and $v = \mathbf{h}+\mathbf{r}-\mathbf{t}_n = (0.3880, 0.7405)$, the per-coordinate updates with $\eta = 0.05$ are:

coordinate $i$	$\Delta \mathbf{h} = -\eta(2u_i-2v_i)$	$\Delta \mathbf{r} = -\eta(2u_i-2v_i)$	$\Delta \mathbf{t} = +\eta\,2u_i$	$\Delta \mathbf{t}_n = -\eta\,2v_i$
$1$	$-0.1221$	$-0.1221$	$+0.1609$	$-0.0388$
$2$	$-0.1369$	$-0.1369$	$+0.2109$	$-0.0741$

Applying them, alice moves to $(0.6783, 0.4626)$, the advises arrow shortens to $(0.3735, 0.4231)$, the true tail bob is pulled inward to $(-0.1520, -0.7389)$, and the fake tail p2 is shoved outward to $(0.8692, 0.3448)$. Recomputing after the step, the true squared distance falls from $7.04$ to $4.09$ and the corrupt one from $0.70$ to $0.33$, so the pair's objective $\gamma + d^{+} - d^{-}$ drops from $7.34$ to $4.76$ in this single step. One triple, one corruption, one nudge, and the geometry is already less wrong. Repeat that 18 times per epoch for 4000 epochs.

The run: geometry that learned the structure

Does laying down arrows actually recover the world's structure? First, the loss falls. Tracking the mean hinge loss on a fixed set of evaluation corruptions (drawn once from eval_seed = 7, so the yardstick itself never moves) as training proceeds shows the bulk of the learning happening in the first hundred epochs:

epoch	0	1	5	20	100	500	2000
mean hinge loss	$1.522$	$1.319$	$0.802$	$0.224$	$0.132$	$0.062$	$0.089$

The loss does not glide smoothly to zero the way the convex bowl of the Gradient Descent chapter did. It drops steeply through the first hundred epochs, then settles into a noisy band that mostly stays between about $0.05$ and $0.18$. Notice that the tabulated loss actually rises from $0.062$ at epoch 500 to $0.089$ at epoch 2000, a small uphill wobble a smoothly converging optimizer would never show. The reason is that training redraws random corruptions and renormalizes every entity each epoch, so the vectors keep jittering, and even this fixed-corruption yardstick wobbles rather than gliding to zero. That noise is the price of stochastic training on manufactured negatives; the stable signal is the gap between true and false scores, which the script reports alongside the curve. The __main__ block (lines 121 to 130) trains while filling a curve dictionary with the per-epoch loss, prints that curve, then prints the mean score of all true triples against freshly corrupted ones, and finally scores one true advising fact against one false one:

if __name__ == "__main__":  # pragma: no cover
    curve = {}
    E, R, triples, ents = train_transe(loss_curve=curve)
    checkpoints = [0, 1, 5, 20, 100, 500, 2000]
    print("epoch      " + "  ".join(f"{ep:>5}" for ep in checkpoints))
    print("hinge loss " + "  ".join(f"{curve[ep]:5.3f}" for ep in checkpoints))
    true_s, corr_s = mean_true_vs_corrupt(E, R, triples, ents)
    print(f"mean score  true triples = {true_s:.3f}   corrupted = {corr_s:.3f}")
    print(f"score advises(alice, bob)  = {score(E, R, 'alice', 'advises', 'bob'):.3f}  (true)")
    print(f"score advises(alice, erin) = {score(E, R, 'alice', 'advises', 'erin'):.3f}  (false)")

epoch          0      1      5     20    100    500   2000
hinge loss 1.522  1.319  0.802  0.224  0.132  0.062  0.089
mean score  true triples = 1.609   corrupted = 2.737
score advises(alice, bob)  = 0.399  (true)
score advises(alice, erin) = 0.929  (false)

That headline is not a summary of some hidden aggregate; it is the mean of an explicit table. Here are the Euclidean scores $s(h,r,t)$ of all 18 true triples under the trained model, sorted from best-fit to worst:

rank	triple	score	rank	triple	score
1	advises(alice, bob)	$0.399$	10	cites(p2, p1)	$1.568$
2	authored(bob, p1)	$0.841$	11	advises(carol, erin)	$1.598$
3	about(p3, ml)	$1.234$	12	about(p2, nesy)	$1.751$
4	cites(p3, p2)	$1.295$	13	authored(carol, p2)	$1.852$
5	authored(dave, p3)	$1.413$	14	advises(bob, carol)	$1.894$
6	affiliated(carol, cmu)	$1.427$	15	affiliated(dave, cmu)	$2.191$
7	authored(alice, p1)	$1.456$	16	about(p1, logic)	$2.253$
8	advises(bob, dave)	$1.484$	17	affiliated(alice, mit)	$2.385$
9	affiliated(erin, cmu)	$1.513$	18	affiliated(bob, mit)	$2.399$

Add the 18 scores and divide by 18 and you get $28.953 / 18 = 1.609$, exactly the reported mean. The average corrupted triple, by contrast, scores $2.737$: true facts sit noticeably closer to where their translation predicts than random fakes do.

The single-fact check is sharper, and it is the whole point of the chapter. advises(alice, bob) is a fact in the knowledge base, and it scores $0.399$, a tight fit and the best of all 18. advises(alice, erin) is not a fact: the advising chain runs alice → bob → carol → erin, so alice advises bob, not erin. The model has never been told this triple is false, yet it scores $0.929$, more than twice as far. We can see why by reading the geometry directly. The trained arrow is $\mathbf{r}_{\texttt{advises}} = (-1.396, -0.857)$ and $\mathbf{h}_{\texttt{alice}} = (0.698, -0.172)$, so the translated head lands at

$$\mathbf{h}_{\texttt{alice}} + \mathbf{r}_{\texttt{advises}} = (0.698 - 1.396,\; -0.172 - 0.857) = (-0.698,\; -1.029).$$

Ranking every candidate advisee by its distance from that point answers "who does alice advise?" purely from the map:

candidate tail	distance from $(-0.698, -1.029)$	in KB?
bob	$0.399$	yes
erin	$0.929$	no
carol	$1.024$	no

The true advisee is ranked first, by a wide margin, and nobody ever wrote the rule "alice does not advise erin." The map simply placed the arrows so that the true edge is short and the false ones are long.

What geometry buys, and what it costs

Notice what just happened. We asked the model about a triple that appears nowhere in the knowledge base and is derivable by no rule, and it still returned a sensible judgment. That is the whole prize of embeddings: open-world generalization, the ability to guess about facts nobody ever wrote down. The symbolic engine of Part II could not do this. It halted at 47 atoms and fell silent on everything else: ask it about advises(alice, erin) and it answers only "not derivable," never "unlikely." Geometry fills that silence with a number for any triple you can name, which is exactly why knowledge-graph embeddings became a workhorse for link prediction and completion [3].

The cost is written in the same numbers. The score is a distance, never a proof. $0.929$ is bigger than $0.399$, but it is not zero and not infinity: the model is never certain, only more or less confident. Recall the exact result of Part II: forward chaining was both sound and complete for Horn entailment, so everything it derived was genuinely entailed and it missed nothing that followed. What it would never do is guess; it stayed silent about the open world, answering "not derivable" for any fact the rules did not force. The embedding is the mirror image. It is never silent, handing back a number for every triple you can name, but it is never certain, because any single answer can be wrong. The score table itself shows the fallibility plainly: the true triple affiliated(bob, mit) scores $2.399$, worse than the false advises(alice, erin) at $0.929$. A single distance threshold that would accept the false fact would also reject a true one. The model ranks well on average and errs in particular cases. It ranks; it does not prove. This trade, coverage bought with certainty, is the neural pole of the entire neuro-symbolic project: logic gives you guarantees and no guesses, geometry gives you guesses and no guarantees, and the rest of this series is largely the search for a way to have both.

The bridge to Volume 3

The toy above is not a detour; it is the on-ramp to Volume 3, whose whole subject is embedding ontologies. There the entities and relations of a real knowledge graph are learned at full scale, and the geometry grows richer than a single point per symbol. Instead of pinning each concept to a dot, Volume 3 studies region embeddings: representing a class as a box or a ball in space, so that "every researcher is a person" can be drawn as one region containing another. TransE is the seed of all of it, the smallest complete example of "meaning as geometry," and everything Volume 3 builds is a reply to the limits our two-dimensional version runs straight into.

The unsolved part

Here is the honest crack in the foundation, and our own run lets us watch it open. TransE gives each entity a single point and each relation a single arrow, and a single arrow can point in only one direction from any given tail. But bob advises two students, carol and dave, and both triples are true. The one advises arrow added to bob can land near only one of them. The trained numbers show exactly this strain. With $\mathbf{h}_{\texttt{bob}} = (-0.306, -0.952)$ and the shared arrow $\mathbf{r}_{\texttt{advises}} = (-1.396, -0.857)$, the translated head is $\mathbf{h}_{\texttt{bob}} + \mathbf{r}_{\texttt{advises}} = (-1.702, -1.809)$, and the two true tails fall at very different distances from it:

$$s(\texttt{bob}, \texttt{advises}, \texttt{dave}) = 1.484, \qquad s(\texttt{bob}, \texttt{advises}, \texttt{carol}) = 1.894.$$

The arrow chose dave. Its true sibling carol is stranded at $1.894$, the fourteenth-best of eighteen true triples, worse than several corruptions would score. This is a one-to-many mismatch a single-vector model can never fully escape: one head, one relation, but many valid tails, and one arrow cannot reach them all.

The deeper version of the same shortage of room is that a point-and-arrow world cannot faithfully carry a logical hierarchy. Our academic world says every professor and every student is a researcher, and every researcher is a person, a chain of subsumptions we would write researcher ⊑ person (read the "⊑" as "is a kind of," or more formally "is a subclass of"). A hierarchy like that is fundamentally about containment: the persons are a big set, the researchers a subset inside it, the professors a smaller subset still. But a flat translation geometry has no natural notion of one thing being inside another. An arrow can move a point next to another point, yet it cannot express "all of these fall within all of those." Push the arrows to honor one part of the hierarchy and they distort another; the single-vector model simply lacks the room.

That failure is the exact motivation for the boxes-versus-balls question Volume 3 takes up. A box can literally contain a smaller box, and a ball can sit inside a larger ball, so a region embedding can encode researcher ⊑ person as genuine geometric containment in a way a point never can. Which shape honors more of logic, the sharp corners of boxes or the smooth symmetry of balls, and at what cost in trainability, is an open and actively debated question. Our from-scratch TransE is the cleanest way to feel why the question matters: run it, and you can watch a flat geometry try, and fail, to be a hierarchy.

Why it matters

Embeddings are the point where symbols become learnable. Every neural method in the volumes ahead, graph networks, differentiable query answering, the neural half of every hybrid system, begins by turning entities and relations into vectors, because gradient descent can only pull on numbers. This chapter showed the smallest honest version of that move, derived its loss and gradient from scratch, and traced a single step of it by hand. Just as importantly, it exposed the price: a learned geometry generalizes past what anyone wrote down, but it trades the certainty that logic gave us for a distance that is only ever a hint. Holding both facts in mind at once, that geometry generalizes and geometry can be wrong, is the mental posture the whole neuro-symbolic field is built on. You now have both poles in hand: the sound-but-silent reasoning of Part II and the fluent-but-fallible geometry of Part III.

Key terms

• Embedding: a map from symbols to vectors chosen so that geometric relationships (distance, direction) encode meaning.
• Vector, coordinate, dimension: a vector is an ordered list of $d$ real numbers; each entry is a coordinate named by an index $i$; the count $d$ (here $2$) is the dimension.
• Distributed representation: a scheme where a symbol's meaning is spread across all of its coordinates rather than stored in one slot, so nearby patterns mean similar things.
• TransE (translating embeddings): a model in which every relation is a translation vector, so a true triple $(h, r, t)$ satisfies $\mathbf{h} + \mathbf{r} \approx \mathbf{t}$.
• Triple: a binary fact written $(\text{head}, \text{relation}, \text{tail})$, for example $(\texttt{alice}, \texttt{advises}, \texttt{bob})$.
• Score: the Euclidean distance $\lVert \mathbf{h} + \mathbf{r} - \mathbf{t}\rVert$; a penalty where lower means more plausible.
• Corrupted triple: a manufactured negative example, made by replacing a true triple's tail with a random wrong entity.
• Margin-ranking (hinge) loss: $\max(0, \gamma + d^{+} - d^{-})$, which forces each true triple to score below its corruption by at least the margin $\gamma$ and vanishes once it does.
• Renormalization: rescaling every entity vector to unit length each epoch, so the model cannot cheat by inflating vectors to infinity.
• Stochastic gradient descent (SGD): the downhill-by-small-steps optimizer that nudges the vectors one triple at a time.
• Open-world generalization: an embedding's ability to assign a plausibility to triples that appear nowhere in the data and follow from no rule.
• Never silent, never certain: the embedding's trade-off: it scores every triple you can name (never silent), but any single answer can be wrong (never certain), the mirror image of forward chaining, which was sound and complete for what the rules entail yet silent about anything they did not force.
• Region embedding (box, ball): Volume 3's richer geometry that represents a class as a region, so subsumption researcher ⊑ person becomes genuine containment.

Where this leads

We now hold the two halves of the field in our hands at once: the exact, provable, but silent reasoning of logic, and the approximate, generalizing, but uncertain geometry of embeddings. They are two ways of representing the same academic world, and they disagree about what representation even is: a set of true sentences, or a cloud of points. The next chapter, Two Cultures: Symbols versus Vectors, puts the two traditions side by side, names what each does that the other cannot, and sets up the central tension the rest of the series exists to resolve: how to build a system that reasons like logic and learns like geometry, at the same time.

---

Companion code: examples/logic/embeddings.py implements the whole model in pure Python: _binary_triples extracts the 18 triples, score measures plausibility as a translation distance, train_transe learns the vectors by margin-ranking SGD with per-epoch renormalization (and, given a loss_curve dict, records the per-epoch loss against a fixed evaluation set), and mean_true_vs_corrupt reports the true-versus-corrupt gap. Run python3 examples/logic/embeddings.py to reproduce every number in this chapter.