Translational Models: TransE and Its Family

📍 Where we are: Part I · Knowledge Graph Embeddings — Chapter 2. Link Prediction built the scoreboard: a score function, the filtered ranking protocol, and the frozen-random baseline that any model must beat. This chapter builds the first model that actually plays.

The previous chapter ended with a challenge in numbers: random geometry earns a filtered Mean Reciprocal Rank (MRR) of 0.1062 on our six ranking queries, so any model worth the name must beat that. This chapter answers with the simplest geometric hypothesis anyone has proposed for a knowledge graph, and one of the most influential [1]. Give every entity a point in a vector space. Give every relation a single fixed arrow, a translation. Declare a triple plausible exactly when the head's point, pushed along the relation's arrow, lands near the tail's point. That is the whole model. It has no hidden layers, no nonlinearity, and on our academic world only 288 numbers to learn, and yet training it multiplies the baseline MRR by seven. Because it is so small, we can afford total honesty: every gradient derived by hand and checked against the committed code, every training number quoted from a real run, and the model's signature failure first derived on paper and then pointed to in the output table, where it visibly happens to two students named carol and dave.

The simple version

Imagine a city map where every person, paper, and university is a pin, and every kind of relationship is a fixed compass instruction, the same for everyone: "advises" might mean walk 300 meters northeast. To check a claimed fact, you stand on the head's pin, follow the relation's instruction, and see whose pin you are now closest to. If "start at alice, walk advises" drops you on bob, then "alice advises bob" is plausible. Training is the process of shuffling the pins and tuning the compass instructions until the known facts all work out as short walks. The catch is built into the metaphor: one fixed instruction gives one landing spot per starting pin, so if bob advises two students, both of their pins are dragged toward the same patch of pavement.

What this chapter covers

• The modeling bet, decoded before any math: what "a relation is a displacement of vector space" can express (chains and compositions) and what it structurally cannot (a 1-to-many relation forces its tails together, derived with the triangle inequality).
• The formal setup: embeddings in $\mathbb{R}^{16}$, the score $s(h,r,t) = -\lVert \mathbf{e}_h + \mathbf{w}_r - \mathbf{e}_t \rVert$, the margin ranking loss with $\gamma = 1$, and uniform negative sampling that resamples known-true corruptions, quoted from transe.py.
• Every gradient by hand: $\partial \lVert \mathbf{v} \rVert / \partial \mathbf{v} = \mathbf{v}/\lVert \mathbf{v} \rVert$ derived from the chain rule on $(\mathbf{v} \cdot \mathbf{v})^{1/2}$, then all six partial derivatives of the hinge, the zero-norm guard, and the stochastic gradient descent update, each matched to a line range of loss_and_grads.
• The constraint that stops cheating: why entities are renormalized to the unit sphere after every epoch: without it, inflating all norms drives the loss to zero while learning nothing.
• The committed run as a worked trace: the loss at epochs 1/10/100/500/1000, trained versus baseline evaluation (0.7778 versus 0.1062), and the top of the ranking table for the query (bob, advises, ?), every number real.
• The failure, measured on our own graph: carol and dave compete for the single target point $\mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}}$, and the ranking table shows exactly who wins and by how much.
• The family as fixes: TransH, TransR, and RotatE, each a one-idea repair of a limitation of pure translation, with all four score functions in one table.

The bet: a relation is a displacement

Every model in this volume begins with a bet about what a relation is, geometrically. TransE's bet is the boldest because it is the smallest: a relation is a displacement, one fixed vector added to wherever you start [1]. Before any symbol appears, decode what that commits us to. In our graph the relation advises will be one arrow, learned once, and that single arrow must simultaneously carry alice to bob, bob to carol, and carol to erin, because all three facts are in the training set. The relation is not a lookup table of who advises whom; it is a rule of motion that every advising pair must obey at once.

That commitment buys real expressive power. A displacement composes: if you can walk advises and then walk advises again, the two arrows add, and the composite arrow $2\mathbf{w}_{\text{advises}}$ is a candidate geometry for Volume 2's derived grandAdvisor role. Chains of 1-to-1 facts are the model's home territory: a chain of people, each advising the next, embeds as a sequence of points marching along one arrow.

The same commitment forbids other things, and we can prove one immediately, using nothing but subtraction. Suppose the model fits two facts exactly: bob advises carol and bob advises dave, so

$$\mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}} = \mathbf{e}_{\text{carol}} \qquad \text{and} \qquad \mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}} = \mathbf{e}_{\text{dave}}.$$

The left sides are the same vector, therefore the right sides are equal: $\mathbf{e}_{\text{carol}} = \mathbf{e}_{\text{dave}}$. Exact fit forces the two advisees onto the same point, erasing everything that distinguishes them. The approximate version is just as damning and follows from the triangle inequality (the fact that a detour is never shorter than the straight path, $\lVert \mathbf{a} + \mathbf{b} \rVert \le \lVert \mathbf{a} \rVert + \lVert \mathbf{b} \rVert$, where $\lVert \mathbf{v} \rVert$ is the length of a vector $\mathbf{v}$: the square root of the sum of its squared coordinates). Write the two residuals, the leftover error vectors of the two facts, and suppose training has made both short, with lengths at most some small $\varepsilon$:

$$\lVert \mathbf{e}_{\text{carol}} - \mathbf{e}_{\text{dave}} \rVert = \lVert (\mathbf{e}_{\text{carol}} - \mathbf{e}_{\text{bob}} - \mathbf{w}_{\text{advises}}) + (\mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}} - \mathbf{e}_{\text{dave}}) \rVert \le \varepsilon + \varepsilon = 2\varepsilon.$$

The first step inserts $-\mathbf{e}_{\text{bob}} - \mathbf{w}_{\text{advises}} + \mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}}$ (which is zero, so nothing changed) and regroups. Look at the two grouped terms before invoking the inequality. The second, $\mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}} - \mathbf{e}_{\text{dave}}$, is exactly the (bob, advises, dave) residual, with length at most $\varepsilon$ by assumption. The first, $\mathbf{e}_{\text{carol}} - \mathbf{e}_{\text{bob}} - \mathbf{w}_{\text{advises}}$, is the negation of the (bob, advises, carol) residual, and flipping every coordinate's sign changes nothing inside a sum of squares, so $\lVert -\mathbf{v} \rVert = \lVert \mathbf{v} \rVert$ and its length is also at most $\varepsilon$. The triangle inequality then bounds the sum of the two terms by $\varepsilon + \varepsilon$. The conclusion: the better TransE fits a 1-to-many relation, the closer its tails are forced together, at distance no more than twice the residual bound. This is not a training difficulty to be patched with more epochs; it is a theorem about the score function. Hold onto it, because at the end of this chapter we will watch it happen to carol and dave in the committed output, and the whole "family" of successor models exists to escape it [2].

Formal setup: points, one arrow per relation, a score

Now the symbols, each decoded before use. Fix an embedding dimension $d$, the number of coordinates each vector gets; the companion code uses $d = 16$ (transe.py lines 42–46). The space $\mathbb{R}^d$ is the set of all lists of $d$ real numbers, so $\mathbb{R}^{16}$ is the space of 16-number lists. Each of the 13 entities $h$ (mnemonic: head) or $t$ (tail) gets an entity embedding $\mathbf{e}_h \in \mathbb{R}^d$, a point. Each of the 5 relations $r$ gets a relation embedding $\mathbf{w}_r \in \mathbb{R}^d$, the displacement arrow (the code stores these as rows of a (13, 16) matrix ent and a (5, 16) matrix rel). The parameter count is exactly $13 \times 16 + 5 \times 16 = 208 + 80 = 288$ real numbers: the entire model.

The model's claim "the triple $(h, r, t)$ is true" becomes the vector equation $\mathbf{e}_h + \mathbf{w}_r \approx \mathbf{e}_t$. To grade how nearly it holds, define the residual $\mathbf{v} = \mathbf{e}_h + \mathbf{w}_r - \mathbf{e}_t$ and measure its length with the Euclidean norm (written $\lVert \cdot \rVert$, the square root of the sum of squared coordinates). That length is the triple's distance, and the score is its negation, so that higher means more plausible, matching the convention the evaluation protocol of the last chapter demands:

$$d(h, r, t) = \lVert \mathbf{e}_h + \mathbf{w}_r - \mathbf{e}_t \rVert = \sqrt{\sum_{k=1}^{d} \left( e_{h,k} + w_{r,k} - e_{t,k} \right)^2 }, \qquad s(h, r, t) = -\,d(h, r, t),$$

where the index $k$ runs over the $d$ coordinates and $e_{h,k}$ is the $k$-th coordinate of $\mathbf{e}_h$. In the code this is a one-liner, the closure handed to kg.evaluate (transe.py lines 171–176):

def make_score_fn(ent: np.ndarray, rel: np.ndarray):
    """The scoring closure ``kg.evaluate`` expects: s(h,r,t) = -‖e_h+e_r-e_t‖₂
    over entity/relation *names* (higher = more plausible)."""
    def score(h: str, r: str, t: str) -> float:
        return -float(np.linalg.norm(ent[E_ID[h]] + rel[R_ID[r]] - ent[E_ID[t]]))
    return score

TransE in one picture: one shared arrow per relation moves every head toward its tail (left), training compares positive and corrupted distances through a margin (middle), and a 1-to-many relation drags its tails toward one target point (right), the failure the rest of the family repairs. Original diagram by the authors, created with AI assistance.

The margin ranking loss and negative sampling

A score alone does not train anything; we need a loss whose descent makes true triples score higher than false ones. The subtlety, inherited from the open-world discussion of the last chapter, is that a knowledge graph supplies no false triples. TransE's answer is to manufacture them: for each training positive, corrupt it into a negative by swapping one end for a random entity, and then demand that the positive beat the negative by a fixed gap [1]. Write $d_{\text{pos}}$ for the distance of the true triple and $d_{\text{neg}}$ for the distance of its corruption. The margin ranking loss for the pair is

$$L = \max\!\left(0,\; \gamma + d_{\text{pos}} - d_{\text{neg}}\right),$$

with the margin $\gamma = 1$ here (transe.py line 43). Read it as a demand with a tolerance: we want $d_{\text{neg}} \ge d_{\text{pos}} + \gamma$, the fake fact at least $\gamma$ farther than the real one. When the demand is met, $\gamma + d_{\text{pos}} - d_{\text{neg}} \le 0$, the $\max$ clamps to zero, and the pair contributes nothing: the model is not rewarded for over-separating. When the demand is violated the loss is positive and grows linearly with the violation. A loss of this clamped shape is called a hinge, and its one-sidedness is the point: it spends capacity only on ordering errors, which is exactly what the ranking evaluation measures.

The corruption step has a trap worth quoting the code for. Our graph is tiny and dense, so a "random" corruption of a true triple is sometimes another true triple: corrupt the tail of (bob, advises, carol) to dave and you have manufactured (bob, advises, dave), which is a held-out true fact, not a negative. Training on it as a negative would push a true fact's distance up. The sampler therefore resamples until the corruption is not a known triple (transe.py lines 114–125):

def _sample_negative(pos: tuple[int, int, int],
                     rng: np.random.Generator) -> tuple[int, int, int]:
    """One uniform negative for ``pos``: flip a coin to corrupt head or tail,
    draw the replacement entity uniformly, and resample while the corruption
    happens to be a known-true triple (so "negatives" are really negative)."""
    h, r, t = pos
    side = int(rng.integers(2))  # 0 = corrupt the head, 1 = corrupt the tail
    while True:
        e = int(rng.integers(len(ENTITIES)))
        cand = (e, r, t) if side == 0 else (h, r, e)
        if cand not in KNOWN_IDS:
            return cand

One honest caveat, flagged already in the last chapter's methodology section: KNOWN_IDS is built from all 18 triples, test included, so the sampler never corrupts into a test fact. On a benchmark that would be a form of leakage; here it is the standard filtered convention applied consistently, and it is disclosed.

The gradients, derived by hand

The companion code uses no automatic differentiation; every derivative is written out, and a separate script (torch_check.py) replays the same function under autograd to certify them. So the derivation below is not decoration; it is the training algorithm. Six embedding rows touch the loss: the head, relation, and tail of the positive triple, and the head, relation, and tail of the negative. We need $\partial L / \partial \mathbf{e}$ for each.

Step 1: the derivative of a norm. Everything reduces to one fact: how the length $\lVert \mathbf{v} \rVert$ of a vector changes as the vector changes. Write the norm as a composition of a dot product and a square root, $\lVert \mathbf{v} \rVert = (\mathbf{v} \cdot \mathbf{v})^{1/2} = \left( \sum_{k=1}^{d} v_k^2 \right)^{1/2}$, and differentiate with respect to one coordinate $v_j$ by the chain rule. The outer function is $u^{1/2}$ with derivative $\tfrac{1}{2} u^{-1/2}$; the inner function is $u = \sum_k v_k^2$, whose derivative with respect to $v_j$ keeps only the $k = j$ term, giving $2 v_j$:

$$\frac{\partial \lVert \mathbf{v} \rVert}{\partial v_j} = \frac{1}{2} \left( \sum_{k=1}^{d} v_k^2 \right)^{-1/2} \cdot 2 v_j = \frac{v_j}{\lVert \mathbf{v} \rVert}.$$

Stacking the $d$ coordinates back into a vector,

$$\frac{\partial \lVert \mathbf{v} \rVert}{\partial \mathbf{v}} = \frac{\mathbf{v}}{\lVert \mathbf{v} \rVert},$$

the unit residual: the direction of $\mathbf{v}$ at length one. Geometrically this says a vector's length grows fastest if you extend the vector along itself, at rate exactly 1, which is why the gradient has unit length. The formula fails only at $\mathbf{v} = \mathbf{0}$, where the norm has a kink (a corner, like $\lvert x \rvert$ at $x = 0$) and no derivative exists; there the code uses the subgradient $\mathbf{0}$, a legitimate stand-in at a kink, guarded by if d_pos > 0.0 (transe.py lines 92–94).

Step 2: through the hinge. The loss $L = \max(0, \gamma + d_{\text{pos}} - d_{\text{neg}})$ has two regimes. If the margin is satisfied, $L = 0$ identically in a neighborhood, so every partial derivative is zero; the code returns six zero vectors (transe.py lines 87–90) and the training loop skips the update entirely (the if loss > 0.0 guard on line 150). If the margin is violated, the $\max$ is the identity on its second argument, so

$$\frac{\partial L}{\partial d_{\text{pos}}} = +1, \qquad \frac{\partial L}{\partial d_{\text{neg}}} = -1 .$$

(At the exact boundary $\gamma + d_{\text{pos}} - d_{\text{neg}} = 0$ the hinge itself has a kink; the code's loss == 0.0 test puts the boundary in the zero-gradient regime, again a valid subgradient choice.)

Step 3: through the residual. The positive residual is $\mathbf{v}_{\text{pos}} = \mathbf{e}_h + \mathbf{w}_r - \mathbf{e}_t$. Nudge the head embedding by $\Delta$ and the residual moves by exactly $\Delta$; nudge the tail and it moves by $-\Delta$. In matrix language, $\partial \mathbf{v}_{\text{pos}} / \partial \mathbf{e}_h = I$ (the identity matrix), $\partial \mathbf{v}_{\text{pos}} / \partial \mathbf{w}_r = I$, and $\partial \mathbf{v}_{\text{pos}} / \partial \mathbf{e}_t = -I$. Chaining the three steps for the positive head, with $\mathbf{u}_{\text{pos}} = \mathbf{v}_{\text{pos}} / \lVert \mathbf{v}_{\text{pos}} \rVert$ the unit residual:

$$\frac{\partial L}{\partial \mathbf{e}_h} = \frac{\partial L}{\partial d_{\text{pos}}} \cdot \frac{\partial d_{\text{pos}}}{\partial \mathbf{v}_{\text{pos}}} \cdot \frac{\partial \mathbf{v}_{\text{pos}}}{\partial \mathbf{e}_h} = (+1) \cdot \mathbf{u}_{\text{pos}} \cdot I = +\,\mathbf{u}_{\text{pos}} .$$

Repeating the identical chain for all six rows (the negative side picks up the $-1$ from Step 2) gives the complete gradient set:

$$\frac{\partial L}{\partial \mathbf{e}_{h}^{\text{pos}}} = +\mathbf{u}_{\text{pos}},\quad \frac{\partial L}{\partial \mathbf{w}_{r}^{\text{pos}}} = +\mathbf{u}_{\text{pos}},\quad \frac{\partial L}{\partial \mathbf{e}_{t}^{\text{pos}}} = -\mathbf{u}_{\text{pos}},\qquad \frac{\partial L}{\partial \mathbf{e}_{h}^{\text{neg}}} = -\mathbf{u}_{\text{neg}},\quad \frac{\partial L}{\partial \mathbf{w}_{r}^{\text{neg}}} = -\mathbf{u}_{\text{neg}},\quad \frac{\partial L}{\partial \mathbf{e}_{t}^{\text{neg}}} = +\mathbf{u}_{\text{neg}} .$$

This is, symbol for symbol, the dictionary the code returns (transe.py lines 101–108):

    grads = {
        "h_pos": u_pos.copy(),
        "r_pos": u_pos.copy(),
        "t_pos": -u_pos,
        "h_neg": -u_neg,
        "r_neg": -u_neg.copy(),
        "t_neg": u_neg.copy(),
    }

Step 4: the update, and what it does geometrically. Stochastic gradient descent applies $\theta \leftarrow \theta - \eta\, \partial L / \partial \theta$ with learning rate $\eta = 0.05$ to each involved row (transe.py lines 156–161):

                ent[h_p] -= LR * g["h_pos"]
                rel[r_p] -= LR * g["r_pos"]
                ent[t_p] -= LR * g["t_pos"]
                ent[h_n] -= LR * g["h_neg"]
                rel[r_n] -= LR * g["r_neg"]
                ent[t_n] -= LR * g["t_neg"]

Read the signs as forces. The positive head moves by $-\eta\, \mathbf{u}_{\text{pos}}$, straight against its residual, and the positive tail moves by $+\eta\, \mathbf{u}_{\text{pos}}$, along it: both motions shorten $\mathbf{v}_{\text{pos}}$, pulling the true fact together. The negative head moves by $+\eta\, \mathbf{u}_{\text{neg}}$ and the negative tail by $-\eta\, \mathbf{u}_{\text{neg}}$: both lengthen $\mathbf{v}_{\text{neg}}$, pushing the fake fact apart. When a row appears in both triples (the uncorrupted slots, always including the relation), it simply receives both updates, which is precisely the sum of the two gradients that the multivariable chain rule prescribes for a shared parameter.

The sphere constraint: closing the inflation escape

One line of the training loop remains unexplained, and it guards the whole enterprise. After every epoch, each entity embedding is rescaled to length one, back onto the unit sphere (transe.py lines 164–167):

        # The TransE constraint: entities back to the unit sphere ‖e‖₂ = 1
        # after every epoch (stops the loss cheating by inflating norms).
        norms = np.linalg.norm(ent, axis=1, keepdims=True)
        ent /= np.where(norms == 0.0, 1.0, norms)

Why is this necessary? Because the margin loss has a degenerate way to reach zero that involves no learning at all: scale everything up. Suppose at some point every positive is closer than its negative by even a hair, $d_{\text{neg}} - d_{\text{pos}} = \delta$ for some tiny $\delta \gt 0$. Multiply every entity and relation embedding by a constant $c \gt 1$. Every residual $\mathbf{v} = \mathbf{e}_h + \mathbf{w}_r - \mathbf{e}_t$ becomes $c\,\mathbf{v}$, so every distance scales to $c\,d$, and the hinge argument becomes

$$\gamma + c\,d_{\text{pos}} - c\,d_{\text{neg}} = \gamma - c\,\delta,$$

which reaches zero (and goes negative beyond) as soon as $c \ge \gamma / \delta$. The loss reaches exactly zero while the ordering of candidates, the only thing the evaluation measures, has not changed at all: scaling by $c$ multiplies every score by $c$ and preserves every rank. The margin $\gamma$ was supposed to be a real gap; inflation makes any nonzero gap look like a real one. Worse, the gradient dynamics actively seek this escape: the negative-side updates push entity rows apart every step, so without a constraint the embedding cloud drifts outward epoch after epoch, the loss decays for free, and training pressure on the positive residuals evaporates. The original recipe therefore constrains entity norms, and this implementation renormalizes after each epoch [1]. Projecting back to the sphere deletes the radial degree of freedom; the only way left to satisfy the margin is to arrange the points, which is the learning we wanted. Relation vectors stay unconstrained, since they must be free to span the gaps between sphere points.

The sphere also quietly reshapes what a translation can do. If a fact fits exactly on the sphere, $\mathbf{e}_t = \mathbf{e}_h + \mathbf{w}_r$ with $\lVert \mathbf{e}_h \rVert = \lVert \mathbf{e}_t \rVert = 1$, then squaring the tail's norm gives $1 = \lVert \mathbf{e}_h + \mathbf{w}_r \rVert^2 = \lVert \mathbf{e}_h \rVert^2 + 2\,\mathbf{e}_h \cdot \mathbf{w}_r + \lVert \mathbf{w}_r \rVert^2$, and using $\lVert \mathbf{e}_h \rVert^2 = 1$ and solving,

$$\mathbf{e}_h \cdot \mathbf{w}_r = -\tfrac{1}{2} \lVert \mathbf{w}_r \rVert^2 .$$

Every head of relation $r$ must have the same projection onto $\mathbf{w}_r$: exact-fitting heads live on one hyperplane slice of the sphere. That identity has an immediate payoff for chains. In our graph, alice advises bob and bob advises carol; suppose both links fit exactly on the sphere. Then $\mathbf{e}_{\text{bob}} = \mathbf{e}_{\text{alice}} + \mathbf{w}_{\text{advises}}$, and taking the dot product of both sides with $\mathbf{w}_{\text{advises}}$ gives $\mathbf{e}_{\text{bob}} \cdot \mathbf{w}_{\text{advises}} = \mathbf{e}_{\text{alice}} \cdot \mathbf{w}_{\text{advises}} + \lVert \mathbf{w}_{\text{advises}} \rVert^2$. But alice and bob are now both exact-fitting heads of advises, so the hyperplane identity forces both dot products to equal $-\tfrac{1}{2} \lVert \mathbf{w}_{\text{advises}} \rVert^2$, and the two statements agree only if $\lVert \mathbf{w}_{\text{advises}} \rVert^2 = 0$. A nonzero translation can never fit two chained hops exactly on the sphere. Constraints of this kind are why, in practice, trained residuals hover near small nonzero values rather than vanishing; keep that in mind when reading the scores below.

The committed run: a worked trace

Everything above is assembled in train (transe.py lines 130–168): initialize entity and relation rows uniformly in $[-6/\sqrt{d},\, 6/\sqrt{d}]$ (the original recipe's scale, lines 137–141), then for 1000 epochs sweep the 15 training triples, sample one negative each, call loss_and_grads, apply the six-row update, and renormalize entities. The run is deterministic (NumPy default_rng(0)), so the numbers below are not one lucky draw; they are the committed output of python3 transe.py, reproducible to the digit. The mean margin loss over the 15 positives falls like this:

epoch	mean margin loss $\frac{1}{15}\sum L$
1	1.5827
10	0.6812
100	0.1594
500	0.0602
1000	0.0072

The first row is worth decoding. At epoch 1 the mean loss 1.5827 exceeds the margin $\gamma = 1$. That cannot happen if the positives are winning: a pair whose positive is already closer than its negative ($d_{\text{pos}} \le d_{\text{neg}}$) contributes at most $\gamma$, so a mean of $\gamma + 0.58$ forces $d_{\text{pos}} \gt d_{\text{neg}}$ on typical pairs. At initialization the true triple is, on average, farther than its corruption, because random points know nothing about the graph. By epoch 10 the mean violation is under the margin; by epoch 1000 it is 0.0072, meaning almost every (positive, negative) pair satisfies $d_{\text{neg}} \ge d_{\text{pos}} + 1$ outright and contributes zero. Training has essentially saturated the hinge.

Did the geometry generalize, or only memorize? The evaluation from the last chapter answers with the six filtered ranking queries over the three held-out triples, against the frozen random baseline (seed 1, never trained, transe.py lines 179–186). The committed output:

filtered ranking over 6 queries (3 test triples × tail/head)
  model            MRR    H@1    H@3    H@10   ranks
  random (seed 1)  0.1062 0.0000 0.0000 0.8333 [10, 10, 12, 7, 10, 9]
  TransE trained   0.7778 0.6667 1.0000 1.0000 [3, 3, 1, 1, 1, 1]
  per query (filtered rank, random → trained):
    (bob, advises, ?dave)        10 →  3
    (?bob, advises, dave)        10 →  3
    (bob, authored, ?p1)         12 →  1
    (?bob, authored, p1)          7 →  1
    (erin, affiliated, ?cmu)     10 →  1
    (?erin, affiliated, cmu)      9 →  1

Check the headline number by hand from the ranks, since Mean Reciprocal Rank is just the average of reciprocal ranks: $\text{MRR} = \frac{1}{6}\left( \frac{1}{3} + \frac{1}{3} + 1 + 1 + 1 + 1 \right) = \frac{1}{6} \cdot \frac{14}{3} = \frac{14}{18} = 0.7\overline{7} \approx 0.7778$, and Hits@1 is $4/6 = 0.6667$: four of the six held-out queries are answered perfectly, at rank 1, by 288 numbers that were never shown those facts. Facts like (erin, affiliated, cmu) are recovered because erin's one training edge, (carol, advises, erin), pins her near carol, a cmu student, and the shared affiliated arrow does the rest. The module's competency checks assert this outcome (trained MRR at least 0.5, beating the baseline; transe.py lines 220–223), so the claim is enforced by an exit code, not by prose.

But four perfect answers is not the interesting part of that table. The interesting part is the pair of 3s, and they are exactly where the theorem from the start of this chapter said they would be.

The 1-to-many failure, measured on our own graph

The two rank-3 queries are the two ends of (bob, advises, dave), and bob is the one entity in our graph with two advisees: carol (a training triple) and dave (held out). Both facts want the same thing, $\mathbf{e}_{\text{advisee}} \approx \mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}}$: one target point, two claimants. The module scores every one of the 13 entities as the tail for the query (bob, advises, ?) and prints the top five; this is the committed output:

query (bob, advises, ?) — top 5 tails by s = -‖e_bob + e_advises - e_t‖
  (gold answers carry their filtered rank: other known-true answers
   are skipped, the same protocol as the MRR/Hits numbers above)
  rank  tail    score
     1  carol   -0.3681  * gold (train), filtered rank 1
     2  bob     -0.4794
     3  erin    -0.7101
     4  dave    -0.7409  * gold (test), filtered rank 3
     5  alice   -0.8321

Read it row by row, because every line is the geometry talking.

Row 1, carol at distance 0.3681. The trained advisee, and she owns the target point. But notice the distance is 0.3681, not near zero, even after the hinge saturated: the shared advises arrow must also carry alice→bob and carol→erin, and the chain argument at the end of the sphere section proved that a nonzero translation cannot fit two chained hops exactly. The residual that remains is the compromise between three facts pulling one arrow in three directions.

Row 2, bob himself at 0.4794. This row is a free measurement of the relation vector. Score bob as his own tail and the entity terms cancel: $s(\text{bob}, \text{advises}, \text{bob}) = -\lVert \mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}} - \mathbf{e}_{\text{bob}} \rVert = -\lVert \mathbf{w}_{\text{advises}} \rVert$. So the table tells us $\lVert \mathbf{w}_{\text{advises}} \rVert = 0.4794$: the advises arrow is short, less than half the radius of the entity sphere. It has to be. A long arrow could fit one hop, but a chain of hops on a unit sphere cannot keep moving far in one fixed direction, so training shrinks the arrow until the whole chain is tolerably wrong rather than one link exactly right. A short arrow, in turn, means every entity is a mediocre candidate for its own advisee, which is why bob outranks three real people here.

Rows 3 and 4, erin at 0.7101 and dave at 0.7409. Here is the collapse, measured. Dave is a true advisee of bob, but the single target point $\mathbf{e}_{\text{bob}} + \mathbf{w}_{\text{advises}}$ is already spoken for by carol, and dave's other edges (he authored p3, he is affiliated with cmu) anchor him elsewhere. Our derivation bounded $\lVert \mathbf{e}_{\text{carol}} - \mathbf{e}_{\text{dave}} \rVert \le 2\varepsilon$ if both residuals were driven to $\varepsilon$; the optimizer, unable to make both small while respecting dave's other facts, made carol's small (0.3681) and left dave's at 0.7409. The 1-to-many tension is resolved by sacrificing the held-out fact. Erin edging out dave is the same geometry from another angle: erin is carol's advisee, so training pulls $\mathbf{e}_{\text{erin}} \approx \mathbf{e}_{\text{carol}} + \mathbf{w}_{\text{advises}}$, and since carol sits near bob's target point, erin sits near the target point plus one more short arrow, close enough to intrude. The filtered protocol (kg.py lines 79–104) removes carol, a known-true answer, from dave's candidate list, so dave's filtered rank is 3, behind bob and erin: exactly the 10 → 3 improvement, and no better, that the per-query table reported. Trained TransE is dramatically better than chance on this query and structurally unable to be perfect on it. That is what "the failure mode is a theorem" looks like in a ranking table.

The family: targeted fixes for provable failures

Everything wrong above is wrong with the score function, not with gradient descent, so the successors keep the training recipe (margin loss, negative sampling, norm constraints) and re-engineer the geometry. Each fix answers a limitation you can now state precisely, with one honest distinction: two of them are theorems derived in this chapter (the 1-to-many collapse above, and the symmetry collapse below), while TransR's motivation is a design argument about capacity, not a derived impossibility.

model	relation parameters	score $s(h,r,t)$	the failure it fixes
TransE [1]	$\mathbf{w}_r \in \mathbb{R}^d$	$-\lVert \mathbf{e}_h + \mathbf{w}_r - \mathbf{e}_t \rVert$	(the baseline)
TransH [2]	$\mathbf{n}_r, \mathbf{d}_r \in \mathbb{R}^d$, $\lVert \mathbf{n}_r \rVert = 1$	$-\lVert \mathbf{e}_h^{\perp} + \mathbf{d}_r - \mathbf{e}_t^{\perp} \rVert$ with $\mathbf{e}^{\perp} = \mathbf{e} - (\mathbf{n}_r^{\top}\mathbf{e})\,\mathbf{n}_r$	1-to-many collapse
TransR [3]	$M_r \in \mathbb{R}^{k \times d}$, $\mathbf{w}_r \in \mathbb{R}^k$	$-\lVert M_r \mathbf{e}_h + \mathbf{w}_r - M_r \mathbf{e}_t \rVert$	one geometry for all relations
RotatE [4]	$\mathbf{r} \in \mathbb{C}^d$, $\lvert r_k \rvert = 1$	$-\lVert \mathbf{e}_h \circ \mathbf{r} - \mathbf{e}_t \rVert$, $\mathbf{e} \in \mathbb{C}^d$	symmetric relations (keeping antisymmetry, inversion, composition)

Three reading notes on the table. First, the TransH and TransR papers state their scores with the squared norm, $\lVert \cdot \rVert_2^2$; the square is dropped here for uniformity across rows, and nothing is lost, because squaring is a monotone map on nonnegative distances and therefore preserves every ranking. Second, $\mathbf{n}_r^{\top}\mathbf{e}$ is transpose notation for the dot product of the unit normal with $\mathbf{e}$ (multiply matching coordinates, add them up): a single number, the length of $\mathbf{e}$'s shadow along the normal. Third, $\lvert r_k \rvert$ in the RotatE row is the modulus of a complex number, its length: the distance from zero in its plane, so the constraint $\lvert r_k \rvert = 1$ puts each coordinate of $\mathbf{r}$ on the unit circle.

TransH: project, then translate [2]. Each relation gets a hyperplane, described by its unit normal $\mathbf{n}_r$ (a length-one vector perpendicular to the plane), plus a translation $\mathbf{d}_r$ that lives inside the plane. Before translating, both entities are projected onto the hyperplane: $\mathbf{e}^{\perp} = \mathbf{e} - (\mathbf{n}_r^{\top}\mathbf{e})\,\mathbf{n}_r$ subtracts off the component of $\mathbf{e}$ along the normal, keeping only the part parallel to the plane. Rerun our collapse derivation and watch it break: exact fit for two advisees now forces $\mathbf{e}_{\text{carol}}^{\perp} = \mathbf{e}_{\text{dave}}^{\perp}$, equality of projections only. Carol and dave may differ freely in the discarded direction $\mathbf{n}_{\text{advises}}$, so they remain distinct points, distinguishable by every other relation, while looking identical to advises. The 1-to-many theorem dies because projection is deliberately lossy.

TransR: a separate space per relation [3]. TransH still makes all relations act inside one shared $d$-dimensional space. TransR gives each relation its own $k$-dimensional space and a projection matrix $M_r$ (a $k \times d$ grid of learned weights mapping entity space to relation space); entities are mapped by $M_r \mathbf{e}$, then translated. The motivation is that different relations care about different features of an entity: affiliated should read bob's institutional coordinates and ignore his research ones, while authored should do the opposite. A full matrix per relation can stretch, rotate, and collapse dimensions independently for each relation, at a real cost: $M_r$ adds $k \times d$ parameters per relation, versus $d$ extra for TransH, so TransR trades the parameter frugality that made TransE trainable on 15 triples for expressiveness that needs far more data.

RotatE: translate angles instead of positions [4]. There is a failure the projection fixes never touch. Take a symmetric relation, one where $(a, r, b)$ and $(b, r, a)$ hold together, and fit both exactly with a translation: $\mathbf{e}_a + \mathbf{w}_r = \mathbf{e}_b$ and $\mathbf{e}_b + \mathbf{w}_r = \mathbf{e}_a$. Add the two equations: $\mathbf{e}_a + \mathbf{e}_b + 2\mathbf{w}_r = \mathbf{e}_a + \mathbf{e}_b$, hence $2\mathbf{w}_r = \mathbf{0}$, hence $\mathbf{w}_r = \mathbf{0}$, and substituting back, $\mathbf{e}_a = \mathbf{e}_b$. Translation can model symmetry only by making the relation vanish and the entities coincide. RotatE moves the arithmetic to the complex numbers: entities are vectors in $\mathbb{C}^d$ (each coordinate a complex number, a point in a plane), and a relation is a coordinate-wise rotation, a complex vector $\mathbf{r}$ with every modulus $\lvert r_k \rvert = 1$ (each coordinate a length-one complex number, a point on the unit circle), applied by elementwise multiplication $\mathbf{e}_h \circ \mathbf{r}$. Multiplying by a unit complex number rotates that coordinate by a phase angle $\theta_k$. Now rerun the symmetry argument, where exact fit both ways means $\mathbf{e}_a \circ \mathbf{r} = \mathbf{e}_b$ and $\mathbf{e}_b \circ \mathbf{r} = \mathbf{e}_a$. Apply $\mathbf{r}$ to both sides of the first equation and substitute the second: $\mathbf{e}_a \circ \mathbf{r} \circ \mathbf{r} = \mathbf{e}_b \circ \mathbf{r} = \mathbf{e}_a$, which coordinate by coordinate reads $r_k^2\, e_{a,k} = e_{a,k}$, that is $(r_k^2 - 1)\, e_{a,k} = 0$. On every coordinate where $e_{a,k} \neq 0$ (a generic-position assumption worth stating: a trained entity has no exactly-zero coordinate) this forces $r_k^2 = 1$, so $r_k = \pm 1$: phases of 0 or $\pi$. A rotation by $\pi$ is its own inverse without being the identity, so symmetric relations exist without collapsing the entities. Be precise about what is new here: pure translation already handled the other three classic patterns. Antisymmetry costs only $\mathbf{w}_r \neq \mathbf{0}$ (if $\mathbf{e}_b = \mathbf{e}_a + \mathbf{w}_r$ exactly, the reversed triple's residual is $\mathbf{e}_b + \mathbf{w}_r - \mathbf{e}_a = 2\mathbf{w}_r$, so the reverse scores strictly worse), inversion is $\mathbf{w}_{r_2} = -\mathbf{w}_{r_1}$, and composition is vector addition itself. Rotation keeps all three (inversion as the complex conjugate $\mathbf{r}_2 = \bar{\mathbf{r}}_1$, rotating back; antisymmetry as any phase other than 0 and $\pi$; composition because rotations compose by adding phases) and adds the one pattern translation provably could not carry. One family member, four relational patterns, each an algebraic identity you can verify in a line [4].

The unsolved part

The family fixed the failures we could derive, and that is precisely what should worry you. Each repair was reactive: a theorem exposed a pattern translation cannot carry, and a new score function was engineered for that pattern. Nothing in this chapter says how many more such theorems are waiting, and no member of the family comes with a guarantee of the form "this geometry can represent every finite knowledge graph faithfully." The honest statement is that each model carves out a class of relational patterns it handles (chains, hierarchies, symmetries) and stays silent about the rest; the field found the failure modes empirically, one benchmark at a time, before deriving them. There is also a guarantee we quietly lack on the training side: the margin loss is not convex in the embeddings, so unlike the single neuron of Volume 1 there is no theorem that stochastic gradient descent finds the best arrangement, only the observed fact that on this graph, with this seed, it found a good one. A principled account of which score functions can express which graphs, rather than a catalog of patches, is where this volume is headed: the question gets a name and some sharp answers in the chapter on the expressiveness ceiling.

Why it matters

TransE is the hinge between the two halves of this series. Volume 2 ended with reasoners that are sound, complete, and silent about anything unproven; this chapter built the first system in the series that guesses, and every design choice ahead is a variation on its template: a geometric type for entities, a geometric operation for relations, a differentiable score, and gradient descent on ranked comparisons. The ontology-embedding chapters later in this volume will replace points with balls and boxes so that Volume 2's concepts, not just its individuals, get geometry; the query-embedding chapters will chain translations and their successors into multi-hop questions; and Volume 4's neuro-symbolic integrators will need exactly what we practiced here, gradients flowing through a truth-like score. Just as important is the epistemic lesson, which is the series' recurring one: the model's central weakness was not discovered by running it, it was derived before training and then confirmed in the output table, rank for rank. Geometry makes a model's promises and its failures provable, and that, more than any single MRR number, is why embedding research is mathematics and not alchemy.

Key terms

• Translational model — a knowledge graph embedding whose relation operation is vector addition: a triple is plausible when $\mathbf{e}_h + \mathbf{w}_r \approx \mathbf{e}_t$.
• Residual — the error vector $\mathbf{v} = \mathbf{e}_h + \mathbf{w}_r - \mathbf{e}_t$ of a triple; its Euclidean norm is the triple's distance, and the score is the negative distance.
• Margin ranking loss (hinge) — $L = \max(0, \gamma + d_{\text{pos}} - d_{\text{neg}})$: zero once the true triple beats its corruption by the margin $\gamma$, linear in the violation otherwise.
• Negative sampling — manufacturing false triples by corrupting the head or tail of a true one with a uniformly random entity, resampling any corruption that is itself a known-true triple.
• Unit residual $\mathbf{v}/\lVert \mathbf{v} \rVert$ — the gradient of the Euclidean norm, derived from the chain rule on $(\mathbf{v} \cdot \mathbf{v})^{1/2}$; every gradient in TransE is $\pm$ a unit residual.
• Norm-inflation escape — scaling all embeddings by $c$ scales every distance by $c$, driving any correctly ordered pair past the margin without changing a single rank; the unit-sphere renormalization of entities removes this direction.
• 1-to-many collapse — the theorem that a well-fit translation forces all tails of a shared (head, relation) pair within $2\varepsilon$ of one another; observed as carol (rank 1) crowding out dave (filtered rank 3).
• TransH / TransR / RotatE — the family: per-relation hyperplane projection, per-relation projection matrices into relation-specific spaces, and per-coordinate complex rotation, fixing the 1-to-many collapse, the one-geometry-for-all limitation, and symmetric relations respectively (rotation keeps translation's antisymmetry, inversion, and composition while adding symmetry).

Where this leads

Translation is one algebraic choice: relations act on entity space by addition. The next chapter, Bilinear Models: DistMult and ComplEx, makes the other elementary choice and lets relations act by multiplication, scoring triples with a weighted product of head and tail coordinates instead of a distance. The trade is instructive: multiplication buys back some of what addition lost, but DistMult, its simplest form, pays with a new theorem-grade failure, a score that is provably identical in both directions, which forces every relation to be symmetric; and its repair, ComplEx, reaches for the same tool RotatE did, the complex numbers. Different algebra, same story: the geometry makes a promise, the derivation finds the crack, and a sharper geometry answers.

---

Companion code: examples/neural/transe.py implements everything derived here, with loss_and_grads (lines 57–109) as the single source of truth for the mathematics, and examples/neural/kg.py supplies the 18 triples, the deterministic 15/3 split (lines 65–74), and the filtered evaluation (rank_of and evaluate, lines 79–122). Run python3 examples/neural/transe.py to reproduce every number in this chapter; its asserts (lines 220–223) fail the run if the trained model stops beating the baseline, and torch_check.py certifies the manual gradients against autograd.