Preface — Packing Meaning into Vectors

📍 Where we are: At the door of Volume 3 — Neural Representation. Volume 2 closed with a verdict earned over seven parts: symbols buy you theorems, provenance, and the right to abstain, but a symbolic reasoner cannot learn from data, cannot rank a guess, and falls silent about every fact nobody wrote down. This volume builds the other pillar: the vectors, regions, and networks that learn the unwritten, and the instruments that measure what that learning silently gives up.

Welcome back. Volume 2 ended with an honest asymmetry. The librarian of its preface is flawless on her own catalog: every conclusion a theorem, every contradiction caught, every unknown answered with a plain "not shown". That last phrase is the wound. Suppose the card recording that bob advises dave had never been filed: ask her, and she has nothing to offer, not even a ranked hunch. Real knowledge bases are radically incomplete, and a system that can only prove can never guess. This volume takes the guessing half seriously, and its one enabling move is to stop treating a symbol as an opaque name and embed it: assign it a position in a geometric space, so that "how plausible is this unwritten fact" becomes a number you can compute, differentiate, and train. The world does not change. We re-read the very same academic knowledge base a third way, as a graph to be packed into vectors, and by the end you will know exactly what a vector can hold, what it can only approximate, and how to catch it lying.

The simple version

Imagine the librarian's colleague: a bookseller who has shelved so many books that he can place a new one by feel. He has no catalog cards. What he has is a shop where position is meaning: mystery near crime, memoir near biography, the distance between two shelves itself a fact about the books on them. Hand him a title he has never seen and he does not say "not shown"; he walks to a shelf and says "here, probably, and if not here, then one of these three". He learned that skill by rearranging: every time two books were borrowed together he nudged them closer, until the whole shop drifted into shape. He is fast, he generalizes, and he is sometimes wrong in ways he cannot explain, because the arrangement is his knowledge and there is no card to point at. This volume is how the bookseller works: how the shop gets arranged (training), what "near" means (score functions), what shapes a shelf can take (points, boxes, balls, curved rooms), and the tests that reveal what his arrangement quietly forgot.

What this volume covers

Volume 3 is one continuous climb in six parts, from scoring a single missing edge to auditing everything geometry can promise. Each part answers one question:

Part	Theme	The question it answers
I — Knowledge Graph Embeddings	link prediction; TransE; DistMult and ComplEx	How does a fact become a vector, and a guess become a rank?
II — Region and Geometric Embeddings	balls and cones; Query2Box boxes; Beta distributions; boxes versus balls	What shape must a concept be, so that logic's ⊓ and ∃ become geometry's intersection and translation?
III — Embedding Ontologies	EL Embeddings; BoxEL, Box²EL, TransBox; mOWL; hyperbolic space	Can the geometry carry a real ontology faithfully, and how do you probe its soundness?
IV — Graph Neural Networks and Their Ceiling	message passing; GCN and R-GCN; 1-WL, C², graded modal logic	What can a network that only passes messages along edges provably never distinguish?
V — Attention and Transformers	scaled dot-product attention; vector symbolic architectures; soft unification	Can vectors address, bind, and match symbols, softly and by degree?
VI — The Verdict	what vectors hold, and what leaks	What did all this geometry genuinely buy you?

Who this volume is for

This is written for a reader one notch past Volume 2, a junior ramping toward a senior. It assumes you are comfortable with vectors, matrices, dot products, and the gradient descent built from scratch in Volume 1, and it re-teaches each piece of machinery the first time it matters. Nothing arrives as a black box. Every model in the volume is implemented in the companion suite examples/neural/ in plain NumPy, every gradient is derived by hand on the page and in a comment above the exact line that computes it, and an optional independent oracle (torch_check.py, the analogue of Volume 2's HermiT cross-check) recomputes three of the suite's hand-derived loss gradients (TransE's margin hinge, DistMult's logistic loss, and the box inclusion hinge) with torch.autograd and confirms agreement to machine precision. The whole suite is sealed by validate.py, an acceptance harness whose 15 competency checks pass 15 of 15 on the committed code; every number quoted in every chapter reproduces byte-identically with python3 <module>.py.

The thesis: a hunch you can rank

Here is the conviction the volume turns on. Neural representation buys you plausible answers about facts nobody wrote down, at the price of guarantees about the facts somebody did. Both clauses will be made precise and both will be measured. The reward is real. Give each entity a vector and each relation a geometric operation, score a candidate triple by how well the geometry closes (for TransE, the score is the negated length $-\lVert \mathbf{e}_h + \mathbf{w}_r - \mathbf{e}_t \rVert$, read "how far the head's point, translated by the relation's arrow, lands from the tail's point"), and train the arrangement so that the 15 known facts score higher than corrupted ones. On our world, that turns a frozen-random arrangement whose filtered Mean Reciprocal Rank (MRR) is 0.1062 (barely better than alphabetical guessing) into a trained one at 0.7778, ranking the truly held-out fact advises(bob, dave) at or near the top without ever having seen it.

The price is just as real. A score is not a proof: there is no derivation to hand you, no notion of entailment, and nothing that stops the geometry from quietly scoring a logically impossible fact above a true one. Where Volume 2 could state soundness as a theorem, Volume 3 can only probe it: train the embedding, then interrogate the geometry against the symbolic gold standard and count the disagreements. That probe, run honestly, is the methodological heart of the volume.

The running example, re-read as a knowledge graph

We keep the world down to the last fact and change only the lens. The 18 binary facts of Volume 1 become 18 (head, relation, tail) triples over 13 entities and 5 base relations (about, advises, affiliated, authored, cites; the sixth role, grandAdvisor, is derived by Volume 2's role chain, so it is never asserted as data). The Volume 2 ABox supplies each entity's concept, giving the node labels the graph-network chapters need: 2 professors, 3 students, 3 papers, 2 institutions, 3 topics. A deterministic split holds out 3 triples as the exam, chosen so every entity and relation still appears in the 15 training triples, and the standard filtered ranking protocol turns those 3 into 6 ranking queries scored by Mean Reciprocal Rank and Hits@k, the fraction of queries whose true answer lands in the top k. All of this is fixed once, in examples/neural/kg.py, so every model in the volume trains on exactly the same data and is judged by exactly the same judge.

The ontology chapters raise the stakes: they re-use Volume 2's EL++ TBox itself, the 14 axioms normalized to 16 normal forms, as training data for geometry, and then ask whether the trained balls and boxes reproduce the 8 gold subsumptions and respect the 2 deliberately unsatisfiable concepts that Volume 2's completion algorithm certified.

One academic world re-read a third way: a knowledge graph embedded into geometry, its guesses ranked against held-out facts and its shapes probed against Volume 2's exact classification, with the six-part climb of the volume laid out beneath. Original diagram by the authors, created with AI assistance.

Points first: facts as arrows, guesses as ranks

Part I builds the smallest working unit of the whole pillar: one entity, one vector; one relation, one operation; one candidate fact, one score. It defines link prediction and the filtered MRR/Hits@k protocol precisely, derives every gradient of TransE's margin loss by hand, and then meets the first structural wall: a pure translation cannot model a symmetric relation, and a bilinear model like DistMult cannot help modeling every relation as symmetric, scoring cites(p1, p2) and cites(p2, p1) identically to the last digit. ComplEx repairs this with one move into complex numbers, and the repair is a first taste of the volume's recurring lesson: what a model can represent is decided by its geometry, before any training begins.

Then regions: concepts and queries as shapes

A point can say "this entity", but it cannot say "all students". Part II gives concepts extent: a concept becomes a region, membership becomes containment, and the logical vocabulary of Volume 2 starts compiling into geometry, with ⊓ as intersection and ∃ as translation. Query2Box answers multi-hop conjunctive queries ("papers authored by someone alice advises") by translating and intersecting boxes; Beta embeddings buy the one operator boxes cannot close under, negation, by trading regions for probability distributions. And one committed near-miss is kept on display deliberately: the two-hop box query ranks the held-out advisee third, which is exactly what honest generalization from an incomplete graph looks like.

Geometry on trial: embedding real ontologies

Part III is where the two pillars touch for the first time. EL Embeddings translate the TBox's normal forms into differentiable losses over balls, all except the one role chain (the axiom composing advises with itself to yield grandAdvisor), for which their loss vocabulary has no term, so that axiom is dropped, not approximated; BoxEL, Box²EL, and TransBox refit the same idea with boxes, whose intersections are exact where ball intersections are not; mOWL packages the tooling. The chapters train these models on 15 of our 16 normal forms, skipping the chain out loud, and then run the soundness probe: the trained geometry recovers all 8 gold subsumptions with 0 false positives and 0 false negatives, and the disjointness axiom shows up as boxes physically pulled apart. But the final loss does not reach zero, and that residue is the exact distance between "trained to satisfy" and "guaranteed to satisfy". The part closes in curved space: at equal dimension, a hyperbolic embedding of the concept hierarchy tracks graph distance better than a Euclidean one (Spearman correlation 0.8136 against 0.7539 at dimension 2), because trees need room that flat space does not have.

The ceiling: what message passing cannot see

Part IV turns from single facts to the graph as a whole. A graph neural network computes each node's representation by repeatedly aggregating its neighbors' representations, and R-GCN makes the aggregation relation-aware. The remarkable result of this part is negative and exact: message passing is bounded by the 1-dimensional Weisfeiler-Leman color refinement, which means there are simple graph pairs, two triangles versus one hexagon, that no such network can ever tell apart, and the logical counterpart of the ceiling is precisely the two-variable counting logic C². A neural architecture, characterized by a logic: this is the volume's deepest bridge, and it comes with a proof.

Attention, binding, and soft unification

Part V asks whether the mechanisms of symbolic reasoning themselves survive translation into vectors. Attention is content-based addressing: a softmax over dot products retrieves by relevance, and we derive its Jacobian and its saturation behavior in full. Vector symbolic architectures show that binding and superposition let a single 1024-dimensional vector store all 18 triples of our graph and answer all 18 tail queries (given the head and the relation, retrieve the tail) correctly, with a capacity that degrades gracefully rather than failing at once. Soft unification closes the part by making Volume 1's unification differentiable: a proof can now step through a synonym (matching supervises against advises at similarity 0.9783) where the hard prover simply fails, at the cost of admitting that "match" is now a matter of degree.

The verdict, promised in advance

Part VI weighs it all. The gains are measured, not asserted: ranks lifted, queries answered, hierarchies compressed, gold standards recovered. The losses are measured too: no entailment, no abstention, soundness only as far as the probe reached, expressiveness capped by theorems. The honest summary is that vectors and symbols fail in opposite places, which is not a defeat but a design brief, and Volume 4 is that brief executed.

Where this leads

The climb starts at the bottom, with the question every embedding model exists to answer: given 15 known facts, how plausible is a 16th? The first chapter, Link Prediction: Completing the Graph, makes that question exact. It defines the triple, the corruption of a triple, the filtered ranking protocol, and the two numbers (Mean Reciprocal Rank and Hits@k) by which every model in this volume will be judged, then establishes the baseline every later chapter must beat: a frozen random arrangement scores a filtered MRR of 0.1062 on our six queries, and everything ahead is the story of earning the distance above it.