Two Cultures: Symbols versus Vectors

📍 Where we are: Part IV · The Neuro-Symbolic Idea — Chapter 12. The previous chapter, Embeddings, turned the symbols alice, advises, and p1 into points on a map and let distance stand in for truth. Now we hold that geometry up against the exact reasoning of Part II and ask the sharper question: these are two ways of representing the same academic world, so what does each one do that the other cannot?

Two chapters, two answers to what looks like one question. In Part II we asked logic whether a sentence follows from what we know; in Embeddings we asked geometry whether one dot sits near another. Both were reading the same tiny world, alice, bob, advises, the citation chain p3 → p2 → p1, and both are called "artificial intelligence," yet they disagree at the root about what an answer even is: a proof, or a distance. This chapter puts the two traditions side by side, names what each buys and what each costs, and shows why their weaknesses are precise opposites, the fact that makes combining them worth a whole series. We will not leave any claim as an assertion: we will run the prover and print its proof tree, watch the forward-chaining engine climb from 23 facts to 47 atoms in three waves, and recompute a TransE score by hand from the learned coordinates until it matches the number the code prints to three decimals.

The simple version

Imagine two experts judging whether a rumor is true. The first is a lawyer: she will say "yes" only when she can walk you through an airtight chain of evidence, step by step — and if a single document is missing, she says nothing at all, not even "probably." The second is a seasoned detective: he has seen thousands of cases and will always give you a hunch, even a percentage, for a rumor he has never heard before — but he can be fluently, confidently wrong. The lawyer is symbolic AI. The detective is neural AI. Neither alone is enough, and the useful surprise of this chapter is that their blind spots are exactly opposite — which is the whole reason to try putting them in the same room.

What this chapter covers

• Two traditions — the physical symbol system program and the connectionist program, and how each of our companion files descends from one of them.
• The contrast, in five rows — exact versus robust, explainable versus opaque, brittle-to-missing-data versus data-hungry, hand-authored versus learned, and silent-on-failure versus answers-anyway.
• The symbolic culture, worked — the forward-chaining fixpoint climbing [23, 41, 47, 47] wave by wave, and the backward-chaining prover returning an auditable proof tree, both grounded line by line in forward_chain.py and sld.py.
• The neural culture, worked — the TransE score recomputed by hand from the learned two-dimensional coordinates, plus the full table of advises scores that shows the geometry getting the structure mostly right and, on one probe, confidently wrong.
• Where each one breaks — the prover falling silent on a missing or misspelled fact, and the scorer handing back a low, confident number for a triple that is simply false.
• Failures in a mirror — soundness and completeness stated with the turnstiles ⊢ (derives) and ⊨ (entails), set against "never silent, never certain," and why the two profiles are complementary.
• The interface problem — the unsolved handoff between an exact prover and a differentiable model that the rest of the series exists to work on.

Two traditions that carved up the field

For most of its history, artificial intelligence was two rival programs that barely spoke the same language. The first is symbolic AI, and its founding creed is the physical symbol system hypothesis: a system that manipulates symbols, tokens like alice and advises, by explicit rules "has the necessary and sufficient means for general intelligent action" [1]. On this view, to think is to manipulate symbols: reasoning is matching patterns and firing rules, and intelligence is what emerges when you do enough of it well. Everything in Part II descends directly from this idea. The forward-chaining engine and the backward-chaining prover we are about to revisit are physical symbol systems in miniature.

The second program is connectionism, crystallized in the two-volume Parallel Distributed Processing [2]. Its bet is the opposite: intelligence is not rules over symbols but the emergent behavior of many simple units, loosely neurons, adjusting the strengths of their connections in response to data. There are no hand-written rules to read off; the knowledge lives, spread out, in the weights. The TransE (translating embeddings) model of the last chapter is a tiny connectionist system: nobody wrote down that alice advises bob, yet training learned an arrangement of points in which that fact fits and its opposite does not.

Before we contrast them, it is worth watching the symbolic side actually run, because the phrase "manipulates symbols by rules" hides real machinery. Our knowledge base kb.py asserts 23 base facts (its FACTS list, lines 39 to 69) and 7 rules (its RULES list, lines 73 to 89). A base fact is a tuple such as ("advises", "alice", "bob"): a predicate name followed by its arguments. By the file's convention (is_var, line 25) a term that starts with an uppercase letter such as X is a variable, a blank to be filled; anything lowercase such as alice or p1 is a fixed constant. A rule such as

(("grandAdvisor", "X", "Z"), [("advises", "X", "Y"), ("advises", "Y", "Z")])

reads "for any X, Y, Z: if X advises Y and Y advises Z, then X grand-advises Z." Forward chaining computes everything the rules can derive from the facts by repeatedly applying the immediate-consequence operator, written $T_P$ and implemented as t_p in forward_chain.py (lines 42 to 49). One application of $T_P$ does exactly one thing: fire every rule whose body is currently satisfied and add all the resulting heads to the set of known atoms. You then apply $T_P$ to the enlarged set, and again, until a round adds nothing new. That stable set, where applying $T_P$ once more changes nothing (nxt == current, line 61), is the least fixpoint, and Part II established that it equals the least Herbrand model: the exact meaning of the program.

The vertical bars $\lvert\cdot\rvert$ below denote the size of a set, its number of elements, and $M_k$ denotes the set of atoms known after wave $k$ (so $M_0$ is the base facts and $\lvert M_0\rvert = 23$). Running least_fixpoint with tracing on (lines 52 to 64) prints the size after each round as [23, 41, 47, 47]. Here is what each wave $k$ actually derives, decomposed by predicate:

wave $k$	model size $\lvert M_k\rvert$	atoms added	which predicates fired
0 (base)	23	—	the asserted facts, straight from FACTS
1	41	+18	researcher (5), grandAdvisor (3), colleague (8), citesTransitively (2)
2	47	+6	person (5), citesTransitively (1)
3	47	+0	nothing new: the fixpoint is reached

The waves are not padding; each one is forced by a dependency the round before could not satisfy. Wave 1 fires every rule whose body is already among the base facts: the two class rules turn all five researchers on (professor or student implies researcher), the grandAdvisor rule composes the advising chain into grandAdvisor(alice, carol), grandAdvisor(alice, dave), grandAdvisor(bob, erin), the colleague rule pairs up everyone sharing an institution (and because colleague(X, Y) is not declared symmetric, it fires for both orderings, so mit yields the 2 ordered pairs among alice and bob while cmu yields the 6 ordered pairs among carol, dave, and erin, giving 8 in all, twice the 4 unordered pairs a reader might first count), and the base case of citation-transitivity records the two direct one-hop links citesTransitively(p2, p1) and citesTransitively(p3, p2). Wave 2 can now fire two rules that needed wave 1's output: person(X) requires researcher(X), which did not exist until wave 1, so all five person atoms appear only now; and the recursive citation rule citesTransitively(A, C) :- cites(A, B), citesTransitively(B, C) can finally chain cites(p3, p2) with the freshly derived citesTransitively(p2, p1) to conclude the two-hop citesTransitively(p3, p1). Wave 3 finds no new body satisfied, so $T_P$ has reached its fixpoint and the engine halts with 47 atoms, 24 of them derived. The recursion and the class chain each cost exactly one extra round, which is precisely why the climb takes three waves and not one.

The contrast, in five rows

The two programs are not just different techniques; they trade off the same virtues in opposite directions. Line them up and the pattern is stark:

Dimension	Symbolic (rules, proofs)	Neural (vectors, gradients)
What an answer is	exact and certain: yes or no	graded and approximate: a number
Explanation	an auditable proof trace	a distance, with no reasons attached
Missing or noisy data	brittle: no fact, no proof	robust: still returns a guess
Where knowledge comes from	hand-authored rules and facts	learned from data
How it fails	stays silent: it will not guess	answers anyway: it can be confidently wrong

Read top to bottom, each row is a coin with two faces. The prover's certainty is exactly what stops it venturing a guess, and the scorer's willingness to guess is exactly what forbids it certainty. A proof is a chain of reasons a human can check; a learned distance is a verdict with the reasoning dissolved into coordinates. This is not a scorecard where one column wins; it is a single trade nobody has escaped.

The same question, two ways

Nothing makes the trade concrete like asking one question in both dialects. Take the plainest fact in the world: does alice advise bob?

The symbolic answer: a proof that carries its own receipt

The symbolic answer comes from sld.py, our backward-chaining prover. SLD resolution, Selective Linear resolution for Definite clauses, is the strategy underneath Prolog: it starts from the goal and works backward, asking "which rule could conclude this, and what would its body then need?" The engine's one primitive is unification (unify in unify.py, line 23): matching two atoms by finding a substitution, a dictionary that maps variables to terms, that makes them identical. To match advises(alice, Y) against the fact advises(alice, bob), unification binds the variable Y to the constant bob and records {Y: bob}. The public verdict is a single Boolean (sld.py, lines 78 to 79):

def provable(goal: tuple, facts=None, rules=None) -> bool:
    return prove(goal, facts, rules) is not None

Call it on our question and it returns True. But the Boolean is the least of it: prove (lines 63 to 75) returns a proof tree, a Proof object (lines 19 to 34) whose render method prints the reasoning as an indented trace. advises(alice, bob) is asserted directly, so its proof is a single leaf. The more telling case is a fact the rules must derive. Asking for grandAdvisor(alice, carol) makes the prover show its work; this is the actual printed output of python3 sld.py:

proof of grandAdvisor(alice, carol):
  grandAdvisor(alice, carol)
    advises(alice, bob)
    advises(bob, carol)

Trace how the engine (_solve, lines 37 to 60) built that. The goal is grandAdvisor(alice, carol). It scans the rules and finds the head grandAdvisor(X, Z); after standardizing the rule's variables apart (rename, unify.py line 51, so two uses of one rule never collide), it unifies that head with the goal, binding {X: alice, Z: carol}. The rule body, under that substitution, becomes the two subgoals advises(alice, Y) and advises(Y, carol). Solving the first against the facts binds Y to bob; solving the second, now advises(bob, carol), succeeds against another base fact. Each subgoal that bottoms out on an asserted fact becomes a leaf, and the tree is assembled on the way back up. The indented tree is the explanation: alice advises bob, bob advises carol, therefore alice grand-advises carol.

The prover handles genuine recursion the same way. Asking citesTransitively(p3, p1) about the chain p3 → p2 → p1 returns a proof that is one level deeper, again the real printed output:

proof of citesTransitively(p3, p1):
  citesTransitively(p3, p1)
    cites(p3, p2)
    citesTransitively(p2, p1)
      cites(p2, p1)

The recursive rule matched citesTransitively(p3, p1), reduced it to the base fact cites(p3, p2) plus the smaller subgoal citesTransitively(p2, p1), and that subgoal in turn reduced through the rule's base case to the fact cites(p2, p1). Every step is a rule you can name and a fact you can check. This is the symbolic culture's signature: the answer arrives with its own receipt.

The neural answer: a distance you compute

Now the same question in the neural dialect, from embeddings.py. There is no proof; there is a distance. TransE places every entity and relation as a vector, a short list of $d$ numbers read as a point in $d$-dimensional space (here $d = 2$, so a point in the plane), and it models a relation as a translation: a true triple $(h, r, t)$, meaning head entity $h$ stands in relation $r$ to tail entity $t$, should satisfy $h + r \approx t$, where $+$ adds the vectors coordinate by coordinate and $\approx$ reads "is approximately equal to." The plausibility of a triple is the distance by which that translation misses the tail (score, lines 79 to 82, built on _dist2, lines 32 to 33):

def score(E, R, h, r, t) -> float:
    """Plausibility as distance: lower means the translation fits better."""
    dim = len(E[h])
    return math.sqrt(_dist2([E[h][i] + R[r][i] for i in range(dim)], E[t]))

In symbols, writing $\mathbf{h}$, $\mathbf{r}$, $\mathbf{t}$ for the learned vectors and $h_i$ for the $i$-th coordinate of $\mathbf{h}$, that function computes

$$\text{score}(h, r, t) \;=\; \bigl\lVert (\mathbf{h} + \mathbf{r}) - \mathbf{t} \bigr\rVert \;=\; \sqrt{\sum_{i=1}^{d} \bigl(h_i + r_i - t_i\bigr)^2}.$$

The double bars $\lVert \cdot \rVert$ denote a vector's length (its norm), the ordinary straight-line distance from the origin; the symbol $\sqrt{\phantom{x}}$ is the square root; and the sign $\sum_{i=1}^{d}$ (a capital Greek sigma) means "add up the term that follows as the index $i$ runs from $1$ to $d$." So the score is nothing more than the Euclidean distance between the translated head $\mathbf{h} + \mathbf{r}$ and the true tail $\mathbf{t}$: lower means a better fit.

These numbers are not asserted; they are trained, and we can recompute one entirely by hand. Running python3 embeddings.py (a from-scratch two-dimensional TransE, 18 triples over 13 entities and 5 relations, trained by margin-ranking SGD for 4000 epochs with the random seed fixed, so the run is reproducible) settles the relevant coordinates to

$$\mathbf{h} = \text{alice} = (0.698,\, -0.172), \quad \mathbf{r} = \text{advises} = (-1.396,\, -0.857), \quad \mathbf{t} = \text{bob} = (-0.306,\, -0.952).$$

Now apply the formula in three steps. First translate the head by the relation, coordinate by coordinate:

$$\mathbf{h} + \mathbf{r} = \bigl(0.698 + (-1.396),\; -0.172 + (-0.857)\bigr) = (-0.698,\, -1.029).$$

Second, subtract the tail to get the miss vector:

$$(\mathbf{h} + \mathbf{r}) - \mathbf{t} = \bigl(-0.698 - (-0.306),\; -1.029 - (-0.952)\bigr) = (-0.392,\, -0.077).$$

Third, take the length of that miss: square each coordinate, add, and take the root:

$$\sqrt{(-0.392)^2 + (-0.077)^2} = \sqrt{0.1537 + 0.0059} = \sqrt{0.1596} = 0.399.$$

That is exactly the number the file prints, score advises(alice, bob) = 0.399, reproduced from the learned coordinates with no machinery hidden. The model answers "does alice advise bob?" with 0.399, a short distance, a plausible fit. It has never seen the word "true"; it simply arranged the arrows so that alice-plus-advises lands close to bob. Two verdicts on one fact: the prover says a certain yes and shows you why; the embedding says 0.399, a graded plausibility with no reasons to inspect.

One question, two cultures: the symbolic side returns a certain yes carried by a proof tree you can audit, while the neural side returns a distance that grades plausibility for any triple you can name but never reaches certainty. Original diagram by the authors, created with AI assistance.

Where each one breaks

The contrast turns from interesting to important the moment you ask each culture something it handles badly, because each one fails in a way the other never would.

The prover breaks on absence. Ask whether alice advises erin. The advising chain runs alice → bob → carol → erin, so alice does not directly advise erin, and the prover is right to reject it: provable(("advises", "alice", "erin")) returns False, because prove searched every rule and fact, found no way to make the goal match, and returned None. But look at how it fails. It does not say "unlikely" or "0.9"; it reports that nothing derivable exists. That silence is identical whether the fact is genuinely false, merely absent from an incomplete knowledge base, or lost to a single typo. Misspell the relation once, ask provable(("advize", "alice", "bob")), and the answer is again a flat False: no rule head and no fact uses the predicate advize, so unification fails at the first step and the proof collapses without a warning, even though the fact is plainly true under its correct spelling. Symbolic reasoning is brittle: it is only ever as complete and as clean as the facts and rules a human hand-authored, and outside that closed world it does not degrade gracefully. It goes quiet, which is fragile in exactly the places real data is noisy, sparse, and misspelled.

The scorer breaks on certainty. The embedding has the opposite problem: it is never silent. Every triple you can name has a score, because the formula $\lVert(\mathbf{h} + \mathbf{r}) - \mathbf{t}\rVert$ is defined for any three vectors. Ask about advises(alice, erin), a triple that appears nowhere and follows from no rule, and it cheerfully returns 0.929. On this particular pair the geometry is right: 0.929 is larger than the 0.399 it gives the true fact, so it ranks them correctly, and averaged over all 18 true triples the score is 1.609 against 2.737 for randomly corrupted ones (mean_true_vs_corrupt, lines 85 to 96). The gap is real; the model genuinely learned the world's structure.

But the same run also shows, in miniature, the exact danger the neural culture carries, and we should be honest that it is not merely hypothetical at scale. Compute the whole family of advises scores from the learned coordinates and a false triple slips below a true one:

triple advises(h, t)	in the world?	score (lower = more plausible)
alice → dave	false (alice advises only bob)	0.286
alice → bob	true (asserted fact)	0.399
alice → erin	false	0.929
alice → carol	false	1.024
bob → dave	true (asserted fact)	1.484
carol → erin	true (asserted fact)	1.598
bob → erin	false	1.809
bob → carol	true (asserted fact)	1.894

Read the top two rows. The false triple advises(alice, dave) scores 0.286, more plausible than the true advises(alice, bob) at 0.399. The model would rank a fact that is simply not in the world above one that is. This is a hallucination in miniature: a confident, low-distance verdict for something false, with no internal alarm. It is not a bug in the code; it is the mechanism working as designed. Margin-ranking training (the update in train_transe, guarded at line 65) only ever pushes each true triple to sit a fixed margin below a randomly sampled corruption; it never promises a globally correct ordering, and squared distance is monotonic in the reported root, so the ranking it enforces is only "true beats random noise on average," not "every true triple beats every false one." Nothing in the geometry forbids a specific false triple from landing close. The neural culture is robust to noise and gaps precisely because it will always answer, and unreliable for the very same reason. It is also data-hungry: the geometry is only as good as the examples it was trained on, where the prover needs no training data at all, just the rules. Trained on messy data at web scale, the setting Volume 3 develops, this same mechanism routinely hands a low, confident-looking score to a fact that is flatly false, a fluent wrong answer with no internal signal that it is wrong.

Failures in a mirror

Put the two failure modes next to each other and the shape of the whole field appears. Part II's reasoning was sound and complete for what the rules entail. To state that precisely we need two symbols. The single turnstile $\vdash$ reads "derives": $P \vdash a$ means the engine can produce a proof of atom $a$ from program $P$. The double turnstile $\models$ reads "entails" or "models": $P \models a$ means $a$ is true in every model of $P$, whether or not any machine ever writes the proof down. Soundness is the implication $P \vdash a \Rightarrow P \models a$: everything the engine derives is genuinely true. Completeness is the converse, $P \models a \Rightarrow P \vdash a$: everything true is eventually derived. For our Horn program both hold, because the forward-chaining fixpoint we watched climb to 47 atoms is the least Herbrand model, so $\vdash$ and $\models$ pick out exactly the same ground atoms. The prover proves all truths and only truths, and it is silent about the open world, refusing to guess about anything the rules do not force.

The embedding is the exact photographic negative. It is never silent, scoring any triple you can name, but never certain, because any single score can be wrong, as advises(alice, dave) = 0.286 just showed. One culture gives you guarantees and no guesses; the other gives you guesses and no guarantees.

Because the blind spots are mirror images rather than overlaps, they are candidates to cancel. Where the prover falls silent (a missing fact, a novel entity, a noisy or misspelled input) the scorer still ventures a number. Where the scorer hallucinates (a confident verdict that violates a hard rule) the prover can act as a check that refuses the impossible. This is the whole thesis of neuro-symbolic AI, the "third wave" argued to be the natural resolution of the two traditions: not choosing symbols or vectors, but engineering a system that reasons like the prover and generalizes like the geometry, at the same time [3]. The rest of this series is, in large part, the study of how.

The unsolved part

The complementarity is easy to state and hard to build, and the difficulty lives in one place: the interface. It is not enough to run a prover and an embedding side by side and average their opinions. The value of the prover is a guarantee, soundness, and the value of the embedding is robustness, it never gives up. The trouble is that the most obvious ways of wiring them together destroy exactly the property you were trying to keep. Let a differentiable model propose facts that flow into a proof, and a single hallucinated premise poisons everything it can reach downstream. Suppose the geometry's confidence slips the false triple advises(alice, carol) into the fact base; the same trained model scores it a deceptively low 1.024, below several genuinely true triples. The sound engine will happily chain that rotten premise with the real fact advises(carol, erin), fire the grand-advising rule, and derive the false grandAdvisor(alice, erin), an impeccable proof tree resting on a fabricated leaf. In this tiny world the damage stops there, because an advises fact can feed only the grand-advising rule; person-hood descends from class membership and colleague-pairs from affiliation, so no injected advising fact can reach them, and the flagship hallucination advises(alice, dave) happens to chain to nothing at all (dave advises no one, and no one advises alice). At web scale, where far more rules share predicates, that same one bad premise cascades much further. Either way the guarantee is gone, and worse, it is gone silently, because the poisoned proof looks exactly as trustworthy as a real one. Push the other way, let hard logical constraints clamp the geometry, and the model can lose the very flexibility that let it guess about the open world in the first place; the robustness is gone. How to pass information across that seam, proofs into gradients and gradients into proofs, without the certainty of one side leaking away or the coverage of the other collapsing, is the central open problem, and no single design has settled it. That interface, in its many forms, is what the remaining volumes are about.

Why it matters

This chapter is the hinge of the whole book. Everything before it built the two poles separately: Parts I and II gave us exact, explainable, silent logic; Part III gave us fluent, generalizing, fallible geometry. Naming them as two cultures with mirror-image strengths is what turns "here are two techniques" into "here is a reason to combine them." A practitioner who internalizes the contrast reads the field correctly: a claim that a system is "explainable" should send you looking for a proof trace like the one sld.py prints, not a heat-map; a claim that it "handles noise" should make you ask what happens when it is confidently wrong, the way our toy scored a false triple at 0.286. Once you accept that neither pole is sufficient and that their failures are complementary, the only interesting question left is how to join them, which is precisely the map the next chapter draws.

Key terms

• Symbolic AI — the tradition that treats intelligence as rule-governed manipulation of symbols; exact, explainable, and brittle to anything outside its hand-authored knowledge.
• Physical symbol system hypothesis — the claim that a symbol-manipulating system has the necessary and sufficient means for general intelligent action.
• Connectionism — the tradition, crystallized in Parallel Distributed Processing, that treats intelligence as emergent from many simple units adjusting connection weights on data.
• Immediate-consequence operator $T_P$ and least fixpoint — one round of "fire every satisfied rule, collect the heads"; iterated until nothing new appears, its stable set is the program's meaning, reached here in the waves [23, 41, 47, 47].
• SLD resolution — Selective Linear resolution for Definite clauses, the backward-chaining proof strategy behind Prolog and our sld.py; it returns a proof tree, not just a yes/no.
• Unification and substitution — matching two atoms by finding a variable-to-term dictionary that makes them identical; the one primitive both reasoning engines share.
• Proof trace — the auditable tree of rules and facts that establishes a goal; the symbolic culture's built-in explanation.
• Score (TransE) — the distance $\lVert(\mathbf{h}+\mathbf{r})-\mathbf{t}\rVert$ from head-plus-relation to tail; a graded plausibility where lower is more plausible, defined for any triple you can name.
• Brittleness — the symbolic failure mode: a missing, absent, or misspelled fact yields no proof and no answer at all, rather than a degraded one.
• Hallucination — the neural failure mode: a confident, low-distance score for something that is simply false, such as advises(alice, dave) = 0.286, with no internal signal that it is wrong.
• Soundness ($\vdash \Rightarrow \models$) and completeness ($\models \Rightarrow \vdash$) versus never silent, never certain — the mirror-image profiles of the two cultures: logic proves only truths and all truths but never guesses; geometry always answers but can always be wrong.
• The interface problem — the unsolved task of passing information between an exact prover and a differentiable model without forfeiting the guarantees of one or the robustness of the other.

Where this leads

We now have the tension stated cleanly: two cultures, two answers to advises(alice, bob), a proof tree against a distance of 0.399, two failure modes that are precise opposites, and one interface problem standing between us and having both at once. The obvious next question is how many ways are there to join them?, and it turns out the answer has a well-known shape. The next chapter, The Kautz Taxonomy: Six Ways to Combine, lays out the field-defining map of neuro-symbolic architectures known as the Kautz taxonomy, from a neural front-end feeding a symbolic solver to a symbolic teacher shaping a neural student, giving us the vocabulary to place every hybrid system, including the ones the rest of this series builds, on a single chart.

---

Companion code: examples/logic/forward_chain.py computes the least fixpoint and prints the wave sizes [23, 41, 47, 47]; examples/logic/sld.py is the backward-chaining prover whose prove returns the proof trees shown above; examples/logic/embeddings.py trains the two-dimensional TransE and prints the scores, all in pure standard-library Python over the shared world in examples/logic/kb.py. Run python3 examples/logic/forward_chain.py, python3 examples/logic/sld.py, and python3 examples/logic/embeddings.py to reproduce every number in this chapter, including the advises(alice, dave) = 0.286 hallucination.