NeSy Benchmark Suites and Simulators

📍 Where we are: Part VI · Benchmarks and Evaluation — Chapter 16. The Seven Difficulty Axes turned "harder" into seven independent generator knobs and measured two systems one factor at a time; this chapter widens the lens from one instrument to the field's benchmark suites, extracts the four design commitments they share, and then turns that rubric on the volume's own five instruments.

A benchmark is a task with an answer key. For most of machine learning's history that was enough: publish a test set, freeze it, and let systems race. The previous chapter already showed why a single frozen number under-describes a reasoning system; this chapter is about the sturdier artifact the field converged on around 2024, the benchmark suite: a family of seeded generators, a panel of metrics, and a fixed evaluation protocol, shipped together as one instrument. The shift is not cosmetic. A single benchmark measures a task; a suite institutionalizes a methodology, and the modern neuro-symbolic (NeSy) suites agree, with striking consistency, on what that methodology must contain. This chapter extracts their four shared design commitments, surveys the exemplar suites against them, and then does what a methods chapter owes its reader: it builds one. The companion's suite_harness.py re-runs the five measuring instruments this volume has constructed, under one seeded protocol, into one consolidated report card, and grades itself against the same rubric it applies to everyone else.

The simple version

Imagine a driving examiner who tests every candidate on the same memorized route. Within a year, driving schools teach exactly that route: the three turns, the one parallel park, the traffic light that is always green at 9 a.m. Pass rates soar; driving ability does not. A good examiner does four things instead. She generates a fresh route for each candidate from a route-making recipe, so there is nothing to memorize (and writing down the recipe's dice rolls lets her replay any route exactly). She knows the ground truth of what each route actually tests, because she designed the recipe. She grades more than arrival: mirror checks, signaling, hesitation at the roundabout. And she can dial the route's difficulty: night driving, rain, a highway merge. A benchmark suite is that examiner, and this chapter is her rulebook.

What this chapter covers

• The design shift: what breaks when a static test set meets a field that trains on everything: leakage, saturation (derived, with the standard-error arithmetic that says exactly when a test set stops discriminating), and Goodhart's law.
• The four commitments: known ground truth for the latent property under test, metrics beyond accuracy, controllable difficulty, and seeded procedural generation, stated up front as the chapter's rubric and used to organize everything after.
• The survey against the rubric: the concept-quality suite with its exact shortcut counter, the urban-logic simulator, the procedural pattern curricula, the temporal-constraint streams, and the ontology-aligned vision resource, each read as one commitment made institutional.
• The harness, this chapter's artifact: suite_harness.py re-runs the volume's five instruments under one protocol and emits a single report card, with every headline number re-derived and asserted equal to the owning module's committed value.
• The report card as a mini state of the field: the consolidated system-by-instrument table, with two rows read aloud through the Parts of this volume that produced them.
• The honest checklist: which of the four commitments the mini-suite satisfies fully, partially, or not at all, printed yes/no per property, because a benchmark chapter that graded itself leniently would be its own counterexample.

From benchmark to suite: three ways a static test set dies

A frozen test set fails in three distinct ways, and it is worth separating them because the four commitments answer them one by one.

The first is leakage. Modern systems are trained on web-scale corpora, and a published test set is, by publication, on the web. Once test items (or paraphrases of them) enter the training distribution, the benchmark measures recall of the benchmark. Nothing in the score distinguishes reasoning from retrieval, and no amount of statistical care at evaluation time can repair contamination that happened at training time.

The second is saturation, and this one we can make exact. Suppose a test set has $m$ items ($m$ is the number of test questions) and a system answers each independently correctly with probability $p$ (its true accuracy). Each scored item is then a Bernoulli draw: a random variable $X$ that takes the value $1$ (question answered correctly) with probability $p$ and the value $0$ otherwise. One draw's variance takes two lines. Write $E[X]$ for the expected value of $X$, its probability-weighted average, so $E[X] = 1 \cdot p + 0 \cdot (1-p) = p$; because $X$ takes only the values $0$ and $1$, $X^2 = X$, so $E[X^2] = p$ as well; and variance is the expected square minus the squared expectation, $\mathrm{Var}(X) = E[X^2] - (E[X])^2 = p - p^2 = p(1-p)$. The measured accuracy $\hat{p}$ is the mean of the $m$ draws. Variances of independent random variables add, so the sum of the $m$ draws has variance $m\,p(1-p)$, and dividing that sum by $m$ to form the mean divides the variance by $m^2$ (rescaling a random variable by a constant $c$ rescales its variance by $c^2$), leaving $\mathrm{Var}(\hat{p}) = m\,p(1-p)/m^2 = p(1-p)/m$. The standard error (the standard deviation of the estimate, the square root of this variance) is therefore

$$\mathrm{se}(\hat{p}) \;=\; \sqrt{\frac{p(1-p)}{m}}.$$

Put numbers in. At $p = 0.99$ on $m = 500$ items: $p(1-p) = 0.99 \times 0.01 = 0.0099$; dividing by 500 gives $0.0099 / 500 = 0.0000198$; the square root is $\mathrm{se} \approx 0.00445$, about $0.45$ accuracy points. Comparing two systems doubles the variance (the variance of a difference of independent estimates is the sum of the variances), so the standard deviation of the measured gap is $\sqrt{2} \times 0.00445 \approx 0.0063$, and a resolution threshold of two standard deviations, the conventional bar for calling a gap real rather than noise (about 95 percent of pure-noise fluctuations fall inside it), sits at $2 \times 0.0063 \approx 0.0126$. Read the conclusion plainly: near the ceiling, a 500-item test set cannot statistically distinguish independently evaluated systems within about $1.3$ accuracy points of each other. Leaderboard reorderings inside that band are noise. One concession keeps the arithmetic honest: when two systems are scored on the same frozen items, their errors are correlated, and a paired test over just the items where they disagree can resolve gaps well inside that band, so the $1.3$-point figure is the worst case for unpaired comparison, not a hard floor. A saturated benchmark has not been solved; it has stopped measuring.

The third is Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Every metric is a proxy for a property, and optimization pressure finds the gap between them. Volume 5 has already exhibited the sharpest NeSy instance: Part II proved that label accuracy cannot certify concept meaning, so a system optimized for label accuracy is free to land on any of the reasoning shortcuts counted in advance, and in Part II's measured sweep fifteen of twenty label-perfect seeds did. Goodhart is not an accident of sloppy metric choice; for latent properties it is a theorem.

The suites this chapter surveys answer the three failures with four design commitments, and it is fair to read the wave that crested in 2024, the concept-quality suite rsbench [1], the urban-logic simulator LogiCity [2], the KANDY pattern curricula [3], the LTLZinc temporal-constraint streams [4], and the ontology-aligned vision resource NeSy4VRD [5], as variations on this one rubric:

commitment	what it defends against	how the suites realize it
(1) known ground truth for the latent property under test	Goodhart on labels: accuracy provably cannot certify concepts or rules	gold concepts and the knowledge connecting them to labels, shipped with every generated split; an exact counter for the label-perfect failure modes
(2) metrics beyond accuracy	the failures worth measuring are invisible to accuracy	concept collapse, concept calibration, proof validity, consistency, forgetting
(3) controllable difficulty	saturation: a fixed set stops discriminating at the ceiling	abstraction modes and compositional splits; staged curricula; horizon and supervision-granularity knobs
(4) procedural generation with seeds	leakage and irreproducibility at once	infinite fresh instances from a generator; byte-identical regeneration from a committed seed

Formally, a suite replaces the static pair (test set, metric) with a triple: a generator $G(\theta, s)$ mapping a difficulty configuration $\theta$ (the knob settings) and a random seed $s$ to a fully labeled task instance; a metric panel scoring more than the label; and a protocol fixing how systems meet instances. The previous chapter built the $\theta$ half of this story, seven knobs deep. This chapter is about the rest of the triple, and about what happens when a volume aims the triple at itself.

The chapter in one picture: four design commitments answering three failure modes, five exemplar suites embodying them, and the volume's own five instruments funneled through one seeded harness into a single report card and an honest checklist. Original diagram by the authors, created with AI assistance.

Commitment (1) incarnate: gold concepts and an exact shortcut counter

The concept-quality suite is commitment (1) built into infrastructure [1]. Its premise is the theorem this volume proved in Identifiability: whenever the knowledge admits a nontrivial label-preserving relabeling (as the knowledge behind every task in this suite does, by construction), no label metric on any test set can certify what a model's intermediate concepts mean. If concepts are to be scored, the benchmark itself must ship concept-level ground truth, and rsbench ships it three ways. First, generators with gold latents: task families spanning arithmetic over handwritten digits, propositional logic over latent bits (parity and random conjunctive-normal-form (CNF) formulas), and rendered visual scenes, where every generated example carries not just its label $y$ but the gold concept vector $g$ and the knowledge $\beta_K$ that maps concepts to label, all known by construction because the generator produced them. Second, a shortcut counter: a training-free verifier that counts, before anyone trains anything, how many concept relabelings preserve every label on the observed data, exactly (by model counting) for small tasks and with an $(\epsilon, \delta)$-guaranteed approximate counter at scale, where $\epsilon$ bounds the relative error and $\delta$ the probability of exceeding it. Third, a metric panel that scores concepts separately from labels: concept accuracy, concept collapse, and concept-level calibration error alongside plain label accuracy.

Part II of this volume already built this suite in miniature, and Measuring Shortcuts walked through every piece; here it earns its seat as the exemplar of commitment (1), so we recall only the load-bearing mathematics. The counter's target is $N_{\mathrm{opt}}$, the number of deterministic relabelings $\alpha$ (maps from the set $\mathcal{G}$ of concept worlds to itself) that preserve the label on every observed world. Because the constraint touches each world's image independently, the count factorizes over worlds (rsbench_lite.py lines 164–192):

$$N_{\mathrm{opt}} \;=\; \prod_{g \,\in\, S} \bigl|\{\,c \in \mathcal{G} : \beta_K(c) = \beta_K(g)\,\}\bigr| \;\times\; \prod_{g \,\notin\, S} \lvert \mathcal{G} \rvert,$$

where $S$ is the observed support (the set of concept worlds that actually appear in training; capital $S$, to keep it apart from the random seed $s$ above), the first product ranges over observed worlds (each may be sent anywhere in its own label class), the second over unobserved worlds (each may be sent anywhere at all, since no training point constrains it), and the vertical bars $\lvert \cdot \rvert$ count a set's elements. For the $k$-bit parity task the factors close up: flipping the first bit is a one-to-one pairing between even-parity and odd-parity worlds, so each label class contains exactly half of the $2^k$ worlds, that is $2^{k-1}$ of them, and

$$N_{\mathrm{opt}} \;=\; \bigl(2^{k-1}\bigr)^{\lvert S \rvert} \cdot \bigl(2^{k}\bigr)^{2^k - \lvert S \rvert} \;=\; 2^{(k-1)\lvert S \rvert} \cdot 2^{k(2^k - \lvert S \rvert)}.$$

The companion's generator (rsbench_lite.py lines 123–159) exposes the support as a knob, support_frac, and the committed table traces the closed form exactly. At $k = 3$ with all $\lvert S \rvert = 8$ worlds observed, $N_{\mathrm{opt}} = 2^{2 \times 8} \cdot 2^{3 \times 0} = 2^{16} = 65{,}536$; drop to $\lvert S \rvert = 6$ and it is $2^{12} \cdot 2^{6} = 2^{18} = 262{,}144$; at $\lvert S \rvert = 4$, $2^{8} \cdot 2^{12} = 2^{20} = 1{,}048{,}576$. The committed run prints exactly those numbers, certified against a literal sweep of all $4^4 = 256$ maps at $k = 2$ (rsbench_lite.py lines 195–206):

      task  k  support  frac    N_opt      reasoning shortcuts (N_opt - 1)
      XOR   2    4/4    1.00        16         15
      XOR   2    3/4    0.75        32         31
      XOR   2    2/4    0.50        64         63
      XOR   3    8/8    1.00    65,536     65,535
      XOR   3    6/8    0.75   262,144    262,143
      XOR   3    4/8    0.50 1,048,576  1,048,575
      CNF   3    8/8    1.00    84,375     84,374
      CNF   3    6/8    0.75   600,000    599,999
      CNF   3    4/8    0.50 2,560,000  2,559,999
      the seeded CNF (k=3): (x1 | x2 | x3) & (x2 | x3 | ~x1) & (~x3 | ~x2 | x1)   (5/8 worlds satisfy)

The CNF rows reward the same close reading. The seeded formula is satisfied by 5 of the 8 worlds, so at full support each of the 5 positive worlds contributes a factor 5 and each of the 3 negative worlds a factor 3: $5^5 \cdot 3^3 = 3125 \times 27 = 84{,}375$. The jump to $600{,}000$ is the factor $(8/3)^2 = 64/9$, which betrays which worlds the seeded permutation dropped: both were negative worlds, each factor rising from its class size 3 to the unconstrained 8, and indeed $84{,}375 \times 64 / 9 = 600{,}000$. The next jump multiplies by $(8/5)(8/3) = 64/15$: one dropped world per class, and $600{,}000 \times 64 / 15 = 2{,}560{,}000$. A suite whose ground truth is known by construction supports exactly this kind of audit: every number in the table is a consequence you can re-derive by hand.

Commitment (3) incarnate: simulators, curricula, and streams

If gold latents answer Goodhart, controllable difficulty answers saturation, and three suites make difficulty a first-class generator input in three different senses of "harder."

The urban-logic simulator. LogiCity is a simulated city whose dynamics are governed by first-order-logic rules over agent concepts: which agents exist (cars, pedestrians, ambulances), which predicates describe them, and which rules bind predicates to actions are all user-configurable, so the abstraction level of the world is itself a knob [2]. Two task regimes probe different muscles: long-horizon multi-agent tasks, where an agent must plan through the interacting rule dynamics of other agents, and single-step reasoning tasks that grade whether a model's action, predicted from a rendered scene, respects the governing rules. The evaluation splits are compositional: test episodes use agent compositions, combinations of concept annotations never seen together in training, under the same governing rules, so memorizing configurations cannot substitute for learning the rules that generate them. In the vocabulary of the previous chapter, the abstraction dial turns two of the seven axes at once: a richer rule binds more variables in one firing, which is the query-joins axis, and the visual tasks require the agent concepts those rules mention to be extracted from rendered pixels, which is the grounding-trust axis. The long horizon stretches the completeness knob itself (the derivation to be recovered is a plan through time), and the multi-agent dynamics contribute a kind of difficulty the desk-scale generator does not have: other agents' rule-governed behavior keeps entering each agent's premises, difficulty from interaction rather than from any single query's derivation.

The procedural pattern curricula. KANDY generates curricula of Kandinsky-pattern tasks: binary classifications of scenes built from geometric primitives, where each task's positive class is defined by a ground-truth symbolic rule in a compositional language, and the tasks arrive as a sequence of increasing compositional depth [3]. Two design choices matter here. Because every task ships its gold rule, rule recovery is checkable rather than eyeballed: a NeSy learner claiming to induce symbolic structure can be graded against the rule that actually generated the data (commitment (1) again, at the rule level rather than the concept level). And the supervision schedule is sparse by design: labels arrive for only a fraction of instances, staging the realistic regime where symbolic structure must be inferred rather than fit. On the previous chapter's axes, the staged nesting of pattern constructors is the derivation-depth knob in compositional form, and the sparse supervision turns the grounding-trust axis: the fewer the labels, the more of the latent concept vocabulary must be inferred through the rules rather than fit to annotations. Its genuinely new export is the curriculum ordering itself, difficulty as a trajectory rather than a point.

The temporal-constraint streams. LTLZinc generates tasks whose specification is a linear temporal logic over finite traces (LTLf) formula over constraint-satisfaction problems: at each time step the system perceives inputs (images standing for symbols) that must satisfy step-level constraints, and the finite sequence as a whole must satisfy the temporal formula, the finite-trace semantics being what makes each sequence's accept/reject label well defined [4]. The knobs are the temporal horizon, the constraint semantics at each step, and, distinctively, the supervision granularity: whether the learner sees labels per step or only per sequence, the temporal generalization of the distant supervision that trained DeepProbLog's digit classifier from sums alone. Because the stream of tasks can shift over time, the framework pairs its accuracy metrics with continual-learning metrics, forgetting and transfer, scoring whether competence on earlier constraint regimes survives training on later ones. That is commitment (2) folded into commitment (3): the difficulty knob (a shifting stream) demands a metric family (forgetting) that no static benchmark needs. On the previous chapter's axes, each step's constraint network is a join (several perceived variables bound by one constraint, the query-joins axis), the horizon reincarnates derivation depth as length in time, and the granularity knob turns grounding trust: per-sequence labels are the regime in which the latent step concepts are most weakly pinned. The export is the time axis itself, with forgetting as its native metric.

The grounding frontier: vision aligned to a real ontology

Every suite so far keeps perception simple so that the symbolic ground truth stays exact. NeSy4VRD inverts the trade: real photographs, and a real Web Ontology Language (OWL) ontology [5]. The resource is built around visual relationship detection: annotating images with subject-predicate-object triples such as "person rides horse," a task where the label is a knowledge-graph edge. Its distinctive move is alignment: the annotation vocabulary of object classes and predicates is matched to a companion OWL ontology that gives those classes and predicates types, hierarchies, and property semantics, so an image's annotations load directly into a knowledge graph that Volume 2's machinery can reason over.

That alignment buys something concrete: materialization as supervision. Run an OWL reasoner over the loaded annotations and the ontology entails triples nobody hand-annotated: superclass and superproperty generalizations of each asserted relationship, and the consequences of declared property semantics. Materialization versus Rewriting priced this operation as a scaling strategy; here it plays a data role, densifying the supervision each annotated image yields and supplying evaluation targets that grade semantic families rather than isolated strings. This is where Volume 2 meets pixels, and it sharpens what "NeSy benchmark" means when one side is perception. The symbolic side keeps commitment (1) fully: the ontology is exact ground truth for the relational structure. The perceptual side cannot: no generator produced the photographs, so there is no gold latent for what a pixel region "really" contains, only human annotation with human noise. The suites of the previous sections bought exact latents by synthesizing perception; NeSy4VRD buys real perception by giving up generated latents on the visual half. That trade is not a defect of either design. It is the frontier itself, and the honest checklist at the end of this chapter will pay for our own position on it.

The harness: one protocol, five instruments, one report card

Volume 5 has built five measuring instruments, one per trust failure: shortcut metrics (Part II), calibration (Part III), faithfulness probes (Part I), the difficulty surface over an explicit depth knob (Part V), and entailment-gap consistency probes (the next chapter). The volume's closing claim is that these are not five demos but one evaluation protocol: every instrument pairs two systems that a plain accuracy leaderboard cannot tell apart with a beyond-accuracy metric that separates them decisively. suite_harness.py makes that claim runnable. It imports all five modules, calls each one's run(), which re-executes that instrument's entire experiment (training loops, enumerations, probes) along with its own asserts, and emits one consolidated report card, built row by row in run() (suite_harness.py lines 340–379) and printed by the __main__ block (lines 400–415).

The seed policy is the whole of commitment (4) in one sentence: the harness owns no randomness at all; every module seeds its own generators, so apart from the labeled wall-clock rows, two runs print byte-identical output. The instruments in turn re-derive rather than retype their inputs. The calibration probe retrains Volume 3's ComplEx embedding from its committed seed inside its own run (calibration.py lines 233–241); the faithfulness probe regenerates its 512-example held-out set from data seed 0 (faithfulness.py lines 330–337); the shortcut probe re-runs the Part II training sweep to recover the collapsing model (rsbench_lite.py lines 276–288). Nothing is cached, nothing is read from a results file.

The sharpest design feature is the regression discipline. The harness commits, in source, the headline numbers the instruments' summary lines and tables print, the same digits the volume's evidence chapters quote (suite_harness.py lines 93–130):

    "calibration": {                     # calibration.py, SUMMARY line
        "temperature tau*": "1.4172",
        "eval ECE  before->after": "0.0245->0.0167",
        "eval NLL  before->after": "0.1708->0.1387",
        "eval acc  before=after": "0.9683=0.9683",
        "eval AUC  before=after": "0.9136=0.9136",
    },

and every collector re-derives its instrument's values from the fresh run and asserts equality, string against string, digit for digit (suite_harness.py lines 133–143):

    for key, got in derived.items():
        assert got == want[key], \
            f"{instrument} regression: {key!r} committed {want[key]!r}, " \
            f"re-derived {got!r}"

The design reason deserves stating, because it is the suite-maintenance lesson of this chapter in executable form: suites drift unless they check themselves against their sources. A drive-by edit to any module, any sibling-volume dependency (the Volume 3 trainer, the Volume 4 corpus, the Volume 1 prover), or any seed changes some digit somewhere, and this file is where that digit fails loudly instead of silently corrupting a chapter's quoted evidence. The gate even guards its own coverage: _regress first asserts that the committed and re-derived key sets are identical (suite_harness.py lines 137–139), so a silently dropped check fails too. This is the same role validate.py has played since Volume 1, an exit code as the acceptance verdict, now operating at the level of research claims: python3 suite_harness.py exiting green means every committed number in Volume 5's evidence chain still reproduces, 26 values across five instruments and four sibling volumes of imports.

note

What the miniature honestly simplifies: the five instruments probe five different toy systems (a concept extractor, a link predictor, a salience classifier, a multilayer perceptron (MLP) ladder, and Volume 4's two reasoners), so the report card's rows share a protocol and a moral, not a single system under test. A full-scale suite run points every instrument at the same model (suite_harness.py lines 53–59). The suite-level asserts guard the cross-system margins the volume actually claims, not published effect sizes.

The committed report card, verbatim from the run:

[2] the report card: five instruments, each pairing two systems that plain
    accuracy calls a tie and a beyond-accuracy metric that separates them
    instrument    metric beyond accuracy               statistical read           structural read             separation   what plain accuracy sees
    shortcuts     concept accuracy AccC                label-only        0.7500   concept-sup 10%    1.0000      +0.2500   AccY 1.000 = 1.000
    calibration   10-bin ECE (lower = better)          p = sigma(s) raw  0.0245   p = sigma(s/tau*)  0.0167      -0.0078   acc 0.9683 = 0.9683, AUC 0.9136 = 0.9136
    faithfulness  rationale comprehensiveness          attention top-2   0.2497   occlusion top-2    0.3659      +0.1162   one model, probe acc 0.9609
    difficulty    parity cliff length (train n <= 6)   depth-1 MLP            3   depth-3 MLP             6           +3   parity acc at n=2: 1.00/1.00/1.00
    entailment    worst-family consistency             soft reasoner     0.4859   translate-prove    1.0000      +0.5141   plain acc 0.9896 vs 1.0000

Each row's separation is itself asserted against a per-row margin, in the right direction (suite_harness.py lines 363–367): the report card cannot silently stop separating.

Reading the report card as a research summary

The table is a mini state of the field for the academic world: five instruments, ten system readings, and in every row an accuracy-flavored column that sees nothing. Read two rows aloud.

The shortcuts row is Part II compressed to one line. Two models from the same initialization, trained by the same machinery on the same parity task: one from labels only, one with concept labels on ten percent of points. The final column is what a leaderboard publishes, label accuracy $1.000 = 1.000$, a perfect tie. The metric column is what Part II built: concept accuracy $0.7500$ against $1.0000$, a gap of exactly one quarter, because the label-only model landed on the collapsing relabeling that Part II counted in advance (it maps the two-bit concept state 10, first bit on and second off, onto the state 01 and never predicts 10 at all). Behind the one-line row sit the rest of the panel's committed values, re-derived this run: collapse $0.250$ versus $0.000$, and concept-level expected calibration error (ECE) $0.2467$ versus $0.0013$, a factor of roughly 190, the confident-and-wrong signature that Measuring Shortcuts diagnosed. Ground truth for the latent property (commitment 1) is what makes this row computable; metrics beyond accuracy (commitment 2) are what make it visible.

The difficulty row is Part V compressed to one line. An MLP ladder trained on parity instances of length $n \le 6$, evaluated along the explicit length knob. The accuracy view, parity at $n = 2$: every depth scores $1.00$, a three-way tie. The difficulty surface separates them: the depth-1 model's accuracy cliff arrives at instance length 3, the depth-3 model holds to length 6, separation $+3$, exactly the mechanism The Transformer Depth Ceiling isolates: a fixed budget of combination rounds fails, in the solutions gradient descent actually finds, once instances demand more rounds than the model has. And the surface is honest in the other direction too: past the training horizon, every depth's committed tail accuracy is coin-flip ($0.506/0.500/0.500$), so depth bought the trained range, not extrapolation. Controllable difficulty (commitment 3) is what makes this row exist at all: no single test set at any fixed length could display a cliff.

The remaining rows carry their own Parts in the same way: the calibration row is Part III's temperature repair leaving accuracy and ranking untouched ($\tau^* = 1.4172$, ECE $0.0245 \to 0.0167$) while changing what the confidences mean; the faithfulness row is Part I's attention-versus-occlusion verdict read off one single model; and the entailment row, the largest separation on the card at $+0.5141$ of worst-family consistency against a $1.04$-point accuracy race, is the next chapter's opening exhibit, deliberately previewed here as one row and unpacked there in full.

The honest checklist

The chapter's rubric would be marketing if it were only ever applied to other people's suites. The harness ends by scoring itself, printing a yes/no verdict per property (suite_harness.py lines 303–324, printed as section [4] of the run):

[4] the design-principles checklist: what the exemplar suites
    institutionalize, scored honestly against this mini-suite
    [yes] ground-truth latent structure known by construction
          gold concepts g per split (rsbench_lite); closed-world lfp labels (calibration, entailment_gap); the generator's causal features (faithfulness); exact S_3 products (depth_ceiling)
    [yes] controllable difficulty knobs in the generators
          support_frac + n_distractors (rsbench_lite); instance length n = 2..16 (depth_ceiling); no knob on the other three instruments
    [yes] metrics beyond accuracy
          AccC / Cls(C) / concept ECE; calibration ECE; comprehensiveness & sufficiency; cliff position; per-family consistency
    [yes] one seeded protocol, byte-identical reruns
          every module seeds its own rngs; the harness adds no randomness and re-derives, never caches
    [ no] multiple model families per instrument
          one extractor family each; rsbench's harness compares six
    [ no] perceptual concept extraction
          Gaussian corner clusters and token counts, not MNIST digits, Blender scenes, or LogiCity's simulated city
    [ no] public scale: big OOD splits, (eps,delta)-guaranteed counting
          exact enumeration and 10^2-10^3-point probes throughout
    score: 4 yes / 3 no — the no-rows are the
    distance between a runnable miniature and a citable benchmark suite

Fold the seven printed properties back onto the chapter's four commitments and the grades come out: commitment (1), yes, gold everything at toy scale, from generated concepts to closed-world least-fixpoint labels to exact group-theoretic products; commitment (2), yes, five metric families, one per instrument; commitment (3), partial, and the justification line says exactly where: difficulty knobs exist on two instruments (the observed support and distractor count; the instance length), the previous chapter's seven-axis generator extends that, but there is no perceptual difficulty and no continual stream, which is why the perception and scale rows below it say no; commitment (4), yes, seeded generation throughout, byte-identical reruns asserted by the regression gate itself. The concessions are not humility theater; they are load-bearing and enforced. The harness asserts that the checklist concedes at least two properties (suite_harness.py lines 371–372):

    assert sum(1 for _, has, _ in CHECKLIST if not has) >= 2, \
        "the checklist stopped conceding anything"

A future edit that quietly upgrades every row to yes fails the build with the message "the checklist stopped conceding anything." That is the meta-point, and it is the chapter's real export: the checklist format, four commitments scored yes, partial, or no with a one-line justification each, is a rubric a reader can hold any suite against, including the five surveyed above, including this one, including the reader's own next benchmark. The closing summary line makes the whole chapter one string:

SUMMARY suite_harness: checks=26/26 accC_gap=+0.250 ece_gap=-0.0078 comp_gap=+0.1162 cliff_gap=+3 cons_gap=+0.5141 checklist=4y/3n

The unsolved part

Three open problems sit on top of even a well-built suite, and the field has first drafts for none of them.

The first is maintenance. Procedural generation defeats leakage of instances, but not adaptation to the generator. Every generator has quirks: a distribution of clause lengths, a renderer's texture statistics, a curriculum's fixed stage boundaries, and systems under sustained optimization pressure learn the quirks exactly as earlier systems learned static test sets. Goodhart operates one level up: the suite's recipe becomes the memorized route. The remedy, revising generators as systems adapt, is an arms race with a measurement cost: every revision silently changes what the numbers mean, breaking comparability with every result published on the previous version. Our own harness illustrates the discipline that at least makes drift visible (26 asserted digits fail loudly when anything moves), but visibility is not immunity: the asserts protect the suite from its authors, not from the systems it measures.

The second is cross-suite comparability. The five suites of this chapter each control difficulty, but in incommensurable units: an abstraction mode of the simulator, a curriculum stage of the pattern generator, a temporal horizon of the constraint streams. There is no shared difficulty calibration, no exchange rate saying that expert-mode city navigation "equals" stage-nine pattern induction, and consequently no way to say which suite is the harder examiner or whether a system strong on one should be expected to be strong on another. Psychometrics solved the analogous problem for human test items decades ago (item-response models put examinees and items on one scale); nothing of the kind exists for reasoning benchmarks, and until it does, "state of the art across benchmarks" is a phrase without a unit.

The third is the perception gap at desk scale, and it is this volume's own concession written as a general truth. Our suite is symbolic-first by construction: its "perception" is Gaussian clusters and token counts, because those are what a NumPy companion can regenerate deterministically in seconds. The visual suites' whole contribution, rendered scenes, simulated cities, real photographs against a real ontology, lives in rendering and annotation pipelines that no textbook companion can honestly carry. The checklist's perceptual no-row is therefore not a missing feature but a boundary of the genre: the properties this volume can verify end where the pixels begin.

Why it matters

This chapter is where the volume's five Parts pay into one account. Since Volume 1, the series' method has been a harness whose exit code is the acceptance verdict; what has changed is the altitude of the claims the harness guards, from "the forward-chainer reaches its fixpoint" to "the confident model's concepts are wrong in a counted way." The suite is that method applied to research claims, and the four commitments are the method stated so it can leave the book. For a reader heading into their own research, the export is concrete: before trusting a benchmark result, yours or anyone's, ask the four questions. Is the ground truth for the property under test known by construction, or only the labels? Do the metrics see the failure you care about, or only accuracy? Can the difficulty be dialed, or will the set saturate? Can anyone regenerate the instances from a seed, or is the test set already in someone's training corpus? Suites that answer all four exist now; the checklist for grading the rest fits in a page of source. And the regression discipline generalizes past benchmarks entirely: any research artifact with committed numbers can carry a file that re-derives and asserts them, which is the cheapest reproducibility instrument this series knows.

Key terms

• Benchmark suite: a family of seeded task generators, a metric panel, and a fixed evaluation protocol shipped as one instrument; a single benchmark measures a task, a suite institutionalizes a methodology.
• Leakage: contamination of a training corpus with test items (or their paraphrases), after which the benchmark measures recall of itself; the failure seeded procedural generation defends against.
• Saturation: the regime where measured accuracies approach the ceiling and the test set's standard error $\sqrt{p(1-p)/m}$ makes nearby systems statistically indistinguishable.
• Goodhart's law: when a measure becomes a target, it ceases to be a good measure; for latent properties like concept meaning, Part II's identifiability theorem makes it unavoidable rather than accidental whenever the knowledge admits a nontrivial label-preserving relabeling.
• Procedural generation: producing task instances from a generator $G(\theta, s)$ of difficulty configuration $\theta$ and seed $s$; fresh instances on demand, byte-identical regeneration from a committed seed.
• Compositional split: a train/test division in which test instances combine familiar components (concepts, rules) in arrangements never seen together in training.
• Supervision granularity: the level at which labels attach (per step versus per sequence, per concept versus per label); a suite knob that controls how distant the supervision is.
• Materialization as supervision: running a reasoner over ontology-aligned annotations so entailed triples densify the supervision and evaluation targets beyond what was hand-annotated.
• Regression discipline: committing every headline number in source and re-deriving it on every run, asserting equality digit for digit, so drift anywhere in the evidence chain fails loudly.
• Report card: the consolidated system-by-instrument table a suite emits under one protocol; here, five instruments each pairing an accuracy tie with a beyond-accuracy separation.
• Honest checklist: a suite's self-grade against the design commitments, printed yes/no per property with justifications, with the concessions enforced by assertion.

Where this leads

The report card's largest separation, the entailment row, was deliberately left folded: a soft reasoner and a translate-and-prove pipeline sit $1.04$ accuracy points apart while their worst-family consistencies sit $51.41$ points apart. The next chapter, The Entailment-Benchmark Gap, unpacks that row into a property the suites of this chapter do not yet test: entailment is closed under contraposition, irrelevant expansion, paraphrase, and De Morgan rewriting, a system that computes entailment inherits those closures as theorems, and a system that predicts labels does not. Measuring the difference requires probe families linked by logic rather than instances sampled independently, which is a fifth design commitment waiting to be institutionalized.

---

Companion code: examples/frontier/suite_harness.py re-runs the volume's five instruments (rsbench_lite.py, calibration.py, faithfulness.py, depth_ceiling.py, entailment_gap.py) under one seeded protocol, passes all 26 committed headline numbers through the regression gate, and prints the report card and the checklist quoted in this chapter. Run python3 examples/frontier/suite_harness.py; apart from the labeled wall-clock rows, the output is byte-identical run to run, and the exit code is the verdict.