Doing the Next Research: Blind Spots and First Steps

📍 Where we are: Part VIII · The Research Runway — Chapter 22. Open Problems mapped the eight gaps G1–G8 and left them deliberately open; this chapter is the method for picking one up: the discipline the series has been demonstrating for five volumes, made explicit, and a worked plan for starting on a gap Monday morning.

Every chapter so far has taught a result. This one teaches the procedure that produced the results, because the procedure is the part the reader takes into their own research. The claim of this chapter is that research method is mostly discipline under uncertainty, and that this series has been demonstrating one particular discipline all along without naming it. It has five parts: build the exact reference before the learned system, write every claim as an experiment before building anything, make every run deterministic and every quoted number an executable assertion, audit your own work with the named failure modes of your field, and publish the ledger including the columns that hurt. None of these is deep. All of them are habits, and habits are exactly what survives contact with a problem nobody has solved before.

The simple version

Imagine a pilot's routine, not a genius's flash of insight. Before takeoff the pilot files a flight plan that says where the aircraft is going and how arrival will be verified; the instruments were calibrated on the ground against a reference that is known to be right; the flight recorder logs everything, so any claim about what happened can be replayed rather than remembered; the pre-flight checklist walks through the known ways aircraft fail, one by one, and each item is checked, not felt; and after an incident the report states plainly what went wrong, because a report that hid the failure would get the next crew killed. Doing research well is this routine transplanted: the reference implementation is the ground calibration, the claims table is the flight plan, the seeded and assert-guarded companion suite is the flight recorder, the failure-mode checklist is the walk-around, and the honest ledger is the incident report. Nothing here requires brilliance. It requires refusing to skip steps precisely when you are most tempted to.

What this chapter covers

• The hidden curriculum made explicit: three moments the reader already lived, Volume 2's HermiT cross-check, Volume 4's controlled three-geometry comparison, and this volume's trace-versus-MinA (minimal axiom set) check, each named as an instance of a general discipline.
• Oracle-first: why the exact reference comes before the learned system, what it honestly costs (this series lives on a 13-entity world for a reason), and the recipe when no oracle exists at target scale: build one at toy scale and plant the ground truth.
• Claims as experiments: the capstone's matrix generalized into a pre-registration discipline, one real row walked in full, and the toy-verified versus cited-only column as intellectual honesty made typographic.
• Determinism as epistemology: seeds, byte-identical reruns, asserts on semantic properties, and the suite_harness.py regression discipline that re-derives every committed headline number, digit for digit, on every run.
• The adversarial self-audit: this volume's five named failure modes turned into a pre-publication checklist, each row pointing at the committed instrument this volume built for it.
• The Monday plan: a complete miniature research plan for one gap, G5 cross-ontology transfer, from instrument-readiness check to publication ledger, small enough to fork.
• The blind-spot inventory: what this series' own method systematically under-weights, published on purpose, because a methods chapter that exempted its own book would fail its own audit.

The hidden curriculum, made explicit

Method is easiest to teach by receipts, so start with three moments the reader has already lived, and name the pattern each one instantiates.

The first is Volume 2's oracle cross-check. The from-scratch EL completion reasoner was never asked to be trusted on its own word: examples/symbolic/reasoners.py drives HermiT, a complete OWL 2 DL reasoner, through owlready2 and checks the hand-built completion's subsumptions and unsatisfiable concepts against it, class for class. The pattern: never let a system you built grade its own homework. An independent implementation of the same semantics, ideally one you did not write, is the cheapest strong evidence there is.

The second is Volume 4's three-geometry comparison. When the complex-query chapters compared GQE's (Graph Query Embedding's) points, Query2Box's boxes, and BetaE's beta distributions, clqa_models.py ran all three under one protocol: one query bank, one training budget, one evaluation, so that every observed difference was attributable to the geometry and nothing else; the acceptance harness asserts that the resulting coverage table matches the structural theory cell for cell (examples/integration/validate.py, lines 85–87). The pattern: a comparison is an experiment only if everything but the variable of interest is controlled, and the control must be checked by code, not by intention.

The third is this volume's trace-versus-MinA cross-check. The SATORI-lite kernel records a derivation trace for every atom it derives; Chapter 1's justifications.py computes true minimal axiom sets (MinAs) with Reiter's hitting-set tree, a wholly independent instrument. The claims evaluation asserts that the kernel's discretized trace contains a true MinA for every tested entailment (satori_eval.py, lines 494–510), and the committed run prints the receipt:

  TBox face — the discretized trace vs justifications.py:
    Professor ⊑ Researcher: trace axioms {1} ⊇ MinA {1}   (all MinAs: {1} {4, 5})
    TenuredStudentAdvisor ⊑ ⊥: trace axioms {6, 10, 11, 12} ⊇ MinA {6, 10, 11, 12}   (all MinAs: {6, 10, 11, 12})

The pattern: an explanation claim is checked against an explanation oracle, not against how convincing the explanation looks. Three moments, three instances of the same underlying stance: every claim gets an instrument, and the instrument is independent of the thing it measures. The machine-learning literature has named the failure patterns that flourish when this stance is absent: explanation by suggestion, gains without identified sources, mathematics as decoration [1]. The five disciplines below are the constructive complement, and the rest of this chapter walks them one at a time.

discipline	the moment already lived	the committed receipt
oracle-first	Volume 1's chainer, Volume 2's completion, Volumes 4–5's symbolic executors	every soundness probe and crisp-limit assert in five volumes
claims-as-experiments	the capstone's C1–C8 matrix	satori_eval.py claims table: 5 toy-verified, 3 cited-only
determinism-as-epistemology	byte-identical companion runs, exit-code verdicts	suite_harness.py: 26 headline values re-derived digit for digit
adversarial self-audit	shortcuts, calibration, faithfulness, truncation, consistency chapters	one committed instrument per named failure mode
honest ledgers	every volume's verdict chapter	buys / pays / guards, unpaid debts listed

Five disciplines, each introduced by a receipt the reader already lived, converging into one forkable Monday-morning plan, with the series' own blind-spot inventory printed underneath.

Original diagram by the authors, created with AI assistance.

Oracle-first: the reference before the learner

The first discipline is an ordering constraint. Before any learned system exists, build the oracle: an exact, exhaustively checkable implementation of the semantics your learned system will later be measured against. Volume 1 built the forward chainer before any embedding existed; Volume 2 built the EL completion before any box was drawn around a concept; Volumes 4 and 5 built symbolic query executors before any soft kernel touched a query. The argument for the ordering is falsifiability. A learned system evaluated without an exact reference can only be scored against labels, and labels grade answers, not reasoning. "The model works" is then an unfalsifiable sentence, because there is no committed procedure whose failure would refute it. With an oracle, every soft claim acquires a crisp shadow that can be checked: the kernel's answers must equal the executor's in the crisp limit (all 36 queries, asserted in satori_eval.py, lines 424–431), truncation must cost exactly the atoms the oracle derives beyond the horizon, an explanation must contain an oracle-verified MinA. Every soundness probe, every recall-versus-budget table, and every crisp-limit assert in five volumes is downstream of an oracle that existed first.

The cost must be priced honestly: oracles constrain scale. An exact reference is exhaustive by nature, closures are enumerated, MinAs are searched, query banks are executed symbolically, and exhaustiveness is exponential company. This is the reason the entire series lives on a 13-entity academic world with 23 base facts (18 of them graph edges) and a 14-axiom TBox. The choice was not modesty; it was the largest world on which every claim could still be checked against an exact reference in seconds. The trade runs one way at research time: a claim verified on a small world is worth more than a number reported on a large one that nothing can check, because the first is evidence and the second is testimony.

The practical recipe when no oracle exists at your target scale has two moves, and both appear in this volume's companion as worked patterns.

Move one: build the oracle at toy scale. Shrink the problem until exact semantics is computable, then demand exact agreement there. The toy does not prove the method works at scale; it proves the method is not already broken at any scale, which is the cheaper and more frequently failed test.

Move two: plant the ground truth. When the latent structure you claim to recover has no natural oracle, construct worlds where you know it by construction. The planted-rule generator is the pattern in miniature: the grandAdvisor rule is hidden from the learner, a softmax router over all 25 length-2 chain templates is trained only on query answers, and the acceptance criterion is not a score but an exact identity, the decoded rule must equal the planted rule (satori_eval.py, lines 467–471):

    alpha, rlosses = train_router(feats, G)
    top = int(np.argmax(alpha))
    decoded = ("chain", *TEMPLATES[top], "grandAdvisor", False, False)
    planted = satori_lite.compile_horn([kb.RULES[3]])[0]  # the hidden rule
    assert decoded == planted, f"decoded {decoded} ≠ planted {planted}"

The same move powers the shortcut metrics: rsbench_lite generates tasks whose gold concepts are known per split, so concept quality is measured against planted truth rather than eyeballed. Planting is what turns "the model learned something" into "the model recovered this, exactly, and here is the assert that would have failed otherwise."

Claims as experiments: the pre-registered matrix

The second discipline governs what you write down before you build. The capstone froze SATORI's eight claims into a matrix, and the generalization is a format: every claim is a row with five fields,

$$\langle\; \text{claim},\ \text{metric},\ \text{baseline},\ \text{instrument},\ \text{phase} \;\rangle,$$

where the claim is a falsifiable sentence, the metric is the number that decides it, the baseline is what the number must beat (and, when honesty demands, what it must not exceed), the instrument is the committed code that produces the number, and the phase says when in the project the row comes due. A claim written this way is a bet. A claim missing any field is a vibe: without a metric it cannot be decided, without a baseline it cannot be contextualized, without an instrument it cannot be reproduced, and without a phase it can be deferred forever.

Walk one real row. Claim C1, multi-hop EPFO (existential positive first-order) answering, decomposes as: claim, the Gödel kernel answers existential-positive queries over a calibrated neural adjacency; metric, filtered mean reciprocal rank (MRR) on the bank's 17 hard answers, a hard answer being an entity that answers its query on the full graph but that no traversal of the training graph alone can reach; baselines, a random scorer (the uninformed floor), the symbolic traversal of the training graph (zero on every hard answer by definition), and the full-graph oracle (the ceiling); instrument, godel_eval over the calibrated adjacency (satori_eval.py, lines 100–151) under Volume 4's frozen protocol; phase, toy-verified now. The committed asserts are the pre-registration made executable (satori_eval.py, lines 448–450): the kernel must beat the random floor by at least 0.25 MRR, beat traversal by at least 0.30, and, remarkably, must stay below 0.8, because a 16-dimensional toy imputer that looked closer to the oracle than that would signal leakage, not brilliance. The committed run resolves the bet: kernel 0.5680, random 0.2301, traversal 0.1488, oracle 1.0000. Pre-registering the upper bound alongside the lower ones is the discipline at full strength: the claims table constrains the result in both directions before the result exists.

What the format prevents is the pair of failures that pre-registration was invented for: moving goalposts (redefining success after seeing the numbers) and post-hoc metric shopping (scanning metrics until one flatters). Both are live risks, not hypothetical ones; the re-evaluation literature showed that knowledge-graph-embedding leaderboards reordered substantially when training protocols were equalized, which means the original orderings were partly measurements of protocol choice, the methods lesson Volume 3 inherited [2]. What the format costs is public failure: a pre-registered row that comes up short is visible. That cost is the reason for the scoping rule, write rows only against instruments that exist or that you commit to building; and when the instrument does not exist, building it is a publishable research contribution in its own right, which is precisely how the rsbench suite earned its NeurIPS slot [3].

The final column is where honesty becomes typographic. satori_eval.py runs the largest honest subset of the matrix and prints, per claim, either toy-verified with its number or cited-only with the reason (satori_eval.py, lines 628–651). One symbol in the table needs its decode: τ is the kernel's temperature, the parameter of the softmin and softmax that smooth its AND and OR, and τ→0 is the crisp limit in which both collapse back to the exact Gödel min and max. The committed table:

[5] the claims table (C4/C6/C7 stay with the paper: stated, not stretched)
  C1 multi-hop EPFO answering   toy-verified  MRR 0.568 vs random 0.230 / oracle 1.000
  C2 router learns rules        toy-verified  advises∘advises recovered, α = 0.972, exact decode
  C3 faithful proof traces      toy-verified  comp 8/8, suff 4/4, trace ⊇ MinA 2/2
  C4 GPU-batched scalability    cited-only    proxy gpu_fixpoint.py; needs PyReason/Scallop at scale
  C5 calibration + abstention   toy-verified  ECE 0.019; coverage 0.821 @ acc-on-answered 1.000
  C6 ontology grounding Δ MRR   cited-only    ablation degenerate here; Box2EL / TransBox carry it
  C7 cross-ontology transfer    cited-only    needs a 2nd ontology; RRN is the failing baseline
  C8 crisp soundness (τ→0)      toy-verified  satori_lite re-run: Horn 47/47, EL 46/46, complete iff L ≥ 3

Five rows carry numbers; three rows carry reasons. The cited-only rows are not weaknesses of the table, they are the table working: each states exactly what evidence is missing and where the missing evidence currently lives. Keep C7 in view; it becomes the Monday plan.

Determinism as epistemology: a paper that checks itself

The third discipline sounds like engineering and is actually epistemology. The engineering rules are four, and every companion suite in this series obeys them. Seed every source of randomness, so that "the run" is a well-defined object rather than a family of anecdotes. Demand two-run byte-equality of the full output, so that any nondeterminism announces itself instead of hiding in a fourth decimal place. Write asserts on semantic properties, exact rule recovery, soundness against the oracle, margins between systems, monotone recall, rather than on raw floats plucked from one lucky run, so the checks state what the experiment means. And exclude wall-clock timing from every assert, because timing is the one number a machine is entitled to change (suite_harness.py, lines 61–63 and 352–353).

The epistemic reading is what the rules buy. A deterministic, assert-guarded companion is a paper that checks itself: every number quoted in prose is simultaneously a standing regression test, an executable statement that fails loudly the moment it stops being true. This is what allowed five volumes to quote each other without drift. When Volume 5's calibration chapter cites Volume 3's ComplEx scores, it does not trust a two-volume-old copy-paste; it re-imports the trainer, re-runs it under the committed seed, and asserts the numbers. The reproducibility movement institutionalized the community-scale version of this, checklists, seeds, and code submission at review time [4]; the companion suites are the same policy enforced continuously rather than once at submission.

The volume-scale enforcement lives in suite_harness.py. Its COMMITTED dictionary freezes all 26 headline values that the evidence chapters quote, at each owning module's own printed precision, and its regression gate re-derives every one from scratch on every run, no caching, and asserts equality digit for digit (suite_harness.py, lines 133–143):

def _regress(instrument: str, derived: dict[str, str]) -> None:
    """The regression gate for one instrument: every re-derived headline
    string must equal its committed value exactly.  Committed and derived
    must also cover the SAME keys, so a silently dropped check fails too."""
    want = COMMITTED[instrument]
    assert set(derived) == set(want), \
        f"{instrument}: check set drifted ({set(derived) ^ set(want)})"
    for key, got in derived.items():
        assert got == want[key], \
            f"{instrument} regression: {key!r} committed {want[key]!r}, " \
            f"re-derived {got!r}"

Note the first assert: the set of checks is guarded too, so a check cannot be silently deleted when it becomes inconvenient. The committed run's ledger shows the discipline at work; here is the calibration block of the output:

    calibration (calibration.py):
      temperature tau*                         1.4172                       = re-derived  OK
      eval ECE  before->after                  0.0245->0.0167               = re-derived  OK
      eval NLL  before->after                  0.1708->0.1387               = re-derived  OK
      eval acc  before=after                   0.9683=0.9683                = re-derived  OK
      eval AUC  before=after                   0.9136=0.9136                = re-derived  OK

and the run's one-line verdict, SUMMARY suite_harness: checks=26/26 ..., is the volume's evidence chain certifying itself: a drive-by edit to any module, any sibling-volume dependency, or any seed surfaces as a failed equality in this file. The same idea at acceptance granularity is Volume 4's validate.py, one named competency check per chapter, each calling its module's run() with all committed asserts live, and the process exit code as the acceptance verdict: 0 if and only if all 21 checks pass (examples/integration/validate.py, lines 31–125). Continuous verification is then literally one command.

The adversarial self-audit: every failure mode maps to a test you ran

The fourth discipline turns this volume's syllabus into a pre-publication checklist. Volume 5 spent its first six parts naming the ways a reasoning system can be wrong while looking right; the audit discipline says: before you publish, run your own system against every named failure mode of your field, using the instrument built for it. For this volume's five, the checklist rows and their committed instruments are:

failure mode	the pre-publication question	the committed instrument
reasoning shortcuts	does anything with latent concepts get the right labels from the wrong concepts?	rsbench_lite.py: shortcut counting, AccC (concept accuracy), Cls(C) (concept collapse, one minus the normalized entropy of the predicted-concept occurrence vector), concept ECE (expected calibration error) against planted concepts
miscalibration	do the confidences mean anything, before you quote them?	calibration.py: 10-bin ECE, temperature fit, before/after ledger
unfaithful explanation	do the explanations move with the mechanism, before you show them?	faithfulness.py: erasure-based comprehensiveness and sufficiency, random-rationale control
silent truncation	does completeness survive the compute budget you actually run at?	satori_lite.py recall-versus-budget: complete iff the unroll depth covers the closure
inconsistency	does "reasoning" survive logically equivalent rephrasings?	entailment_gap.py: contraposition, expansion, paraphrase, De Morgan probes

Each row exists because plain accuracy cannot see the failure: the harness's report card pairs, per instrument, two systems that accuracy calls a tie and the beyond-accuracy metric that separates them decisively (label accuracy 1.000 = 1.000 while concept accuracy splits 0.750 versus 1.000; plain accuracy 0.9896 versus 1.0000 while worst-family consistency splits 0.486 versus 1.000). The suite's asserts guard those separations as margins, not vibes (suite_harness.py, lines 363–367). The meta-rule generalizes past this volume: enumerate the named failure modes of your subfield, and for each one either point at the test you ran or say out loud that you did not run it. Auditing your own work adversarially before reviewers do is not masochism; it is the difference between a paper that survives scrutiny and a paper that has not yet received any [1]. And when your field's checklist has a row with no instrument, you have found a research contribution waiting to be built, which is how the shortcut-measurement suite itself came to exist [3].

Honest ledgers: report the columns that hurt

The fifth discipline is a publication format. Every volume of this series closes with a verdict chapter whose form is a ledger: per method, what it buys, what it pays, and the committed guard that certifies the row. The format's load-bearing property is that the pays column has equal standing with the buys column. This volume's own numbers supply the habit's texture: the kernel's headline MRR of 0.5680 ships in the same table as its worst per-type number (ip, the intersection-then-projection query type, at 0.111) and the unclosed gap to the oracle's 1.0; the abstention result advertises accuracy 1.0 on answered queries and, in the same line, the price, coverage 0.8205, with 84 of 468 decisions refused; the mini-suite's design checklist scores itself 4 yes and 3 no against the properties real benchmark suites institutionalize, and the harness asserts that the concessions never quietly disappear (suite_harness.py, lines 369–372: "the checklist stopped conceding anything" is a failure condition). A ledger whose pays column is empty is marketing. A ledger that guards its own concessions with an assert is measurement.

The Monday plan: attacking G5 end to end

Now the method runs forward. Suppose the reader wants to start a real project Monday morning, on a real gap from the open problems map. The five disciplines compose into a sequence of seven steps; here they are, worked end to end on G5, non-transferability: reasoning systems that must be retrained per domain, embeddings that are transductive, the gap whose capstone row C7 is still cited-only.

Step 1: pick by instrument-readiness. Among the gaps you find interesting, prefer the one whose instruments already exist, because instrument-building time is the silent budget killer. G5 scores well. The claims row already names the failing baseline (per-domain retraining, the pattern of the RRN, the Recursive Reasoning Network) and states exactly what is missing: a second ontology (satori_eval.py, lines 644–645). The evaluation protocol exists (Volume 4's frozen query bank and filtered MRR). A transfer result at toy scale even exists as precedent: Volume 4's ultra_lite.py transferred a 12-parameter structural scorer zero-shot to a hospital world at MRR 0.83 against a random floor of 0.22, invariant under relation renaming (examples/integration/validate.py, lines 91–94). What has never been tested is whether the reasoning kernel with its TBox, not just a link scorer, transfers.

Step 2: scope a toy with an oracle. The target scale (SNOMED CT to Gene Ontology transfer, say) admits no oracle you can enumerate. So scope down: a second 13-to-15-entity world with its own EL TBox, disjoint vocabulary, comparable shape. The hospital world is the natural candidate, entities for clinicians, patients, wards, and procedures, roles like supervises and treats mirroring advises and cites. The oracle stack is entirely reusable: Volume 2's el_completion.py computes the new TBox's classification, reasoners.py cross-checks it against HermiT, Volume 1's chainer computes the Horn closure, and the Volume 4 executor computes exact query answers. Two afternoons of work, none of it novel, all of it load-bearing.

Step 3: plant the ground truth. Hide one chain rule in the hospital world, the analogue of grandAdvisor (for instance, a supervising-clinician rule composed from two supervises edges), and generate the query bank from the closed world exactly as query_dag.py did. Now every transfer claim has a planted, exact target: rule recovery is an identity check, not a score.

Step 4: pre-register the claims table. Written before any code, honestly small:

claim	metric	baseline / bound	instrument	phase
T1: the kernel schooled on the academic world answers hospital queries zero-shot	filtered MRR on the hospital bank's hard answers	above the random floor +0.25; below retrained-in-domain (the honest ceiling)	godel_eval on the hospital adjacency, protocol frozen from Volume 4	week 1
T2: rule recovery transfers	decoded rule == planted hospital rule, exactly	uniform router (chance = 1/25)	the C2 router re-instantiated on hospital templates	week 2
T3: crisp soundness survives the move	thresholded kernel == symbolic executor on all hospital queries at τ→0	exact equality, no tolerance	the C1 sanity check, new world	week 1
T4: confidences stay meaningful	hospital ECE ≤ 2× academic ECE (0.0186)	in-domain calibration	calibration.py's ECE helper on transferred degrees	week 2
T5: the failure boundary is real	at least one pre-named condition where transfer breaks (e.g. an unseen role signature)	none: this row exists to be filled honestly	ablation over vocabulary overlap	week 3

T5 deserves a comment: a transfer paper without a demonstrated failure boundary is a paper that did not look for one. Pre-registering the search for the boundary makes finding it a success rather than an embarrassment.

Step 5: build in order. Oracle first (the hospital TBox, its completion, the HermiT cross-check), generator second (the planted rule and query bank), baseline third (per-domain retraining, so the comparison exists before the system does), system fourth (the transferred kernel), audit fifth (the five-row checklist above, run on your own result). The ordering is the point: each stage is measured by the stages before it, so at no moment does an unmeasurable artifact exist.

Step 6: run the self-audit. Shortcuts: could the router decode the right rule from spurious co-occurrence in the tiny bank? Run the counting kit. Calibration: quote no confidence before T4's ECE exists. Faithfulness: erase trace leaves in the hospital world before claiming the traces explain anything. Truncation: verify the unroll depth covers the hospital closure's actual depth. Consistency: run the entailment probes on the new bank.

Step 7: publish the ledger. Buys: one cited-only row of the capstone matrix converted to toy-verified, with committed numbers. Pays: 15 entities is not SNOMED; one target domain is not a distribution over domains; the planted rule is one rule. Guards: the T1–T5 asserts, seeded, byte-identical, exit code 0. That is a small paper, and it is a real one: fork the plan, swap the gap, and the skeleton holds for G7 (run the same seven steps with calibration.py, abstention.py, and rsbench_lite.py as the ready instruments) or any gap whose instruments you can check off.

The blind-spot inventory, turned on this series

A methods chapter that exempted its own book would fail its own audit, so the last table is the one this series owes about itself. The five disciplines above are real, and they systematically under-weight five things, precisely because of what they optimize for.

Perception. Oracle-first favors worlds where an oracle can exist, and those worlds are symbolic-first. Our entities arrive pre-symbolized; nothing in five volumes ever had to decide whether a pixel region is alice. The vision-facing NeSy (neuro-symbolic) frontier, scene graphs, visual question answering, perceptual concept extraction, appears in this series only through citations, and the mini-suite concedes it explicitly: its concept extractors see Gaussian corner clusters and token counts, "not MNIST digits, Blender scenes, or LogiCity's simulated city" (suite_harness.py, lines 319–321).

Human factors. Every explanation in this series was evaluated by a machine: erasure probes, MinA containment, sufficiency scores. No human ever read one of our justifications and reported whether it helped, and no abstention policy was ever tested on a person deciding whether to trust a deferral. A faithfulness score is not a usefulness score; the two can and do diverge, and only one of them appears anywhere in our evidence chain.

Adversarial settings. There is no attacker in any threat model of this series. Our failure modes are statistical (shortcuts, miscalibration) or structural (truncation, inconsistency), never strategic. A knowledge base with a poisoned fact, a query crafted to exploit the softmax phantom, a calibration attack that reshapes confidences without touching accuracy: none of our instruments would notice, because none of them was pointed that way.

Compute realities. A NumPy desk is not a datacenter. The scaling chapters derived work-depth counts and cited the systems results, but every scaling number in five volumes above toy scale is cited, not measured; the capstone's C4 row says so in print. Batched-fixpoint behavior at 10⁹ facts, memory walls, communication costs: our method demonstrates the shape of these questions and has measured none of them.

Survivorship in the citations. We taught what worked. The reference lists of five volumes are populated by methods that survived; the graveyard of reasonable ideas that failed quietly, negative results that never reached print, is invisible in exactly the way survivorship bias predicts. A reader who calibrates "how hard is this field" on our bibliographies alone will underestimate it.

The committed miniature of this practice is the harness's own design checklist, which scores the mini-suite against the properties the exemplar benchmark suites institutionalize and prints its concessions with the same typography as its wins:

    [ no] multiple model families per instrument
          one extractor family each; rsbench's harness compares six
    [ no] perceptual concept extraction
          Gaussian corner clusters and token counts, not MNIST digits, Blender scenes, or LogiCity's simulated city
    [ no] public scale: big OOD splits, (eps,delta)-guaranteed counting
          exact enumeration and 10^2-10^3-point probes throughout
    score: 4 yes / 3 no — the no-rows are the
    distance between a runnable miniature and a citable benchmark suite

Publishing the inventory strengthens the work rather than weakening it, for the same reason the cited-only column strengthened the claims table: a stated limitation is a scoped claim, and a scoped claim is one that can actually be true. The alternative is not a book without blind spots; it is a book whose blind spots the reader must discover the hard way.

The unsolved part

The method itself has an unmeasured input: taste. Every discipline in this chapter operates after a problem has been chosen; none of them helps choose it. Instrument-readiness is a tiebreaker among gaps you already find promising, not a theory of which gaps matter, and the honest observation is that the choice of problem dominates the outcome of research more than any discipline applied downstream of it. Worse, the disciplines have a selection bias of their own, the same one the blind-spot inventory just named: oracle-first pulls toward problems where exact references can exist, claims-as-experiments pulls toward claims that resolve within a project's horizon, and both quietly push the hardest questions (perception grounded in the physical world, explanations judged by humans, robustness against adaptive adversaries) to the cited-only column indefinitely. Whether a method this committed to checkability can be extended to domains where ground truth is contested, expensive, or human, without collapsing back into vibes, is an open problem about method, and this chapter has no instrument for it.

Why it matters

This chapter is the series cashing out as practice. A reader who retains every theorem in five volumes but none of the habits will reproduce neither; a reader who retains only the habits will rediscover the theorems they need, because the habits force contact with exact references and committed evidence. The five disciplines are also, deliberately, cheap: nothing in this chapter required a cluster, a grant, or a team, only a fixed ordering of work (oracle, generator, baseline, system, audit) and the stomach to write claims down before results exist. For the reader heading into their own research, the transfer is direct: the G1–G8 map says where the open ground is, this chapter says how to move on it, and the companion suites are a working, forkable demonstration that the method runs end to end at desk scale. What the method cannot supply, problem taste, adversaries, humans, datacenters, it now at least names, which is what an instrument-first culture owes its blind spots.

Key terms

• Oracle-first: the ordering discipline that builds an exact reference implementation of the target semantics before any learned system, so every later claim has a committed procedure that could falsify it.
• Planted ground truth: latent structure (a rule, a concept assignment) inserted by construction into a generated world, so recovery can be checked as an exact identity rather than eyeballed.
• Claims-as-experiments matrix: the pre-registration format ⟨claim, metric, baseline, instrument, phase⟩; a claim missing a field is not yet an experiment.
• Toy-verified / cited-only: the claims-table column that separates what the committed companion actually demonstrates from what the surrounding literature is trusted for, with the reason stated per row.
• Determinism-as-epistemology: seeded runs, two-run byte-equality, asserts on semantic properties, timing excluded; the practice that makes every quoted number a standing regression test.
• Regression discipline: re-deriving every committed headline value from scratch and asserting digit-for-digit equality on every run, with the check set itself guarded against silent deletion (suite_harness.py).
• Adversarial self-audit: running your own system against every named failure mode of your field before publication, each mapped to its committed instrument.
• Honest ledger: the buys / pays / guards publication format, in which costs have equal standing with gains and the concessions are themselves assert-guarded.
• Instrument-readiness: the project-selection heuristic that prefers gaps whose measuring instruments already exist, because instrument-building is the silent budget of research.
• Blind-spot inventory: the published list of what a method systematically under-weights; a stated limitation is a scoped claim, and a scoped claim can be true.

Where this leads

The runway is built: the gaps are mapped, the instruments are committed, and the method for connecting a gap to an instrument now fits on one page. One obligation remains, and it is the oldest one in the series. Volume 1 opened by asking what symbols and vectors could possibly want from each other; five volumes of oracles, gradients, circuits, kernels, and audits later, the question is finally answerable with evidence instead of position. The Honest Verdict closes the series by answering it: what, on the committed record, symbols and vectors actually owe each other.

---

Companion code: examples/frontier/suite_harness.py is the volume's continuous-verification instrument (five instruments re-run under one seeded protocol, 26 committed headline values re-derived and asserted digit for digit, the design checklist with its guarded concessions), and examples/frontier/satori_eval.py runs the honest subset of the claims matrix, including the pre-registered C1 asserts, the planted-rule identity check, and the trace-versus-MinA cross-check quoted in this chapter. Run python3 suite_harness.py and python3 satori_eval.py in examples/frontier/ to reproduce every number above; apart from labelled timing rows, both runs are byte-identical across executions.