Abstention: Knowing When You Do Not Know

📍 Where we are: Part III · Calibration, Uncertainty, and Abstention — Chapter 8. Calibration repaired the link predictor's probabilities with one fitted temperature; this chapter spends those probabilities, and two older currencies from Volumes 2 and 4, on the right to say "I don't know."

A classifier that must answer every question has no way to be ignorant. Whatever it does not know, it converts into an error, and the error looks exactly like every other answer until someone checks. The oldest fix in pattern recognition is to add one more output, the reject option: let the system decline a question, and price the declining [1]. This chapter builds that option in full on the academic world: first the score-based machinery (selective prediction, the risk-coverage curve, its area, and the two-line derivation of the cost-optimal threshold, all on the calibrated probabilities the previous chapter fitted), then two abstentions that are not scores at all. An open-world abstention reads Volume 2's annotated intervals, where "I don't know" is a structural fact about the knowledge base, and a structural abstention re-quotes Volume 4's translate-then-prove pipeline, where every answer is a proof and every failure a visible Error. The chapter's verdict is one committed table comparing all three on the same terms: coverage, accuracy on the answered, and the rate of silent wrong answers, the quantity this series has been engineering toward zero since Volume 1.

The simple version

Imagine an exam with negative marking: a wrong answer costs a point, a blank costs a tenth of a point. Suddenly the smart move on a shaky question is to leave it blank, and your grade splits into two numbers: how many questions you attempted, and how well you did on those. Now notice there are three different reasons to leave a blank. "I studied this but I'm not sure" is a confidence blank; you set a personal threshold and skip anything below it. "The textbook genuinely never settles this" is a different blank; no threshold tuning changes it, because the material itself is undecided. And "I cannot even parse this question" is a third; you could not attempt it if you wanted to. This chapter measures the first kind exactly, then shows that the other two, though coarser, have something the first never has: when they fail, they fail out loud.

What this chapter covers

• Selective prediction, precisely: the predictor plus selection-function pair, coverage as the fraction answered, selective risk as the error rate on the answered, and the curve construction decoded (sort by confidence, sweep the threshold, one point per cut) before any formula.
• The curve on real scores: the exact risk-coverage sweep over the 126 calibrated questions, the committed curve table read at three anchors, the asserted half-coverage improvement, and the area under the risk-coverage curve (AURC) as the area summary, with what a single number hides.
• The Chow rule, derived: with abstention priced at $c_r$ and errors at 1, the expected-cost comparison solved in two lines for the threshold $\theta = 1 - c_r$, and why calibration is the precondition.
• Two other species of silence: the open-world interval policy, where an atom annotated $[0,1]$ is structurally undecided and no calibration set exists, and the structural policy, where a parser's Error verdict makes silent wrongness impossible by construction.
• The three-policy verdict: one committed table, coverage against accuracy-on-answered against silent-wrong rate: thresholds trade smoothly but err silently; intervals and structure are coarser but fail loudly.
• Abstention across the volume: shortcut-aware abstention on concept uncertainty, verifier-budget abstention anticipated for Part V, and the SATORI claim that is exactly calibrated abstention on open-world queries.

The missing output

Start with the definitions, because the whole subject is a pair of numbers and the pair must be exact. A selective classifier is a pair $(f, g)$: the predictor $f$ maps a question $x$ to an answer, and the selection function $g$ maps the same question to $1$ ("answer") or $0$ ("abstain"). The system outputs $f(x)$ when $g(x) = 1$ and stays silent when $g(x) = 0$ [2]. Two quantities grade the pair. Write $\mathbb{E}[\cdot]$ for the expectation, the average of the bracketed quantity over questions drawn from the task's distribution, and write $\ell(f(x), y)$ for the zero-one loss, equal to $1$ when the answer $f(x)$ differs from the true answer $y$ and $0$ when it matches. Write capital $X$ for a question drawn at random from the task's distribution and $Y$ for its true answer; the capital letters mark the randomly drawn versions of $x$ and $y$. Then

$$\phi(g) \;=\; \mathbb{E}\!\left[\, g(X) \,\right], \qquad R(f, g) \;=\; \frac{\mathbb{E}\!\left[\, \ell(f(X), Y)\, g(X) \,\right]}{\mathbb{E}\!\left[\, g(X) \,\right]}.$$

The first number, the coverage $\phi$ (the Greek letter phi), is the fraction of questions the system answers: the average of a $0/1$ indicator is exactly the proportion of $1$s. The second, the selective risk $R$, is the error rate among the answered questions: the numerator averages the loss but multiplies it by $g(X)$, so abstained questions contribute nothing, and dividing by the answered fraction renormalizes over just the answered set. On a finite benchmark of $N$ questions (here $N$ will be $126$), with $A$ the set of answered questions, the empirical versions are $\phi = \lvert A \rvert / N$ and $R = \frac{1}{\lvert A \rvert} \sum_{i \in A} \ell_i$, where $\lvert A \rvert$ counts the answered questions and the summation $\sum_{i \in A}$ adds the $0/1$ errors over exactly those.

Nothing yet says how to choose $g$. The standard construction attaches a confidence function $\kappa$ (the Greek letter kappa) that scores every question, and answers exactly the questions scoring at least a threshold $\theta$ (theta): $g_\theta(x) = 1$ if $\kappa(x) \ge \theta$, else $0$. Plotting selective risk against coverage as $\theta$ sweeps from high to low gives the risk-coverage curve, the standard evaluation object of selective prediction for deep models [3]. The construction is worth decoding before any formula, because it is just a sort. Rank the $N$ questions by confidence, most confident first. Any threshold admits some prefix of that ranking, say the $k$ most confident questions, and produces one point of the curve:

$$\text{coverage}(k) \;=\; \frac{k}{N}, \qquad \text{risk}(k) \;=\; \frac{1}{k} \sum_{i \le k} \ell_{(i)},$$

where the parenthesized subscript $(i)$ means "the $i$-th question in the sorted order", so the sum counts the errors among the $k$ answered. Sweeping $k$ from $N$ down to $1$ yields the whole curve, all $N$ points, exactly. The confidence signal is usually the cheapest thing available: the model's own top predicted probability, the maximum softmax probability, the field's default baseline for spotting a model's likely mistakes and, famously, an imperfect one [4]. For a binary question with calibrated probability $p$ of "true", the analogue is $\kappa = \max(p, 1-p)$: the probability the model assigns to whichever answer it is about to give.

The risk-coverage curve, on real scores

The companion computes this sweep exactly, in three lines (examples/frontier/abstention.py, lines 145–147 inside risk_coverage):

    order = np.argsort(-conf, kind="stable")
    risks = np.cumsum(wrong[order].astype(float)) / np.arange(1, len(conf) + 1)
    return {"order": order, "risks": risks, "aurc": float(risks.mean())}

The questions are the previous chapter's, unchanged: the 126 labelled triples of calibration.build_labelled (the 18 true kg triples plus 108 seeded corruptions), scored by Volume 3's retrained ComplEx and read through the fitted temperature as $p = \sigma(s / \tau^{*})$, where $s$ is the raw ComplEx plausibility score, $\sigma$ is the sigmoid function $\sigma(z) = 1/(1 + e^{-z})$, which squashes any real score $z$ into a probability strictly between $0$ and $1$, and $\tau^{*} = 1.4172$ is the fitted temperature (examples/frontier/abstention.py, lines 116–125, calling calibration.run and calibration._sigmoid, examples/frontier/calibration.py lines 85–88). The prediction is the argmax, $\hat{y} = 1$ exactly when $p \ge \tfrac12$, and the confidence is $\kappa = \max(p, 1-p)$. Answering everything, the model is right on 123 of 126 questions, and the three errors are precisely the held-out kg.TEST facts, the true triples ComplEx never trained on; the companion asserts this identity (line 324). The committed run places them on the confidence ranking:

    triple                        gold   p=sigma(s/tau)  conf=max(p,1-p)  rank/126
    (bob, advises, dave)          True       0.4491         0.5509       126
    (bob, authored, p1)           True       0.0012         0.9988        30
    (erin, affiliated, cmu)       True       0.0419         0.9581       114

Two errors sit near the bottom of the ranking, where a confidence threshold will catch them. One does not: (bob, authored, p1) is a true fact the model denies at confidence $0.9988$, rank 30 of 126. Hold that row; it is the villain of this chapter. The committed curve, one row per coverage anchor:

    coverage    k    conf cut   selective risk
       1.0     126    0.5509     0.0238
       0.9     114    0.9581     0.0175
       0.8     101    0.9714     0.0099
       0.7      89    0.9834     0.0112
       0.6      76    0.9878     0.0132
       0.5      63    0.9909     0.0159
       0.4      51    0.9943     0.0196
       0.3      38    0.9979     0.0263
       0.2      26    0.9990     0.0000
       0.1      13    0.9996     0.0000
    AURC = (1/N) sum_k risk(k) = 0.0125   (flat always-answer line: 0.0238)

Read it at three anchors. At full coverage the selective risk is the plain error rate, $3/126 = 0.0238$. At half coverage, answering only the 63 most confident questions, the risk drops to $1/63 = 0.0159$: abstention genuinely buys accuracy on the answered, and the companion asserts the strict improvement rather than hoping for it (examples/frontier/abstention.py, lines 327–328). At coverage $0.2$ the risk is exactly zero: the 26 most confident answers contain no error at all. The scalar summary at the bottom is the AURC, the area under the risk-coverage curve [5]: $\mathrm{AURC} = \frac{1}{N} \sum_{k=1}^{N} \text{risk}(k)$, the average of the selective risks over all $N$ coverage levels, the mean height of the curve along the coverage axis. Here it is $0.0125$, against the flat always-answer line at $0.0238$: an uninformative confidence ordering would hold the full-coverage risk at every cut, so an AURC below that line certifies the ordering is informative on average (asserted, lines 331–332).

Now the honesty. A single number hides two things. First, two systems' curves can cross: one better at high coverage, the other at low, with the AURCs equal or reversed at the crossing's expense; whenever a deployment has a target coverage, read the curve there, not the area. Second, this very curve is not monotone. The risk falls to $0.0099$ at coverage $0.8$, then climbs back to $0.0263$ at coverage $0.3$ before hitting zero. The culprit is the held row: (bob, authored, p1) is wrong at rank 30, so every cut that keeps the top 30 questions while discarding the safe bulk below concentrates that one error into a smaller denominator ($1/38 = 0.0263$ at $k = 38$); the error is only shed once the cut drops below $k = 30$, first visible among the tabulated anchors at $k = 26$. A confidence ordering is only as good as the probabilities behind it, which is why this chapter had to come after the calibration chapter, and why the next section's rule names calibration as its precondition.

One trade, three species: the threshold buys smooth coverage but leaves silent errors above the gate; the interval and structural policies answer less and fail out loud. Original diagram by the authors, created with AI assistance.

The Chow rule, derived in two lines

Where exactly should the threshold sit? The question only has content once silence has a price, so price it: an error costs $1$, an abstention costs $c_r$ with $0 \lt c_r \lt \tfrac12$ (the subscript $r$ is for "reject"; a reject cost above $\tfrac12$ would make abstaining worse than guessing at even odds, so the option would never fire). Consider one binary question whose calibrated posterior is $p$: the probability, given everything the model has seen, that the queried fact is true. If the system answers, it answers the argmax, so the probability its answer is wrong is $1 - \max(p, 1-p)$: when $p \ge \tfrac12$ it answers "true" and is wrong with probability $1 - p$, when $p \lt \tfrac12$ it answers "false" and is wrong with probability $p$, and in both cases the wrong-answer probability is whichever of $p$ and $1-p$ is smaller, one minus whichever is larger. The whole derivation is now a two-line cost comparison:

$$\mathbb{E}[\text{cost} \mid \text{answer}] \;=\; 1 \cdot \bigl(1 - \max(p, 1-p)\bigr), \qquad \mathbb{E}[\text{cost} \mid \text{abstain}] \;=\; c_r .$$

Abstaining is the cheaper action exactly when $c_r \lt 1 - \max(p, 1-p)$. Solve for the confidence: add $\max(p, 1-p)$ to both sides and subtract $c_r$ from both, giving $\max(p, 1-p) \lt 1 - c_r$. So the expected-cost-minimizing rule answers the argmax and rejects exactly when the top-class probability falls below the threshold $\theta = 1 - c_r$ [1]. With $K$ classes the same comparison replaces $\max(p, 1-p)$ by the largest of the $K$ posteriors and the threshold $\theta = 1 - c_r$ is unchanged; only the never-fire bound moves, from $c_r \ge \tfrac12$ to $c_r \ge 1 - 1/K$, because the top posterior can now be as small as $1/K$ rather than $\tfrac12$. The threshold is not a tuning knob; it is the price of silence, read off the cost model. The companion fixes $c_r = 0.10$, hence $\theta = 0.90$ (examples/frontier/abstention.py, lines 87–88), and the policy is one comprehension over the calibrated confidences (lines 170–172):

    verdicts: list[bool | None] = [
        (bool(v) if c >= theta else None)
        for v, c in zip(q["yhat"] == 1.0, q["conf"])]

Note what the rule consumes: probabilities, not scores. The statement "answering costs $1 - \max(p, 1-p)$ in expectation" is true only if $p$ actually is the probability of truth, a property the previous chapter's temperature repair approximates on average (it shrank the binned confidence-accuracy gap) but cannot guarantee per instance, as the confidence-$0.9988$ error above already shows. The distinction is sharp enough to derive. The confidence is $\kappa = \max(\sigma(z), 1 - \sigma(z))$ with $z = s/\tau$. Two facts about the sigmoid do the work. First, $1 - \sigma(z) = \sigma(-z)$: compute $1 - \sigma(z) = 1 - \tfrac{1}{1 + e^{-z}} = \tfrac{e^{-z}}{1 + e^{-z}}$, then multiply numerator and denominator by $e^{z}$ to get $\tfrac{1}{e^{z} + 1} = \sigma(-z)$. Second, $\sigma$ is strictly increasing: as $z$ grows, $e^{-z}$ strictly shrinks, so the denominator $1 + e^{-z}$ shrinks and the quotient $1/(1 + e^{-z})$ grows. Together these give $\kappa = \max(\sigma(z), \sigma(-z)) = \sigma(\max(z, -z)) = \sigma(\lvert z \rvert) = \sigma(\lvert s \rvert / \tau)$. For any temperature $\tau \gt 0$ this increases strictly with $\lvert s \rvert$, so the confidence ordering, and with it the curve and the AURC, is the same at every temperature; what the temperature moves is where a fixed probability level like $\theta = 0.90$ lands on that ordering (examples/frontier/abstention.py, lines 112–115). An uncalibrated model handed to Chow's rule sweeps out the same curve but stops at the wrong point on it: the threshold is a promise about error probability, and only a calibrated $p$ keeps the promise.

And the committed verdict on our questions is deliberately uncomfortable:

[3] policy (a) — Chow's optimum reject rule on the calibrated p
    reject iff max(p,1-p) < 1 - c_r: two of the three errors sit ABOVE
    theta = 0.90 (conf 0.9988 and 0.9581 -> answered, silently wrong);
    only (bob, advises, dave) at conf 0.5509 is rejected

The rule does its job: it keeps coverage at $124/126 = 0.9841$ and rejects the genuinely uncertain question. But the model is confidently wrong twice, and no threshold a defensible reject cost can justify will catch a wrong answer wearing a confidence of $0.9988$: rejecting it means pricing an abstention below $0.0012$ of an error and answering barely a fifth of the questions. The companion asserts the failure by name: (bob, authored, p1) is answered False at confidence $0.9988$ even though it is true (lines 337–340). This is the score policy's honest cost, and the reason the chapter does not end here.

A different species: open-world intervals

Everything so far treats "I don't know" as a property of the model: a score too close to the middle. Volume 2 taught a different reading, in which not-knowing is a property of the knowledge. Recall the machinery: an annotated knowledge base (KB) attaches to every fact a value from a commutative semiring, a rule body combines its facts' annotations with the semiring product $\otimes$, alternative derivations of the same fact combine with the semiring sum $\oplus$, and the annotated least fixpoint propagates the labels through the rules (examples/symbolic/annotated.py, Semiring at lines 38–47, provenance_lfp at lines 79–96). The companion instantiates this engine, unchanged, with an interval semiring: the carrier is the set of intervals $[l, u]$ with $0 \le l \le u \le 1$, read as "this atom's truth degree lies somewhere between $l$ and $u$", and the operations act componentwise as the direct product of two copies of Volume 2's $(\max, \min)$ confidence semiring (examples/frontier/abstention.py, lines 203–207):

$$[l_1, u_1] \oplus [l_2, u_2] = [\max(l_1, l_2),\, \max(u_1, u_2)], \qquad [l_1, u_1] \otimes [l_2, u_2] = [\min(l_1, l_2),\, \min(u_1, u_2)],$$

with zero $[0,0]$ ("known false") and one $[1,1]$ ("known true"); both semiring laws hold because they hold in each component. The knowledge base is the academic world rewritten as source-annotated assertions: a curated registry asserts the 16 non-citation triples with reliability $[0.95, 1.00]$, a web miner asserts the 2 citation edges at an honest $[0.35, 0.85]$, and an author's CV independently confirms one citation at $[0.70, 1.00]$ (lines 94–98). One rule threads reliability through every claim, holds(X, R, Y) ← saidBy(S, X, R, Y) ∧ reliable(S) (lines 212–213), so a fact's interval is its sources' claims discounted by their reliabilities ($\otimes$) and joined across independent sources ($\oplus$). A queried atom the fixpoint never derives gets the ignorance interval $[0, 1]$: nothing asserted, nothing refuted, the open-world assumption as an annotation (lines 245–257). Type signatures induced from the data play the ontology's part: a query violating its relation's domain or range meets the near-zero interval $[0.00, 0.02]$, the ontology refuting what no source ever said (the signatures are induced at lines 232–242; TYPE_BAD is declared at line 100 and the meet applied at lines 256–257).

The decision rule reads the interval, and only the interval. In general form, with an assertion threshold $\theta_{\text{hi}}$ and a denial threshold $\theta_{\text{lo}}$: assert the fact when $l \ge \theta_{\text{hi}}$ (even the worst case is strong), deny it when $u \le \theta_{\text{lo}}$ (even the best case is weak), and abstain between. The companion takes the tightest symmetric instance, $\theta_{\text{hi}} = \theta_{\text{lo}} = \tfrac12$: answer only when the entire interval sits on one side of $\tfrac12$ (lines 260–270). The committed mini-run on the same 126 questions:

    query                          [l, u]          verdict   why
    (alice, affiliated, mit)       [0.95, 1.00]    True      registry [0.95,1.00], types ok
    (p3, cites, p2)                [0.35, 0.85]    abstain   webminer only [0.35,0.85]: straddles 1/2
    (p2, cites, p1)                [0.70, 1.00]    True      webminer + authorcv: join = [0.70,1.00]
    (bob, advises, bob)            [0.00, 1.00]    abstain   never asserted, types ok: ignorance [0,1]
    (alice, advises, cmu)          [0.00, 0.02]    False     type violation: meet with [0.00,0.02]

Each row is a different way of knowing or not knowing. The registry fact is asserted because even its lower bound clears $\tfrac12$. The scraped-only citation is true in the gold standard, yet the policy abstains, because a $[0.35, 0.85]$ source does not determine it; the companion asserts this abstention specifically (lines 348–350). The doubly-sourced citation shows $\oplus$ earning its keep: the CV's $[0.70, 1.00]$ joins the miner's noisy interval and lifts the lower bound past the threshold. And the fourth row is the conceptual point of the section. The abstention on (bob, advises, bob) is not low confidence. There is no score, no calibration set, no fitted temperature, and no tunable threshold behind it; the value $\tfrac12$ is fixed by the semantics of truth degrees, not by a validation sweep. The interval $[0, 1]$ is a structural fact about what the knowledge base does not determine: no source spoke, no rule fired, and the open world declines to convert that silence into a "false". Where the Chow rule says "I am not sure", the interval policy says "nothing I was given settles this", and those are different sentences.

Structural abstention: the parser that says Error

The third species comes from Volume 4, re-quoted rather than rebuilt, because it needs something bare triple-queries do not have: a translation stage that can fail. The translate-then-prove pipeline parses controlled English with a strict grammar, compiles the parsed theory, and proves the question with Volume 1's backward chainer; any sentence the grammar cannot place raises a ParseError, and the pipeline's contract is four words long: parse or be silent (examples/integration/translate_prove.py, lines 217–228, inside pipeline_answer):

    try:
        theory = parse_theory(theory_sentences)
        person, pred, neg = parse_statement(question)
    except ParseError:
        if not use_repair:
            return None
        try:
            theory = parse_theory([repair(s) for s in theory_sentences])
            person, pred, neg = parse_statement(repair(question))
        except ParseError:
            return None                      # abstain: visibly, not wrongly
    return prove_statement(theory, person, pred, neg)

On its committed corpus of 1500 controlled-English questions, 288 of them seeded out-of-grammar by a synonym corruption, the one-round repair pass lifts coverage from $0.8080$ to $0.9293$, and everything that parses is proved: accuracy on parsed inputs is exactly $1.0$ at every depth, asserted per depth (lines 299–301), and the no-silent-errors property is checked question by question, not through the aggregate (lines 305–309). The committed failure-mode contrast against the soft reasoner that always answers, on the identical evaluation set:

      system                       answers      silent errors      overall acc
      soft reasoner (always)       100.0%       yes (quietly wrong)   0.8773
      translate-then-prove         92.9%        never (asserted)     0.9293

Here coverage is below one, accuracy-on-answered is exactly one, and the silent-wrong rate is zero, not because the corpus was kind but because the architecture leaves no other option: every delivered answer is a proof, so the only possible failure is a visible Error. And the Error is not a bare flag. It carries its reason, the exact token the grammar could not place ("unknown predicate 'referenced'"), which changes what a deployment can do with it. A Chow rejection says "confidence 0.55"; an operator can only shrug or re-ask. A structural abstention names the word that broke the parse; an operator can fix the word, extend the lexicon, or route the question to a human with the diagnosis attached. An abstention the system can explain is worth more than a threshold trip, because it is the beginning of a repair rather than the end of a transaction.

Three policies, one table

The chapter's spine is the committed table that puts all three species side by side. The three summary numbers are computed by the same three formulas for every policy (defined in _tally, examples/frontier/abstention.py, lines 177–190; policy (c) aggregates them from its per-depth table inside structural_policy, lines 284–291): with $A$ the answered questions out of $N$,

$$\text{coverage} = \frac{\lvert A \rvert}{N}, \qquad \text{acc-on-answered} = \frac{\#\{i \in A : \text{verdict}_i = \text{gold}_i\}}{\lvert A \rvert}, \qquad \text{silent-wrong} = \frac{\#\{i \in A : \text{verdict}_i \ne \text{gold}_i\}}{N},$$

where $\#\{\cdot\}$ counts the elements of the set described. An abstention is visible by definition, so only answered-and-wrong questions are silent errors, and the three numbers obey the identity $\text{silent-wrong} = \text{coverage} \times (1 - \text{acc-on-answered})$: the fraction answered times the error rate among them is the fraction of all questions answered wrongly. The committed run:

    policy                               n     coverage  acc-on-answered  silent-wrong
    (a) Chow threshold on sigma(s/tau)   126     0.9841       0.9839         0.0159
    (b) open-world interval [l,u]        126     0.8651       1.0000         0.0000
    (c) structural (translate-prove)    1500     0.9293       1.0000         0.0000

Policies (a) and (b) answer the same 126 knowledge-graph questions; policy (c) is Volume 4's committed exhibit on its own 1500 questions, quoted through structural_policy (lines 275–291), because structural abstention needs a parser that can fail. Read the table as a graded ladder of guarantees:

	(a) score threshold	(b) open-world interval	(c) structural
what it consumes	a calibrated probability $p = \sigma(s/\tau^{*})$	the fixpoint interval $[l, u]$	the parse verdict
the dial	$\theta = 1 - c_r$, continuous: any coverage is reachable	none: $\tfrac12$ is fixed by the semantics	none: the grammar is the gate
coverage	$0.9841$, and dialable to any level	$0.8651$, fixed by what the KB determines	$0.9293$, fixed by corpus and grammar
silent-wrong	$0.0159$	$0.0000$, empirical	$0.0000$, by construction under its own semantics
its abstention says	"confidence below $0.90$"	"the sources do not determine this"	"out of grammar: here is the token"

The honest reading runs both ways. The threshold's virtue is the smooth dial: pick any target coverage and the sorted confidences hand you the cut, which neither other policy can do; their coverages are whatever the knowledge and the grammar fix them to be, and moving them means changing the KB or the lexicon, not turning a knob. The threshold's vice is the row's last number: $0.0159$ of all questions are answered wrongly with no signal whatsoever, two of them at confidence above $0.95$. The interval row's zero deserves its precise epistemic label: it is empirical. The companion asserts it (lines 344–346), but it holds because this KB's sources and signatures happen to be sound; a lying source with a confident interval would produce a wrong assertion, and no theorem forbids it. The structural row's zero is different in kind: asserted question by question inside translate_prove, it holds for any corpus read under the pipeline's own grammar-plus-closed-world semantics, because an answer is a proof of the parsed theory and the only failure mode is a visible Error; a corpus whose gold labels follow different semantics (an open-world reading of negation, say, or a synonym the grammar accepts in another sense) could still be silently mis-answered. Thresholds trade smoothly but can be silently wrong; intervals and structure are coarser but their failures are loud. And loud failure is a design property this series has been buying since Volume 1's prover first said "no proof found" instead of guessing.

Where abstention meets the rest of the volume

Three threads leave this chapter and reappear later. First, shortcut-aware abstention. Every policy above gates on the final answer's uncertainty, but Part II showed that a neuro-symbolic model can be confidently right about a label while wrong about the concepts underneath it; a shortcut-aware system should abstain when the concepts are uncertain even if the label is not. That is the BEARS proposal (the name is short for "BE Aware of Reasoning Shortcuts"): ensemble the concept posteriors so shortcut ambiguity surfaces as concept-level uncertainty an abstention gate can read [6]. The reasoning-shortcuts chapter built the disease; concept-calibrated abstention is its symptomatic treatment. Second, budget abstention, anticipated for Part V: a bounded prover that exhausts its depth or time budget has learned nothing about the query's truth, and the honest output is "budget exhausted", a fourth verdict alongside true, false, and unparseable; Reinforcement and Verifiers lives this distinction: its budget sweep treats a goal underived within the budget as unreached, never as false. Third, the capstone: SATORI's claim C5 in Part VII is, stated plainly, calibrated abstention on open-world queries, the (a) and (b) rows of this chapter's table fused into one system; the claims chapter evaluates it with precisely the instruments built here.

The unsolved part

Three open problems, in increasing order of discomfort. The first is distribution shift. The threshold $\theta$ is a promise about error probability under the calibration that held when it was set. Move the questions (new entities, new relations, a drifted corpus) and both layers break at once: the probabilities decalibrate, so $\theta = 1 - c_r$ no longer marks the cost-optimal cut, and the risk-coverage curve itself moves, so a threshold chosen for coverage $0.9$ may deliver a very different coverage at a selective risk far off the curve it was read from. Selective prediction under shift is an active, unsettled line of work; the interval and structural policies are less exposed only because their gates never consulted a distribution in the first place. The second is abstention for generative outputs. Coverage and selective risk presuppose a countable set of questions with checkable answers. What is the coverage of a proof sketch, a program, a paragraph? If a generator abstains on one step of a twelve-step derivation, has it answered or not? Candidate readings exist (per-claim coverage, per-token entropy gates, abstain-and-cite), but there is no agreed analogue of the risk-coverage curve for open-ended generation, and Part V's verifier-gated systems will feel the gap directly. The third is the one the field keeps deferring because it is not mathematical: the human on the other end. A system that abstains hands a decision to a person, and the deployed pattern across domains is that frequent abstainers get overridden, routed around, or switched off; a wrong answer in a fluent voice is acted on, while an honest "I don't know" is treated as friction. Loud failure only helps if someone listens, and making abstention consumable, trusted, and cheap to act on is a human-factors problem no threshold derivation touches.

Why it matters

This chapter is where the volume's trust story acquires its second number. Accuracy alone cannot distinguish a system that is wrong 2% of the time silently from one wrong 2% of the time visibly, and everything a deployment does downstream (audit, escalation, automation level) depends on which one it has. The instruments built here, coverage, selective risk, the curve, AURC, the Chow threshold, the silent-wrong rate, are the standard kit for making that distinction measurable on your own system, and the three-policy table is the template: put your score gate, your knowledge-level unknowns, and your structural failures on the same axes and read what each buys. For the series arc, the chapter closes a loop opened long ago. Volume 1's backward chainer returned "no proof" rather than a guess; Volume 2's open-world semantics made "unknown" a value rather than an absence; Volume 4's parser turned translation failure into a verdict. Those were abstentions before this chapter had the word, and the committed table is the receipt that their shared design property, failures that announce themselves, survives quantification. The researcher's takeaway is architectural, not numeric: where you can, build systems whose ignorance is structural, because a threshold can only lower the volume of silent errors, while a proof obligation can eliminate them.

Key terms

• Selective prediction / selective classification: the setting in which a predictor may abstain; a pair $(f, g)$ of predictor and selection function, graded by coverage and selective risk.
• Coverage: the fraction of questions the system answers; empirically $\lvert A \rvert / N$ for answered set $A$ among $N$ questions.
• Selective risk: the error rate among the answered questions only.
• Risk-coverage curve: selective risk plotted against coverage as the confidence threshold sweeps; one point per cut of the confidence-sorted question list.
• AURC (area under the risk-coverage curve): the average selective risk over all $N$ coverage levels, $\frac{1}{N}\sum_k \text{risk}(k)$; a summary that can hide curve crossings and non-monotonicity.
• Chow's rule / reject option: the expected-cost-optimal policy with errors costing 1 and abstentions $c_r$: answer the argmax, reject exactly when the top-class probability falls below $\theta = 1 - c_r$.
• Maximum softmax probability: the model's own top predicted probability, the default confidence signal; here the binary analogue $\max(p, 1-p)$.
• Open-world (interval) abstention: declining because the knowledge base does not determine the fact: the fixpoint interval $[l, u]$ straddles the decision level, with $[0,1]$ the ignorance interval of a never-derived atom.
• Structural abstention: declining because the input never entered the formal system: a parser's Error verdict, which makes silent wrongness impossible since every delivered answer is a proof.
• Silent-wrong rate: the fraction of all questions answered incorrectly, $\text{coverage} \times (1 - \text{acc-on-answered})$; the number that separates loud failure from quiet failure.

Where this leads

Part III has priced individual answers: what a probability must mean, and when to withhold one. Part IV changes the question from whether to answer to what it costs to answer at scale. The academic-world closure Volume 1 computed in three waves becomes, at a million facts, a genuine systems decision: precompute every consequence once and store it, or leave the data alone and rewrite each query through the rules at ask time. Materialization versus Rewriting builds both on the running example, counts what each buys and pays, and opens the volume's second movement: reasoning as a workload.

---

Companion code: examples/frontier/abstention.py builds the question set from calibration.py's calibrated scores, computes the exact risk-coverage sweep and AURC, implements Chow's rule, runs the interval policy on Volume 2's annotated fixpoint engine, and quotes Volume 4's structural pipeline, with every claim in this chapter guarded by an assert. Run python3 examples/frontier/abstention.py to reproduce every number; the run is deterministic end to end.