Calibration: Confidence That Means Something

📍 Where we are: Part III · Calibration, Uncertainty, and Abstention — Chapter 7. Measuring Shortcuts: rsbench closed Part II by measuring whether a model's internal concepts are right; Part III opens by asking a humbler question about its outputs: when the model says 0.97, is it right 97 times in 100?

Every volume of this series ended with a number attached to a claim: Volume 3 attached ranking scores to triples, Volume 4 attached fuzzy degrees to formulas and proof scores to derivations, and none of those numbers ever promised to be a probability. The moment a downstream consumer (a human curator, an abstention gate, a verifier deciding which derivations to check first) reads the number as a probability, it takes on an obligation it was never trained to meet. Calibration is the name of that obligation, and this chapter is its audit. We define the promise exactly, build the two standard instruments, the reliability table and the expected calibration error (ECE), from their definitions, run them on the series' own knowledge-graph model, read the miscalibration the committed run actually shows, and repair it with the smallest possible intervention: a single scalar, fitted on held-out data, that provably cannot change a single decision the model was already making.

The simple version

Imagine a weather forecaster who says "70 percent chance of rain" on many different days. You audit her the obvious way: collect every day she said 70 percent and check how often it actually rained. If it rained on about 70 of each 100 such days, her "70 percent" means something; you can plan a picnic around it. If it rained on only 40, she is overconfident, and the number is decoration. Notice what the audit does not ask: whether she is a good forecaster. A cautious forecaster who always says "50 percent" in a city where it rains half the time is perfectly calibrated and perfectly useless. Calibration audits the honesty of the confidence, not the sharpness of the forecast, and this chapter runs exactly that audit on a neural model's scores, then adjusts the model's tone of voice (not its opinions) until the numbers mean what they say.

What this chapter covers

• The promise, defined: why the series' scores (margins, degrees, proof scores) are not probabilities, and what the conditional-probability statement $P(\text{correct} \mid \text{confidence} = p) = p$ says, every symbol decoded before any estimator appears.
• The reliability table from its definition: ten equal-width confidence bins, per-bin count, mean confidence, and observed accuracy; the committed BEFORE table read row by row, and why a table beats a curve at this scale.
• ECE and the Brier score, derived then computed: the count-weighted mean absolute gap, the equal-width estimator's own bias named, and the classic three-line decomposition of squared error into calibration plus refinement.
• Why ranking scores miscalibrate: the objective-mismatch argument made precisely on the companion's own training setup, with the model's two evaluation-split errors identified by name.
• Temperature scaling as the minimal repair: one scalar $\tau$ fitted by held-out negative log-likelihood via a deterministic golden-section search, the committed AFTER table with ECE strictly improved, and the two-line invariance theorem asserted exactly in code.
• What one scalar cannot fix, and what sits beyond it: residual per-bin gaps in the committed run, the Platt and isotonic alternatives, the overfitting caution, and calibration's place between the shortcut lesson and the abstention gate.

A score is not a probability

Recall what the series actually produced. Link Prediction scored every candidate triple with a real number and was judged entirely by rank: filtered mean reciprocal rank (MRR) and Hits@k care only about which score is larger, never about the score's value. Bilinear Models made the score concrete: ComplEx's $s = \mathrm{Re}(\langle e_h, w_r, \overline{e_t} \rangle)$, read as the real part ($\mathrm{Re}$) of a three-way product $\langle \cdot, \cdot, \cdot \rangle$ that multiplies, coordinate by coordinate, the head entity's complex embedding vector $e_h$, the relation's vector $w_r$, and the complex conjugate (the overline) of the tail entity's embedding $e_t$, then sums the coordinates, exactly as built in that chapter. The result is a real number on no particular scale; the committed run below will show these scores ranging from $-16.9$ to $+7.5$. Volume 4's T-norms produced satisfaction degrees in $[0,1]$ that compose by minimum or product under fuzzy semantics, a truth-degree algebra with no frequency reading, and Neural Theorem Proving scored a proof by the weakest soft unification along its best derivation, again a maximum of similarities, not a chance of anything. Each number orders candidates. None of them counts anything.

The probability reading is a different and stronger contract, and modern neural networks are known to violate it even when their accuracy is excellent [1]. State it exactly. Fix an audited system that, for each input, announces a confidence $\hat{p} \in [0,1]$ (read "p-hat": the model's own claimed probability, a number between zero and one) alongside its prediction, and let $Y \in \{0, 1\}$ be the outcome variable, equal to $1$ when the prediction is correct and $0$ when it is not. Draw an audited case at random. The system is perfectly calibrated when, for every value $p$ that the confidence can take,

$$P\big(\,Y = 1 \;\big|\; \hat{p} = p\,\big) \;=\; p.$$

Decode the notation piece by piece. $P(\cdot)$ is a probability over the random draw of a case. The vertical bar $\mid$ is conditioning: the expression $P(A \mid B)$ means "the probability of the event $A$ among only the cases where $B$ holds," so the left-hand side reads "among all the cases where the model announced confidence exactly $p$, the fraction that were actually correct." The equation demands that this fraction equal the announced number itself, for every announced number. That is the forecaster's audit from the analogy, written as a conditional probability, and it is a promise about frequencies, checkable by counting, which is what makes it an empirical instrument rather than a philosophical stance. Two readings coexist in the literature: the binary-forecast reading used in this chapter, where $\hat{p}$ is the claimed probability that a specific proposition is true and $Y$ marks whether it is, and the top-label reading used for multiclass classifiers, where $\hat{p}$ is the probability assigned to the predicted class [1]. Our audit is a binary one, true triple or not, so the first reading applies directly.

One caution before the estimators, echoing the analogy: calibration is necessary for trust, not sufficient for usefulness. The always-say-the-base-rate forecaster is calibrated and uninformative. The instruments below therefore come in pairs: the reliability table and ECE measure the promise, and the Brier score, derived later, measures the promise and the sharpness together, in a way whose algebra separates the two exactly.

The audit: real scores on the running example

The companion examples/frontier/calibration.py audits a real model, not a synthetic score stream: Volume 3's ComplEx link predictor, retrained byte-identically by calling bilinear.train_complex(seed=0) (calibration.py line 241), the same 16-real-dimension model whose scoring function $s = \mathrm{Re}(\langle e_h, w_r, \overline{e_t} \rangle)$ is committed at bilinear.py lines 99–115. The labelled set is the academic world read as binary classification: all 18 known-true triples of kg.TRIPLES as positives, plus 6 seeded corruptions of each as negatives, 108 in total, each corruption redrawn until it is neither a known-true triple nor a duplicate, so the 126 labelled triples are distinct and the negatives are genuinely false in the closed world of the running example (build_labelled, calibration.py lines 93–116). A deterministic stratified split then divides the 126 into a calibration half of 63, which will fit the repair, and an evaluation half of 63, on which every BEFORE/AFTER calibration metric quoted below is measured, with both halves carrying the same 1-positive-to-6-negatives class ratio (split_indices, lines 119–130).

One more design choice needs stating because it makes the audit fair. ComplEx was trained with the logistic loss $L = \mathrm{softplus}(-y \cdot s)$, where $y \in \{-1, +1\}$ is the label, $\ln$ is the natural logarithm, $e^{z}$ is the exponential function (the constant $e \approx 2.718$ raised to the power $z$), and $\mathrm{softplus}(z) = \ln(1 + e^{z})$ (bilinear.py lines 132–135). Under that loss the model's own implied probability that a triple is true is exactly $\sigma(s)$, the sigmoid $\sigma(z) = 1/(1 + e^{-z})$ of the score, the squashing function from Volume 1 that maps any real number into $(0, 1)$; the loss is precisely the negative log-likelihood of the label under that reading. So the BEFORE column is not our invention imposed on the model; it is the model's native probability claim, read off its own training objective. The committed run opens with the raw material:

[1] the raw scores, read as probabilities
    positives score in [  -9.492,   7.508], negatives in [ -16.895,  -1.588]
    the held-out kg.TEST positives (true facts never trained on):
    triple                        score      p=σ(s)   p=σ(s/τ)  split
    (bob, advises, dave)           -0.290     0.4281     0.4491  eval
    (bob, authored, p1)            -9.492     0.0001     0.0012  eval
    (erin, affiliated, cmu)        -4.436     0.0117     0.0419  cal
    (bob, authored, p1) is TRUE but gets p = 0.0001: confidently
    wrong — the failure a calibration metric exists to expose

Read the middle row twice. The triple (bob, authored, p1) is true in the running example; it was simply held out of training back in Volume 3's split. The model assigns it probability $0.0001$: not unsure, but confidently wrong. Ranking metrics can partially forgive this (the triple merely ranks low); a probability reading cannot, because a system that says "one in ten thousand" about a true fact will silently discard it from any pipeline that trusts the number. The instruments below turn this anecdote into a measurement.

The reliability table, built from its definition

The definition conditions on $\hat{p} = p$, an event with probability zero for continuous confidences, so no finite audit can check it value by value. The standard estimator discretizes: partition $[0,1]$ into $M = 10$ equal-width bins (the letter $M$ is the bin count, the companion's N_BINS at calibration.py line 75), assign each prediction to the bin containing its confidence, and compare each bin's average confidence against its observed frequency of correctness [2]. Bin $b$, for $b = 0, 1, \ldots, M-1$, covers the interval $[\,b/M,\ (b+1)/M\,)$, with the last bin also including $1.0$; the bin index of a prediction with confidence $p_i$ is $\min(\lfloor p_i \cdot M \rfloor,\ M - 1)$, where $\lfloor \cdot \rfloor$ is the floor function, the largest integer not exceeding its argument (reliability_bins, lines 135–153). Within bin $b$, three statistics summarize the audit, with $i$ indexing the predictions that landed there and $n_b$ counting them:

$$\mathrm{conf}_b = \frac{1}{n_b} \sum_{i \in b} p_i, \qquad \mathrm{acc}_b = \frac{1}{n_b} \sum_{i \in b} y_i,$$

the bin's mean announced confidence and its observed accuracy, the empirical frequency of the label $1$ among its members ($\sum_{i \in b}$ adds one term per prediction in the bin). Perfect calibration makes the two columns agree in every bin, up to sampling noise. Here is the committed BEFORE table, the evaluation half of the split under the model's native reading $p = \sigma(s)$:

[2] BEFORE — reliability diagram of p = σ(s), 10 equal-width bins
    (Naeini et al. 2015; table form of Niculescu-Mizil & Caruana 2005)
    bin         count   conf    acc     gap(acc-conf)
    [0.0,0.1)      54   0.0040 0.0185   +0.0145
    [0.1,0.2)       1   0.1697 0.0000   -0.1697
    [0.4,0.5)       1   0.4281 1.0000   +0.5719
    [0.9,1.0]       7   0.9976 1.0000   +0.0024
    (63 points; empty bins [2, 3, 5, 6, 7, 8] omitted)
    ECE = Σ_b (n_b/N)·|acc_b − conf_b| = 0.0245   Brier = 0.0216   NLL = 0.1708
    acc(0.5) = 0.9683   AUC = 0.9136

Two acronyms in the print's summary lines should not have to wait for their sections: NLL is the negative log-likelihood, the fitting objective defined fully in the temperature-scaling section below, and AUC is the area under the receiver operating characteristic (ROC) curve, a pure ranking statistic defined in the invariance section; both reappear exactly where they do their work.

Teach the reading. In a plotted reliability diagram, each occupied bin is a point at $(\mathrm{conf}_b, \mathrm{acc}_b)$ and perfect calibration is the diagonal; a point below the diagonal, accuracy short of confidence, negative gap in the last column, is overconfidence in the plain sense: the model promised more than it delivered. The $[0.1, 0.2)$ row is one such point: a single negative triple announced at $0.1697$ and false, gap $-0.1697$. At the low end of the probability-of-truth reading the roles flip: the $[0.0, 0.1)$ row's positive gap ($\mathrm{conf} = 0.0040$ against $\mathrm{acc} = 0.0185$) means the model's near-zero announcements were too extreme downward, too sure of falsehood, which is the same disease seen from the other side. Fold both ends into the top-label reading (confidence in whichever side you assert) and both rows say one thing: the probabilities are pushed too far from one half. Hold that diagnosis; the fitted temperature will quantify it with a single number.

Now the honest part, and the reason this chapter prints a table rather than a smooth reliability curve. Two of the four occupied rows contain exactly one point each, so their "accuracy" is a single Bernoulli draw, 0 or 1, and their gaps ($-0.1697$ and $+0.5719$) are mostly sampling noise wearing the costume of a measurement. Even the full rows carry visible error bars. The $[0.0,0.1)$ row holds $n_b = 54$ points at $\mathrm{acc}_b = 0.0185$; the standard error of a frequency (the typical size of the random fluctuation in an estimated fraction, shrinking as the square root of the count grows) is $\sqrt{\mathrm{acc}_b(1 - \mathrm{acc}_b)/n_b} = \sqrt{0.0185 \times 0.9815 / 54} \approx 0.018$, wider than the row's entire gap of $+0.0145$. The $[0.9,1.0]$ row shows seven successes in seven trials. If the bin's true accuracy were some value $p$, the probability of seeing seven correct in seven independent cases would be $p \cdot p \cdots p = p^{7}$; the 95 percent lower bound is then the smallest $p$ for which seven-for-seven is not yet a below-5-percent fluke, that is, the $p$ solving $p^7 = 0.05$, giving $p = 0.05^{1/7} = e^{\ln(0.05)/7} \approx 0.652$, so "accuracy 1.0000" there certifies only "probably above about two thirds." A curve drawn through such bins would manufacture precision the data does not contain; a table with a count column keeps the uncertainty in view. Reliability estimates are only as good as their bin counts, and the counts should be printed next to the estimates.

There is a second honest reading of the table's good news. The headline accuracy at the $0.5$ threshold, $0.9683$, is $61$ of $63$: flattering, until you notice who the winners and losers are. The seven confident hits in $[0.9, 1.0]$ are all training triples, edges the model memorized during its 1000 epochs; the evaluation split's only two errors are exactly the two held-out kg.TEST positives it contains, (bob, advises, dave) at $0.4281$, the lone occupant of the $[0.4,0.5)$ bin, and (bob, authored, p1) at $0.0001$, hiding inside the $[0.0,0.1)$ bin as its one true member ($0.0185 \times 54 = 1$). The audit's positives are mostly facts the model has seen, which flatters its accuracy; the facts it has not seen are precisely where its confidence fails. The companion's docstring says this in as many words (calibration.py lines 43–50), and the chapter repeats it because a calibration audit that hides its own sampling regime is exactly the kind of unexamined score this volume exists to retire.

ECE, derived, then the Brier score, decomposed

A table is for reading; comparisons need one number. The expected calibration error (ECE) collapses the table into the count-weighted mean absolute gap [2]:

$$\mathrm{ECE} \;=\; \sum_{b=0}^{M-1} \frac{n_b}{N}\,\big|\,\mathrm{acc}_b - \mathrm{conf}_b\,\big|,$$

where $N$ is the total number of audited predictions (here $63$), $n_b/N$ is the fraction of them that landed in bin $b$, and $|\cdot|$ is the absolute value, the gap's size with its sign dropped. Each choice in the formula is doing a job. The absolute value prevents overconfident bins and underconfident bins from cancelling into a false zero. The weight $n_b/N$ makes the number an expectation: it is the average calibration gap experienced by a randomly drawn prediction, so a wild gap in a one-point bin counts for exactly one point's worth. Empty bins contribute nothing. The companion's implementation is the formula verbatim (ece, calibration.py lines 156–163):

def ece(p: np.ndarray, y: np.ndarray) -> float:
    """Expected Calibration Error (Naeini et al. 2015):
        ECE = Σ_b (n_b / N) · |acc_b − conf_b|
    the count-weighted mean absolute gap between each bin's observed accuracy
    and its mean confidence (empty bins contribute 0)."""
    rows = reliability_bins(p, y)
    return float(sum(r["n"] / len(p) * abs(r["acc"] - r["conf"])
                     for r in rows if r["n"] > 0))

Run the formula by hand on the committed BEFORE table, one term per occupied row:

$$\mathrm{ECE} = \tfrac{54}{63}(0.0145) + \tfrac{1}{63}(0.1697) + \tfrac{1}{63}(0.5719) + \tfrac{7}{63}(0.0024) = 0.0124 + 0.0027 + 0.0091 + 0.0003 = 0.0245,$$

matching the committed print to all four digits. The weighting is visible in the arithmetic: the spectacular $0.5719$ gap contributes less than the modest $0.0145$ gap, because one sits on a single point and the other on fifty-four.

The estimator has known biases of its own, and they should be named rather than discovered later. Binning with equal-width bins, as here and as standard [2], lets the data pile into a few bins and leave others empty; the alternative, equal-mass binning, fixes each bin's count instead of its width, trading empty bins for data-dependent bin edges. Neither escapes the underlying tension: within a bin, oppositely-signed miscalibration cancels before the absolute value is taken, biasing ECE downward, while sampling noise in $\mathrm{acc}_b$ inflates the per-bin gaps, biasing it upward, and the balance shifts with $M$ and $N$. ECE is a biased estimator whose value depends on its own binning knob; at our scale that is visible in the table itself, and the unsolved-part section returns to it.

The Brier score predates all of this machinery by decades: it is simply the mean squared error of the probability forecast against the outcome [3],

$$\mathrm{BS} \;=\; \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2,$$

with $p_i$ the announced probability and $y_i \in \{0,1\}$ the outcome (brier, calibration.py lines 166–169). Its virtue is what its algebra separates. Group the sum by the reliability table's bins, and idealize the forecast within bin $b$ as the single value $\mathrm{conf}_b$ (exact when all of a bin's forecasts coincide, an approximation at the bin mean otherwise). Insert $\mathrm{acc}_b$ by adding and subtracting it inside the square, then expand:

$$\mathrm{BS} = \frac{1}{N} \sum_b \sum_{i \in b} \big[(\mathrm{conf}_b - \mathrm{acc}_b) + (\mathrm{acc}_b - y_i)\big]^2 = \frac{1}{N} \sum_b \Big[ n_b(\mathrm{conf}_b - \mathrm{acc}_b)^2 + 2(\mathrm{conf}_b - \mathrm{acc}_b)\!\sum_{i \in b}(\mathrm{acc}_b - y_i) + \sum_{i \in b}(\mathrm{acc}_b - y_i)^2 \Big].$$

The cross term dies by the definition of $\mathrm{acc}_b$ as the bin's mean label: $\sum_{i \in b}(\mathrm{acc}_b - y_i) = n_b\,\mathrm{acc}_b - n_b\,\mathrm{acc}_b = 0$. The last term simplifies because $y_i$ is binary, so $y_i^2 = y_i$: expanding the square gives $\sum_{i \in b}(\mathrm{acc}_b - y_i)^2 = n_b\,\mathrm{acc}_b^2 - 2\,\mathrm{acc}_b \cdot n_b\,\mathrm{acc}_b + n_b\,\mathrm{acc}_b = n_b\,\mathrm{acc}_b(1 - \mathrm{acc}_b)$. What remains is the classic two-part decomposition, whose two terms the forecast-verification literature named calibration and refinement [4]:

$$\mathrm{BS} \;=\; \underbrace{\sum_b \frac{n_b}{N}\,(\mathrm{conf}_b - \mathrm{acc}_b)^2}_{\text{calibration}} \;+\; \underbrace{\sum_b \frac{n_b}{N}\,\mathrm{acc}_b\,(1 - \mathrm{acc}_b)}_{\text{refinement}}.$$

The first term is ECE's squared-gap sibling: zero exactly when every bin keeps the promise. The second term is refinement, the average Bernoulli variance within bins: it is small when the bins are pure, each bin nearly all-correct or all-wrong, which is a property of how well the confidence sorts cases, not of whether its values are honest. One score, two ingredients, separated by three lines of algebra. The committed BEFORE values are $\mathrm{BS} = 0.0216$ against $\mathrm{ECE} = 0.0245$, and the decomposition predicts something we will verify shortly: an intervention can improve the calibration term while nudging refinement the other way, so Brier and ECE need not move together.

Why ranking scores miscalibrate

It is tempting to treat the BEFORE table's gaps as bad luck. They are not; they are the training objective showing through, and the argument deserves precision because our setup is the favourable case. Training objectives leave characteristic, measured distortions in the probabilities a learner announces: margin maximizers push probability mass toward the middle, a sigmoid-shaped, too-moderate distortion, while naive Bayes pushes it toward the extremes; neither kind of score distribution makes good probabilities without a post-hoc map, and this is a systematic finding across learning methods, not folklore [5]. Knowledge-graph embedding (KGE) models sit at the far end of that spectrum, and their miscalibration is likewise a measurement, not a guess [6]: the canonical KGE pipeline optimizes ranking losses against sampled corruptions and reports only rank metrics, so nothing in training ever anchors the score's scale, only its order.

Our ComplEx is friendlier than canonical, which sharpens the point. It was trained with a likelihood, the logistic loss, so $\sigma(s)$ is a genuine implied probability. But a probability of what, measured where? Three mismatches separate the training objective from the audit. First, the negative distribution: training drew exactly one uniform corruption per positive with its own seed (bilinear.py lines 215–227 and 267–286), while the audit poses six per positive from a different seed, so the population of "false" statements the model was calibrated against is not the population it is now being examined on. Second, the class prior: the training stream was balanced one-to-one, the audit is one-to-six, and a probability calibrated under one prior is miscalibrated under another even if everything else transfers. Third, and dominant at this scale: 1000 epochs of stochastic gradient descent (SGD) over 15 positive triples is memorization, and memorization drives the training loss toward zero by pushing $|s|$ ever larger on the training set, saturating the sigmoid toward 0 and 1. That is the same mechanism identified for modern networks at scale, where capacity plus a negative-log-likelihood objective yields well-ranked but overconfident probabilities [1]. The aggregate direction of our model's failure is therefore predictable before fitting anything: probabilities too extreme, both ends pushed away from one half. The repair we now fit will return a single number that says exactly that.

Temperature scaling: the one-scalar repair

Temperature scaling replaces the probability reading $p = \sigma(s)$ by $p = \sigma(s / \tau)$, one shared scalar $\tau \gt 0$ (the Greek letter tau, the "temperature") dividing every score before the sigmoid [1]. A $\tau$ above one shrinks each $s/\tau$ toward zero and therefore each probability toward one half, softening an overconfident model; a $\tau$ below one does the opposite. The fit minimizes the negative log-likelihood (NLL) of the calibration half of the split. Write the label in signed form, $y^{\pm} = 2y - 1 \in \{-1, +1\}$, so one formula can cover both label values. Two identities do the work, and each is two lines of algebra. First, substitute the definition $\sigma(z) = 1/(1 + e^{-z})$ into $-\ln \sigma(z)$: the logarithm of a reciprocal flips its sign, so $-\ln \sigma(z) = \ln(1 + e^{-z})$, which is $\mathrm{softplus}(-z)$ by the definition of softplus. Second, put $1 - \sigma(z)$ over the common denominator $1 + e^{-z}$: the numerator is $(1 + e^{-z}) - 1 = e^{-z}$, and multiplying the numerator and the denominator of $e^{-z}/(1 + e^{-z})$ by $e^{z}$ gives $1/(e^{z} + 1) = \sigma(-z)$, so $1 - \sigma(z) = \sigma(-z)$ and therefore, by the first identity applied at $-z$, $-\ln(1 - \sigma(z)) = -\ln \sigma(-z) = \mathrm{softplus}(z)$. Now check the per-example negative log-likelihood on both label values with $z = s/\tau$: a positive example ($y = 1$, so $y^{\pm} = +1$) contributes $-\ln \sigma(s/\tau) = \mathrm{softplus}(-s/\tau)$, and a negative example ($y = 0$, so $y^{\pm} = -1$) contributes $-\ln(1 - \sigma(s/\tau)) = \mathrm{softplus}(+s/\tau)$. Both cases are the single expression $\mathrm{softplus}(-y^{\pm} s / \tau)$, and

$$\mathrm{NLL}(\tau) \;=\; \frac{1}{N} \sum_{i=1}^{N} \mathrm{softplus}\!\Big(\!-\frac{y^{\pm}_i\, s_i}{\tau}\Big),$$

computed from the raw scores for numerical stability (nll_at_tau, calibration.py lines 172–178). Note the continuity with training: this is the very loss ComplEx was trained under, now with every parameter frozen except one.

One dimension deserves a deterministic optimizer, and the companion uses golden-section search over the bracket $[0.05, 20]$ (fit_tau, lines 193–228). The method needs the objective to be unimodal on the bracket, decreasing then increasing, and here that is a theorem rather than a hope: in the inverse temperature $\beta = 1/\tau$, each term $\mathrm{softplus}(-y^{\pm}_i \beta s_i)$ is a convex function of $\beta$ (softplus is convex, and composing a convex function with a linear function of $\beta$ preserves convexity), a nonnegative average of convex functions is convex, and a convex function of one variable is unimodal; the map $\tau \mapsto 1/\tau$ (the arrow $\mapsto$ reads "maps to": the function sending each $\tau$ to $1/\tau$) is strictly monotone on $\tau \gt 0$, and a monotone change of variable preserves unimodality. Golden-section then works by bracket shrinking: keep two interior probes at fractions set by $\varphi = (\sqrt{5} - 1)/2 \approx 0.618$ of the current bracket, discard the end that provably cannot contain the minimum, and repeat; each iteration multiplies the bracket width by $\varphi$. The committed trace confirms the geometry: after 10 iterations the width is $(20 - 0.05) \cdot \varphi^{10} = 19.95 \times 0.00813 = 0.1622$, exactly the printed 1.622e-01, and 80 iterations leave $19.95 \cdot \varphi^{80} \approx 4 \times 10^{-16}$, far below any printed digit, with the whole fit a fixed count of pure-function probes, hence bit-deterministic:

[3] temperature scaling (Guo et al. 2017): fit τ by NLL on the
    calibration split, golden-section over [0.05, 20.0]
    iter   bracket width   best NLL
      10       1.622e-01   0.063194
      20       1.319e-03   0.063191
      30       1.072e-05   0.063191
      40       8.718e-08   0.063191
    τ* = 1.4172  (τ > 1: the raw reading was OVERconfident,
    the direction Guo et al. report for modern networks; Tabacof &
    Costabello measure the same uncalibrated reading for KGE models)
    NLL on the calibration split: 0.0734 (τ=1) → 0.0632 (τ*)

The fitted temperature is $\tau^{\ast} = 1.4172$. That single number is the audit's aggregate verdict: above one, so the model's native reading was overconfident, precisely the direction the objective-mismatch argument predicted, and every raw score must be shrunk by a factor of about $1.42$ before its sigmoid means what it says. The companion asserts the direction rather than narrating it: assert tau > 1.0 (calibration.py line 291). Applying $p = \sigma(s/\tau^{\ast})$ to the untouched evaluation half gives the committed AFTER table:

[4] AFTER — the same evaluation split under p = σ(s/τ*)
    bin         count   conf    acc     gap(acc-conf)
    [0.0,0.1)      54   0.0156 0.0185   +0.0029
    [0.2,0.3)       1   0.2459 0.0000   -0.2459
    [0.4,0.5)       1   0.4491 1.0000   +0.5509
    [0.9,1.0]       7   0.9860 1.0000   +0.0140
    (63 points; empty bins [1, 3, 5, 6, 7, 8] omitted)
    ECE   0.0245 → 0.0167   (strictly better, asserted)
    NLL   0.1708 → 0.1387   (better on data τ never saw)
    Brier 0.0216 → 0.0221   (a hair worse here: 63 points is a tiny sample,
    and Brier mixes calibration with sharpness — reported, not hidden)

Rerun the hand computation on the new rows: $\tfrac{54}{63}(0.0029) + \tfrac{1}{63}(0.2459) + \tfrac{1}{63}(0.5509) + \tfrac{7}{63}(0.0140) = 0.0025 + 0.0039 + 0.0087 + 0.0016 = 0.0167$. The improvement is where the mass is: the 54-point bin's gap fell from $+0.0145$ to $+0.0029$, and the 7-point bin's confidence relaxed from $0.9976$ to a humbler $0.9860$ under a ceiling of observed perfection. Both gains are on data the fit never touched, and the companion refuses to publish without them: assert after["ece"] < before["ece"] (calibration.py lines 294–295). Meanwhile the Brier score moved a hair the wrong way, $0.0216$ to $0.0221$, exactly the divergence the decomposition licensed: shrinking probabilities toward one half improved the calibration term but paid a little refinement, and on 63 points the balance tipped negative. The companion prints the regression instead of hiding it, which is the reporting norm this volume is trying to make reflexive.

One scalar, fitted on held-out likelihood, moves every bar toward the diagonal while a monotonicity theorem guarantees that not one ranking or threshold decision moves. Original diagram by the authors, created with AI assistance.

The invariance theorem, in two lines

Temperature scaling's safety guarantee is a theorem small enough to state and prove completely, and important enough to deserve the ceremony. Claim: for any $\tau \gt 0$, the map $s \mapsto s/\tau$ preserves every strict order and every sign. Proof. Take any two scores with $s_1 \gt s_2$. Multiplying both sides of an inequality by the positive constant $1/\tau$ preserves it, so $s_1/\tau \gt s_2/\tau$; and $\sigma$ is strictly increasing (its derivative $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ is positive everywhere, as derived in Gradient Descent), so $\sigma(s_1/\tau) \gt \sigma(s_2/\tau)$. For signs, $s/\tau$ has the sign of $s$ because $1/\tau \gt 0$, and $\sigma(z) \ge 0.5$ exactly when $z \ge 0$, so $p \ge 0.5$ before scaling if and only if $p \ge 0.5$ after. $\square$

The consequences enumerate themselves. Every pairwise comparison between probabilities is unchanged, so the full sorted order of all 126 triples is unchanged, so every ranking metric of Volume 3, filtered rank, MRR, Hits@k, is unchanged; the AUC (the area under the receiver operating characteristic curve), being computable as exactly the fraction of positive-negative score pairs ordered correctly, ties counting one half (the Mann-Whitney form, auc at calibration.py lines 181–188), is unchanged; and every decision taken by thresholding at one half is unchanged, so accuracy is unchanged. The companion does not trust the theorem to survive floating point on its own; it asserts each consequence exactly, not approximately (calibration.py lines 296–304):

    # (4) Order preservation, decision level: s/τ has the sign of s, so every
    #     0.5-threshold decision — hence accuracy — is bit-identical.
    assert np.array_equal(p_before >= 0.5, p_after >= 0.5)
    assert before["acc"] == after["acc"]
    # (5) Order preservation, ranking level: s ↦ s/τ is strictly increasing,
    #     so the full sorted order and the pairwise AUC are bit-identical.
    assert np.array_equal(np.argsort(p_before, kind="stable"),
                          np.argsort(p_after, kind="stable"))
    assert before["auc"] == after["auc"]

and the committed run confirms: acc 0.9683 = 0.9683 AUC 0.9136 = 0.9136 argsort(p) identical over all 126 triples. The full ledger, instrument by instrument:

instrument	what it measures	BEFORE	AFTER	can $\tau$ move it?
ECE	mean absolute reliability gap	0.0245	0.0167	yes: the target
NLL	log-likelihood of the labels	0.1708	0.1387	yes: the fitting objective
Brier	squared error = calibration + refinement	0.0216	0.0221	yes, either direction
accuracy at 0.5	threshold decisions	0.9683	0.9683	never (sign-preserving)
AUC	pairwise ranking	0.9136	0.9136	never (order-preserving)

This is why temperature scaling is the minimal repair and the standard first move [1]: it spends one degree of freedom, purchased entirely inside the safe subspace of monotone transforms, where nothing the model decides can be damaged, only how loudly it announces its decisions.

What one scalar cannot fix

The same theorem that makes $\tau$ safe bounds what it can do. Look back at the AFTER table's residuals. The single false triple once announced at $0.1697$ now sits at $0.2459$: pulling all probabilities toward one half moved this one upward, and its bin gap grew from $-0.1697$ to $-0.2459$. And (bob, advises, dave), true at $0.4281$ before, is true at $0.4491$ after: closer to one half, still on the wrong side of it. It will always be on the wrong side of it, for every $\tau$, because $s = -0.290$ is negative and the map preserves signs; the invariance that protects every correct decision equally protects every incorrect one. A miscalibration whose shape varies across the score range, overconfident here, underconfident there, wrong-side-of-the-threshold somewhere, is beyond a one-parameter family: temperature can only compress or stretch the whole score axis around zero at a single rate.

The menu beyond spends more degrees of freedom, at rising risk. Platt scaling fits an affine map $\sigma(as + b)$, two parameters, so it can also shift scores and thus move the threshold, repairing a global bias at the price of the invariance guarantee [5]. Isotonic regression fits an arbitrary nondecreasing step function from scores to probabilities, as many effective parameters as the data supports; it keeps rankings (the map is still monotone) but can flatten distinct scores to ties, and it overfits small calibration sets readily [5]. For multiclass systems, vector and matrix scaling give each class its own temperature or an affine map on the whole logit vector, and were found to underperform plain temperature scaling at scale precisely because the extra freedom overfits [1]. The through-line is the overfitting caution: a calibrator is a model, trained on data, and it can memorize its calibration split exactly as the original network memorized its training set. That is why fit_tau sees only the 63 calibration triples and every reported audit metric comes from the other 63 (calibration.py lines 261–272), and why the strict-improvement assert is meaningful: it certifies generalization of the repair, not fit.

Between shortcuts and abstention

Part III's placement is an argument, not an accident. The previous chapter measured whether a model's internal concepts are right; this chapter measured whether its output confidence is honest; the next chapter will spend that honest confidence, trading coverage for risk by refusing to answer below a threshold. The order matters in both directions. Downstream, a selective predictor is only as good as the confidence it thresholds: an abstention gate over the BEFORE column would keep (bob, authored, p1) silently discarded at $p = 0.0001$, wrong with maximal conviction, and no threshold can save a gate whose input lies. Upstream, the shortcut lesson bounds what today's audit certifies. A model that has learned the wrong concepts can still pass every reliability check, because calibration conditions on confidence, not on reasons: it certifies the honesty of the announcement, not the semantics behind it. A calibrated wrong-concept model is calibrated about the wrong thing. Closing that gap is a live research line: making neuro-symbolic models' concept-level confidence track their reasoning-shortcut ambiguity, so that uncertainty lands on the concepts that are actually unidentifiable, is exactly the program of BEARS in the shortcut literature [7]. And there is a structural alternative to the scalar confidence altogether, one this series already built: Volume 2's Annotated Logic attached interval annotations to facts, a lower and an upper bound with open-world semantics, representing "somewhere between 0.6 and 1, pending evidence" as a first-class object rather than collapsing ignorance and measured frequency into one number. The SATORI chapters closing this volume will lean on precisely that representation when scalar confidence runs out of room.

The unsolved part

Three honest gaps, in rising order of difficulty. First, the estimator itself: ECE on small samples is biased, its value depends on the bin count and placement, within-bin cancellation pulls it down while per-bin noise pushes it up, and the equal-width scheme this chapter implements is a convention, not an optimum; the binning question was already the motivating problem for the Bayesian binning line of work that named ECE [2], and no single estimator has settled it since. Our 63-point audit, with its singleton bins and its seven-for-seven bin certifying only "above about two thirds," is the problem in miniature. Second, distribution shift: calibration is a property of a distribution, the one the audit sampled from, and it degrades silently when deployment drifts, with no in-model signal that the promise has lapsed; a model calibrated on yesterday's queries makes today's announcements with yesterday's honesty, and detecting the lapse requires labels that deployment, by definition, does not have. Third, and deepest for this volume: for structured outputs, even defining the event in $P(\text{correct} \mid \text{confidence} = p)$ is unsettled. What is "correct with probability $p$" for a ranking over thirteen entities, for a proof whose steps can be individually sound while the conclusion misses, for a conjunctive query returning a set of answers? The binary audit of this chapter worked because a triple is true or false; the reasoning systems this volume cares about emit objects with internal structure, and a calibration theory for them, what to condition on, what counts as the outcome, what the reliability table's rows even are, is an open research edge, not a solved chapter waiting to be written up.

Why it matters

The series has been climbing toward systems whose answers can be trusted, not merely scored, and calibration is the property that lets a number cross a system boundary. Inside one model, a score only has to order candidates, and Volumes 3 and 4 legitimately never asked for more. The moment the score is consumed elsewhere, by the abstention gate of the next chapter, by a human deciding which derivations to audit, by SATORI's verifier deciding what to prove, the consumer needs the number to mean the same thing every time, and $P(\text{correct} \mid \text{conf} = p) = p$ is exactly that contract. This chapter's protocol is deliberately portable to your own research: any system in this book that emits a score in $[0,1]$, a fuzzy satisfaction degree, a query-embedding answer weight, a proof confidence, can be audited with thirty lines that bin, count, and compare, repaired with one held-out scalar, and guarded with asserts that the repair changed no decision. Printing the reliability table next to the accuracy number, counts included, regressions included, is cheap, and it is the difference between reporting a model and reporting a model you may believe.

Key terms

• Calibration: the conditional-probability promise $P(\text{correct} \mid \text{confidence} = p) = p$; among all announcements at confidence $p$, a fraction $p$ are correct.
• Reliability table (reliability diagram): the binned estimator of that promise; per confidence bin, the count $n_b$, mean confidence $\mathrm{conf}_b$, and observed accuracy $\mathrm{acc}_b$, with deviation from $\mathrm{conf}_b = \mathrm{acc}_b$ the miscalibration.
• Expected calibration error (ECE): the count-weighted mean absolute gap $\sum_b (n_b/N)\,\lvert \mathrm{acc}_b - \mathrm{conf}_b \rvert$; a biased, binning-dependent, and indispensable summary.
• Equal-width vs equal-mass binning: fixing the bins' intervals versus fixing their counts; the estimator's main knob, and a source of its bias either way.
• Brier score: the mean squared error $\frac{1}{N}\sum_i (p_i - y_i)^2$ of the forecast against the outcome; decomposes exactly into a calibration term plus a refinement term.
• Refinement: the within-bin Bernoulli variance $\sum_b (n_b/N)\,\mathrm{acc}_b(1 - \mathrm{acc}_b)$; small when confidence sorts cases into pure bins, independent of whether its values are honest.
• Negative log-likelihood (NLL): the fitting objective $\frac{1}{N}\sum_i \mathrm{softplus}(-y^{\pm}_i s_i / \tau)$; the same loss the model trained under, reused with one free parameter.
• Temperature scaling: the one-scalar repair $p = \sigma(s/\tau)$, fitted by held-out NLL; $\tau \gt 1$ softens overconfidence, $\tau \lt 1$ sharpens underconfidence.
• Order preservation: the two-line theorem that $s \mapsto s/\tau$ with $\tau \gt 0$ preserves every strict inequality and every sign, so rankings, AUC, and threshold decisions are untouched.
• Golden-section search: the deterministic one-dimensional minimizer used to fit $\tau$; requires unimodality, shrinks the bracket by $\varphi \approx 0.618$ per iteration.

Where this leads

A calibrated confidence is raw material; the next chapter builds the machine that spends it. Abstention gives the model a third output, "I decline to answer," and turns the calibrated probability into a dial: sweep a confidence threshold, refuse everything below it, and trade coverage (how often the system answers) against risk (how often an answer is wrong). The risk-coverage curve, the selective-prediction guarantees that live on it, and the question of when refusing is worth more than guessing are where the audit of this chapter becomes an operating policy.

---

Companion code: examples/frontier/calibration.py builds the labelled set from the running example, scores it with Volume 3's retrained ComplEx (examples/neural/bilinear.py), implements the reliability table, ECE, Brier, and AUC from their definitions, fits the temperature by golden-section search on the calibration half, and asserts the chapter's four claims: the fit lowers its own objective, the measured direction is overconfidence, ECE strictly improves on held-out data, and no decision, rank, or AUC moves. Run python3 examples/frontier/calibration.py to reproduce every number in this chapter byte for byte.