A Chatbot Confidence Score That Never Drops Is a Bug

“What’s the capital of France?” Intent: check_balance. Confidence: 0.96.

That’s a reconstruction, not a log line, but it’s faithful to the intent service behind a chatbot at a large Canadian bank. The service read a customer’s message, picked a task from a fixed list (check balance, block card, dispute a charge), and returned a score for how sure it was. It was sure of everything. Nearly every message came back above 0.9, including the ones that made no sense. When a chatbot confidence score is always high, it carries no information: the routing threshold never fires, off-topic questions go down a banking flow, and the dashboard stays green. Our own QA test couldn’t fail either:

from intent_client import classify


def test_intent_confidence_is_high():
    result = classify("I lost my card")
    assert result.intent == "block_card"
    assert result.confidence > 0.8  # passed on every run, for every input

It would have passed for “asdf jkl;” too, and nobody had written that version. A test that only checks for a high number can’t tell a confident model from an honest one. The model had a bug, and so did our test.

I wanted to know which fixes actually work, so I rebuilt the problem on open data and tried them in order. Three were dead ends. The thing that finally made the score drop wasn’t a model change at all.

What does a confidence score actually promise?

A track record. Of all the messages scored 0.9, about 90% should be classified correctly. When that holds, the model is calibrated; when it doesn’t, the number is decoration.

A forecaster who says “90% chance of rain” every day is right on the rainy days and useless on all of them. The metaphor is literal: the standard scoring rule for probabilistic predictions comes from Glenn Brier’s 1950 paper in Monthly Weather Review, written to grade rain forecasts. Vendors admit as much in the fine print. Amazon’s Lex V2 docs call its intent scores “comparative values” you shouldn’t treat as absolute.

The standard suspect is the model itself. Softmax, the last step, forces the scores to sum to 1 across the intents you defined, so “none of these” has nowhere to go. And Guo et al.’s “On Calibration of Modern Neural Networks” (ICML 2017) showed that deeper networks trained harder grow overconfident. That was my hypothesis going in.

Can you reproduce an always-confident intent model on open data?

Not the in-scope part. On CLINC150 with a shallow TF-IDF + neural-network classifier, scores on real banking-style questions were honest. The wall showed up only on out-of-scope messages, and it grew the longer I trained.

CLINC150 (Larson et al., EMNLP 2019) has 150 intents across 10 domains plus a separate set of out-of-scope queries. I trained a TF-IDF (1–2 word n-grams) + scikit-learn MLPClassifier with 256 hidden units, seed 0, on the 15,000 in-scope training queries. It trained in 145 seconds on my laptop. Then I bucketed the 4,500 test queries by confidence, the same table you’d build from one day of production logs:

import numpy as np


def reliability_table(confidence, correct, n_bins=15):
    """Bucket predictions by stated confidence; compare to actual accuracy."""
    confidence = np.asarray(confidence, dtype=float)
    correct = np.asarray(correct, dtype=float)
    bins = np.minimum((confidence * n_bins).astype(int), n_bins - 1)
    rows, ece = [], 0.0
    for b in range(n_bins):
        in_bucket = bins == b
        if not in_bucket.any():
            continue
        said, was = confidence[in_bucket].mean(), correct[in_bucket].mean()
        rows.append((b / n_bins, int(in_bucket.sum()), float(said), float(was)))
        ece += in_bucket.mean() * abs(said - was)  # weight the gap by bucket size
    return rows, ece

The ece it returns is Expected Calibration Error: the average gap between “said” and “right”, weighted by bucket size. Zero is perfect.

In-scope: the wall refused to appear

In-scope accuracy was 90.0% and ECE was 0.0197. The top bucket held 3,377 of 4,500 queries at a mean confidence of 0.992, and 98.0% of them were right. That’s not a liar; that’s a model that’s genuinely sure about easy questions.

Out-of-scope: the tail grows with training

The 1,000 out-of-scope test queries told a different story. Their median top score was 0.255, but 11.8% scored 0.8 or higher on some real intent. Training to 200 epochs (749 seconds) barely moved the in-scope numbers: 91.9% accuracy, ECE 0.0238. Out-of-scope confidence crept up the whole way:

So Guo’s mechanism is real here, just not where I looked first. On this shallow model, training harder doesn’t wreck in-scope calibration. It makes off-topic messages look more like real intents. A logistic regression with weak regularization showed the same thing: ECE 0.0351, but 25.2% of out-of-scope queries at 0.8 or above.

Dead end 1: does temperature scaling fix an overconfident intent model?

Not this one. Temperature scaling divides the raw scores by one fitted number, T, so it can’t change which intent wins. On my 40-epoch model the fit came out at T = 0.889: the model was slightly under-confident, so the “fix” made everything more confident.

import numpy as np
from scipy.optimize import minimize_scalar
from scipy.special import log_softmax, softmax


def fit_temperature(val_logits, val_labels):
    """Find the single number T that makes validation-set confidence honest."""
    def nll(t):
        log_probs = log_softmax(val_logits / t, axis=1)
        return -log_probs[np.arange(len(val_labels)), val_labels].mean()

    return minimize_scalar(nll, bounds=(0.05, 10.0), method="bounded").x


def calibrate(logits, t):
    probs = softmax(logits / t, axis=1)
    return probs  # top-scoring intent is unchanged; only the score softens

T is fit on in-scope validation data, so it optimizes for the questions the model already handles. Test ECE got slightly worse (0.0197 → 0.0265). The out-of-scope median rose from 0.255 to 0.323, and the share at 0.8 or higher went from 11.8% to 15.9%. Accuracy was identical and no prediction changed.

On the 200-epoch model, T was 1.127, so scaling did soften the scores. After scaling, all three models landed in the same place: 15.9–16.4% of out-of-scope queries still scored 0.8 or higher. Whatever the training regime, roughly one off-topic query in six stays confident. Calibration can tune in-scope scores. It can’t invent a “none of these” bucket.

Where does a confidence score that never drops come from?

Often from code between the model and the API. One post-processing step, renormalizing the top few intents so they sum to 1, produced more confidence inflation in my experiment than 160 extra epochs of training. It needs no retraining and it’s easy to miss in review.

Older Rasa versions did this: the classifier renormalized the top ranking_length intents (10 by default). Here’s the step:

import numpy as np


def renormalize_top_k(probs, k):
    """Keep the k highest scores per row and rescale them to sum to 1."""
    top_k = np.argsort(-probs, axis=1)[:, :k]
    rows = np.arange(len(probs))[:, None]
    out = np.zeros_like(probs)
    out[rows, top_k] = probs[rows, top_k]
    return out / out.sum(axis=1, keepdims=True)

Applied to the 40-epoch model’s output, same model, no retraining:

After top-3 renormalization, “write me a poem about mortgages” scored 0.856 and France scored 0.546. I can’t prove this is what the bank’s service did; I never saw its code. I can prove that one innocent-looking line is enough to produce the symptom, and it’s the first thing I’d check now.

Dead end 2: can a confidence threshold alone catch out-of-scope messages?

Only by escalating real customers. A threshold is one dial: every notch that catches more off-topic messages also escalates more correct, answerable questions. There’s no setting where both numbers look good.

On the raw 40-epoch scores:

At 0.95, one in five correct answers goes to a human. Push this far and you’ve built the opposite failure, the one I wrote about in a safer model quietly refusing your real users.

Dead end 3: does an explicit out-of-scope intent fix it?

Not with 100 examples. I retrained with an oos class using CLINC150’s 100 out-of-scope training queries. It caught 13.7% of out-of-scope test queries, and 131 of the 1,000 still went to a real intent at a calibrated 0.8 or higher.

The misses are the instructive part:

• “give me the weather forecast for today” → weather, 1.000
• “ignore call” → make_call, 1.000
• “deny incoming phone call” → make_call, 1.000

These aren’t nonsense. They sit right next to a real intent, and the model reads them as that intent. Larson et al. found the same pattern: their best out-of-scope recall, 66%, came from an explicit class trained on 250 out-of-scope queries instead of 100. In-scope accuracy held at 90.2%, so the class cost nothing. It just didn’t buy much.

What should the chatbot do with the confidence score?

Route three ways, and require a clear winner before acting. Act when the calibrated score is high and the top intent beats the runner-up by a clear margin. Confirm (“Did you mean…?”) when it’s plausible but close. Escalate below a floor. This was the only change that made a real dent, and it still leaves a hole.

import numpy as np

ACT_AT = 0.80          # tune on your own labeled data, not a vendor default
MIN_MARGIN = 0.15      # winner must beat the runner-up by this much
ESCALATE_BELOW = 0.40


def route(probs, intents):
    """probs: calibrated probabilities for one message, aligned with intents."""
    second, top = np.argsort(probs)[-2:]
    confidence, margin = probs[top], probs[top] - probs[second]

    if intents[top] == "out_of_scope" or confidence < ESCALATE_BELOW:
        return "escalate", None
    if confidence >= ACT_AT and margin >= MIN_MARGIN:
        return "act", intents[top]
    return "confirm", (intents[top], intents[second])  # "Did you mean X or Y?"

On the calibrated 200-epoch model, out-of-scope traffic split 16.4% act / 30.7% confirm / 52.9% escalate. In-scope traffic split 87.7% act / 9.0% confirm / 3.3% escalate, and 97.2% of the acted-on messages were right. Here are my probe messages through the same model:

Gibberish is the easy case. The messages that get through are the near-misses: close enough to a real intent that the model is sure, wrong enough that acting on them is a bug. For a bank I’d also force confirmation on money-movement intents, whatever the score.

How do you test a confidence score in CI?

Not with ECE alone. Every finished model in this experiment passed an ece <= 0.05 assertion, including the one that acts on 16.4% of off-topic traffic. Even the top-2 renormalized version, at 34.6% of out-of-scope at 0.8 or higher, scored 0.0473. An ECE-only gate is another test that can’t fail for the failure you care about.

Keep ECE as a drift tripwire on a nightly schedule, the same pattern as a scheduled real-model eval gate. Gate on behaviour instead:

from pathlib import Path

import pytest

from intent_client import classify  # returns intent, confidence, routed action

OOS_HOLDOUT = (Path(__file__).parent / "data/oos_holdout.txt").read_text().splitlines()
NEAR_DOMAIN = ["write me a poem about mortgages", "can you help me with my thing", "what's the capital of France?"]
OOS_ACT_BUDGET = 0.20  # baseline was 16.4%; ratchet down, never up


@pytest.mark.parametrize("text", NEAR_DOMAIN)
def test_near_domain_message_is_never_acted_on(text):
    assert classify(text).action != "act"


def test_share_of_held_out_oos_routed_to_act_stays_in_budget():
    acted = sum(classify(text).action == "act" for text in OOS_HOLDOUT)
    assert acted / len(OOS_HOLDOUT) <= OOS_ACT_BUDGET

In the experiment, my first version of this gate (one ECE check plus the five probe messages above) ran 6 tests against the 200-epoch model and failed 2, the poem and “my thing”, on both raw and calibrated scores. That’s the first test in this whole story that goes red for the right reason. The budget test passes today at 16.4%. Its job is to fail when the next retrain or post-processing change pushes more off-topic traffic into “act”. Feed both lists from production logs: your customers will write stranger near-misses than you can invent. The held-out set has to be big enough to measure a rate; 30 labeled cases can’t tell you whether a model got worse.

What I’d do differently

• Check what sits between the model and the API before touching the model. Ten messages’ raw output next to the API response would have shown whether the wall came from the model or from the code after it.
• Write every confidence assertion in both directions. High on known-good messages, never acted on for near-domain and out-of-scope ones.
• Log the runner-up intent and the margin from day one. The top score alone hides a collapse; the margin shows it.

Start with one day of logs and ten raw outputs

Pull one day of intent-service traffic, label a few hundred messages, and run reliability_table on them. Then take ten of those messages and put the model’s raw output next to what the API returned. If the raw scores spread out and the API’s don’t, your bug is in the code between them, not the model. Want to reproduce my numbers first? The full scripts are at /code/clinc150-calibration.py and /code/clinc150-calibration-pass2.py; they expect data/data_full.json from the CLINC150 repo.

FAQ