In our previous Dynamic AI Security blog and the underlying CAMLIS 2025 paper, we described a release platform built to move new protections into production without disrupting customer workflows. Such a platform is a requirement for security systems because the constant evolution and adaptation of adversaries require a similar response loop from vendors like Cisco.

Rapid detection-model churn creates potential downstream disruptions for customers, who suddenly, and without knowing it, start using a newer version of the model that behaves differently from the previous iteration. One major variable between deployments is model aggression: the newer model is better but is also deployed more aggressively, breaking workflows that seemed fine only a few hours ago.

Preserving the aggression level between releases is something we focus on during every model update. If a customer chooses a blocking tier that flags about 1 in 1,000 requests, a model retrain should not silently turn that into 1 in 200 or 1 in 20,000. The detector may improve underneath, but the customer’s false-positive budget should remain the same.

Different downstream users also operate at different points on that tradeoff. A SOC running aggressive blocking sits far from a tool that only enriches logs, so it is not enough to preserve one threshold across releases. The whole range of operating points, from very aggressive to very conservative, has to carry the same meaning from one model version to the next.

This is a common problem for anyone deploying detection models, so we are open-sourcing our solution for FPR calibration that can be applied before a model is released to minimize the chance of customer disruption. The method works offline on benign scores and ships a bounded sklearn artifact along with the model. The code is at github.com/cisco-ai-defense/fpr-model-calibration, and the paper describing the technical details can be downloaded here.

Why false-positive rate is the right contract

False-positive rate (FPR) is the fraction of benign traffic that would be flagged at a threshold. For a model score threshold, FPR estimates how much legitimate activity the threshold will interrupt in production.

FPR calibration differs from probability calibration, which estimates Pr(attack | score). For many security models, that probability depends on an attack distribution that is rare, adversarial, and rapidly shifting. Attackers change tactics when detectors improve. The positive class a model sees during training is therefore a record of past attacks, not a stable sample of future attacks.

FPR calibration depends only on benign traffic. In many production security settings, benign traffic is more plentiful, easier to measure, and tied directly to false-positive harm. If the calibrated score says a request is a 1-in-1,000 benign event, the product team can reason about alert volume without needing to know tomorrow’s attack prevalence.

What a calibrated score means

The calibrator maps raw model scores onto a fixed score contract. The calibrated score contract maps common operating tiers to target FPRs:

The scale is logarithmic because production FPR decisions are logarithmic. Moving from 1% to 0.1% and from 0.1% to 0.01% are both tenfold reductions in benign alerts. A linear score axis would compress the low-FPR region covered by the 0.50, 0.70, and 0.85 operating tiers.

With the score contract in place, policy thresholds stay stable across model releases. A policy can block at 0.50, alert at 0.30, and enrich logs at 0.10. When the model team ships a new detector version, it ships a new calibrator with it. The policy thresholds keep their FPR meaning even though the raw model scores underneath changed.

How much data is enough?

One common gotcha when estimating the performance of detection models is just how much data you actually need to properly calibrate, or even measure, a model. While attacks can seem to be everywhere in public test sets, in practice they are very rare, usually below 0.1% of traffic. At those rates, the model has to be extremely accurate to keep the false-positive rate practical, and calibrating it requires a lot more benign data than one would expect.

A normal-approximation rule of thumb gives about 16 / p benign samples for plus-or-minus 50% relative precision at 95% confidence, where p is the target FPR. For common operating points, the rough sample counts are:

Sample size dominates low-FPR error in practice, and more benign data is the only path to tighter estimates.

Validation on a public benchmark

We validated the method on the public Credit Card Fraud Detection benchmark (284,807 transactions, 492 fraud cases), fitting the calibrator on a held-out benign subset:

The takeaway is simple: as long as the benign distribution stays fairly constant between calibration and production, a model can be calibrated very accurately.

What changes for product teams

An FPR-calibrated release includes the detector, the calibrator, and either calibrated-score serving or raw thresholds derived from the calibrator. Policy thresholds keep their FPR meaning, customers keep their false-positive budget, and the model can improve underneath.

The same contract also makes detector scores easier to compare across categories. If a prompt-injection detector and a data-leakage detector both emit calibrated score 0.50, each score means the same thing about benign rarity. Compound policies still need their own FPR measurement, but their inputs no longer mix unrelated raw score scales.

Getting started

Fit the calibrator with fit_calibration_pipeline:

from fpr_model_calibration import fit_calibration_pipeline
import joblib

pipeline = fit_calibration_pipeline(benign_scores, n_knots=10000)
joblib.dump(pipeline, “calibration.pkl”)

Production inference calls the serialized sklearn pipeline:

pipeline = joblib.load(“calibration.pkl”)
calibrated = pipeline.predict(raw_scores.reshape(-1, 1))

FPR calibration gives model releases a stable score contract without replacing fresh benign data, drift monitoring, or detection-quality evaluation. For security systems that retrain under adversarial pressure, that contract lets detectors improve while policy thresholds keep their FPR meaning.

Link to the open source GitHub repo can be found here:
https://github.com/cisco-ai-defense/fpr-model-calibration

and the preprint:
https://arxiv.org/abs/2607.05481