Anomaly detection that operations teams will actually use

The detector that got muted

The story is almost always the same. A detector goes live. For two weeks people look at it. Then the alerts get routed to a channel, the channel gets muted, someone writes a mail rule, and eighteen months later the model is still running, still scoring, still costing money, and nobody has opened its output since spring. When something bad finally happens, the post-incident review finds that the detector caught it — on the day it happened, in a queue nobody was reading.

It is tempting to call that a discipline failure. It is not. It is a rational response to a channel with a low hit rate, and the best-documented version of it is not in software at all. It is in hospitals.

The Joint Commission's Sentinel Event Alert 50, issued April 8, 2013, states that "the number of alarm signals per patient per day can reach several hundred depending on the unit within the hospital," and that "it is estimated that between 85 and 99 percent of alarm signals do not require clinical intervention." The consequence it describes is the part worth memorizing: clinicians "become desensitized or immune to the sounds, and are overwhelmed by information — in short, they suffer from 'alarm fatigue.'" And then, in response to the noise, "clinicians may turn down the volume of the alarm, turn it off, or adjust the alarm settings outside the limits that are safe and appropriate for the patient."

These are trained professionals in an environment where the cost of a miss is a death, and they still turn the alarms off. Your fraud analysts and on-call engineers will not behave better. The lever is not the humans; it is the share of alerts worth acting on.

The arithmetic that mutes a detector

The reason that share is so hard to move is arithmetic, and it was worked out precisely by Stefan Axelsson in "The Base-Rate Fallacy and the Difficulty of Intrusion Detection" (ACM Transactions on Information and System Security, 2000). The paper is about intrusion detection, but the result applies to any detector hunting something rare.

His setup, in the 1999 conference version: ten audit records per intrusion, two intrusions a day, a million audit records a day — so the prior probability that any given record is part of an intrusion is 2 × 10⁻⁵. He then asks the only question an operator cares about, which he calls the Bayesian detection rate: given an alarm, what is the chance it is real?

The answer is bleak in a specific, useful way. Even at a perfect detection rate of 1.0, you need a false alarm rate "on the order of 1 × 10⁻⁵, or 1/100,000 per event" before roughly two thirds of alarms are genuine. At a more realistic detection rate of 0.7, same false alarm rate, "the value of the Bayesian detection rate is about 58%, nearing fifty-fifty."

Then the sentence that should govern your roadmap: "even quite a large difference in the detection rate does not substantially alter the Bayesian detection rate, which instead is dominated by the false alarm rate."

Read that as a budgeting instruction. A quarter spent lifting recall from 0.70 to 0.90 barely changes what the operator experiences. A quarter spent cutting the false-positive rate tenfold decides whether the queue survives. Teams get this backwards constantly, because recall is what gets reported upward and precision is what gets felt downward. If the vocabulary here is fuzzy, our guide to precision, recall, and false-extraction rate is the prerequisite.

Baselines and thresholds are two different decisions

Most teams collapse two decisions into one knob, then wonder why turning it never helps. The decisions are: what is normal for this particular entity, and how far from normal is far enough to interrupt a person. The first is a modelling problem, the second a management problem. Different owners, different failure modes.

On the first, NIST Special Publication 800-94 is still the clearest short description in the literature. Anomaly-based detection, it says, uses "profiles that represent the normal behavior of such things as users, hosts, network connections, or applications," developed "by monitoring the characteristics of typical activity over a period of time." Then it names the two ways profiles go wrong, and you will hit both.

• Static profiles rot. "Because systems and networks change over time, the corresponding measures of normal behavior also change; a static profile will eventually become inaccurate, so it needs to be regenerated periodically." Nobody schedules this. It is the most common reason a detector that was good in March is noise by September.
• Dynamic profiles can be walked. A continuously updating profile is "susceptible to evasion attempts from attackers," because an attacker "can perform small amounts of malicious activity occasionally, then slowly increase the frequency and quantity of activity. If the rate of change is sufficiently slow, the IDPS might think the malicious activity is normal behavior and include it in its profile." NIST adds that absorbing bad activity into a profile "is a common problem."

A third failure is less dramatic and more expensive. NIST's example: "if a particular maintenance activity that performs large file transfers occurs only once a month, it might not be observed during the training period; when the maintenance occurs, it is likely to be considered a significant deviation from the profile and trigger an alert." Monthly close, quarterly reindex, the annual enrollment surge — each a legitimate, high-amplitude event your training window never saw, and each will page someone. Treat seasonality as part of the baseline rather than as an annoyance; our time-series analysis guide covers the decomposition side.

Now the threshold, where the trap is that a strict-sounding rule is not strict at scale. The NIST/SEMATECH e-Handbook of Statistical Methods gives the number for the classic three-sigma limit under normality: "the 3σ in one direction is 0.00135, or in both directions 0.0027."

An illustrative calculation, not a measurement: score 2,000 entities once a minute and you have 2,880,000 scored points a day. At 0.0027 per point, a perfectly healthy system produces about 7,776 alerts a day out of pure noise — roughly four per entity, before anything has gone wrong. To get to twenty noise alerts a day across the fleet you need a per-point false-positive probability near 7 × 10⁻⁶, which lands right where Axelsson said it would.

That is why "just use three sigma" produces a muted channel, and why the threshold cannot be one global number. Set it per segment — per entity class, per business hour, per volume band — because a busy production cluster and a quiet lab host do not have the same variance, and one number serving both is simultaneously too loud in one place and blind in the other.

Why the benchmark number lied

Before you buy or build, be suspicious of the score. The field has spent several years auditing its own evaluation practices, and the findings are unkind.

Start with point adjustment, a convention where detecting any single point inside a labelled anomaly segment counts as detecting the whole segment. Kim, Choi, Choi, Lee and Yoon examined it in "Towards a Rigorous Evaluation of Time-series Anomaly Detection" (AAAI 2022) and found the protocol "has a great possibility of overestimating the detection performance; that is, even a random anomaly score can easily turn into a state-of-the-art TAD method." They also report that untrained models score comparably to published methods once point adjustment is removed. If a vendor shows you an F1 score on time-series data, ask whether it was point-adjusted.

Then the datasets. Renjie Wu and Eamonn Keogh, in "Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress" (IEEE TKDE), report that "the majority of the individual exemplars in these datasets suffer from one or more of four flaws," and conclude that much apparent progress may be illusory.

And then the algorithms. ADBench (Han, Hu, Huang, Jiang and Zhao, NeurIPS 2022) ran 30 algorithms across 57 datasets. Two findings from the authors' summary should reshape how you spend engineering time: "surprisingly none of the benchmarked unsupervised algorithms is statistically better than others," and "with merely 1% labeled anomalies, most semi-supervised methods can outperform the best unsupervised method."

Three consequences follow.

1. Stop shopping for the algorithm. Isolation Forest versus a transformer is a smaller lever than almost any team believes.
2. Capturing analyst dispositions is the highest-return task on the project. Every time an investigator marks an alert as nothing, that is a label — and one percent of your alerts labelled beats a year of model shopping. You only get those labels if someone built the button.
3. Evaluate the way the operator lives. Precision at the top k they can review in a shift. Recall at a false-positive rate the team can sustain. Time from onset to alert. And walk-forward backtests — a random split on temporal data leaks the future into the past and will flatter you badly.

Designing for the person carrying the pager

The best writing on this problem is not in the anomaly detection literature. It is in site reliability engineering, where the cost of a noisy pager is measured in attrition.

The SRE book, in "Monitoring Distributed Systems," puts the human constraint plainly: "Every time the pager goes off, I should be able to react with a sense of urgency," and a person can sustain that "a few times a day before I become fatigued." The rules follow: "every page should be actionable," and "every page response should require intelligence. If a page merely merits a robotic response, it shouldn't be a page." The Prometheus alerting guidance arrives at the same place: "aim to have as few alerts as possible, by alerting on symptoms that are associated with end-user pain rather than trying to catch every possible way that pain could be caused."

Translated into anomaly detection, that produces a short list of design rules that matter more than model choice.

1. Start from capacity, not accuracy. How many items can this team genuinely review in a shift? That number is the alert budget. A detector with excellent AUC and 500 alerts a day is worthless to a team with room for 20.
2. Every alert carries its own reason. NIST names this directly: with anomaly-based detection "it is often difficult for analysts to determine why a particular alert was generated and to validate that an alert is accurate and not a false positive." A bare score can only be believed or ignored, and it will be ignored. State what was expected, what was observed, over what window, against which peer group.
3. Rank rather than threshold where you can. A ranked queue degrades gracefully — the shift ends and the highest-value items were the ones reviewed. A threshold-triggered stream either floods or goes silent.
4. Separate "page me" from "put it in a queue." Most anomalies are queue-worthy, not page-worthy, and collapsing the two mutes the channel within weeks.
5. Suppress correlated duplicates. One root cause, one alert. A rack failure firing 300 per-host anomalies has spent the day's whole attention budget on a single fact.
6. Build the disposition button before the model. Adjudication capture is your label pipeline. Without it, the ADBench result — 1% labels beating the best unsupervised method — is a finding you can read but never use.
7. Schedule baseline regeneration, and review what it absorbed. Static profiles rot; dynamic profiles can be walked. Both are managed by regenerating on a calendar with a human checking the new normal. Our MLOps guide covers the machinery.
8. Publish the miss in advance. Tuning to a sustainable alert budget means choosing to miss things. Say so in writing, with an estimate, before go-live. A documented trade-off is a design decision; undocumented, the same miss is a scandal — and the reflex response to a scandal is to widen thresholds, which restarts the cycle.

The honest trade-off underneath all eight: a lower alert budget is a decision to catch less. No configuration gives you high recall and a quiet pager on a rare-event problem. The only choice is whether that trade-off gets made deliberately by the people accountable for the mission, or accidentally by a tired analyst at 3 a.m. with the mute button.

The short version

1. Ask the operator-capacity question before the modelling question. Alerts per shift is the specification.
2. Spend on cutting false positives, not on lifting recall. Precision is what the operator feels.
3. Treat the baseline and the threshold as separate, separately owned decisions.
4. Distrust any score not evaluated at your real alert budget, on a walk-forward split, with the point-adjustment question answered.

None of this needs a novel algorithm. It needs a decision, up front, that the design target is a person's attention rather than a metric on a slide. If the detector is one piece of a larger effort that has stalled, the pattern is worth reading too: why AI pilots stall before production.

Sources: The Joint Commission — Sentinel Event Alert 50, "Medical device alarm safety in hospitals," April 8, 2013; Stefan Axelsson, "The Base-Rate Fallacy and the Difficulty of Intrusion Detection," ACM TISSEC 3(3), 2000 (numeric example quoted from the 1999 conference version); NIST SP 800-94, Guide to Intrusion Detection and Prevention Systems, §2.3.2; NIST/SEMATECH e-Handbook of Statistical Methods — control chart limits; Kim et al., "Towards a Rigorous Evaluation of Time-series Anomaly Detection," AAAI 2022; Wu & Keogh, "Current Time Series Anomaly Detection Benchmarks are Flawed," IEEE TKDE; Han et al., "ADBench: Anomaly Detection Benchmark," NeurIPS 2022; Google SRE Book — Monitoring Distributed Systems; Prometheus — alerting best practices. The scaling calculation in the threshold section is our own arithmetic on the NIST figure and is labelled as illustrative. Analysis and framing by Precision AI Academy.

Common questions

Why do operations teams stop responding to anomaly alerts? Because muting is a rational response to a channel with a low hit rate. The Joint Commission's Sentinel Event Alert 50 estimates that between 85 and 99 percent of hospital alarm signals do not require clinical intervention, and reports that in response clinicians may turn down the volume of the alarm, turn it off, or adjust settings outside safe limits. Software teams do the same thing with a mail rule. The fix is to raise the share of alerts worth acting on, not to retrain the humans.

Is a higher detection rate the way to fix a noisy detector? Usually no. Axelsson's base-rate analysis shows that when the events you care about are rare, the chance an alarm is genuine is dominated by the false alarm rate rather than the detection rate. In his worked example, even a perfect detection rate of 1.0 requires a false alarm rate on the order of 1 in 100,000 events before about two thirds of alarms are real.

What is the difference between a baseline and a threshold? A baseline models what normal looks like for a specific entity — a host, a provider, a pump, an account. A threshold is the separate decision about how far from that baseline warrants interrupting a person. NIST SP 800-94 describes baselines as profiles built by monitoring typical activity over a training period, and warns that static profiles eventually become inaccurate while dynamic profiles can be gradually poisoned by an attacker who ramps up slowly.

Why do published anomaly detection benchmark scores overstate real performance? Three documented reasons. Kim et al. (AAAI 2022) showed that the point-adjustment evaluation protocol overestimates performance badly enough that even a random anomaly score can look state of the art. Wu and Keogh found that most exemplars in common benchmark datasets suffer from one or more of four flaws. And ADBench, covering 30 algorithms across 57 datasets, reported that none of the benchmarked unsupervised algorithms is statistically better than the others.