Most alerting configurations in production today were not designed. They were copied. A staging environment’s thresholds, a vendor’s default ruleset, or a previous engineer’s best guess got committed to version control and has been silently misfiring ever since. The result is familiar: a constant drizzle of pages that trained your team to ignore alerts, interrupted by the one critical event that actually mattered and got buried in the noise.

Industry surveys put false-positive alert rates at 60 to 80% across the median team. Research tracking on-call engineers found that 41% have considered leaving due to alert load, and 62% report weekly sleep disruption from overnight pages. This is not a tooling problem you can vendor your way out of. It is a design problem, and it starts with how you connect raw telemetry to thresholds.

This post lays out a structured framework for doing that connection properly: which metrics frameworks to use, how to set thresholds that mean something, and how to wire it together so that an alert tells an engineer what to do rather than what to investigate.

Start with a metrics framework, not a metrics firehose

The first mistake teams make is alerting on everything they can collect. Observability tooling makes it trivially easy to ship thousands of time-series to a backend; it does not tell you which ones matter. Three frameworks have proven durable enough to anchor a data-driven alerting strategy on.

The four golden signals, from Google’s Site Reliability Engineering book, are the minimum viable set for any user-facing system: latency, traffic, errors, and saturation. The key property of this set is that it is symptom-oriented. These signals detect that users are affected before you know why. As the SRE book puts it: if you measure all four golden signals and page a human when one is problematic, your service will be at least decently covered by monitoring. That is a low bar stated plainly, and it is the right starting point.

One subtlety worth preserving: latency of failed requests should be tracked separately from latency of successful ones. An HTTP 500 triggered by a dropped database connection can be served in microseconds; if you average it with successful request latency, you will understate how bad things are.

The USE method, developed by Brendan Gregg, turns the lens inward toward resources. For every resource, check utilization (the percentage of time it was busy), saturation (queued work it cannot yet service), and errors. It works as a checklist: CPUs, memory, network interfaces, storage, thread pools, file descriptors. Its limitation is that managed cloud services hide many of these resources, so USE is most useful for the infrastructure layer you actually control.

The RED method, introduced by Tom Wilkie in 2015, was designed for microservices: Rate (requests per second), Errors (failed requests), and Duration (distribution of response times). Where USE looks inward at a resource, RED looks outward at what requests experience. They are not competing frameworks; they answer different questions. Run RED against your services, USE against your infrastructure, and the four golden signals as your user-experience canary.

Golden signals tell you whether the user experience is good. USE tells you whether the servers are healthy. Most teams treat these as alternatives; the ones with clean alerting use both.

The symptom-versus-cause distinction that changes everything

A well-designed alerting strategy pages on symptoms, not causes. A symptom is something a user experiences: elevated error rate, degraded latency, a broken checkout flow. A cause is the underlying mechanism: high CPU, a saturated disk, a misconfigured connection pool. Causes are what you investigate after you know there is a symptom.

Teams that alert on causes first generate most of their noise. CPU at 82% is not an incident. CPU at 82% causing the p99 latency of your payment API to breach its SLO is an incident. The difference is the user impact, and that is what your threshold should be anchored to.

This is not an argument against infrastructure alerting. Saturation metrics, disk fill rates, and memory pressure all belong in your stack. But they belong at a lower severity, routed to a queue rather than a pager, and investigated during business hours unless they are actively producing user-visible symptoms.

Infrastructure alerting thresholds: from static to dynamic

Static thresholds are the original sin of monitoring. A fixed CPU threshold of 80% will fire during every legitimate traffic spike, every batch job, every deployment rollout. Over time, your team learns to dismiss it, and when the threshold fires because something is actually broken, it looks identical to every false positive that preceded it.

Moving to actionable alert design means replacing or supplementing static values with three better approaches:

  • Percentile-based thresholds: Alert when p95 or p99 latency exceeds a value, not when the mean does. Means hide tail latency, and tail latency is what your worst-affected users experience.
  • Rate-of-change alerting: A metric moving from 40% to 75% in two minutes is more alarming than one sitting at 75% for six hours. Derivatives catch runaway conditions before they breach absolute limits.
  • Baseline-relative thresholds: Alert when a metric deviates from its own historical baseline by a statistically meaningful amount. This handles the weekly traffic cycle that makes a fixed threshold useless on Monday mornings versus Saturday nights.

None of these require AI. They require discipline in how you define the alert rule, and a metrics backend that can evaluate it efficiently.

SLO-based alerting and burn rate: the most underused pattern

The cleanest evolution of infrastructure alerting thresholds is error budget burn rate alerting, which comes from the Google SRE Workbook. The concept is straightforward: your SLO defines how much failure is acceptable over a window. A 99.9% availability SLO gives you 43 minutes of monthly downtime. An error budget is 1 minus the SLO.

Burn rate measures how fast you are consuming that budget relative to the rate that would exhaust it exactly at the end of the window. A burn rate of 1 means you are on track to use the full budget by month end. A burn rate of 10 means you will exhaust it in one-tenth the time. Alerting on elevated burn rate rather than raw error counts gives you two properties that static thresholds cannot: it is proportional to actual user impact, and it has a built-in sense of urgency that scales with how quickly the situation is deteriorating.

This approach also gives teams a principled way to reduce alert fatigue by tying pages to business impact rather than metric values. If the budget is not burning, there is no page. If it is burning fast, the page is warranted and the engineer receiving it has a clear mandate.

Building the metrics-to-alerts pipeline

A working pipeline has four stages: collection, normalization, evaluation, and routing. Collection is the instrumentation layer: exporters, agents, and SDK-level metrics. Normalization is where you apply consistent labeling, drop high-cardinality noise, and enforce retention policies. Evaluation is where your alerting rules live. Routing is where severity, ownership, and escalation paths are defined.

Most teams have collection and a partial version of evaluation. They are missing normalization (which is why the same service appears under six different label values) and routing (which is why every alert goes to the same channel at the same priority).

Routing deserves more attention than it gets. An actionable alert design maps each alert to a severity tier: a user-visible symptom burning error budget at high rate goes to an on-call pager; a saturation warning on a non-critical queue goes to a ticket; a deployment-correlated spike auto-resolves after the rollout window. The routing logic is not complex, but it has to be written down and enforced, or it collapses back into every alert going everywhere.

We run this pattern across our own multi-datacenter infrastructure, including AMS1 in Amsterdam where dense peering with AMS-IX means network telemetry is particularly rich and noisy. Separating infrastructure saturation signals from service-level symptom alerts was what made our on-call rotation sustainable at that site.

Operationalizing the strategy

A data-driven alerting strategy is not a one-time configuration. It requires a regular review cycle: which alerts fired in the last 30 days, which were actioned, which were dismissed, and which fired without a corresponding user impact. That review is where you tighten thresholds, promote some cause-level alerts to symptoms, and retire the ones that never produced a meaningful action.

The Catchpoint SRE Report 2025 found that the median SRE now spends 30% of their time on reactive operations, up from 25% the year before. That trajectory does not reverse by adding more tooling. It reverses when the alerts that land in an inbox are the ones that require a human response, and the rest are handled by automation or not raised at all.

Start with the four golden signals. Layer in USE for the infrastructure you own. Add burn rate alerting for any service with a defined SLO. Build routing tiers before you tune thresholds. And then review the output monthly until the ratio of actionable alerts to noise is one you would be comfortable waking up to at 3 a.m.