Why did that page anyone?

The alert-noise playbook: flap damping, recovery bursts, silence detection, quiet hours, and how to audit your own alerting in an afternoon.

A rebooted server used to mean forty emails. A host whose network flaps used to mean a hundred. Alert noise is not a tuning problem; it is a design problem, and it has four known fixes. This playbook explains each, then gives you a checklist to run against your current monitoring, whatever it is.

Written by Jared DeGraffenried, Guardian Plus. Every claim here is on the site or in the product documentation; where our own answer is a no, it says so.

A host bouncing up and down should be damped, then announced once with its real outage duration. Without damping, every transition is a page, and the person on call learns to ignore the host that matters most.

• Count transitions in a window, not states
• Announce once when it settles, with the total downtime
• Record the damped transitions so the post-mortem still has them

When a box comes back, its checks do not each need to mail you their own recovery notice. Hold recoveries for a short window and send one message per host.

• One recovery per host, not one per check
• Include the checks that recovered and the ones that did not
• Never hold back a critical that fires during the window

A check that simply stops reporting is treated as a problem, not as passing. The most dangerous state in monitoring is a green dashboard fed by nothing.

• Expected-interval per check, with a grace period
• Silence pages the owner of the host, not the owner of the check
• Distinguish an agent that lost its socket from a host that is off

Per tenant, with critical alerts able to break through. A rate limit without a break-through rule hides the outage that matters; quiet hours without one hide it at night.

• Quiet hours per client, in the client’s time zone
• A per-channel rate limit with a critical exception
• Six ways to be told: email, SMS, Slack, Microsoft Teams, Discord, webhooks, with time-delayed escalation

Suppression only works if every suppression is recorded with its reason. Then you can answer the question that actually comes up: why did this not page anyone? In Guardian Plus every withheld notification is written down with why. If your current tool cannot answer that question, its suppression is a liability.

• Pull the last 30 days of notifications. How many were recoveries? How many were repeats of the same host within ten minutes?
• Find the noisiest host. Is it flapping, or genuinely failing?
• List every check with no expected interval. Those are the ones that can go silent unnoticed.
• For each client, write down quiet hours and the one condition that should always break through.
• Pick three alerts from last week and ask: could I explain, from the tool, why each one paged? And for one that did not page, why not?
• Count the notification channels. If it is one, the on-call person has no escalation path.

Start a trial — a specialist installs the first agent with you — and within minutes it will have found what runs on the box, built the checks, and told you something you did not know. Start at https://get.guardianplus.ai/trial, or bring your noisiest host to a walkthrough at https://get.guardianplus.ai/demo.