It is 03:12 and the technician on call has just watched a database server reboot for a scheduled kernel update. Every check on it went critical inside the same minute: the disk, the load average, MySQL replication, the HTTP endpoint, the backup mount. Forty emails. Then the host came back and every check sent its own recovery notice, so forty more. Somewhere in that pile is one message from a different client, whose file server has gone quiet for good, and nobody reads it until the office opens and the client rings. That is what alert noise costs. It is not a tuning problem. It is a design problem with four known fixes, and a harder question at the end.
Because most monitoring treats every check as its own conversation with you. Forty checks on one host means forty state changes on the way down and forty on the way back, and each one is reported as if the others did not exist. The tool is not wrong about any of them. It is wrong about what you needed to know, which was one sentence: this host rebooted and came back in four minutes.
The fix is not to lower thresholds or delete checks. Deleting checks makes an estate quieter and blinder at the same time, and the check you delete is the one that goes silent unnoticed next year. The fix is to change what a notification is: a report about a host rather than a report about a check, damped, held and timed. Four mechanisms do that, and Guardian RMM ships with all four on by default.
A host bouncing between up and down is damped, then announced once with its real outage duration. The rule counts transitions in a window rather than states, announces when the host settles, and records the total downtime in that single notice. Without it, every transition is a page, and the person on call learns to ignore the host that matters most.
The part people miss is what happens to the transitions you did not announce. They are kept. The post-mortem still has all fourteen link drops with their timestamps; the on-call phone just did not buzz fourteen times. Damping changes who reads the detail and when, not whether it exists.
When a box comes back, its checks do not each need to mail you their own recovery notice. Hold recoveries for a short window and send one message per host, listing the checks that recovered and the ones that did not. A recovery burst is good news arriving forty times; one message that says what came back and what is still failing is the same news arriving once, with the useful part at the top.
There is one exception and it is not optional: never hold back a critical that fires during the window. Good news can wait a minute. Bad news that arrives while the good news is being collected must not.
A check that simply stops reporting is treated as a problem, not as passing. A green dashboard fed by nothing is the state to fear in monitoring, because it looks exactly like a healthy estate right up until the client rings. Every check needs an expected interval with a grace period, and silence should page the owner of the host, not the owner of the check.
The design also has to distinguish an agent that lost its socket from a host that is off. The Guardian agent keeps one outbound socket open with a ten-second heartbeat, reconnects with exponential backoff and detects sleep and wake, so a laptop that closed its lid and a server that lost power look different from the platform’s side. That difference is what stops silence detection paging you about every laptop at 18:05. We wrote about the socket itself in the agent never accepts a connection.
Quiet hours are set per client, in the client’s time zone, with critical alerts able to break through. A rate limit without a break-through rule hides the outage that matters; quiet hours without one hide it at night. Both are suppression, and suppression without an exception is a way of not knowing.
There are six ways to be told (email, SMS, Slack, Microsoft Teams, Discord and webhooks) with time-delayed escalation between them. If you count your channels and the answer is one, the on-call person has no escalation path, and quiet hours are the least of the problem. The rule itself, and where everything you suppressed goes, has its own post: quiet hours that still let critical through.
Suppression only works if every suppression is recorded with its reason. Then you can answer the question that actually comes up, which is never “why did that page me?” but “why did this not page anyone?” In Guardian RMM every withheld notification is written down with why: flap damping, a held recovery, quiet hours, a rate limit. If your current tool suppresses and cannot show you the reason afterwards, its suppression is a liability rather than a feature. You have traded forty emails for a silence you cannot explain.
Forty emails per reboot, each opened and dismissed in a few seconds, is a small number until it is a patch window with a dozen hosts and it is the first half hour of a technician’s day, every patch day. The reconciliation comes once a month, when a client asks why they were not told about something and someone spends an afternoon reading the notification log for a page that should have gone. And the outage the client found first is the one that costs the contract.
None of this needs our product. Run the list below against whatever you monitor with today, and the answers will tell you which of the four mechanisms you are missing.
The four mechanisms, the harder question, and the checklist as a PDF you can run against any monitoring tool.