A dental practice closes at 18:00 and its managing partner does not want a text about a printer at 23:00. The MSP sets quiet hours for that client, 18:00 to 07:00. Three weeks later the practice server loses a drive at 01:30. The array degrades, the check goes critical, and the notification sits in the quiet queue with the printer alerts until 07:00. By then the second drive has gone too. Quiet hours did exactly what they were told. The rule was missing its one exception, and the client found the outage first.
Because a suppression rule that cannot tell a printer from a failing array hides the outage that matters at precisely the hour nobody is looking. Quiet hours without a break-through rule hide it at night; a rate limit without one hides it during a storm. Both are useful, and both are a way of not knowing unless they carry an exception.
The alternative, no quiet hours at all, is not safer. It teaches the on-call person to sleep through the phone. The mechanism is fine; it needs four parts, written down per client.
One rule per client, with four parts: the client, the window, the time zone and the single condition that breaks through. Per client rather than global, because the dental practice and the freight company that runs a night shift do not share a night. In the client’s time zone rather than yours, because an MSP with clients across two zones will otherwise suppress one client’s business hours or expose another’s night. The window is the client’s own answer to “when do you not want to hear from us?” The exception is the answer to “what would you ring us about at 01:30?”
Critical, and the work is deciding with the client which checks are allowed to be critical at night. A degraded array is. A printer is not, however loudly it says so. The playbook’s checklist puts it as a single line: for each client, write down quiet hours and the one condition that should always break through. If you cannot write that condition down, you do not have a quiet-hours rule; you have a blindfold with a schedule.
A rate limit is quiet hours for a storm rather than a clock: per channel, it caps how many notices go out in a window. It needs the same exception. A per-channel rate limit with a critical exception lets a flapping host stop filling your inbox without letting a genuine critical wait in the queue behind it. The recovery-burst window follows the same principle: recoveries are held for a short window and sent as one message per host, and a critical that fires during the window is never held. One rule, three places.
There are six ways to be told: email, SMS, Slack, Microsoft Teams, Discord and webhooks, with time-delayed escalation between them. A ladder is the order you climb them and the delay between rungs. A shape that works for most clients: the first notice goes to the channel the team already watches in working hours, Slack or Teams, and to email for the record. If it is still there after the delay you set, SMS to the person on call. After a second delay, a webhook, which is how the alert becomes a ticket or reaches whatever on-call tool you already pay for. Quiet hours hold the lower rungs; the break-through condition goes straight to the rung that wakes someone.
The checklist’s last line is the test: count the channels. If the answer is one, the on-call person has no escalation path, and an alert that is missed once is missed for good.
Into the record, each with its reason. Every withheld notification is written down with why it was withheld: quiet hours, a rate limit, flap damping, a held recovery. That is what makes the rule safe to switch on. When the managing partner asks on Monday why nobody told her about the printer on Saturday night, the answer is in the tool: held, 23:14, quiet hours, delivered 07:00. When she asks why she was told about the array at 01:31, the answer is there too: critical, broke through, SMS. The question and its opposite are the subject of why did that page anyone?
Without the exception: the outage the client found, and in the story above a second drive. With quiet hours but no record: a reconciliation every month or so, when a client asks why they were not told and someone reads a notification log for an afternoon with nothing to show for it. Without quiet hours at all: printer alerts at 23:00, a few minutes per night per technician, and the habit of ignoring the phone that eventually costs the array anyway.
Flap damping, recovery bursts, silence detection and quiet hours, with the checklist to audit your own alerting in an afternoon.