Alerting Without Waking the Whole Team

Published 2026-03-24 by Faraz Rahimi

If every warning is urgent, nothing is. Page on stuck queues and impossible counts, not on routine zeros at 3am.

Alerting Without Waking the Whole Team

Tags: Operations, Alerting, Reliability, Python, unlisted

I page when the exception queue ages past a threshold during operating hours, when a job that always has rows returns none on a weekday, and when write error rates spike.

I do not page on Saturday night because a vendor did not send a file they never send on Saturday.

Quiet hours are a product decision

The warehouse has a clock. Alerts should too. A digest at open is better than a siren at 2am for a report nobody reads until 8.

Make the alert the next action

A link to the failed rows beats a Slack message that says "error." If I cannot click to the queue, I have only created anxiety.

Original post: https://farazrahimi.com/posts/alerting-without-waking-the-whole-team