alt Hacker NewsOK you design a new out of band notification system that somehow has better reliability than email but is also used frequently enough that people will notice when it stops working.
Replies
How about an audible alarm and a flashing light at the dangerous location?
This seems like a solved problem, e.g. for smoke detectors.
Phones are good. So are loud, annoying alarms as well as labeled, illuminated alert lights in the potentially-hazardous area.
The alarms and lights are easy to test, too. Make pressing the self-test button part of the nightly or weekly rounds.
The notification does not need to be out-of-band.
It can be very much in-band, like a fire alarm primarily also is.
Do not design anything. Buy an already designed and approved horn with a flashing light.
This one’s really simple. The current system is a bad one because if the email password changes or internet connectivity gets interrupted or the system sending the email dies (eg: power supply failure), the system fails because on any lack of signal it’s assumed that everything’s good. That’s a terrible design. Here’s a better one:
- External system polls the monitor and records levels.
- Local system reaches out to external system and fetches status.
- Someone has the responsibility to do rounds and record the status indicated by the local system.
External system down or indicates unacceptable levels, monitor unreachable, etc.? Local system shows a failure. Investigate. Local system down? Investigate.
You can even add redundancy by having a manager double check that the responsible employee is checking off the status plus the manager can do a daily sweep as a fallback. Throw something like PagerDuty into the mix and have multi-channel notifications on any failure that escalate over time.