logoalt Hacker News

dncornholiotoday at 2:16 PM1 replyview on HN

> If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well.

Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.


Replies

hnlmorgtoday at 2:33 PM

No, monitoring for HTTP response code is a subset of observability and not one that generally gives you the best insights into which subsystems are misbehaving nor why.

There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the observability platform (think like running a cron-job) and thus you're not tied to user activity to see when problems arise.

There are other metrics outside of HTTP response codes too. Think like free RAM, CPU usage, disk space, etc. This is just naming some obvious ones because these types of metrics are generally bespoke to the type of application your monitoring. And with these types of monitors, you'd not just have an alert when things have failed, but ideally have alerts when an irregular trend is showing that things are likely to fail too. This latter type of monitors helps you get ahead of the problem before it become customer facing.

Then you have more traditional stuff like logs. This will also be bespoke to the application. But you'd expect errors in logs to get surfaced quickly. Assuming Github have good hygiene in what's being logged.

Tie that up with APMs, RUM, and other goodies like that and you'll have diagnostics to investigate issues when they appear.

(this is just a super high level view of observability too)

show 1 reply