>Observability is a lot like imaging or patching: a necessary process to be sure, but do you really need a Cadillac Escalade (New Relic/Datadog/etc) to go to the grocery store when a Honda Accord (self-hosted Grafana + OTel) will do the same job more efficiently for less money?
The way that I've seen it play out is something like this:
1. We should self host something like Grafana and otel.
2. Oh no, the teams don't want to host individual instances of that, we should centralize it!
(2b - optional, but common, Random team gets saddled with this job)
3. Oh no, the centralized team is struggling with scaling issues and the service isn't very reliable. We should outsource it for 10x the cost!
This will happen even if they have a really nice set of deployment infrastructure and patterns that could have allowed them to host observability at the team level. It turns out, most teams really don't need the Escalade, they just need some basic graphs and alerts.Self hosting needs to be more common within organizations.
Another variant of step 2: some individual with a little bit of political Capital sees something new and shiny and figures out how to be the first project internally to use influx, for example, over Prometheus... And now you have a patchwork of dashboards, each broken in their own unique way...