OTel isn’t going well

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away.

It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.

I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.

I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.

What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.

For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.

I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.

OpenTelemtry is the perfect example of an overengineered mess.

While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped.

A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get.

I think OpenTelemetry turned metrics into a farce. In many situations even self-rolled telemetry works better even with the added stuff. The annoying thing is that OpenTelemtry is that big standard now one kind of has to to add compatibility. So please, if you write software, make sure you don't lock yourself into OTel.

I've never found instrumentation to be a huge issue. Sure it takes more effort but you get a lot more value once you understand _business_ events.

Sounds a lot like K8s. It's not a framework you use, it's a framework to build a framework on top of.

I wish the observability vendors would move to using it under the covers so it's easier to mix and match.

I wish the otel support wasn't super buggy in most of the frameworks and backends.

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today:

1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.

2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.

3. Serverless runtimes pay a heavy penalty for cold starts with OTel.

4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.

5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.

6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?

I've found their django instrumentation to be kinda useless for larger apps.

The only choices you get is full auto instrumentation, which breaks most non-trivial apps, or zero assistance/documentation.

There is no in-between where I can inject the functionality required in a way that is compatible with the application.

I disagree. I'm an observability geek, and OTel is... fine.

It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:

1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.

2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).

The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage.

At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!

Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.

(TFA author is not me.)

It is crazy to me how often people don't grok how to design software well.

1. The worst thing you can do is try to stuff too many things into one specification. So you want an API? That's great. What's that? You want a rigid set of types so that any tiny changes over time aren't compatible? You want to try to define every conceivable use case as a new call? You want to combine multiple elements from different domains into one flat set of functions? You don't have any hierarchy or inheritance? You don't support extensions?

2. The second-worst thing you can do is to force a whole lot of different people to go through a single standards body. So you want to support a thousand different 3rd party components. What's that? You want to require everyone get their adapter approved by one group? And there's only one supported adapter per 3rd party component?

If you're trying to feed an entire city, it's logistically incredibly difficult to try to do it all yourself. If instead you just define where food can be dropped off or picked up, and ask volunteers to bring their own food there whenever they can/want, now you don't have a logistical nightmare on your hands anymore. The tech alternative? Add support for "plugins", make the plugin interface incredibly loose/backwards-compatible/layered, and invite people to publish their own plugins. If you under-engineer it, it actually works better.

I wish otel was never there. It is badly designed abstraction and due to otel the code gets very very messy and bad.

[dead]