alt Hacker NewsWhat always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
Replies
Tracing is the most general of them, and the most expensive unless you're careful with the implementation.
Trace spans are time-delimited units of "stuff that happened", with a tree relationship among the spans, and each span can have arbitrary tags (key/value pairs) and events (time/value).
From that, if you chose, you could derive metrics and logs. The trick is to start with tracing and to actually put it in your program, rather than trying to mostly-automatically tack it on later.
You can do that in Lisp, since you can arbitrarily redefine the wrapper to have such or other logic etc.
One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...
I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How would you get, say, Garbage Collector metrics from logs or traces? You cannot.
There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And metrics are something else altogether. Trying to somehow unify them would be a prime example of "wrong abstraction".
> “The next time something like this occurs again, please save me a trace.”
The building blocks for this exist. The observability platform must simply (haha) implement the pattern detectors and use them for sampling decisions.