alt Hacker NewsLogging, metrics and traces are not free, especially if you turn them on at every requests.
Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.
Replies
A very small % of startups gets anywhere near that traffic so why give them angst? Most people can just do this without any issues and learn from it and a tiny fraction shouldn't.
Metrics are usually minimal overheard. Traces need to be sampled. Logs need to be sampled at error/critical levels. You also need to be able to dynamically change sampling and log levels.
100% traces are a mess. I didn’t see where he setup sampling.
You have to do the tracing anyway if you are going to sample based on criteria that isn't available at the beginning of the trace (like an error that occurs later in the request) and tail sample. You can head sample of course, but that's going to be the most coarse sampling you can do and you can't sample based on anything but the initial conditions of the trace.
What we have started doing is still tracing every unit of work, but deciding at the root span the level of instrumentation fidelity we want for the trace based on the initial conditions. Spans are still generated in the lifecycle of the trace, but we discard them at the processor level (before they are batched and sent to the collector) unless they have errors on them or the trace has been marked as "full fidelity".
I am relatively new to the topic. In the sample code of the OP there is no logging right? It's metrics and traces but no logging.
How is logging in OTel?
> Tracing every http 200 at 10k req/sec is not something you should be doing
You don't know if a request is HTTP 200 or HTTP 500 until it ends, so you have to at least collect trace data for every request as it executes. You can decide whether or not to emit trace data for a request based on its ultimate response code, but emission is gonna be out-of-band of the request lifecycle, and (in any reasonable implementation) amortized such that you really shouldn't need to care about sampling based on outcome. That is, the cost of collection is >> the cost of emission.
If your tracing system can't handle 100% of your traffic, that's a problem in that system; it's definitely not any kind of universal truth... !