alt Hacker NewsAs someone who has been on that side of things for 10 years, can we stop building really complicated stuff that requires this pile of shit and dedicated staff to run it.
The whole otel, logging and metrics thing is based around "record everything" and dig through the pile of shit looking for things that might be helpful. Fine until you have 50TB of logs/traces online which is our problem. Oh that and the 50 gig prometheus instance. 99.99999% of all information we record is entirely and utterly useless and costs us money
Make simpler things, log only exceptional things, use the rule of silence.
Replies
How do you implement that? Would you like to explain, I am very interested!
How do you decide what is exceptional when you have only silence to compare to?
I’m having trouble understanding your take here. Why is recording logs with embedded metrics at full resolution a problem, assuming you can afford to?