alt Hacker NewsRelatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more common solution for distributed logging and tracing (yes it doesn’t solve the UI problem for those, but it does solve collecting your logs)
Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.
Replies
We're doing this with a few dozen GiBs of logs a day (rsylog -> central rsylog -> elasticsearch). It works reliably, but the config is an absolute nightmare, documentation is a mixed bag and troubleshooting often involves deep dives into the C code. We're planning to migrate to Alloy+Loki.
People don’t care about syslog. 98% of my colleagues haven’t heard of it.
Since this is an OTel-related submission, you could also use OTel collectors to collect and forward logs to a central OTel collector instance.
> yes it doesn’t solve the UI problem for those, but it does solve collecting your logs
I work for Netdata and over the last couple months, we've developed an external Netdata plugin that can ingest/index OTel logs [1]. The current implementation stores logs in systemd-compatible journal files and our visualization is effectively the same one someone would get when querying systemd journal logs [2]. i > Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.
I've not used rsyslogd specifically, but I don't see how you'd have any issues with the log volume you described.
[1] https://github.com/netdata/netdata/tree/master/src/crates/ne...
[2] https://learn.netdata.cloud/docs/logs/systemd-journal-logs/s...