Because Fluentd and rsyslog read from the journal, and the journal default settings are very low, journal entries can be lost because the journal cannot keep up with the logging rate from system services.

We recommend setting RateLimitInterval=1s and RateLimitBurst=10000 (or even higher if necessary) to prevent the journal from losing entries.

Scaling up systemd-journald

As you scale up your project, the default logging environment might need some adjustments.

For example, if you are missing logs, you might have to increase the rate limits for journald.

Procedure
  1. Update to systemd-219-22.el7.x86_64.

  2. Add the following to the /etc/systemd/journald.conf file:

    # Disable rate limiting
    RateLimitInterval=1s
    RateLimitBurst=10000
    Storage=volatile
    Compress=no
    MaxRetentionSec=30s
  3. Restart the services:

    $ systemctl restart systemd-journald.service
    $ systemctl restart rsyslog.service

    These settings account for the bursty nature of uploading in bulk.

After removing the rate limit, you might see increased CPU utilization on the system logging daemons as it processes any messages that would have previously been throttled.