OpenShift Container Platform uses Fluentd to collect operations and application logs from your cluster which OpenShift Container Platform enriches with Kubernetes Pod and Namespace metadata.

You can configure log rotation, log location, use an external log aggregator, and make other configurations.

Procedures in this topic require your cluster to be in an unmanaged state. For more information, see Changing the cluster logging management state.

Viewing Fluentd pods

You can use the oc get pods -o wide command to see the nodes where the Fluentd pod are deployed.

Procedure

Run the following command in the openshift-logging project:

$ oc get pods -o wide | grep fluentd

NAME                         READY     STATUS    RESTARTS   AGE     IP            NODE                           NOMINATED NODE
fluentd-5mr28                1/1       Running   0          4m56s   10.129.2.12   ip-10-0-164-233.ec2.internal   <none>
fluentd-cnc4c                1/1       Running   0          4m56s   10.128.2.13   ip-10-0-155-142.ec2.internal   <none>
fluentd-nlp8z                1/1       Running   0          4m56s   10.131.0.13   ip-10-0-138-77.ec2.internal    <none>
fluentd-rknlk                1/1       Running   0          4m56s   10.128.0.33   ip-10-0-128-130.ec2.internal   <none>
fluentd-rsm49                1/1       Running   0          4m56s   10.129.0.37   ip-10-0-163-191.ec2.internal   <none>
fluentd-wjt8s                1/1       Running   0          4m56s   10.130.0.42   ip-10-0-156-251.ec2.internal   <none>

Viewing Fluentd logs

How you view logs depends upon the LOGGING_FILE_PATH setting.

  • If LOGGING_FILE_PATH points to a file, the default, use the logs utility, from the project, where the pod is located, to print out the contents of Fluentd log files:

    $ oc exec <any-fluentd-pod> -- logs (1)
    1 Specify the name of a Fluentd pod. Note the space before logs.

    For example:

    $ oc exec fluentd-ht42r -n openshift-logging -- logs

    To view the current setting:

    oc -n openshift-logging set env daemonset/fluentd --list | grep LOGGING_FILE_PATH
  • If you are using LOGGING_FILE_PATH=console, Fluentd writes logs to stdout/stderr`. You can retrieve the logs with the oc logs [-f] <pod_name> command, where the -f is optional, from the project where the pod is located.

    $ oc logs -f <any-fluentd-pod> (1)
    1 Specify the name of a Fluentd pod. Use the -f option to follow what is being written into the logs.

    For example

    $ oc logs -f fluentd-ht42r -n openshift-logging

    The contents of log files are printed out, starting with the oldest log.

Configure Fluentd CPU and memory limits

Each component specification allows for adjustments to both the CPU and memory limits.

Procedure
  1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

    $ oc edit ClusterLogging instance
    $ oc edit ClusterLogging instance
    
    apiVersion: "logging.openshift.io/v1"
    kind: "ClusterLogging"
    metadata:
      name: "instance"
    
    ....
    
    spec:
      collection:
        logs:
          fluentd:
            resources:
              limits: (1)
                cpu: 250m
                memory: 1Gi
              requests:
                cpu: 250m
                memory: 1Gi
    1 Specify the CPU and memory limits as needed. The values shown are the default values.

Configuring Fluentd log location

Fluentd writes logs to a specified file or to the default location, /var/log/fluentd/fluentd.log, based on the LOGGING_FILE_PATH environment variable.

Prerequisite

Set cluster logging to the unmanaged state.

Procedure

To set the output location for the Fluentd logs:

  1. Edit the LOGGING_FILE_PATH parameter in the fluentd daemonset. You can specify a particular file or console:

    spec:
      template:
        spec:
          containers:
              env:
                - name: LOGGING_FILE_PATH
                  value: console (1)
    
    LOGGING_FILE_PATH= (2)
    1 Specify the log output method:
    • use console to use the Fluentd default location. Retrieve the logs with the oc logs [-f] <pod_name> command.

    • use <path-to-log/fluentd.log> to sends the log output to the specified file. Retrieve the logs with the `oc exec <pod_name> — logs command. This is the default setting.

      Or, use the CLI:

      oc -n openshift-logging set env daemonset/fluentd LOGGING_FILE_PATH=console

Configuring Fluentd to send logs to an external log aggregator

You can configure Fluentd to send a copy of its logs to an external log aggregator, and not the default Elasticsearch, using the secure-forward plug-in. From there, you can further process log records after the locally hosted Fluentd has processed them.

The logging deployment provides a secure-forward.conf section in the Fluentd configmap for configuring the external aggregator:

  1. Prerequisite

Set cluster logging to the unmanaged state.

Procedure

To send a copy of Fluentd logs to an external log aggregator:

  1. Edit the secure-forward.conf section of the Fluentd configuration map:

    Sample secure-forward.conf section
    $ oc edit configmap/fluentd -n openshift-logging
    
    <store>
      @type forward
      <server> (1)
        name externalserver1
        host 192.168.1.1
        port 24224
      </server>
      <server> (1)
        name externalserver2
        host 192.168.1.2
        port 24224
      </server>
    </store>
    1 Enter the name, host, and port for your external Fluentd server.
  2. Add certificates to be used in secure-forward.conf to the existing secret that is mounted on the Fluentd pods. The your_ca_cert and your_private_key values must match what is specified in secure-forward.conf in configmap/logging-fluentd:

    $ oc patch secrets/fluentd --type=json \
      --patch "[{'op':'add','path':'/data/your_ca_cert','value':'$(base64 /path/to/your_ca_cert.pem)'}]"
    $ oc patch secrets/fluentd --type=json \
      --patch "[{'op':'add','path':'/data/your_private_key','value':'$(base64 /path/to/your_private_key.pem)'}]"

    Replace your_private_key with a generic name. This is a link to the JSON path, not a path on your host system.

    When configuring the external aggregator, it must be able to accept messages securely from Fluentd.

    • If using Fluentd 1.0 or later, configure the built-in in_forward plug-in with the appropriate security parameters.

      In Fluentd 1.0 and later, in_forward implements the server (receiving) side, and out_forward implements the client (sending) side.

      For Fluentd versions 1.0 or higher, you can find further explanation of how to set up the inforward plugin and the out_forward plugin.

    • If using Fluentd 0.12 or earlier, you must have the fluent-plugin-secure-forward plug-in installed and make use of the input plug-in it provides. In Fluentd 0.12, the same fluent-plugin-secure-forward plugin implements both the client (sending) side and the server (receiving) side.

      For Fluentd 0.12 you can find further explanation of fluent-plugin-secure-forward plug-in in fluent-plugin-secure-forward repository.

      The following is an example of a in_forward configuration for Fluentd 0.12:

      secure-forward.conf: |
        # <store>
        # @type secure_forward
      
        # self_hostname ${hostname}
        # shared_key <SECRET_STRING>
      
        # secure yes
        # enable_strict_verification yes
      
        # ca_cert_path /etc/fluent/keys/your_ca_cert
        # ca_private_key_path /etc/fluent/keys/your_private_key
          # for private CA secret key
        # ca_private_key_passphrase passphrase
      
        <server>
          host server.fqdn.example.com  # or IP
          # port 24284
        </server>
        # <server>
          # ip address to connect
        #   host 203.0.113.8
          # specify hostlabel for FQDN verification if ipaddress is used for host
        #   hostlabel server.fqdn.example.com
        # </server>
        # </store>

Throttling Fluentd logs

For projects that are especially verbose, an administrator can throttle down the rate at which the logs are read in by Fluentd before being processed. By throttling, you deliberately slow down the rate at which you are reading logs, so Kibana might take longer to display records.

Throttling can contribute to log aggregation falling behind for the configured projects; log entries can be lost if a pod is deleted before Fluentd catches up.

Throttling does not work when using the systemd journal as the log source. The throttling implementation depends on being able to throttle the reading of the individual log files for each project. When reading from the journal, there is only a single log source, no log files, so no file-based throttling is available. There is not a method of restricting the log entries that are read into the Fluentd process.

Prerequisite

Set cluster logging to the unmanaged state.

Procedure
  1. To configure Fluentd to restrict specific projects, edit the throttle configuration in the Fluentd ConfigMap after deployment:

    $ oc edit configmap/fluentd

    The format of the throttle-config.yaml key is a YAML file that contains project names and the desired rate at which logs are read in on each node. The default is 1000 lines at a time per node. For example:

throttle-config.yaml: |
  - opensift-logging:
      read_lines_limit: 10
  - .operations:
      read_lines_limit: 100

Configuring Fluentd JSON parsing

You can configure Fluentd to inspect each log message to determine if the message is in JSON format and merge the message into the JSON payload document posted to Elasticsearch. This feature is disabled by default.

You can enable or disable this feature by editing the MERGE_JSON_LOG environment variable in the fluentd daemonset.

Enabling this feature comes with risks, including:

  • Possible log loss due to Elasticsearch rejecting documents due to inconsistent type mappings.

  • Potential buffer storage leak caused by rejected message cycling.

  • Overwrite of data for field with same names.

The features in this topic should be used by only experienced Fluentd and Elasticsearch users.

Prerequisites

Set cluster logging to the unmanaged state.

Procedure

Use the following command to enable this feature:

oc set env ds/fluentd MERGE_JSON_LOG=true (1)
1 Set this to false to disable this feature or true to enable this feature.

Setting MERGE_JSON_LOG and CDM_UNDEFINED_TO_STRING

If you set the MERGE_JSON_LOG and CDM_UNDEFINED_TO_STRING enviroment variables to true, you might receive an Elasticsearch 400 error. The error occurs because when`MERGE_JSON_LOG=true`, Fluentd adds fields with data types other than string. When you set CDM_UNDEFINED_TO_STRING=true, Fluentd attempts to add those fields as a string value resulting in the Elasticsearch 400 error. The error clears when the indices roll over for the next day.

When Fluentd rolls over the indices for the next day’s logs, it will create a brand new index. The field definitions are updated and you will not get the 400 error.

Records that have hard errors, such as schema violations, corrupted data, and so forth, cannot be retried. Fluent sends the records for error handling. If you add a <label @ERROR> section to your Fluentd config, as the last <label>, you can handle these records as needed.

For example:

data:
  fluent.conf:

....

    <label @ERROR>
      <match **>
        @type file
        path /var/log/fluent/dlq
        time_slice_format %Y%m%d
        time_slice_wait 10m
        time_format %Y%m%dT%H%M%S%z
        compress gzip
      </match>
    </label>

This section writes error records to the Elasticsearch dead letter queue (DLQ) file. See the fluentd documentation for more information about the file output.

Then you can edit the file to clean up the records manually, edit the file to use with the Elasticsearch /_bulk index API and use cURL to add those records. For more information on Elasticsearch Bulk API, see the Elasticsearch documentation.

Configuring Fluentd using environment variables

You can use environment variables to modify your Fluentd configuration.

Prerequisite

Set cluster logging to the unmanaged state.

Procedure

Set any of the Fluentd environment variables as needed:

oc set env ds/fluentd <env-var>=<value>

For example:

oc set env ds/fluentd LOGGING_FILE_AGE=30