OpenShift Container Platform cluster logging allows you to configure the Fluentd out_forward plug-in to send logs to external devices.

You can use the log forwarding feature, which can be easier to configure than the plugins. Note that the log forwarding feature is currently in Technology Preview.

Changes introduced by the new log forward feature modified the support for out_forward starting with the OpenShift Container Platform 4.3 release. In OpenShift Container Platform 4.3, you create a ConfigMap to configure out_forward, as described below, instead of editing the secure-forward.conf section in the fluentd ConfigMap. You can add any certificates required by your external devices to a secret, called secure-forward, which is mounted to the Fluentd Pods.

When you update to OpenShift Container Platform 4.3, any existing modifications to the secure-forward.conf section of the fluentd ConfigMap are removed. You can copy your current secure-forward.conf section before updating to use when creating the secure-forward ConfigMap.

Configuring the Fluentd out_forward plug-in to send logs to an external log aggregator

You can configure Fluentd to send a copy of its logs to an external log aggregator, and not the default Elasticsearch, using the forward plug-in. From there, you can further process log records after the locally hosted Fluentd has processed them.

In this documentation, the OpenShift Container Platform cluster is called the sender and the external aggregator is called the receiver.

This legacy out_forward method is deprecated and will be removed in a future release.

The forward plug-ins are supported by Fluentd only. The in_forward plug-in implements the server side (receiver), and out_forward implements the client side (sender).

To configure the out_forward plug-in, create a ConfigMap called secure-forward in the openshift-logging namespace. On the receiver, configure the Fluentd secure-forward.conf file. For more information on using the in_forward plug-in, see the Fluentd documentation.

Changes introduced by the new log forward feature modified the support for out_forward starting with the OpenShift Container Platform 4.3 release. In OpenShift Container Platform 4.3, you create a ConfigMap, as described below, to configure out_forward. Any updates to the secure-forward.conf section of the Fluentd ConfigMap are removed. Before upgrading cluster logging, you can copy your current secure-forward.conf section and use the copied data when you create the secure-forward ConfigMap.

Additionally, you can add any certificates required by your configuration to a secret named secure-forward that will be mounted to the Fluentd Pods.

Sample secure-forward.conf
<store>
  @type forward
  <security>
    self_hostname ${hostname} # ${hostname} is a placeholder.
    shared_key "fluent-receiver"
  </security>
  transport tls
  tls_verify_hostname false           # Set false to ignore server cert hostname.

  tls_cert_path '/etc/ocp-forward/ca-bundle.crt'
  <buffer>
    @type file
    path '/var/lib/fluentd/secureforwardlegacy'
    queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }"
    chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }"
    flush_interval "#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}"
    flush_at_shutdown "#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}"
    flush_thread_count "#{ENV['FLUSH_THREAD_COUNT'] || 2}"
    retry_max_interval "#{ENV['FORWARD_RETRY_WAIT'] || '300'}"
    retry_forever true
    # the systemd journald 0.0.8 input plugin will just throw away records if the buffer
    # queue limit is hit - 'block' will halt further reads and keep retrying to flush the
    # buffer to the remote - default is 'exception' because in_tail handles that case
    overflow_action "#{ENV['BUFFER_QUEUE_FULL_ACTION'] || 'exception'}"
  </buffer>
  <server>
    host fluent-receiver.openshift-logging.svc  # or IP
    port 24224
  </server>
</store>
Sample secure-forward ConfigMap based on the configuration
apiVersion: v1
data:
 secure-forward.conf: "<store>
     \ @type forward
     \ <security>
     \   self_hostname ${hostname} # ${hostname} is a placeholder.
     \   shared_key \"fluent-receiver\"
     \ </security>
     \ transport tls
     \ tls_verify_hostname false           # Set false to ignore server cert hostname.
     \ tls_cert_path '/etc/ocp-forward/ca-bundle.crt'
     \ <buffer>
     \   @type file
     \   path '/var/lib/fluentd/secureforwardlegacy'
     \   queued_chunks_limit_size \"#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }\"
     \   chunk_limit_size \"#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }\"
     \   flush_interval \"#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}\"
     \   flush_at_shutdown \"#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}\"
     \   flush_thread_count \"#{ENV['FLUSH_THREAD_COUNT'] || 2}\"
     \   retry_max_interval \"#{ENV['FORWARD_RETRY_WAIT'] || '300'}\"
     \   retry_forever true
     \   # the systemd journald 0.0.8 input plugin will just throw away records if the buffer
     \   # queue limit is hit - 'block' will halt further reads and keep retrying to flush the
     \   # buffer to the remote - default is 'exception' because in_tail handles that case
     \   overflow_action \"#{ENV['BUFFER_QUEUE_FULL_ACTION'] || 'exception'}\"
     \ </buffer>
     \ <server>
     \   host fluent-receiver.openshift-logging.svc  # or IP
     \   port 24224
     \ </server>
     </store>"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-01-15T18:56:04Z"
  name: secure-forward
  namespace: openshift-logging
  resourceVersion: "19148"
  selfLink: /api/v1/namespaces/openshift-logging/configmaps/secure-forward
  uid: 6fd83202-93ab-d851b1d0f3e8
Procedure

To configure the out_forward plug-in:

  1. Create a configuration file named secure-forward.conf for the out_forward parameters:

    1. Configure the secrets and TLS information:

       <store>
        @type forward
      
        self_hostname ${hostname} (1)
        shared_key <SECRET_STRING> (2)
      
        transport tls (3)
      
        tls_verify_hostname true (4)
        tls_cert_path <path_to_file> (5)
      1 Specify the default value of the auto-generated certificate common name (CN).
      2 Enter the Shared key between nodes
      3 Specify true to enable TLS validation.
      4 Set to true to verify the server cert hostname. Set false to ignore server cert hostname.
      5 Specify the path to private CA certificate file as /etc/ocp-forward/ca_cert.pem.

      To use mTLS, see the Fluentd documentation for information about client certificate and key parameters and other settings.

    2. Configure the name, host, and port for your external Fluentd server:

        <server>
          name (1)
          host (2)
          hostlabel (5)
          port (3)
        </server>
        <server> (4)
          name
          host
        </server>
      1 Optionally, enter a name for this receiver.
      2 Specify the host name or IP of the receiver.
      3 Specify the port of the receiver.
      4 Optionally, add additional receivers. If you specify two or more receivers, out_secure_forward uses these server nodes in a round-robin order.

      For example:

        <server>
          name externalserver1
          host 192.168.1.1
          hostlabel externalserver1.example.com
          port 24224
        </server>
        <server>
          name externalserver2
          host externalserver2.example.com
          port 24224
        </server>
        </store>
  2. Create a ConfigMap named secure-forward in the openshift-logging namespace from the configuration file:

    $ oc create configmap secure-forward --from-file=secure-forward.conf -n openshift-logging
  3. Optionally, import any secrets required for the receiver:

    $ oc create secret generic secure-forward --from-file=<arbitrary-name-of-key1>=cert_file_from_fluentd_receiver --from-literal=shared_key=value_from_fluentd_receiver

    For example:

    $ oc create secret generic secure-forward --from-file=ca-bundle.crt=ca-for-fluentd-receiver/ca.crt --from-literal=shared_key=fluentd-receiver
  4. Refresh the fluentd Pod to apply the secure-forward secret and secure-forward ConfigMap:

    $ oc delete pod --selector logging-infra=fluentd
  5. Configure the secure-forward.conf file on the receiver to accept messages securely from OpenShift Container Platform.

    When configuring the recevier, it must be able to accept messages securely from OpenShift Container Platform.

You can find further explanation of how to set up the inforward plug-in and the out_forward plug-in.