Overview

As an OpenShift administrator, you may want to view the logs from all containers in one user interface. The currently supported method for aggregating container logs in OpenShift Enterprise is using a centralized file system. Additional supported methods are planned for inclusion in future releases.

As packaging improvements are made, these instructions will be simplified.

Using a Centralized File System

This method reads all container logs and forwards them to a central server for storage on the file system.

Installing fluentd (td-agent) on Nodes

Perform the following steps on each node to install and configure fluentd (td-agent):

  1. Run the following commands:

    # export RPM=td-agent-2.2.0-0.x86_64.rpm
    # curl https://packages.treasuredata.com/2/redhat/7/x86_64/$RPM \
        -o /tmp/$RPM
    # yum localinstall /tmp/$RPM
    # /opt/td-agent/embedded/bin/gem install fluent-plugin-kubernetes
    # mkdir -p /etc/td-agent/config.d
    # chown td-agent:td-agent /etc/td-agent/config.d
  2. Create a directory to house the logs:

    # mkdir -p /var/log/td-agent/tmp
    # chown td-agent:td-agent /var/log/td-agent/tmp

    To allow td-agent access to the containers logs, create the /etc/sysconfig/td-agent file and ensure it contains the following:

    DAEMON_ARGS=
    TD_AGENT_ARGS="/usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --use-v1-config"
  3. Add the following line to the /etc/td-agent/td-agent.conf file:

      @include config.d/*.conf
  4. Create the /etc/td-agent/config.d/kubernetes.conf file with the following contents:

        <source>
          type tail
          path /var/lib/docker/containers/*/*-json.log
          pos_file /var/log/td-agent/tmp/fluentd-docker.pos
          time_format %Y-%m-%dT%H:%M:%S
          tag docker.*
          format json
          read_from_head true
        </source>
    
        <match docker.var.lib.docker.containers.*.*.log>
          type kubernetes
          container_id ${tag_parts[5]}
          tag docker.${name}
        </match>
    
        <match kubernetes>
          type copy
          <store>
            type forward
            send_timeout 60s
            recover_wait 10s
            heartbeat_interval 1s
            phi_threshold 16
            hard_timeout 60s
            log_level trace
            require_ack_response true
            heartbeat_type tcp
            <server>
              name logging_name (1)
              host host_name (2)
              port 24224
              weight 60
            </server>
    
            <secondary>
              type file
              path /var/log/td-agent/forward-failed
            </secondary>
          </store>
    
          <store>
            type file
            path /var/log/td-agent/containers.log
            time_slice_format %Y%m%d
            time_slice_wait 10m
            time_format %Y%m%dT%H%M%S%z
            compress gzip
            utc
          </store>
        </match>
    1 The name for the master that will be used during logging.
    2 The IP or a DNS resolvable name used to access the master.
  5. Enable fluentd:

# systemctl enable td-agent
# systemctl start td-agent

Any errors are logged in the /var/log/td-agent/td-agent.log file.

Optional Method to Verify Working Nodes

You can optionally set up the master to be the aggregator to test and verify that the nodes are working properly.

  1. Install fluentd (td-agent) on the master:

    # export RPM=td-agent-2.2.0-0.x86_64.rpm
    # curl https://packages.treasuredata.com/2/redhat/7/x86_64/$RPM \
        -o /tmp/$RPM
    # yum localinstall /tmp/$RPM
    # mkdir -p /etc/td-agent/config.d
    # chown td-agent:td-agent /etc/td-agent/config.d
  2. Ensure port 24224 is open on the master’s firewall to allow the nodes access.

  3. Configure fluentd to aggregate container logs by adding the following line to the /etc/td-agent/td-agent.conf file:

      @include config.d/*.conf
  4. Create the /etc/td-agent/config.d/kubernetes.conf file with the following contents:

        <match kubernetes.**>
            type file
            path /var/log/td-agent/containers.log
            time_slice_format %Y%m%d
            time_slice_wait 10m
            time_format %Y%m%dT%H%M%S%z
            compress gzip
            utc
        </match>
  5. Enable fluentd:

    # systemctl enable td-agent
    # systemctl start td-agent

    Any errors are logged in the /var/log/td-agent/td-agent.log file.

You should now find all the containers' logs available on the master in the /var/log/td-agent/containers.log file.