Chapter 10. Forward logs to S3-compatible object storage

Procedure

1. Create a Secret YAML file that uses the aws_access_key_id and aws_secret_access_key fields to specify your base64-encoded AWS credentials. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: aws-secret
     namespace: <namespace_name>
   data:
     aws_access_key_id: QUtJQUlPU0ZPRE5ON0VYQU1QTEUK
     aws_secret_access_key: d0phbHJYVXRuRkVNSS9LN01ERU5HL2JQeFJmaUNZRVhBTVBMRUtFWQo=

   namespace

   Specify the namespace where your ClusterLogForwarder CR is deployed. The secret must be in the same namespace as the ClusterLogForwarder CR.

2. Create the secret. For example:

   $ oc apply -f aws-secret.yaml

3. Create or edit a YAML file that defines the ClusterLogForwarder CR object and specify the secret name. For example:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
     - name: s3bucket
       type: s3
       s3:
         authentication:
           type: awsAccessKey
           awsAccessKey:
             keyId:
               key: aws_access_key_id
               secretName: aws-secret
             keySecret:
               key: aws_secret_access_key
               secretName: aws-secret
         bucket: <bucket_name>
         keyPrefix: 's3{.log_type||"missing"}'
         region: us-east-1
         url: 'https://s3.us-east-1.amazonaws.com'
     pipelines:
     - name: pipe1
       inputRefs:
       - application
       outputRefs:
       - s3bucket

   serviceAccount.name

   Specify the name of your service account.

   outputs.name

   Specify a name for the output.

   type

   Specify the s3 type.

   secretName

   Specify the name of the secret that has your AWS credentials.

   bucket

   Specify the Amazon S3 bucket name for log storage.

   keyPrefix

   Specify the S3 key prefix for log objects. The value can be a combination of static and dynamic values consisting of field paths separated by "||". Encase dynamic values in single curly brackets "{}". You must follow dynamic values with a static fallback value.

   url

   Optional: Specify the S3 endpoint URL. If you do not specify this, the system uses the default AWS S3 endpoint https://s3.<region>.amazonaws.com based on the region. Use this field for S3-compatible storage endpoints or custom endpoints.

4. Apply the CR object:

   $ oc apply -f <file_name>.yaml

Procedure

1. Create or update a ClusterLogForwarder custom resource (CR):

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <log_forwarder_name>
   spec:
     serviceAccount:
       name: <service_account_name>
     outputs:
      - name: s3-output
        type: s3
        s3:
          keyPrefix: 's3{.log_type||"missing"}'
          bucket: <bucket_name>
          region: us-east-2
          authentication:
            type: iamRole
            iamRole:
              roleARN:
                key: role_arn
                secretName: sts-secret
              token:
                from: serviceAccount
     pipelines:
       - name: to-s3
         inputRefs:
           - infrastructure
           - audit
           - application
         outputRefs:
           - s3-output

   • keyPrefix: The S3 key prefix for log objects. You can use static or dynamic values consisting of field paths separated by || and ending with a static fallback value.
   • bucket: The S3 bucket name for log storage.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml