×

Use this tutorial to deploy the Red Hat OpenShift Logging Operator and configure it to use Security Token Services (STS) authentication to forward logs to CloudWatch.

Prerequisites
  • A Red Hat OpenShift Service on AWS (ROSA) Classic cluster

  • The jq command-line interface (CLI)

  • The Amazon Web Services (AWS) CLI (aws)

Setting up your environment

  1. Configure the following environment variables, changing the cluster name to suit your cluster:

    You must be logged in as an administrator.

    $ export ROSA_CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
    $ export REGION=$(rosa describe cluster -c ${ROSA_CLUSTER_NAME} --output json | jq -r .region.id)
    $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
    $ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
    $ export AWS_PAGER=""
    $ export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/clf-cloudwatch-sts"
    $ mkdir -p ${SCRATCH}
  2. Ensure all fields output correctly before moving to the next section:

    $ echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

Preparing your AWS account

  1. Create an Identity Access Management (IAM) policy for logging:

    $ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaCloudWatch'].{ARN:Arn}" --output text)
    $ if [[ -z "${POLICY_ARN}" ]]; then
    cat << EOF > ${SCRATCH}/policy.json
    {
      "Version": "2012-10-17",
      "Statement": [
         {
               "Effect": "Allow",
               "Action": [
                  "logs:CreateLogGroup",
                  "logs:CreateLogStream",
                  "logs:DescribeLogGroups",
                  "logs:DescribeLogStreams",
                  "logs:PutLogEvents",
                  "logs:PutRetentionPolicy"
               ],
               "Resource": "arn:aws:logs:*:*:*"
         }
    ]
    }
    EOF
    POLICY_ARN=$(aws iam create-policy --policy-name "RosaCloudWatch" \
    --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn --output text)
    fi
    $ echo ${POLICY_ARN}
  2. Create an IAM role trust policy for the cluster:

    $ cat <<EOF > ${SCRATCH}/trust-policy.json
    {
       "Version": "2012-10-17",
       "Statement": [{
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "${OIDC_ENDPOINT}:sub": "system:serviceaccount:openshift-logging:logcollector"
           }
         }
       }]
    }
    EOF
    $ ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
          --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
          --query Role.Arn --output text)
    $ echo ${ROLE_ARN}
  3. Attach the IAM policy to the IAM role:

    $ aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
       --policy-arn ${POLICY_ARN}

Deploying Operators

  1. Deploy the Red Hat OpenShift Logging Operator:

    $  cat << EOF | oc apply -f -
       apiVersion: operators.coreos.com/v1alpha1
       kind: Subscription
       metadata:
         labels:
          operators.coreos.com/cluster-logging.openshift-logging: ""
         name: cluster-logging
         namespace: openshift-logging
       spec:
         channel: stable
         installPlanApproval: Automatic
         name: cluster-logging
         source: redhat-operators
         sourceNamespace: openshift-marketplace
    EOF
  2. Create a secret:

    $  cat << EOF | oc apply -f -
      apiVersion: v1
      kind: Secret
      metadata:
        name: cloudwatch-credentials
        namespace: openshift-logging
      stringData:
        role_arn: $ROLE_ARN
    EOF

Configuring cluster logging

  1. Create a ClusterLogForwarder custom resource (CR):

    $ cat << EOF | oc apply -f -
      apiVersion: "logging.openshift.io/v1"
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        outputs:
          - name: cw
            type: cloudwatch
            cloudwatch:
              groupBy: namespaceName
              groupPrefix: rosa-${ROSA_CLUSTER_NAME}
              region: ${REGION}
            secret:
              name: cloudwatch-credentials
        pipelines:
          - name: to-cloudwatch
            inputRefs:
              - infrastructure
              - audit
              - application
            outputRefs:
              - cw
    EOF
  2. Create a ClusterLogging CR:

    $ cat << EOF | oc apply -f -
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        collection:
          logs:
             type: vector
        managementState: Managed
    EOF

Checking CloudWatch for logs

  • Use either the AWS console or the AWS CLI to validate that there are log streams from the cluster.

    • To validate the logs in the AWS CLI, run the following command:

      $ aws logs describe-log-groups --log-group-name-prefix rosa-${ROSA_CLUSTER_NAME}
      Example output
      {
        "logGroups": [
            {
              "logGroupName": "rosa-xxxx.audit",
              "creationTime": 1661286368369,
              "metricFilterCount": 0,
              "arn": "arn:aws:logs:us-east-2:xxxx:log-group:rosa-xxxx.audit:*",
              "storedBytes": 0
            },
            {
              "logGroupName": "rosa-xxxx.infrastructure",
              "creationTime": 1661286369821,
              "metricFilterCount": 0,
              "arn": "arn:aws:logs:us-east-2:xxxx:log-group:rosa-xxxx.infrastructure:*",
              "storedBytes": 0
            }
        ]
      }

      If this is a new cluster, you might not see a log group for application logs as applications are not yet running.

Cleaning up your resources

  1. Delete the ClusterLogForwarder CR:

    $ oc delete -n openshift-logging clusterlogforwarder instance
  2. Delete the ClusterLogging CR:

    $ oc delete -n openshift-logging clusterlogging instance
  3. Detach the IAM policy to the IAM role:

    $ aws iam detach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
       --policy-arn "${POLICY_ARN}"
  4. Delete the IAM role:

    $ aws iam delete-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch"
  5. Delete the IAM policy:

    Only delete the IAM policy if there are no other resources using the policy.

    $ aws iam delete-policy --policy-arn "${POLICY_ARN}"
  6. Delete the CloudWatch log groups:

    $ aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.audit"
    $ aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.infrastructure"