Chapter 8. Troubleshooting

Procedure

1. Navigate to the Clusters page of the Hybrid Cloud Console.
2. Click the name of your cluster to go to the cluster details page.

3. Click the Cluster history tab.

   Cluster notifications appear under the Cluster history heading.

4. Optional: Filter for relevant cluster notifications.

   Use the filter controls to hide cluster notifications that are not relevant to you, so that you can focus on your area of expertise or on resolving a critical issue. You can filter notifications based on text in the notification description, severity level, notification type, when you received the notification, and the system or person that triggered the notification.

Procedure

1. Check for Karpenter CRDs:

   $ oc get crd | grep karpenter

   Example output

   ec2nodeclasses.karpenter.k8s.aws
   nodeclaims.karpenter.sh
   nodepools.karpenter.sh
   openshiftec2nodeclasses.karpenter.hypershift.openshift.io

2. If CRDs are not present, wait 5 minutes for the Karpenter controller to create them. The controller deployment can take several minutes to complete after enabling the Red Hat build of Karpenter.

3. After 5 minutes, if CRDs are still not present, check whether the Karpenter controller pod is running:

   $ oc get pods -n kube-system | grep karpenter

   The output should show a Karpenter controller pod in Running status.

4. If the controller pod is not running or CRDs are still missing after 5 minutes, escalate to Red Hat Support. Provide the following diagnostic information:

   • Your cluster ID
   • Output of rosa describe cluster -c $CLUSTER_ID -o json
   • Output of oc get pods -n kube-system

Procedure

1. Check the EC2NodeClass status:

   $ oc get ec2nodeclass/default -o json | jq .status.conditions

   This command displays all status conditions for the default EC2NodeClass.

2. Identify which conditions are not ready by looking for entries with "status": "False".

   Common failures are SubnetsReady and SecurityGroupsReady.

3. If SubnetsReady shows False with the message "SubnetSelector did not match any Subnets", private subnets in your VPC are missing the required discovery tag. Continue to the next step.
4. If SecurityGroupsReady shows False with message "SecurityGroupSelector did not match any SecurityGroups", security groups are missing the required discovery tag. The same resolution process applies for both subnets and security groups.

5. Get the Karpenter discovery tag value from the EC2NodeClass specification:

   $ oc get ec2nodeclass/default -o json | jq -r '.spec.subnetSelectorTerms[0].tags["karpenter.sh/discovery"]'

   Example output

   <discovery_tag_value>

   This cluster-specific value must be added as a tag to AWS resources.

6. List private subnets in your cluster VPC:

   $ aws ec2 describe-subnets --filters "Name=vpc-id,Values=<vpc_id>" \
     --query 'Subnets[?MapPublicIpOnLaunch==`false`].[SubnetId,Tags]' --output table

   Replace <vpc_id> with your cluster’s VPC ID.

7. For each private subnet, verify whether the Karpenter discovery tag is present:

   $ aws ec2 describe-subnets --subnet-ids <subnet_id> \
     --query 'Subnets[0].Tags[?Key==`karpenter.sh/discovery`].Value' --output text

   Replace <subnet_id> with the subnet ID to check.

8. If the discovery tag is missing, add it to the subnet:

   $ aws ec2 create-tags --resources <subnet_id> \
     --tags Key=karpenter.sh/discovery,Value=<discovery_tag_value>

   Replace <subnet_id> with the subnet ID and <discovery_tag_value> with the value from step 5.

9. Repeat steps 7 and 8 for all private subnets in your VPC.

10. Verify that the default security group has the discovery tag:

    $ aws ec2 describe-security-groups --group-ids <default_sg_id> \
      --query 'SecurityGroups[0].Tags[?Key==`karpenter.sh/discovery`].Value' --output text

    Replace <default_sg_id> with your cluster’s default security group ID.

11. If the discovery tag is missing from the security group, add it:

    $ aws ec2 create-tags --resources <default_sg_id> \
      --tags Key=karpenter.sh/discovery,Value=<discovery_tag_value>

    Replace <default_sg_id> with the security group ID and <discovery_tag_value> with the value from step 5.

12. Wait 30 to 60 seconds for Karpenter to re-scan AWS resources.

13. Verify that the EC2NodeClass Ready condition is now True:

    $ oc get ec2nodeclass/default -o json | jq '.status.conditions[] | select(.type=="Ready")'

    The output should show "status": "True".

14. If the Ready condition is still False, check for other condition failures in the output and repeat the appropriate steps.

Procedure

1. Identify the IAM role used by the control plane operator:

   $ aws iam list-roles | grep kube-system-control-plane-operator

   The role name follows the pattern <cluster_prefix>-kube-system-control-plane-operator.

2. Optional: Verify the authorization error in AWS CloudTrail:

   1. Navigate to the AWS CloudTrail console.

   2. Search for events matching these criteria:

      • Event name: CreateTags
      • Error code: Client.UnauthorizedOperation
      • User identity: Contains control-plane-operator

      If you find matching events, the error message confirms that the IAM role lacks ec2:CreateTags permission.

3. Create an IAM policy document that allows creating tags:

   $ cat > /tmp/create-tags-policy.json << 'POLICY_EOF'
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowCreateTags",
               "Effect": "Allow",
               "Action": [
                   "ec2:CreateTags"
               ],
               "Resource": "*"
           }
       ]
   }
   POLICY_EOF

4. Attach the policy to the control plane operator IAM role:

   $ aws iam put-role-policy \
     --role-name <cluster_prefix>-kube-system-control-plane-operator \
     --policy-name KarpenterCreateTags \
     --policy-document file:///tmp/create-tags-policy.json

   Replace <cluster_prefix> with your cluster’s prefix from step 1.

5. Verify that the policy is attached to the role:

   $ aws iam get-role-policy \
     --role-name <cluster_prefix>-kube-system-control-plane-operator \
     --policy-name KarpenterCreateTags

   The output shows the policy document you attached.

Procedure

1. List Operator subscriptions:

   $ oc get subs -n <operator_namespace>

2. Use the oc describe command to inspect a Subscription resource:

   $ oc describe sub <subscription_name> -n <operator_namespace>

3. In the command output, find the Conditions section for the status of Operator subscription condition types. In the following example, the CatalogSourcesUnhealthy condition type has a status of false because all available catalog sources are healthy:

   Example output

   Name:         cluster-logging
   Namespace:    openshift-logging
   Labels:       operators.coreos.com/cluster-logging.openshift-logging=
   Annotations:  <none>
   API Version:  operators.coreos.com/v1alpha1
   Kind:         Subscription
   # ...
   Conditions:
      Last Transition Time:  2019-07-29T13:42:57Z
      Message:               all available catalogsources are healthy
      Reason:                AllCatalogSourcesHealthy
      Status:                False
      Type:                  CatalogSourcesUnhealthy
   # ...

   Note

   Default Red Hat OpenShift Service on AWS cluster Operators are managed by the Cluster Version Operator (CVO) and they do not have a Subscription object. Application Operators are managed by Operator Lifecycle Manager (OLM) and they have a Subscription object.

Procedure

1. List the catalog sources in a namespace. For example, you can check the openshift-marketplace namespace, which is used for cluster-wide catalog sources:

   $ oc get catalogsources -n openshift-marketplace

   Example output

   NAME                  DISPLAY               TYPE   PUBLISHER   AGE
   certified-operators   Certified Operators   grpc   Red Hat     55m
   community-operators   Community Operators   grpc   Red Hat     55m
   example-catalog       Example Catalog       grpc   Example Org 2m25s
   redhat-operators      Red Hat Operators     grpc   Red Hat     55m

2. Use the oc describe command to get more details and status about a catalog source:

   $ oc describe catalogsource example-catalog -n openshift-marketplace

   Example output

   Name:         example-catalog
   Namespace:    openshift-marketplace
   Labels:       <none>
   Annotations:  operatorframework.io/managed-by: marketplace-operator
                 target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
   API Version:  operators.coreos.com/v1alpha1
   Kind:         CatalogSource
   # ...
   Status:
     Connection State:
       Address:              example-catalog.openshift-marketplace.svc:50051
       Last Connect:         2021-09-09T17:07:35Z
       Last Observed State:  TRANSIENT_FAILURE
     Registry Service:
       Created At:         2021-09-09T17:05:45Z
       Port:               50051
       Protocol:           grpc
       Service Name:       example-catalog
       Service Namespace:  openshift-marketplace
   # ...

   In the preceding example output, the last observed state is TRANSIENT_FAILURE. This state indicates that there is a problem establishing a connection for the catalog source.

3. List the pods in the namespace where your catalog source was created:

   $ oc get pods -n openshift-marketplace

   Example output

   NAME                                    READY   STATUS             RESTARTS   AGE
   certified-operators-cv9nn               1/1     Running            0          36m
   community-operators-6v8lp               1/1     Running            0          36m
   marketplace-operator-86bfc75f9b-jkgbc   1/1     Running            0          42m
   example-catalog-bwt8z                   0/1     ImagePullBackOff   0          3m55s
   redhat-operators-smxx8                  1/1     Running            0          36m

   When a catalog source is created in a namespace, a pod for the catalog source is created in that namespace. In the preceding example output, the status for the example-catalog-bwt8z pod is ImagePullBackOff. This status indicates that there is an issue pulling the catalog source’s index image.

4. Use the oc describe command to inspect a pod for more detailed information:

   $ oc describe pod example-catalog-bwt8z -n openshift-marketplace

   Example output

   Name:         example-catalog-bwt8z
   Namespace:    openshift-marketplace
   Priority:     0
   Node:         ci-ln-jyryyg2-f76d1-ggdbq-worker-b-vsxjd/10.0.128.2
   ...
   Events:
     Type     Reason          Age                From               Message
     ----     ------          ----               ----               -------
     Normal   Scheduled       48s                default-scheduler  Successfully assigned openshift-marketplace/example-catalog-bwt8z to ci-ln-jyryyf2-f76d1-fgdbq-worker-b-vsxjd
     Normal   AddedInterface  47s                multus             Add eth0 [10.131.0.40/23] from openshift-sdn
     Normal   BackOff         20s (x2 over 46s)  kubelet            Back-off pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          20s (x2 over 46s)  kubelet            Error: ImagePullBackOff
     Normal   Pulling         8s (x3 over 47s)   kubelet            Pulling image "quay.io/example-org/example-catalog:v1"
     Warning  Failed          8s (x3 over 47s)   kubelet            Failed to pull image "quay.io/example-org/example-catalog:v1": rpc error: code = Unknown desc = reading manifest v1 in quay.io/example-org/example-catalog: unauthorized: access to the requested resource is not authorized
     Warning  Failed          8s (x3 over 47s)   kubelet            Error: ErrImagePull

   In the preceding example output, the error messages indicate that the catalog source’s index image is failing to pull successfully because of an authorization issue. For example, the index image might be stored in a registry that requires login credentials.

Procedure

1. List Operators running in the cluster. The output includes Operator version, availability, and up-time information:

   $ oc get clusteroperators

2. List Operator pods running in the Operator’s namespace, plus pod status, restarts, and age:

   $ oc get pod -n <operator_namespace>

3. Output a detailed Operator pod summary:

   $ oc describe pod <operator_pod_name> -n <operator_namespace>

Procedure

1. Switch into a project:

   $ oc project <project_name>

2. List pods running within the namespace, as well as pod status, error states, restarts, and age:

   $ oc get pods

3. Determine whether the namespace is managed by a deployment configuration:

   $ oc status

   If the namespace is managed by a deployment configuration, the output includes the deployment configuration name and a base image reference.

4. Inspect the base image referenced in the preceding command’s output:

   $ skopeo inspect docker://<image_reference>

5. If the base image reference is not correct, update the reference in the deployment configuration:

   $ oc edit deployment/my-deployment

6. When deployment configuration changes on exit, the configuration will automatically redeploy. Watch pod status as the deployment progresses, to determine whether the issue has been resolved:

   $ oc get pods -w

7. Review events within the namespace for diagnostic information relating to pod failures:

   $ oc get events

Procedure

1. Query logs for a specific pod:

   $ oc logs <pod_name>

2. Query logs for a specific container within a pod:

   $ oc logs <pod_name> -c <container_name>

   Logs retrieved using the preceding oc logs commands are composed of messages sent to stdout within pods or containers.

3. Inspect logs contained in /var/log/ within a pod.

   1. List log files and subdirectories contained in /var/log within a pod:

      $ oc exec <pod_name>  -- ls -alh /var/log

      Example output

      total 124K
      drwxr-xr-x. 1 root root   33 Aug 11 11:23 .
      drwxr-xr-x. 1 root root   28 Sep  6  2022 ..
      -rw-rw----. 1 root utmp    0 Jul 10 10:31 btmp
      -rw-r--r--. 1 root root  33K Jul 17 10:07 dnf.librepo.log
      -rw-r--r--. 1 root root  69K Jul 17 10:07 dnf.log
      -rw-r--r--. 1 root root 8.8K Jul 17 10:07 dnf.rpm.log
      -rw-r--r--. 1 root root  480 Jul 17 10:07 hawkey.log
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 lastlog
      drwx------. 2 root root   23 Aug 11 11:14 openshift-apiserver
      drwx------. 2 root root    6 Jul 10 10:31 private
      drwxr-xr-x. 1 root root   22 Mar  9 08:05 rhsm
      -rw-rw-r--. 1 root utmp    0 Jul 10 10:31 wtmp

   2. Query a specific log file contained in /var/log within a pod:

      $ oc exec <pod_name> cat /var/log/<path_to_log>

      Example output

      2023-07-10T10:29:38+0000 INFO --- logging initialized ---
      2023-07-10T10:29:38+0000 DDEBUG timer: config: 13 ms
      2023-07-10T10:29:38+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
      2023-07-10T10:29:38+0000 INFO Updating Subscription Management repositories.
      2023-07-10T10:29:38+0000 INFO Unable to read consumer identity
      2023-07-10T10:29:38+0000 INFO Subscription Manager is operating in container mode.
      2023-07-10T10:29:38+0000 INFO

   3. List log files and subdirectories contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> ls /var/log

   4. Query a specific log file contained in /var/log within a specific container:

      $ oc exec <pod_name> -c <container_name> cat /var/log/<path_to_log>

Procedure

1. Switch into the project that contains the pod you would like to access. This is necessary because the oc rsh command does not accept the -n namespace option:

   $ oc project <namespace>

2. Start a remote shell into a pod:

   $ oc rsh <pod_name>

   where:

   <pod_name>

   If a pod has multiple containers, oc rsh defaults to the first container unless -c <container_name> is specified.

3. Start a remote shell into a specific container within a pod:

   $ oc rsh -c <container_name> pod/<pod_name>

4. Create a port forwarding session to a port on a pod:

   $ oc port-forward <pod_name> <host_port>:<pod_port>

   where:

   <pod_name> <host_port>:<pod_port>

   Enter Ctrl+C to cancel the port forwarding session.

Procedure

1. Start a debug pod with root access, based on a deployment.

   1. Obtain a project’s deployment name:

      $ oc get deployment -n <project_name>

   2. Start a debug pod with root privileges, based on the deployment:

      $ oc debug deployment/my-deployment --as-root -n <project_name>

2. Start a debug pod with root access, based on a deployment configuration.

   1. Obtain a project’s deployment configuration name:

      $ oc get deploymentconfigs -n <project_name>

   2. Start a debug pod with root privileges, based on the deployment configuration:

      $ oc debug deploymentconfig/my-deployment-configuration --as-root -n <project_name>

      Note

      You can append -- <command> to the preceding oc debug commands to run individual commands within a debug pod, instead of running an interactive shell.

Procedure

1. Copy a file to a pod:

   $ oc cp <local_path> <pod_name>:/<path> -c <container_name>

   -c <container_name> refers to the desired container in a pod. If you do not specify a container with the -c option, then the first container in a pod is selected.

2. Copy a file from a pod:

   $ oc cp <pod_name>:/<path> -c <container_name> <local_path>

   -c <container_name> refers to the desired container in a pod. If you do not specify a container with the -c option, then the first container in a pod is selected.

   Note

   For oc cp to function, the tar binary must be available within the container.

Procedure

1. To determine where in the S2I process a failure occurs, you can observe the state of the pods relating to each of the following S2I stages:

   1. During the build configuration stage, a build pod is used to create an application container image from a base image and application source code.
   2. During the deployment configuration stage, a deployment pod is used to deploy application pods from the application container image that was built in the build configuration stage. The deployment pod also deploys other resources such as services and routes. The deployment configuration begins after the build configuration succeeds.
   3. After the deployment pod has started the application pods, application failures can occur within the running application pods. For instance, an application might not behave as expected even though the application pods are in a Running state. In this scenario, you can access running application pods to investigate application failures within a pod.

2. When troubleshooting S2I issues, follow this strategy:

   1. Monitor build, deployment, and application pod status.
   2. Determine the stage of the S2I process where the problem occurred.
   3. Review logs corresponding to the failed stage.

Procedure

1. Watch the pod status throughout the S2I process to determine at which stage a failure occurs:

   $ oc get pods -w

   Use the -w flag to monitor pods for changes until you quit the command using Ctrl+C.

2. Review a failed pod’s logs for errors.

   • If the build pod fails, review the build pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-build

     Note

     Alternatively, you can review the build configuration’s logs using oc logs -f bc/<application_name>. The build configuration’s logs include the logs from the build pod.

   • If the deployment pod fails, review the deployment pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-deploy

     Note

     Alternatively, you can review the deployment configuration’s logs using oc logs -f dc/<application_name>. This outputs logs from the deployment pod until the deployment pod completes successfully. The command outputs logs from the application pods if you run it after the deployment pod has completed. After a deployment pod completes, its logs can still be accessed by running oc logs -f pod/<application_name>-<build_number>-deploy.

   • If an application pod fails, or if an application is not behaving as expected within a running application pod, review the application pod’s logs:

     $ oc logs -f pod/<application_name>-<build_number>-<random_string>

Procedure

1. List events relating to a specific application pod. The following example retrieves events for an application pod named my-app-1-akdlg:

   $ oc describe pod/my-app-1-akdlg

2. Review logs from an application pod:

   $ oc logs -f pod/my-app-1-akdlg

3. Query specific logs within a running application pod. Logs that are sent to stdout are collected by the OpenShift Logging framework and are included in the output of the preceding command. The following query is only required for logs that are not sent to stdout.

   1. If an application log can be accessed without root privileges within a pod, concatenate the log file as follows:

      $ oc exec my-app-1-akdlg -- cat /var/log/my-application.log

   2. If root access is required to view an application log, you can start a debug container with root privileges and then view the log file from within the container. Start the debug container from the project’s DeploymentConfig object. Pod users typically run with non-root privileges, but running troubleshooting pods with temporary root privileges can be useful during issue investigation:

      $ oc debug dc/my-deployment-configuration --as-root -- cat /var/log/my-application.log

      Note

      You can access an interactive shell with root access within the debug pod if you run oc debug dc/<deployment_configuration> --as-root without appending -- <command>.

4. Test application functionality interactively and run diagnostic tools, in an application container with an interactive shell.

   1. Start an interactive shell on the application container:

      $ oc exec -it my-app-1-akdlg /bin/bash

   2. Test application functionality interactively from within the shell. For example, you can run the container’s entry point command and observe the results. Then, test changes from the command line directly, before updating the source code and rebuilding the application container through the S2I process.

   3. Run diagnostic binaries available within the container.

      Note

      Root privileges are required to run some diagnostic binaries. In these situations you can start a debug pod with root access, based on a problematic pod’s DeploymentConfig object, by running oc debug dc/<deployment_configuration> --as-root. Then, you can run diagnostic binaries as root from within the debug pod.

5. If diagnostic binaries are not available within a container, you can run a host’s diagnostic binaries within a container’s namespace by using nsenter. The following example runs ip ad within a container’s namespace, using the host`s ip binary.

   1. Enter into a debug session on the target node. This step instantiates a debug pod called <node_name>-debug:

      $ oc debug node/my-cluster-node

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Note

      Red Hat OpenShift Service on AWS 4 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the Red Hat OpenShift Service on AWS API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. Determine the target container ID:

      # crictl ps

   4. Determine the container’s process ID. In this example, the target container ID is a7fe32346b120:

      # crictl inspect a7fe32346b120 --output yaml | grep 'pid:' | awk '{print $2}'

   5. Run ip ad within the container’s namespace, using the host’s ip binary. This example uses 31150 as the container’s process ID. The nsenter command enters the namespace of a target process and runs a command in its namespace. Because the target process in this example is a container’s process ID, the ip ad command is run in the container’s namespace from the host:

      # nsenter -n -t 31150 -- ip ad

      Note

      Running a host’s diagnostic binaries within a container’s namespace is only possible if you are using a privileged container such as a debug node.

Procedure

1. Ensure that your project and resources are not excluded from user workload monitoring. The following examples use the ns1 project.

   1. Verify that the project does not have the openshift.io/user-monitoring=false label attached:

      $ oc get namespace ns1 --show-labels | grep 'openshift.io/user-monitoring=false'

      Note

      The default label set for user workload projects is openshift.io/user-monitoring=true. However, the label is not visible unless you manually apply it.

   2. Verify that the ServiceMonitor and PodMonitor resources do not have the openshift.io/user-monitoring=false label attached. The following example checks the prometheus-example-monitor service monitor.

      $ oc -n ns1 get servicemonitor prometheus-example-monitor --show-labels | grep 'openshift.io/user-monitoring=false'

   3. If the label is attached, remove the label:

      Example of removing the label from the project

      $ oc label namespace ns1 'openshift.io/user-monitoring-'

      Example of removing the label from the resource

      $ oc -n ns1 label servicemonitor prometheus-example-monitor 'openshift.io/user-monitoring-'

      Example output

      namespace/ns1 unlabeled

2. Check that the corresponding labels match in the service and ServiceMonitor resource configurations. The following examples use the prometheus-example-app service, the prometheus-example-monitor service monitor, and the ns1 project.

   1. Obtain the label defined in the service.

      $ oc -n ns1 get service prometheus-example-app -o yaml

      Example output

        labels:
          app: prometheus-example-app

   2. Check that the matchLabels definition in the ServiceMonitor resource configuration matches the label output in the previous step.

      $ oc -n ns1 get servicemonitor prometheus-example-monitor -o yaml

      Example output

      apiVersion: v1
      kind: ServiceMonitor
      metadata:
        name: prometheus-example-monitor
        namespace: ns1
      spec:
        endpoints:
        - interval: 30s
          port: web
          scheme: http
        selector:
          matchLabels:
            app: prometheus-example-app

      Note

      You can check service and ServiceMonitor resource labels as a developer with view permissions for the project.

3. Inspect the logs for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. List the pods in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get pods

      Example output

      NAME                                   READY   STATUS    RESTARTS   AGE
      prometheus-operator-776fcbbd56-2nbfm   2/2     Running   0          132m
      prometheus-user-workload-0             5/5     Running   1          132m
      prometheus-user-workload-1             5/5     Running   1          132m
      thanos-ruler-user-workload-0           3/3     Running   0          132m
      thanos-ruler-user-workload-1           3/3     Running   0          132m

   2. Obtain the logs from the prometheus-operator container in the prometheus-operator pod. In the following example, the pod is called prometheus-operator-776fcbbd56-2nbfm:

      $ oc -n openshift-user-workload-monitoring logs prometheus-operator-776fcbbd56-2nbfm -c prometheus-operator

      If there is a issue with the service monitor, the logs might include an error similar to this example:

      level=warn ts=2020-08-10T11:48:20.906739623Z caller=operator.go:1829 component=prometheusoperator msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=eagle/eagle namespace=openshift-user-workload-monitoring prometheus=user-workload

4. Review the target status for your endpoint on the Metrics targets page in the Red Hat OpenShift Service on AWS web console UI.

   1. Log in to the Red Hat OpenShift Service on AWS web console and go to Observe Targets.
   2. Locate the metrics endpoint in the list, and review the status of the target in the Status column.
   3. If the Status is Down, click the URL for the endpoint to view more information on the Target Details page for that metrics target.

5. Configure debug level logging for the Prometheus Operator in the openshift-user-workload-monitoring project.

   1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

   2. Add logLevel: debug for prometheusOperator under data/config.yaml to set the log level to debug:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheusOperator:
            logLevel: debug
      # ...

   3. Save the file to apply the changes. The affected prometheus-operator pod is automatically redeployed.

   4. Confirm that the debug log-level has been applied to the prometheus-operator deployment in the openshift-user-workload-monitoring project:

      $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

      Example output

              - --log-level=debug

      Debug level logging will show all calls made by the Prometheus Operator.

   5. Check that the prometheus-operator pod is running:

      $ oc -n openshift-user-workload-monitoring get pods

      Note

      If an unrecognized Prometheus Operator loglevel value is included in the config map, the prometheus-operator pod might not restart successfully.

   6. Review the debug logs to see if the Prometheus Operator is using the ServiceMonitor resource. Review the logs for other related errors.

Procedure

1. In the Red Hat OpenShift Service on AWS web console, go to Observe Metrics.

2. Enter a Prometheus Query Language (PromQL) query in the Expression field. The following example queries help to identify high cardinality metrics that might result in high disk space consumption:

   • By running the following query, you can identify the ten jobs that have the highest number of scrape samples:

     topk(10, max by(namespace, job) (topk by(namespace, job) (1, scrape_samples_post_metric_relabeling)))

   • By running the following query, you can pinpoint time series churn by identifying the ten jobs that have created the most time series data in the last hour:

     topk(10, sum by(namespace, job) (sum_over_time(scrape_series_added[1h])))

3. Investigate the number of unbound label values assigned to metrics with higher than expected scrape sample counts:

   • If the metrics relate to a user-defined project, review the metrics key-value pairs assigned to your workload. These are implemented through Prometheus client libraries at the application level. Try to limit the number of unbound attributes referenced in your labels.
   • If the metrics relate to a core Red Hat OpenShift Service on AWS project, create a Red Hat support case on the Red Hat Customer Portal.

4. Review the TSDB status using the Prometheus HTTP API by following these steps when logged in as a cluster administrator:

   1. Get the Prometheus API route URL by running the following command:

      $ HOST=$(oc -n openshift-monitoring get route prometheus-k8s -ojsonpath='{.status.ingress[].host}')

   2. Extract an authentication token by running the following command:

      $ TOKEN=$(oc whoami -t)

   3. Query the TSDB status for Prometheus by running the following command:

      $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/status/tsdb"

      Example output

      "status": "success","data":{"headStats":{"numSeries":507473,
      "numLabelPairs":19832,"chunkCount":946298,"minTime":1712253600010,
      "maxTime":1712257935346},"seriesCountByMetricName":
      [{"name":"etcd_request_duration_seconds_bucket","value":51840},
      {"name":"apiserver_request_sli_duration_seconds_bucket","value":47718},
      ...

Procedure

1. In the ROSA CLI, link your ocm-role resource to your Red Hat organization by using your Amazon Resource Name (ARN):

   Note

   You must have Red Hat Organization Administrator privileges to run the rosa link command. After you link the ocm-role resource with your AWS account, it takes effect and is visible to all users in the organization.

   $ rosa link ocm-role --role-arn <arn>

   For example:

   I: Linking OCM role
   ? Link the '<AWS ACCOUNT ID>` role with organization '<ORG ID>'? Yes
   I: Successfully linked role-arn '<AWS ACCOUNT ID>' with organization account '<ORG ID>'

2. In the ROSA CLI, link your user-role resource to your Red Hat user account by using your Amazon Resource Name (ARN):

   $ rosa link user-role --role-arn <arn>

   For example:

   I: Linking User role
   ? Link the 'arn:aws:iam::<ARN>:role/ManagedOpenShift-User-Role-125' role with organization '<AWS ID>'? Yes
   I: Successfully linked role-arn 'arn:aws:iam::<ARN>:role/ManagedOpenShift-User-Role-125' with organization account '<AWS ID>'

Procedure

1. Delete the stack:

   $ rosa init --delete

2. Reinitialize your account:

   $ rosa init

Procedure

1. Run the following command to create the user-role IAM resource:

   $ rosa create user-role

2. After you see that the role has been created, you can delete the cluster. The following confirms that the role was created and linked:

   I: Successfully linked role ARN <user role ARN> with account <account ID>