Deploy the monitoring stack

Deploy the Cluster Observability Operator and create MonitoringStack and ServiceMonitor resources so that Prometheus scrapes metrics from Red Hat Edge Manager components.

Before you begin

  • A Red Hat OpenShift Container Platform cluster version 4.19 or later and cluster-admin permissions.
  • Red Hat Edge Manager deployed in <project_name> (for example, by using the Software Catalog or an equivalent Helm-based installation).
  • The OpenShift CLI (oc) installed and authenticated to the cluster.

Procedure

  1. Install the Cluster Observability Operator from the Red Hat Operator catalog by creating a Subscription resource: 
    $ oc create -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-observability-operator
  namespace: openshift-operators
spec:
  channel: stable
  name: cluster-observability-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF
  2. Verify the installation is complete. The following command returns when the operator status reaches Succeeded: 
    $ oc wait --for=jsonpath='{.status.phase}'=Succeeded \
  csv -l operators.coreos.com/cluster-observability-operator.openshift-operators \
  -n openshift-operators --timeout=300s
  3. Create a MonitoringStack resource to deploy a Prometheus instance in <project_name>: 
    $ oc create -f - <<EOF
apiVersion: monitoring.rhobs/v1alpha1
kind: MonitoringStack
metadata:
  name: flightctl-monitoring-stack
  namespace: <project_name>
spec:
  alertmanagerConfig:
    disabled: true
  logLevel: info
  prometheusConfig:
    persistentVolumeClaim:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi
    replicas: 1
  resourceSelector:
    matchLabels:
      app: flightctl-monitoring
  retention: 365d
EOF
  4. Track the pod’s transition from Pending to Ready (which confirms the container has started and the Prometheus service is initialized) by running the following command: 
    $ oc wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus \
  -n <project_name> --timeout=300s
  5. Create ServiceMonitor resources so that Prometheus scrapes metrics from the Red Hat Edge Manager API and Telemetry Gateway: 
    $ oc create -f - <<EOF
apiVersion: monitoring.rhobs/v1
kind: ServiceMonitor
metadata:
  name: flightctl-api
  namespace: <project_name>
  labels:
    app: flightctl-monitoring
spec:
  selector:
    matchLabels:
      flightctl.service: flightctl-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
    - port: db-metrics
      path: /metrics
      interval: 30s
---
apiVersion: monitoring.rhobs/v1
kind: ServiceMonitor
metadata:
  name: flightctl-telemetry-gateway
  namespace: <project_name>
  labels:
    app: flightctl-monitoring
spec:
  selector:
    matchLabels:
      flightctl.service: flightctl-telemetry-gateway
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
EOF

Results

Confirm that Prometheus is scraping Red Hat Edge Manager targets:

  1. Open a tunnel to the monitoring stack

    Run this command to connect your local machine to the Prometheus service inside the cluster. Leave this terminal open while you verify targets.

    $ oc port-forward -n <project_name> svc/flightctl-monitoring-stack-prometheus 9090:9090
  2. Check the health of your targets (in a new terminal window)

    In a separate terminal window, run this command to verify that the API and Telemetry Gateway are being successfully monitored. The command uses jq, which must be installed on the workstation where you run it.

    $ curl -s http://localhost:9090/api/v1/targets | \
  jq -r '.data.activeTargets[] | select(.labels.job | contains("flightctl")) | {job: .labels.job, health: .health}'
  3. Close the connection

    Upon verifying that the intended targets have reached a healthy state, return to the initial terminal and use Ctrl+C to end the port-forwarding process and close the connection.

    You should see healthy targets for both the API and the Telemetry Gateway.