Configure metrics collection and alerting on the control plane

The Telemetry service (ceilometer, prometheus) is enabled by default in a RHOSO deployment. You can configure observability by editing the Telemetry service in your OpenStackControlPlane custom resource (CR) file.

Before you begin

Procedure

  1. Open your OpenStackControlPlane CR file, openstack_control_plane.yaml, on your workstation.
  2. Configure the Telemetry service, telemetry, as required for your environment: 
      telemetry:
    enabled: true
    template:
      metricStorage:
        enabled: true
        dashboardsEnabled: true
        dataplaneNetwork: ctlplane
        networkAttachments:
          - ctlplane
        monitoringStack:
          alertingEnabled: true
          scrapeInterval: 30s
          storage:
            strategy: persistent
            retention: 24h
            persistent:
              pvcStorageRequest: 20G
      autoscaling:
        enabled: false
        aodh:
          notificationsBus:
            cluster: rabbitmq-notification
          databaseAccount: aodh
          databaseInstance: openstack
          secret: osp-secret
        heatInstance: heat
      ceilometer:
        enabled: true
        notificationsBus:
          cluster: rabbitmq-notification
        secret: osp-secret
      logging:
        enabled: false
        annotations:
          metallb.universe.tf/address-pool: internalapi
          metallb.universe.tf/allow-shared-ip: internalapi
          metallb.universe.tf/loadBalancerIPs: 172.17.0.80
      cloudkitty:
        enabled: false
        messagingBus:
          cluster: rabbitmq
        s3StorageConfig:
          schemas:
          - effectiveDate: "2024-11-18"
            version: v13
          secret:
            name: cloudkitty-loki-s3
            type: s3
    • metricStorage.monitoringStack.scrapeInterval: Specifies the interval at which new metrics are gathered. Changing this interval can affect performance.
    • metricStorage.monitoringStack.storage.retention: Specifies the length of time that telemetry metrics are stored. The duration affects the amount of storage required.
    • storage.persistent.pvcStorageRequest: Specifies the amount of storage to allocate to the Prometheus time series database.
    • autoscaling.enabled: Set to true to enable autoscaling. The autoscaling field must be present even when autoscaling is disabled. For more information about autoscaling, see Activate heat and telemetry services for instance autoscaling.
    • ceilometer.enabled: Set to false to disable the ceilometer service. If you do not disable ceilometer, then a Prometheus metrics exporter is created and exposed from inside the cluster at the following URL: http://ceilometer-internal.openstack.svc:3000/metrics
    • logging.enabled: Set to true to enable observability logging. For more information about configuring observability logging, see Enable observability logging.
    • cloudkitty.enabled: Set to true to enable the Rating service (cloudkitty). For more information about configuring chargeback and rating capabilities, see Enable the Rating service on the control plane.
    • aodh.notificationsBus.cluster and ceilometer.notificationsBus.cluster: Set to a dedicated RabbitMQ cluster for notifications as in this example or to a combined RabbitMQ cluster for both RPC and notifications as in the default RHOSO deployment that uses the combined rabbitmq cluster.
  3. Update the control plane: 
    $ oc apply -f openstack_control_plane.yaml -n openstack
  4. Wait until RHOCP creates the resources related to the OpenStackControlPlane CR. Run the following command to check the status: 
    $ oc get openstackcontrolplane -n openstack
NAME 						STATUS 	MESSAGE
openstack-control-plane 	Unknown 	Setup started

    The OpenStackControlPlane resources are created when the status is "Setup complete".

    Tip
    Append the -w option to the end of the get command to track deployment progress.
  5. Optional: Confirm that the control plane is deployed by reviewing the pods in the openstack namespace for each of your cells: 
    $ oc get pods -n openstack

    The control plane is deployed when all the pods are either completed or running.

Results

  1. Access the remote shell for the OpenStackClient pod from your workstation: 
    $ oc rsh -n openstack openstackclient
  2. Confirm that you can query prometheus and that the scrape endpoints are active: 
    $ openstack metric query up --disable-rbac -c container -c instance -c value

    Example output:

    +-----------------+------------------------+-------+
| container       | instance               | value |
+-----------------+------------------------+-------+
| alertmanager    | 10.217.1.112:9093      | 1     |
| prometheus      | 10.217.1.63:9090       | 0     |
| proxy-httpd     | 10.217.1.52:3000       | 1     |
|                 | 192.168.122.100:9100   | 1     |
|                 | 192.168.122.101:9100   | 1     |
+-----------------+------------------------+-------+
    Note
    Each entry in the value field should be "1" when there are active workloads scheduled on the cluster, except for the prometheus container. The prometheus container reports a value of "0" due to TLS, which is enabled by default.
  3. You can find the openstack-telemetry-operator dashboards by clicking Observe and then Dashboards in the RHOCP console. For more information about RHOCP dashboards, see Reviewing monitoring dashboards as a cluster administrator in the RHOCP Monitoring Guide.