Monitor network metrics

In Red Hat OpenStack Services on OpenShift (RHOSO) environments, you can use Red Hat OpenShift Container Platform (RHOCP) observability features and the RHOSO Telemetry service to access metrics for many aspects of your RHOSO deployment.

The Telemetry service includes Ceilometer and Prometheus.

Prometheus collects metrics from various components. It scrapes exporter endpoints on data plane nodes and control plane pods. You can configure Prometheus to drive a variety of functions including data logging, metrics viewing and queries on the RHOCP Observability dashboard, alarms, and autoscaling.

The RHOSO OpenStack network exporter (openstack-network-exporter) is a Prometheus exporter that runs on data plane nodes and control plane pods and collects OVN, OVS, and OVS-DPDK networking metrics to be scraped by Prometheus.

The RHOSO OpenStack network exporter collects a variety of metrics by default. You can access them through the RHOCP console dashboards. You can also scrape the network exporter endpoints with tools such as curl.

In addition to the default metrics collection, you can enable and disable the collection of extended metrics sets.

Available network metrics
Copy link

The network exporter exposes OVN Controller metrics, OVS metrics, and OVS-DPDK PMD metrics. For example:

OVN southbound database server metrics examples
  • ovn_raft_cluster_server_vote: A metric with a constant '1' value labeled by database name, cluster uuid, server uuid and vote.
  • ovn_raft_cluster_inbound_connections_total: total Number of inbound connections to the server labeled by database name, cluster uuid, and server uuid.
  • ovn_raft_cluster_leader: Value 1.0 if the server is the cluster leader for the given database or 0.0 if it is not, labeled by database name, cluster uuid, and server uuid.
OVN northd metrics examples
  • ovn_northd_pstream_open_total: Number of times that passive connections were opened for the remote peer to connect.
  • ovn_northd_txn_aborted_total: Number of times the OVSDB transaction has been aborted.
OVN Controller metrics examples
  • ovnc_lflow_run: The number of times ovn-controller translated the Logical_Flow table in the OVN SB database into OpenFlow flows in the interval.
  • ovnc_txn_error: number of times the OVSDB transaction has thrown an error.
OVN logical router and logical router port metrics examples
  • ovnc_router_port_traffic_pkts: Number of packets transmitted and received by a logical router port labeled by the logical data path number and the logical port number.
  • ovnc_router_port_traffic_bytes: Number of bytes transmitted and received by a logical router port labeled by the logical data path number and the logical port number
OVS metrics examples
  • ovs_bridge_port_count: Number of ports in a bridge, labeled by bridge name
  • ovs_bridge_flow_count: Number of OpenFlow rules configured on a bridge, labeled by bridge name
OVS-DPDK PMD metrics examples
  • ovs_pmd_tx_packets: Rate of PMD packets transmits during the interval
  • ovs_pmd_rx_packets: Rate of PMD packets received during the interval

You can view all of the available metrics and descriptions by running curl commands on a Compute node or control plane pod. For more information, see Query live network metrics with curl.

Access network metrics snapshots with the RHOCP console
Copy link

In Red Hat OpenStack Services on OpenShift (RHOSO) environments, you can view changes to a subset of the available networking metrics in charts on the RHOCP Console.

Before you begin
Copy link

Procedure
Copy link

  1. On the RHOCP Console, go to Observe > Dashboards.
  2. Choose Dashboard > Openstack/Openstack Network Exporter.
  3. (Optional) Select a Compute node from the Instance menu.
  4. (Optional) Select a Time Range and Refresh Interval.

Query live network metrics with curl
Copy link

The OpenStack network exporter pushes metrics to endpoints. You can log in to a Compute node and use the curl command to get metrics from those endpoints.

Before you begin
Copy link

  • The administrator has created a project for you and has provided you with a clouds.yaml file for you to access the cloud.
  • The python-openstackclient package is installed on your workstation. 
    $ dnf list installed python-openstackclient
  • You can log in to Compute nodes.

Procedure
Copy link

  1. Log in to a Compute node.
  2. Example: To view all available metrics, run the command: 
    curl -k https://localhost:9105/metrics | more
  3. Example: To view link state metrics, run the command: 
    curl -k -s  https://localhost:9105/metrics | grep ovs_interface_link_state
    Sample output
    # HELP ovs_interface_link_state The link state of the interface. Possible values are: up(1), down(0) or unknown(-1).
# TYPE ovs_interface_link_state gauge
ovs_interface_link_state{bridge="br-ex",interface="br-ex",port="br-ex",type="internal"} 1
ovs_interface_link_state{bridge="br-ex",interface="eth1",port="eth1",type="system"} 1
ovs_interface_link_state{bridge="br-ex",interface="patch-provnet-e0353205-c937-4ebb-af35-a7db0d85c9d3-to-br-int",port="patch-provnet-e0353205-c937-4ebb-af35-a7db0d85c9d3-to-br-int",type="patch"} 1
ovs_interface_link_state{bridge="br-ex",interface="vlan20",port="vlan20",type="internal"} 1
ovs_interface_link_state{bridge="br-ex",interface="vlan21",port="vlan21",type="internal"} 1
ovs_interface_link_state{bridge="br-ex",interface="vlan22",port="vlan22",type="internal"} 1
ovs_interface_link_state{bridge="br-ex",interface="vlan23",port="vlan23",type="internal"} 1
ovs_interface_link_state{bridge="br-int",interface="br-int",port="br-int",type="internal"} 0
ovs_interface_link_state{bridge="br-int",interface="ovn-3cc1e2-0",port="ovn-3cc1e2-0",type="geneve"} 1
ovs_interface_link_state{bridge="br-int",interface="ovn-737fe9-0",port="ovn-737fe9-0",type="geneve"} 1
ovs_interface_link_state{bridge="br-int",interface="ovn-7488f5-0",port="ovn-7488f5-0",type="geneve"} 1
ovs_interface_link_state{bridge="br-int",interface="ovn-87e045-0",port="ovn-87e045-0",type="geneve"} 1
ovs_interface_link_state{bridge="br-int",interface="ovn-acb76a-0",port="ovn-acb76a-0",type="geneve"} 1
ovs_interface_link_state{bridge="br-int",interface="patch-br-int-to-provnet-e0353205-c937-4ebb-af35-a7db0d85c9d3",port="patch-br-int-to-provnet-e0353205-c937-4ebb-af35-a7db0d85c9d3",type="patch"} 1
ovs_interface_link_state{bridge="br-int",interface="tap32b26818-e0",port="tap32b26818-e0",type="system"} 1
ovs_interface_link_state{bridge="br-int",interface="tapba83f025-aa",port="tapba83f025-aa",type="system"} 1

Enable and disable extended PMD metrics
Copy link

Enable extended poll mode driver (PMD) metrics to debug your RHOSO networks, and then disable the metrics when you are finished.

Before you begin
Copy link

  • You can log in to Compute nodes as root.

About this task
Copy link

Note

Extended PMD metrics collection is disabled by default because PMD metrics collection can increase CPU load and impact performance.

Procedure
Copy link

  1. Log in to a Compute node.
  2. Reset and enable extended PMD metrics: 
    # ovs-appctl dpif-netdev/pmd-stats-clear
# ovs-vsctl set Open_vSwitch . other_config:pmd-perf-metrics=true
  3. Disable extended PMD metrics: 
    # ovs-vsctl set Open_vSwitch . other_config:pmd-perf-metrics=false