1. Documentation
    2. history bug_report picture_as_pdf
×

Using metrics with dashboards and alerts

The Network Observability Operator uses the flowlogs-pipeline to generate metrics from flow logs. You can utilize these metrics by setting custom alerts and viewing dashboards.

Viewing Network Observability metrics dashboards

On the Overview tab in the OpenShift Container Platform console, you can view the overall aggregated metrics of the network traffic flow on the cluster. You can choose to display the information by node, namespace, owner, pod, and service. You can also use filters and display options to further refine the metrics.

Procedure

  1. In the web console Observe → Dashboards, select the Netobserv dashboard.

  2. View network traffic metrics in the following categories, with each having the subset per node, namespace, source, and destination:

    • Byte rates

    • Packet drops

    • DNS

    • RTT

  3. Select the Netobserv/Health dashboard.

  4. View metrics about the health of the Operator in the following categories, with each having the subset per node, namespace, source, and destination.

    • Flows

    • Flows Overhead

    • Flow rates

    • Agents

    • Processor

    • Operator

Infrastructure and Application metrics are shown in a split-view for namespace and workloads.

Network Observability metrics

Metrics generated by the flowlogs-pipeline are configurable in the spec.processor.metrics.includeList of the FlowCollector custom resource to add or remove metrics.

You can also create alerts by using the includeList metrics in Prometheus rules, as shown in the example "Creating alerts".

When looking for these metrics in Prometheus, such as in the Console through Observe → Metrics, or when defining alerts, all the metrics names are prefixed with `netobserv_. For example, `netobserv_namespace_flows_total. Available metrics names are as follows.

includeList metrics names

Names followed by an asterisk * are enabled by default.

  • namespace_egress_bytes_total

  • namespace_egress_packets_total

  • namespace_ingress_bytes_total

  • namespace_ingress_packets_total

  • namespace_flows_total *

  • node_egress_bytes_total

  • node_egress_packets_total

  • node_ingress_bytes_total *

  • node_ingress_packets_total

  • node_flows_total

  • workload_egress_bytes_total

  • workload_egress_packets_total

  • workload_ingress_bytes_total *

  • workload_ingress_packets_total

  • workload_flows_total

PacketDrop metrics names

When the PacketDrop feature is enabled in spec.agent.ebpf.features (with privileged mode), the following additional metrics are available:

  • namespace_drop_bytes_total

  • namespace_drop_packets_total *

  • node_drop_bytes_total

  • node_drop_packets_total

  • workload_drop_bytes_total

  • workload_drop_packets_total

DNS metrics names

When the DNSTracking feature is enabled in spec.agent.ebpf.features, the following additional metrics are available:

  • namespace_dns_latency_seconds *

  • node_dns_latency_seconds

  • workload_dns_latency_seconds

FlowRTT metrics names

When the FlowRTT feature is enabled in spec.agent.ebpf.features, the following additional metrics are available:

  • namespace_rtt_seconds *

  • node_rtt_seconds

  • workload_rtt_seconds

Creating alerts

You can create custom alerting rules for the Netobserv dashboard metrics to trigger alerts when some defined conditions are met.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role or with view permissions for all projects.

  • You have the Network Observability Operator installed.

Procedure

  1. Create a YAML file by clicking the import icon, +.

  2. Add an alerting rule configuration to the YAML file. In the YAML sample that follows, an alert is created for when the cluster ingress traffic reaches a given threshold of 10 MBps per destination workload.

    apiVersion: monitoring.openshift.io/v1
kind: AlertingRule
metadata:
  name: netobserv-alerts
  namespace: openshift-monitoring
spec:
  groups:
  - name: NetObservAlerts
    rules:
    - alert: NetObservIncomingBandwidth
      annotations:
        message: |-
          {{ $labels.job }}: incoming traffic exceeding 10 MBps for 30s on {{ $labels.DstK8S_OwnerType }} {{ $labels.DstK8S_OwnerName }} ({{ $labels.DstK8S_Namespace }}).
        summary: "High incoming traffic."
      expr: sum(rate(netobserv_workload_ingress_bytes_total     {SrcK8S_Namespace="openshift-ingress"}[1m])) by (job, DstK8S_Namespace, DstK8S_OwnerName, DstK8S_OwnerType) > 10000000      (1)
      for: 30s
      labels:
        severity: warning
    1 The netobserv_workload_ingress_bytes_total metric is enabled by default in spec.processor.metrics.includeList.

  3. Click Create to apply the configuration file to the cluster.

Additional resources