Chapter 2. Monitor collector metrics

The Vector log collector exposes Prometheus metrics that track log collection performance and health. In deployments with many inputs, outputs, and pipelines, these metrics can exhibit high cardinality. High cardinality increases resource consumption in the Red Hat OpenShift Logging monitoring stack.

Understanding how ClusterLogForwarder configuration affects metrics cardinality helps you balance tenant isolation requirements with monitoring stack stability.

Metrics cardinality refers to the number of unique time series created by a metric. Each unique combination of metric name and label values creates a separate time series. Prometheus must store and query each time series. For example, if a metric has labels component_id and namespace, and there are 100 unique component_id values and 50 unique namespace values, the potential cardinality is 100 × 50 = 5,000 time series for that metric.

High cardinality metrics require the following resources: more memory in Prometheus pods to store time series, more CPU to process queries across many series, more storage to persist historical data, and longer query response times.

The Vector log collector creates internal components for each stage of log processing. Each component is identified by a component_id label in Vector metrics. The number of components grows with the complexity of your ClusterLogForwarder configuration.

A ClusterLogForwarder with the following configuration:

Creates these types of components:

Each input typically requires multiple transformation stages before reaching an output. This creates 5-10 components per input-output path. In this example, you might have 30-50 unique component_id values.

In multitenant environments with many separate inputs, outputs, or pipelines for tenant isolation, the number of components can grow to hundreds:

This configuration can create 400-500 unique component_id values or more.

The following Vector metrics use the component_id label and are most affected by high cardinality:

Histogram metrics are particularly problematic. They create multiple time series per unique label combination (one per bucket plus sum and count).

A single component_id value in a histogram metric with 8 buckets creates:

Total: 10 time series per component_id.

With 450 unique component_id values:

High cardinality from Vector metrics affects the Red Hat OpenShift Logging monitoring stack in several ways:

Consider reviewing your ClusterLogForwarder configuration if:

You can check the current cardinality of Vector metrics by using the Prometheus API. See "Troubleshooting high collector metrics cardinality" for diagnostic procedures.

You can configure the cluster monitoring operator to use metrics collection profiles. These profiles control how many metrics Prometheus collects from the log collector. The minimal profile reduces metrics cardinality by collecting only the metrics needed for alerts, dashboards, and telemetry.

Metrics collection profiles provide a supported method to reduce Prometheus resource consumption without manually editing ServiceMonitor resources. The Red Hat OpenShift Logging Operator automatically reverts manual edits to ServiceMonitor resources.

When you use the minimal metrics collection profile, Prometheus collects only the metrics required for platform functionality: alerts, console dashboards, telemetry, and Horizontal and Vertical Pod Autoscaling. This significantly reduces metrics cardinality compared to the full profile.

In multitenant Red Hat OpenShift Logging clusters, configure logging to isolate logs between tenants while minimizing the impact on the monitoring stack. Balance tenant isolation requirements with the metrics cardinality that your configuration creates.

Instead of creating separate inputs for each tenant or namespace, use label selectors to route logs from multiple sources through a single input.

The following configuration example shows an anti-pattern with separate inputs per namespace:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: tenant-logs
  namespace: openshift-logging
spec:
  inputs:
    - name: tenant-a-logs
      type: application
      application:
        namespaces:
          - tenant-a
    - name: tenant-b-logs
      type: application
      application:
        namespaces:
          - tenant-b
    # ... 98 more tenant inputs
  outputs:
    - name: tenant-a-splunk
      type: splunk
      # ...
    - name: tenant-b-splunk
      type: splunk
      # ...
  pipelines:
    - name: tenant-a-pipeline
      inputRefs:
        - tenant-a-logs
      outputRefs:
        - tenant-a-splunk
    - name: tenant-b-pipeline
      inputRefs:
        - tenant-b-logs
      outputRefs:
        - tenant-b-splunk

Each tenant input creates multiple Vector components. This increases cardinality. This configuration with 100 tenants creates approximately 400-500 unique component_id values.

The following configuration example shows the recommended approach with a single input and filtering:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: tenant-logs
  namespace: openshift-logging
spec:
  inputs:
    - name: all-tenants
      type: application
      application:
        selector:
          matchLabels:
            tenant-logging: "enabled"
  filters:
    - name: tenant-a-filter
      type: drop
      drop:
        - test:
            - field: .kubernetes.namespace_name
              notMatches: "^tenant-a$"
    - name: tenant-b-filter
      type: drop
      drop:
        - test:
            - field: .kubernetes.namespace_name
              notMatches: "^tenant-b$"
  outputs:
    - name: tenant-a-splunk
      type: splunk
      # ...
    - name: tenant-b-splunk
      type: splunk
      # ...
  pipelines:
    - name: tenant-a-pipeline
      inputRefs:
        - all-tenants
      filterRefs:
        - tenant-a-filter
      outputRefs:
        - tenant-a-splunk
    - name: tenant-b-pipeline
      inputRefs:
        - all-tenants
      filterRefs:
        - tenant-b-filter
      outputRefs:
        - tenant-b-splunk

A single input for all tenant logs reduces the component count. Use pod labels to control which pods are included in log collection. This configuration creates far fewer Vector components because there is only one input source.

If multiple tenants send logs to the same destination system (for example, the same Splunk instance or LokiStack), use a single output rather than creating separate outputs per tenant.

The following configuration example shows an anti-pattern with separate outputs per tenant to the same destination:

spec:
  outputs:
    - name: tenant-a-splunk
      type: splunk
      splunk:
        url: https://splunk.example.com:8088
        token:
          secretName: tenant-a-splunk-token
    - name: tenant-b-splunk
      type: splunk
      splunk:
        url: https://splunk.example.com:8088
        token:
          secretName: tenant-b-splunk-token

Both outputs point to the same Splunk instance. This creates duplicate components.

The following configuration example shows the recommended approach with a single output and tenant identification:

spec:
  outputs:
    - name: shared-splunk
      type: splunk
      splunk:
        url: https://splunk.example.com:8088
        token:
          secretName: splunk-token
  pipelines:
    - name: tenant-a-pipeline
      inputRefs:
        - all-tenants
      filterRefs:
        - tenant-a-filter
      outputRefs:
        - shared-splunk
    - name: tenant-b-pipeline
      inputRefs:
        - all-tenants
      filterRefs:
        - tenant-b-filter
      outputRefs:
        - shared-splunk

A single output reduces the component count. Multiple pipelines can share the same output. Tenant isolation is maintained by the namespace information in the log records. Use Splunk, LokiStack, or other destination capabilities to filter and route logs by tenant.

Each pipeline creates additional components for routing and processing. Where possible, combine pipelines that share inputs and outputs.

Separate pipelines are necessary when:

Pipelines can be combined when:

Creating multiple ClusterLogForwarder custom resources increases the overall component count. Each ClusterLogForwarder instance deploys a separate collector pod with its own set of components.

Use multiple ClusterLogForwarder instances when:

A single ClusterLogForwarder is sufficient when:

The following example shows a multitenant logging configuration that balances tenant isolation with low metrics cardinality:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: multitenant-logging
  namespace: openshift-logging
spec:
  serviceAccount:
    name: logcollector
  inputs:
    - name: application-logs
      type: application
      application:
        selector:
          matchExpressions:
            - key: logging-enabled
              operator: In
              values:
                - "true"
    - name: infrastructure-logs
      type: infrastructure
  filters:
    - name: add-tenant-label
      type: detectMultilineException
  outputs:
    - name: tenant-lokistack
      type: lokiStack
      lokiStack:
        target:
          name: logging-loki
          namespace: openshift-logging
    - name: compliance-s3
      type: s3
      s3:
        authentication:
          type: awsAccessKey
          awsAccessKey:
            keyId:
              key: aws_access_key_id
              secretName: s3-secret
            keySecret:
              key: aws_secret_access_key
              secretName: s3-secret
        bucket: audit-logs
        keyPrefix: compliance/
        region: us-east-1
  pipelines:
    - name: application-to-loki
      inputRefs:
        - application-logs
      filterRefs:
        - add-tenant-label
      outputRefs:
        - tenant-lokistack
    - name: infrastructure-to-loki
      inputRefs:
        - infrastructure-logs
      outputRefs:
        - tenant-lokistack
    - name: audit-to-s3
      inputRefs:
        - audit
      outputRefs:
        - compliance-s3

This configuration uses:

This configuration creates approximately 40-60 component_id values, compared to 400-500 values for a per-tenant input/output design.

In LokiStack, tenant isolation is achieved through the namespace labels in the logs. Use LogQL queries to filter logs by tenant:

{kubernetes_namespace_name="tenant-a"}

Use the following rough estimates to predict the cardinality impact of your ClusterLogForwarder configuration:

For example, a ClusterLogForwarder with:

Estimated total: ~70 component_id values

Each histogram metric creates 10 time series per component_id:

These estimates are approximate. Use the diagnostic procedures in "Troubleshoot the collector" to measure actual cardinality.