Chapter 13. Policy controller advanced configuration

The governance framework ManagedClusterAddOn supports policy-evaluation-concurrency, client-qps, and client-burst annotations to control evaluation concurrency and API client rate on each managed cluster.

The policy-evaluation-concurrency annotation controls how many policies the governance framework evaluates concurrently on each managed cluster. Apply this annotation to the ManagedClusterAddOn named governance-policy-framework in the managed cluster namespace on the hub cluster. The default value is 2. Valid values are nonzero integers specified as quoted strings.

The following YAML example sets concurrency to 2 on the managed cluster named cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: governance-policy-framework
  namespace: cluster1
  annotations:
    policy-evaluation-concurrency: "2"
spec:
  installNamespace: open-cluster-management-agent-addon

The client-qps and client-burst annotations control the rate of API client requests the governance framework makes on each managed cluster. Apply these annotations to the ManagedClusterAddOn named governance-policy-framework in the managed cluster namespace on the hub cluster. Valid values are nonzero integers specified as quoted strings.

The following YAML example sets QPS to 30 and burst to 45 on the managed cluster named cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: governance-policy-framework
  namespace: cluster1
  annotations:
    client-qps: "30"
    client-burst: "45"
spec:
  installNamespace: open-cluster-management-agent-addon

The configuration policy controller ManagedClusterAddOn supports a policy-evaluation-concurrency annotation to control how many configuration policies are evaluated concurrently on each managed cluster.

The policy-evaluation-concurrency annotation controls how many configuration policies the controller evaluates concurrently on each managed cluster. Apply this annotation to the ManagedClusterAddOn named config-policy-controller in the managed cluster namespace on the hub cluster. The default value is 2. Valid values are nonzero integers specified as quoted strings.

Note: Increased concurrency values increase CPU and memory utilization on the config-policy-controller pod, Kubernetes API server, and OpenShift API server.

The following YAML example sets concurrency to 5 on the managed cluster named cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    policy-evaluation-concurrency: "5"
spec:
  installNamespace: open-cluster-management-agent-addon

The configuration policy controller ManagedClusterAddOn supports client-qps and client-burst annotations to control the rate of requests to the Kubernetes API server on each managed cluster.

The client-qps and client-burst annotations control the rate of requests the configuration policy controller makes to the Kubernetes API server on each managed cluster. Apply these annotations to the ManagedClusterAddOn named config-policy-controller in the managed cluster namespace on the hub cluster. By default, the request rate scales with the policy-evaluation-concurrency setting at 30 queries per second (QPS) with a burst value of 45. Valid values are nonzero integers specified as quoted strings.

Note: Increased request rates improve configuration policy controller responsiveness and increase CPU and memory utilization on the Kubernetes API server and OpenShift API server.

The following YAML example sets QPS to 20 and burst to 100 on the managed cluster named cluster1:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    client-qps: "20"
    client-burst: "100"
spec:
  installNamespace: open-cluster-management-agent-addon

You can adjust policy controller log levels through log-level annotations on ManagedClusterAddOn resources and use recordDiff: Log on ConfigurationPolicy object templates for diff logging.

When you configure and collect debug logs for each policy controller, you can adjust the log level.

Note: Reducing the volume of debug logs means there is less information displayed from the logs.

You can reduce the debug logs emitted by the policy controllers to be display error-only bugs in the logs. To reduce the debug logs, set the debug log value to -1 in the annotation. See what each value represents:

To receive the second level of debugging information for the Kubernetes configuration controller, add the log-level annotation with the value of 2 to the ManagedClusterAddOn custom resource. By default, the log-level is set to 0, which means you receive informative messages. View the following example:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: cluster1
  annotations:
    log-level: "2"
spec:
  installNamespace: open-cluster-management-agent-addon

Additionally, for each spec.object-template[] in a ConfigurationPolicy resource, you can set the parameter recordDiff to Log. The difference between the objectDefinition and the object on the managed cluster is logged in the config-policy-controller pod on the managed cluster. View the following example:

This ConfigurationPolicy resource with recordDiff: Log:

apiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
  name: my-config-policy
spec:
  object-templates:
  - complianceType: musthave
    recordDiff: Log
    objectDefinition:
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: my-configmap
      data:
        fieldToUpdate: "2"

If the ConfigMap resource on the cluster lists fieldToUpdate: "1", then the diff appears in the config-policy-controller pod logs with the following information:

Logging the diff:
--- default/my-configmap : existing
+++ default/my-configmap : updated
@@ -2,3 +2,3 @@
 data:
-  fieldToUpdate: "1"
+  fieldToUpdate: "2"
 kind: ConfigMap

+ * Avoid logging the difference for a secure object. The difference is logged in plain text.

After you apply configuration changes, the ManifestApplied condition on ManagedClusterAddOn confirms the controller received the update.