Chapter 4. Sharding clusters across Argo CD Application Controller replicas

Procedure

1. In the Administrator perspective of the web console, go to Operators Installed Operators.
2. Click Red Hat OpenShift GitOps from the installed operators and go to the Argo CD tab.
3. Click the Argo CD instance where you want to enable the round-robin sharding algorithm, for example, openshift-gitops.

4. Click the YAML tab and edit the YAML file as shown in the following example:

   Example Argo CD instance with round-robin sharding algorithm enabled:

   apiVersion: argoproj.io/v1beta1
   kind: ArgoCD
   metadata:
     name: openshift-gitops
     namespace: openshift-gitops
   spec:
     controller:
       sharding:
         enabled: true
         replicas: 3
         algorithm: round-robin
       logLevel: debug

   where:

   spec.controller.sharding.enabled

   Specifies the value of the sharding.enabled parameter to enable sharding.

   spec.controller.sharding.replicas

   Specifies the number of controller replicas, for example, 3.

   spec.controller.sharding.algorithm

   Specifies the cluster distribution algorithm, for example, round-robin.

   spec.controller.env.value

   Specifies the sharding algorithm to round-robin.

   spec.controller.logLevel

   Specifies the log level as debug so that you can verify to which shard each cluster is attached.

5. Click Save.

   A success notification alert, openshift-gitops has been updated to version <version>, appears.

   Note

   If you edit the default openshift-gitops instance, the Managed resource dialog box is displayed. Click Save again to confirm the changes.

6. Verify that the sharding is enabled with round-robin as the sharding algorithm by performing the following steps:

   1. Go to Workloads StatefulSets.
   2. Select the namespace where you installed the Argo CD instance from the Project drop-down list.
   3. Click <instance_name>-application-controller, for example, openshift-gitops-application-controller, and go to the Pods tab.
   4. Observe the number of created application controller pods. It should correspond with the number of set replicas.

   5. Click on the controller pod you want to examine and go to the Logs tab to view the pod logs.

      Example controller pod logs snippet:

      time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=openshift-gitops version=v2.9.2+c5ea5c4
      time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1"
      time="2023-12-13T09:05:34Z" level=info msg="Using filter function:  round-robin"
      time="2023-12-13T09:05:34Z" level=info msg="Using filter function:  round-robin"
      time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"

      Look for the "Using filter function: round-robin" message.

   6. In the log Search field, search for processed by shard to verify that the cluster distribution across shards is even, as shown in the following example.

      Important

      Ensure that you set the log level to debug to observe these logs.

      Example controller pod logs snippet:

      time="2023-12-13T09:05:34Z" level=debug msg="ClustersList has 3 items"
      time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id= and name=in-cluster to cluster's map"
      time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 and name=in-cluster2 to cluster's map"
      time="2023-12-13T09:05:34Z" level=debug msg="Adding cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w and name=in-cluster3 to cluster's map"
      time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id= will be processed by shard 0"
      time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 will be processed by shard 1"
      time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w will be processed by shard 2"

      In this example, 3 clusters are attached consecutively to shard 0, shard 1, and shard 2.

      Note

      If the number of clusters "C" is a multiple of the number of shard replicas "R", then each shard must have the same number of assigned clusters "N", which is equal to "C" divided by "R". The previous example shows 3 clusters and 3 replicas; therefore, each shard has 1 cluster assigned.

Procedure

1. Enable sharding and set the number of replicas to the wanted value by running the following command:

   $ oc patch argocd <argocd_instance> -n --patch='{"spec":{"controller":{"sharding":{"enabled":true,"replicas":,"algorithm":"round-robin"}}}}' --type=merge

   Example output:

   argocd.argoproj.io/<argocd_instance> patched

2. Verify that the number of Argo CD Application Controller pods corresponds with the number of set replicas by running the following command:

   $ oc get pods -l app.kubernetes.io/name=<argocd_instance>-application-controller -n <namespace>

   Example output:

   NAME                                        READY   STATUS    RESTARTS   AGE
   <argocd_instance>-application-controller-0   1/1     Running   0          11s
   <argocd_instance>-application-controller-1   1/1     Running   0          32s
   <argocd_instance>-application-controller-2   1/1     Running   0          22s

3. Verify that the sharding is enabled with round-robin as the sharding algorithm by running the following command:

   $ oc logs <argocd_application_controller_pod> -n <namespace>

   Example output snippet:

   time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=<namespace> version=v2.9.2+c5ea5c4
   time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1"
   time="2023-12-13T09:05:34Z" level=info msg="Using filter function:  round-robin"
   time="2023-12-13T09:05:34Z" level=info msg="Using filter function:  round-robin"
   time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"

   Verify that the "Using filter function: round-robin" message appears in the output. This message confirms that the Argo CD Application Controller is using the round-robin sharding algorithm.

4. Verify that the cluster distribution across shards is even by performing the following steps:

   1. Set the log level to debug by running the following command:

      $ oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"logLevel":"debug"}}}' --type=merge

      Example output:

      argocd.argoproj.io/<argocd_instance> patched

   2. View the logs and search for processed by shard to observe to which shard each cluster is attached by running the following command:

      $ oc logs <argocd_application_controller_pod> -n <namespace> | grep "processed by shard"

      Example output snippet:

      time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id= will be processed by shard 0"
      time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 will be processed by shard 1"
      time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w will be processed by shard 2"

      In this example, 3 clusters are attached consecutively to shard 0, shard 1, and shard 2.

      Note

      If the number of clusters "C" is a multiple of the number of shard replicas "R", then each shard must have the same number of assigned clusters "N", which is equal to "C" divided by "R". The previous example shows 3 clusters and 3 replicas; therefore, each shard has 1 cluster assigned.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, go to Operators Installed Operators.
2. From the list of Installed Operators, select the Red Hat OpenShift GitOps Operator, and then click the ArgoCD tab.
3. Select the Argo CD instance name for which you want to enable dynamic scaling of shards, for example, openshift-gitops.

4. Click the YAML tab, and then edit and configure the spec.controller.sharding properties as follows:

   Example Argo CD YAML file with dynamic scaling enabled:

   apiVersion: argoproj.io/v1beta1
   kind: ArgoCD
   metadata:
     name: openshift-gitops
     namespace: openshift-gitops
   spec:
     controller:
       sharding:
         dynamicScalingEnabled: true
         minShards: 1
         maxShards: 3
         clustersPerShard: 1

   where:

   spec.controller.sharding.dynamicScalingEnabled

   Set to true to enable dynamic scaling.

   spec.controller.sharding.minShards

   Specifies the minimum number of shards. The value must be set to 1 or greater.

   spec.controller.sharding.maxShards

   Specifies the maximum number of shards. The value must be greater than the value of minShards.

   spec.controller.sharding.clustersPerShard

   Specifies the number of clusters per shard. The value must be set to 1 or greater.

5. Click Save.

   A success notification alert, openshift-gitops has been updated to version <version>, appears.

   Note

   If you edit the default openshift-gitops instance, the Managed resource dialog box is displayed. Click Save again to confirm the changes.

1. Go to Workloads StatefulSets.
2. Select the namespace where the Argo CD instance is deployed from the Project drop-down list, for example, openshift-gitops.
3. Click the name of the StatefulSet object that has the name of the Argo CD instance, for example openshift-gitops-application-controller.
4. Click the Pods tab, and then verify that the number of pods is equal to or greater than the value of minShards that you have set in the Argo CD YAML file.

Procedure

1. Log in to the cluster by using the oc tool as a user with cluster-admin privileges.

2. Enable dynamic scaling by running the following command:

   $ oc patch argocd <argocd_instance> -n <namespace> --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":<value>,"maxShards":<value>,"clustersPerShard":<value>}}}}'

   Example command:

   $ oc patch argocd openshift-gitops -n openshift-gitops --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1}}}}'

   The example command enables dynamic scaling for the openshift-gitops Argo CD instance in the openshift-gitops namespace. It sets the minimum number of shards to 1, the maximum number of shards to 3, and the number of clusters per shard to 1. The values of minShards and clustersPerShard must be set to 1 or greater. The value of maxShards must be equal to or greater than the value of minShards.

   Example output:

   argocd.argoproj.io/openshift-gitops patched

Verification

1. Check the spec.controller.sharding properties of the Argo CD instance:

   $ oc get argocd <argocd_instance> -n <namespace> -o jsonpath='{.spec.controller.sharding}'

   Example command:

   $ oc get argocd openshift-gitops -n openshift-gitops -o jsonpath='{.spec.controller.sharding}'

   Example output when dynamic scaling of shards is enabled:

   {"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1}

2. Optional: Verify that dynamic scaling is enabled by checking the configured spec.controller.sharding properties in the configuration YAML file of the Argo CD instance in the OpenShift Container Platform web console.

3. Check the number of Argo CD Application Controller pods:

   $ oc get pods -n <namespace> -l app.kubernetes.io/name=<argocd_instance>-application-controller

   Example command:

   $ oc get pods -n openshift-gitops -l app.kubernetes.io/name=openshift-gitops-application-controller

   Example output:

   NAME                                           READY   STATUS    RESTARTS   AGE
   openshift-gitops-application-controller-0      1/1     Running   0          2m

   The number of Argo CD Application Controller pods must be greater than or equal to the value of minShards.