Chapter 9. Advanced managed cluster configuration with SiteConfig resources

Procedure

1. Create a set of extra manifest CRs that the GitOps ZTP pipeline uses to customize the cluster installs.

2. In your custom /siteconfig directory, create a subdirectory /custom-manifest for your extra manifests. The following example illustrates a sample /siteconfig with /custom-manifest folder:

   siteconfig
   ├── site1-sno-du.yaml
   ├── site2-standard-du.yaml
   ├── extra-manifest/
   └── custom-manifest
       └── 01-example-machine-config.yaml

   Note

   The subdirectory names /custom-manifest and /extra-manifest used throughout are example names only. There is no requirement to use these names and no restriction on how you name these subdirectories. In this example /extra-manifest refers to the Git subdirectory that stores the contents of /extra-manifest from the ztp-site-generate container.

3. Add your custom extra manifest CRs to the siteconfig/custom-manifest directory.

4. In your SiteConfig CR, enter the directory name in the extraManifests.searchPaths field, for example:

   clusters:
   - clusterName: "example-sno"
     networkType: "OVNKubernetes"
     extraManifests:
       searchPaths:
         - extra-manifest/
         - custom-manifest/

   • The extra-manifest/ folder contains the manifests copied from the ztp-site-generate container.
   • The custom-manifest/ folder contains your custom manifests.

5. Save the SiteConfig, /extra-manifest, and /custom-manifest CRs, and push them to the site configuration repo.

   During cluster provisioning, the GitOps ZTP pipeline appends the CRs in the /custom-manifest directory to the default set of extra manifests stored in extra-manifest/.

   Note

   As of version 4.14 extraManifestPath is subject to a deprecation warning.

   While extraManifestPath is still supported, we recommend that you use extraManifests.searchPaths. If you define extraManifests.searchPaths in the SiteConfig file, the GitOps ZTP pipeline does not fetch manifests from the ztp-site-generate container during site installation.

   If you define both extraManifestPath and extraManifests.searchPaths in the Siteconfig CR, the setting defined for extraManifests.searchPaths takes precedence.

   It is strongly recommended that you extract the contents of /extra-manifest from the ztp-site-generate container and push it to the Git repository.

Procedure

1. In your /clusterinstance directory, create an extra-manifest subdirectory and save the MachineConfig object to a file in this directory, for example, extra-manifest/10-br-ex-master.yaml.

2. Create or update the kustomization.yaml file to use configMapGenerator to package the MachineConfig object into a ConfigMap:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - site1-sno-du.yaml
   configMapGenerator:
     - name: br-ex-extra-manifest
       namespace: site1-sno-du
       files:
         - extra-manifest/10-br-ex-master.yaml
   generatorOptions:
     disableNameSuffixHash: true

   • The configMapGenerator.namespace value must match the ClusterInstance namespace.
   • Setting generatorOptions.disableNameSuffixHash to true disables the hash suffix so the ConfigMap name is predictable.

3. In your ClusterInstance CR, reference the ConfigMap in the extraManifestsRefs field:

   apiVersion: siteconfig.open-cluster-management.io/v1alpha1
   kind: ClusterInstance
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusterName: "site1-sno-du"
     networkType: "OVNKubernetes"
     extraManifestsRefs:
       - name: br-ex-extra-manifest
     # ...

4. Commit the ClusterInstance CR, MachineConfig object, and kustomization.yaml file to your Git repository and push the changes.

   During cluster provisioning, the SiteConfig Operator applies the MachineConfig object as an extra manifest, and the customized br-ex bridge configuration is applied to each node without requiring a reboot.

Procedure

1. In your SiteConfig CR, set the machine network MTU on the network interface for all nodes.

   The following example configures a VLAN interface with MTU 9100:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusters:
       - clusterName: "site1-sno-du"
         nodes:
           - hostName: "node1.example.com"
             nodeNetwork:
               interfaces:
                 - name: "bond0.120"
                   macAddress: "00:00:00:00:00:00"
               config:
                 interfaces:
                   - name: bond0.120
                     type: vlan
                     state: up
                     mtu: 9100
                     ipv4:
                       enabled: true
                       dhcp: false
                       address:
                         - ip: "192.168.120.15"
                           prefix-length: 25
                     vlan:
                       base-iface: bond0
                       id: 120
     # ...

2. Create a Network CR manifest file named set-cluster-mtu.yaml that sets the cluster network MTU:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     defaultNetwork:
       ovnKubernetesConfig:
         mtu: 8900

   where:

   mtu

   Specifies the cluster network MTU value. This value must be at least 100 bytes less than the machine network MTU. In this example, the value is 200 bytes less than the machine network MTU of 9100 to allow headroom for IPsec and other future requirements.

3. In your siteconfig directory, place the manifest file in a custom extra manifests subdirectory:

   siteconfig/
   ├── site1-sno-du.yaml
   ├── extra-manifest/
   └── custom-manifest/
       └── set-cluster-mtu.yaml

4. In your SiteConfig CR, enter the directory name in the extraManifests.searchPaths field:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     clusters:
       - clusterName: "site1-sno-du"
         networkType: "OVNKubernetes"
         extraManifests:
           searchPaths:
             - extra-manifest/
             - custom-manifest/
     # ...

5. Commit the SiteConfig CR and the set-cluster-mtu.yaml manifest to your Git repository and push the changes.

   During cluster provisioning, the GitOps ZTP pipeline applies the Network CR as an extra manifest, and the CNO uses the specified MTU value instead of auto-calculating it.

Procedure

1. To prevent the GitOps ZTP pipeline from applying the 03-sctp-machine-config-worker.yaml CR file, apply the following YAML in the SiteConfig CR:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "site1-sno-du"
     namespace: "site1-sno-du"
   spec:
     baseDomain: "example.com"
     pullSecretRef:
       name: "assisted-deployment-pull-secret"
     clusterImageSetNameRef: "openshift-4.20"
     sshPublicKey: "<ssh_public_key>"
     clusters:
   - clusterName: "site1-sno-du"
     extraManifests:
       filter:
         exclude:
           - 03-sctp-machine-config-worker.yaml

   The GitOps ZTP pipeline skips the 03-sctp-machine-config-worker.yaml CR during installation. All other CRs in /source-crs/extra-manifest are applied.

2. Save the SiteConfig CR and push the changes to the site configuration repository.

   The GitOps ZTP pipeline monitors and adjusts what CRs it applies based on the SiteConfig filter instructions.

3. Optional: To prevent the GitOps ZTP pipeline from applying all the /source-crs/extra-manifest CRs during cluster installation, apply the following YAML in the SiteConfig CR:

   - clusterName: "site1-sno-du"
     extraManifests:
       filter:
         inclusionDefault: exclude

4. Optional: To exclude all the /source-crs/extra-manifest RAN CRs and instead include a custom CR file during installation, edit the custom SiteConfig CR to set the custom manifests folder and the include file, for example:

   clusters:
   - clusterName: "site1-sno-du"
     extraManifestPath: "<custom_manifest_folder>" 

   1

   
     extraManifests:
       filter:
         inclusionDefault: exclude  

   2

   
         include:
           - custom-sctp-machine-config-worker.yaml

   1

   Replace <custom_manifest_folder> with the name of the folder that contains the custom installation CRs, for example, user-custom-manifest/.

   2

   Set inclusionDefault to exclude to prevent the GitOps ZTP pipeline from applying the files in /source-crs/extra-manifest during installation.

   The following example illustrates the custom folder structure:

   siteconfig
     ├── site1-sno-du.yaml
     └── user-custom-manifest
           └── custom-sctp-machine-config-worker.yaml

Procedure

1. Update the SiteConfig CR to include the bmac.agent-install.openshift.io/remove-agent-and-node-on-delete=true annotation and push the changes to the Git repository:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "cnfdf20"
     namespace: "cnfdf20"
   spec:
     clusters:
       nodes:
       - hostname: node6
         role: "worker"
         crAnnotations:
           add:
             BareMetalHost:
               bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: true
   # ...

2. Verify that the BareMetalHost object is annotated by running the following command:

   oc get bmh -n <managed-cluster-namespace> <bmh-object> -ojsonpath='{.metadata}' | jq -r '.annotations["bmac.agent-install.openshift.io/remove-agent-and-node-on-delete"]'

   Example output

   true

3. Suppress the generation of the BareMetalHost CR by updating the SiteConfig CR to include the crSuppression.BareMetalHost annotation:

   apiVersion: ran.openshift.io/v1
   kind: SiteConfig
   metadata:
     name: "cnfdf20"
     namespace: "cnfdf20"
   spec:
     clusters:
     - nodes:
       - hostName: node6
         role: "worker"
         crSuppression:
         - BareMetalHost
   # ...

4. Push the changes to the Git repository and wait for deprovisioning to start. The status of the BareMetalHost CR should change to deprovisioning. Wait for the BareMetalHost to finish deprovisioning, and be fully deleted.

Verification

1. Verify that the BareMetalHost and Agent CRs for the worker node have been deleted from the hub cluster by running the following commands:

   $ oc get bmh -n <cluster-ns>

   $ oc get agent -n <cluster-ns>

2. Verify that the node record has been deleted from the spoke cluster by running the following command:

   $ oc get nodes

   Note

   If you are working with secrets, deleting a secret too early can cause an issue because ArgoCD needs the secret to complete resynchronization after deletion. Delete the secret only after the node cleanup, when the current ArgoCD synchronization is complete.