1. Documentation
    2. history bug_report picture_as_pdf
×

Topology Aware Lifecycle Manager for cluster updates

You can use the Topology Aware Lifecycle Manager (TALM) to manage the software lifecycle of multiple single-node OpenShift clusters. TALM uses Red Hat Advanced Cluster Management (RHACM) policies to perform changes on the target clusters.

Topology Aware Lifecycle Manager is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

About the Topology Aware Lifecycle Manager configuration

The Topology Aware Lifecycle Manager (TALM) manages the deployment of Red Hat Advanced Cluster Management (RHACM) policies for one or more OpenShift Container Platform clusters. Using TALM in a large network of clusters allows the phased rollout of policies to the clusters in limited batches. This helps to minimize possible service disruptions when updating. With TALM, you can control the following actions:

  • The timing of the update

  • The number of RHACM-managed clusters

  • The subset of managed clusters to apply the policies to

  • The update order of the clusters

  • The set of policies remediated to the cluster

  • The order of policies remediated to the cluster

TALM supports the orchestration of the OpenShift Container Platform y-stream and z-stream updates, and day-two operations on y-streams and z-streams.

About managed policies used with Topology Aware Lifecycle Manager

The Topology Aware Lifecycle Manager (TALM) uses RHACM policies for cluster updates.

TALM can be used to manage the rollout of any policy CR where the remediationAction field is set to inform. Supported use cases include the following:

  • Manual user creation of policy CRs

  • Automatically generated policies from the PolicyGenTemplate custom resource definition (CRD)

For policies that update an Operator subscription with manual approval, TALM provides additional functionality that approves the installation of the updated Operator.

For more information about managed policies, see Policy Overview in the RHACM documentation.

For more information about the PolicyGenTemplate CRD, see the "About the PolicyGenTemplate" section in "Deploying distributed units at scale in a disconnected environment".

Installing the Topology Aware Lifecycle Manager by using the web console

You can use the OpenShift Container Platform web console to install the Topology Aware Lifecycle Manager.

Prerequisites

  • Install the latest version of the RHACM Operator.

  • Set up a hub cluster with disconnected regitry.

  • Log in as a user with cluster-admin privileges.

Procedure

  1. In the OpenShift Container Platform web console, navigate to Operators → OperatorHub.

  2. Search for the Topology Aware Lifecycle Manager from the list of available Operators, and then click Install.

  3. Keep the default selection of Installation mode ["All namespaces on the cluster (default)"] and Installed Namespace ("openshift-operators") to ensure that the Operator is installed properly.

  4. Click Install.

Verification

To confirm that the installation is successful:

  1. Navigate to the Operators → Installed Operators page.

  2. Check that the Operator is installed in the All Namespaces namespace and its status is Succeeded.

If the Operator is not installed successfully:

  1. Navigate to the Operators → Installed Operators page and inspect the Status column for any errors or failures.

  2. Navigate to the Workloads → Pods page and check the logs in any containers in the cluster-group-upgrades-controller-manager pod that are reporting issues.

Installing the Topology Aware Lifecycle Manager by using the CLI

You can use the OpenShift CLI (oc) to install the Topology Aware Lifecycle Manager (TALM).

Prerequisites

  • Install the OpenShift CLI (oc).

  • Install the latest version of the RHACM Operator.

  • Set up a hub cluster with disconnected registry.

  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a Subscription CR:

    1. Define the Subscription CR and save the YAML file, for example, talm-subscription.yaml:

      apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-topology-aware-lifecycle-manager-subscription
  namespace: openshift-operators
spec:
  channel: "stable"
  name: topology-aware-lifecycle-manager
  source: redhat-operators
  sourceNamespace: openshift-marketplace

    2. Create the Subscription CR by running the following command:

      $ oc create -f talm-subscription.yaml
Verification

  1. Verify that the installation succeeded by inspecting the CSV resource:

    $ oc get csv -n openshift-operators
    Example output
    NAME                                                   DISPLAY                            VERSION               REPLACES                           PHASE
topology-aware-lifecycle-manager.4.11.x   Topology Aware Lifecycle Manager   4.11.x                                      Succeeded

  2. Verify that the TALM is up and running:

    $ oc get deploy -n openshift-operators
    Example output
    NAMESPACE                                          NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
openshift-operators                                cluster-group-upgrades-controller-manager        1/1     1            1           14s

About the ClusterGroupUpgrade CR

The Topology Aware Lifecycle Manager (TALM) builds the remediation plan from the ClusterGroupUpgrade CR for a group of clusters. You can define the following specifications in a ClusterGroupUpgrade CR:

  • Clusters in the group

  • Blocking ClusterGroupUpgrade CRs

  • Applicable list of managed policies

  • Number of concurrent updates

  • Applicable canary updates

  • Actions to perform before and after the update

  • Update timing

As TALM works through remediation of the policies to the specified clusters, the ClusterGroupUpgrade CR can have the following states:

  • UpgradeNotStarted

  • UpgradeCannotStart

  • UpgradeNotComplete

  • UpgradeTimedOut

  • UpgradeCompleted

  • PrecachingRequired

After TALM completes a cluster update, the cluster does not update again under the control of the same ClusterGroupUpgrade CR. You must create a new ClusterGroupUpgrade CR in the following cases:

  • When you need to update the cluster again

  • When the cluster changes to non-compliant with the inform policy after being updated

The UpgradeNotStarted state

The initial state of the ClusterGroupUpgrade CR is UpgradeNotStarted.

TALM builds a remediation plan based on the following fields:

  • The clusterSelector field specifies the labels of the clusters that you want to update.

  • The clusters field specifies a list of clusters to update.

  • The canaries field specifies the clusters for canary updates.

  • The maxConcurrency field specifies the number of clusters to update in a batch.

You can use the clusters and the clusterSelector fields together to create a combined list of clusters.

The remediation plan starts with the clusters listed in the canaries field. Each canary cluster forms a single-cluster batch.

Any failures during the update of a canary cluster stops the update process.

The ClusterGroupUpgrade CR transitions to the UpgradeNotCompleted state after the remediation plan is successfully created and after the enable field is set to true. At this point, TALM starts to update the non-compliant clusters with the specified managed policies.

You can only make changes to the spec fields if the ClusterGroupUpgrade CR is either in the UpgradeNotStarted or the UpgradeCannotStart state.
Sample ClusterGroupUpgrade CR in the UpgradeNotStarted state
apiVersion: ran.openshift.io/v1alpha1
kind: ClusterGroupUpgrade
metadata:
  name: cgu-upgrade-complete
  namespace: default
spec:
  clusters: (1)
  - spoke1
  enable: false
  managedPolicies: (2)
  - policy1-common-cluster-version-policy
  - policy2-common-nto-sub-policy
  remediationStrategy: (3)
    canaries: (4)
      - spoke1
    maxConcurrency: 1 (5)
    timeout: 240
status: (6)
  conditions:
  - message: The ClusterGroupUpgrade CR is not enabled
    reason: UpgradeNotStarted
    status: "False"
    type: Ready
  copiedPolicies:
  - cgu-upgrade-complete-policy1-common-cluster-version-policy
  - cgu-upgrade-complete-policy2-common-nto-sub-policy
  managedPoliciesForUpgrade:
  - name: policy1-common-cluster-version-policy
    namespace: default
  - name: policy2-common-nto-sub-policy
    namespace: default
  placementBindings:
  - cgu-upgrade-complete-policy1-common-cluster-version-policy
  - cgu-upgrade-complete-policy2-common-nto-sub-policy
  placementRules:
  - cgu-upgrade-complete-policy1-common-cluster-version-policy
  - cgu-upgrade-complete-policy2-common-nto-sub-policy
  remediationPlan:
  - - spoke1
1 Defines the list of clusters to update.
2 Lists the user-defined set of policies to remediate.
3 Defines the specifics of the cluster updates.
4 Defines the clusters for canary updates.
5 Defines the maximum number of concurrent updates in a batch. The number of remediation batches is the number of canary clusters, plus the number of clusters, except the canary clusters, divided by the maxConcurrency value. The clusters that are already compliant with all the managed policies are excluded from the remediation plan.
6 Displays information about the status of the updates.

The UpgradeCannotStart state

In the UpgradeCannotStart state, the update cannot start because of the following reasons:

  • Blocking CRs are missing from the system

  • Blocking CRs have not yet finished

The UpgradeNotCompleted state

In the UpgradeNotCompleted state, TALM enforces the policies following the remediation plan defined in the UpgradeNotStarted state.

Enforcing the policies for subsequent batches starts immediately after all the clusters of the current batch are compliant with all the managed policies. If the batch times out, TALM moves on to the next batch. The timeout value of a batch is the spec.timeout field divided by the number of batches in the remediation plan.

The managed policies apply in the order that they are listed in the managedPolicies field in the ClusterGroupUpgrade CR. One managed policy is applied to the specified clusters at a time. After the specified clusters comply with the current policy, the next managed policy is applied to the next non-compliant cluster.
Sample ClusterGroupUpgrade CR in the UpgradeNotCompleted state
apiVersion: ran.openshift.io/v1alpha1
kind: ClusterGroupUpgrade
metadata:
  name: cgu-upgrade-complete
  namespace: default
spec:
  clusters:
  - spoke1
  enable: true (1)
  managedPolicies:
  - policy1-common-cluster-version-policy
  - policy2-common-nto-sub-policy
  remediationStrategy:
    maxConcurrency: 1
    timeout: 240
status: (2)
  conditions:
  - message: The ClusterGroupUpgrade CR has upgrade policies that are still non compliant
    reason: UpgradeNotCompleted
    status: "False"
    type: Ready
  copiedPolicies:
  - cgu-upgrade-complete-policy1-common-cluster-version-policy
  - cgu-upgrade-complete-policy2-common-nto-sub-policy
  managedPoliciesForUpgrade:
  - name: policy1-common-cluster-version-policy
    namespace: default
  - name: policy2-common-nto-sub-policy
    namespace: default
  placementBindings:
  - cgu-upgrade-complete-policy1-common-cluster-version-policy
  - cgu-upgrade-complete-policy2-common-nto-sub-policy
  placementRules:
  - cgu-upgrade-complete-policy1-common-cluster-version-policy
  - cgu-upgrade-complete-policy2-common-nto-sub-policy
  remediationPlan:
  - - spoke1
  status:
    currentBatch: 1
    remediationPlanForBatch: (3)
      spoke1: 0
1 The update starts when the value of the spec.enable field is true.
2