Migrate the Red Hat Ceph Storage cluster

In the context of data plane adoption, where the Red Hat OpenStack Platform (RHOSP) services are redeployed in Red Hat OpenShift Container Platform (RHOCP), you migrate a director-deployed Red Hat Ceph Storage cluster by using a process called “externalizing” the Red Hat Ceph Storage cluster.

There are two deployment topologies that include an internal Red Hat Ceph Storage cluster:

  • RHOSP includes dedicated Red Hat Ceph Storage nodes to host object storage daemons (OSDs)
  • Hyperconverged Infrastructure (HCI), where Compute and Storage services are colocated on hyperconverged nodes

In either scenario, there are some Red Hat Ceph Storage processes that are deployed on RHOSP Controller nodes: Red Hat Ceph Storage monitors, Ceph Object Gateway (RGW), Rados Block Device (RBD), Ceph Metadata Server (MDS), Ceph Dashboard, and NFS Ganesha. To migrate your Red Hat Ceph Storage cluster, you must decommission the Controller nodes and move the Red Hat Ceph Storage daemons to a set of target nodes that are already part of the Red Hat Ceph Storage cluster.

Prerequisites
Copy link

  • Before you begin the migration, complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see "Red Hat Ceph Storage prerequisites" in the Adoption overview chapter.

Red Hat Ceph Storage migration considerations for DCN deployments
Copy link

Before you adopt your Distributed Compute Node (DCN) deployments that host Red Hat Ceph Storage clusters on Compute nodes at edge sites so that your architecture runs on Red Hat OpenStack Services on OpenShift (RHOSO) (RHOSO), be aware of important considerations.

Supported edge storage topologies

DCN deployments support the following storage topologies at edge sites:

  • Hyperconverged Infrastructure (HCI): Red Hat Ceph Storage daemons run on Compute nodes at each edge site.
  • Director-deployed dedicated storage: Red Hat Ceph Storage runs on separate storage nodes deployed by director.
  • External Red Hat Ceph Storage cluster: Edge sites connect to pre-existing Red Hat Ceph Storage clusters not managed by director.
Central site Red Hat Ceph Storage migration
For the central site, migrate Red Hat Ceph Storage daemons from the RHOSP Controller nodes by using the same process as a non-DCN deployment. For more information, see Red Hat Ceph Storage daemon cardinality.
Edge site Red Hat Ceph Storage migration

For edge sites that use HCI or director-deployed dedicated storage, the Red Hat Ceph Storage daemons can continue to run on their current nodes without migration. The Compute nodes or dedicated storage nodes at edge sites are not decommissioned during adoption, so the Red Hat Ceph Storage daemons remain operational.

For edge sites that use external Red Hat Ceph Storage clusters, no migration is required because the Red Hat Ceph Storage cluster is not managed by director.

Red Hat Ceph Storage back-end configuration and key distribution

In a DCN deployment, each site has its own Red Hat Ceph Storage cluster with its own configuration file and Red Hat Ceph Storage keyring. These must be stored in Kubernetes secrets and mounted into the appropriate Red Hat OpenStack Services on OpenShift (RHOSO) service pods.

Rather than storing all Red Hat Ceph Storage keys in a single secret accessible to every pod, the recommended approach is to create one secret per site containing only the keys that site actually needs. This limits the security impact if a site is compromised: a pod at an edge site can authenticate only to its local Red Hat Ceph Storage cluster and the central cluster, not to the Red Hat Ceph Storage keyrings of other edge sites.

The key distribution rule for N sites is:

  • The central site (site 0) receives the Red Hat Ceph Storage keys and configuration for all clusters, because central services such as Image service use the split back end and must be able to copy images to and from any site.
  • Each edge site (site 1 through N) receives only the keys for the central cluster and its own local cluster.

    For example, in a three-site deployment with a central site and two edge sites:

    ceph-conf-central  ->  central.conf + central.keyring
                       dcn1.conf    + dcn1.keyring
                       dcn2.conf    + dcn2.keyring

ceph-conf-dcn1     ->  central.conf + central.keyring
                       dcn1.conf    + dcn1.keyring

ceph-conf-dcn2     ->  central.conf + central.keyring
                       dcn2.conf    + dcn2.keyring

    The per-site secrets are created and then mounted into the appropriate pods by using extraMounts propagation labels. The procedure in Configuring a Red Hat Ceph Storage back end covers both creating the secrets and applying the propagation labels so that each pod receives only its site-specific keys.

Red Hat Ceph Storage daemon cardinality
Copy link

Red Hat Ceph Storage 7 and later applies strict constraints in the way daemons can be colocated within the same node. Your topology depends on the available hardware and the amount of Red Hat Ceph Storage services in the Controller nodes that you retire.

The amount of services that you can migrate depends on the amount of available nodes in the cluster.

The following diagrams show the distribution of Red Hat Ceph Storage daemons on Red Hat Ceph Storage nodes where at least 3 nodes are required.

For more information, see Red Hat Ceph Storage: Supported configurations.

  • The following scenario includes only RGW and RBD, without the Red Hat Ceph Storage dashboard: 
    |    |                     |             |
|----|---------------------|-------------|
| osd | mon/mgr/crash      | rgw/ingress |
| osd | mon/mgr/crash      | rgw/ingress |
| osd | mon/mgr/crash      | rgw/ingress |
  • With the Red Hat Ceph Storage dashboard, but without Shared File Systems service (manila), at least 4 nodes are required. The Red Hat Ceph Storage dashboard has no failover: 
    |     |                     |             |
|-----|---------------------|-------------|
| osd | mon/mgr/crash | rgw/ingress       |
| osd | mon/mgr/crash | rgw/ingress       |
| osd | mon/mgr/crash | dashboard/grafana |
| osd | rgw/ingress   | (free)            |
  • With the Red Hat Ceph Storage dashboard and the Shared File Systems service, a minimum of 5 nodes are required, and the Red Hat Ceph Storage dashboard has no failover: 
    |     |                     |                         |
|-----|---------------------|-------------------------|
| osd | mon/mgr/crash       | rgw/ingress             |
| osd | mon/mgr/crash       | rgw/ingress             |
| osd | mon/mgr/crash       | mds/ganesha/ingress     |
| osd | rgw/ingress         | mds/ganesha/ingress     |
| osd | mds/ganesha/ingress | dashboard/grafana       |

Migrate the monitoring stack to new Red Hat Ceph Storage nodes
Copy link

To migrate the monitoring stack to target nodes, add the monitoring label to your existing nodes and update each daemon's configuration. You do not need to migrate node exporters. These daemons are deployed across the nodes that are part of the Red Hat Ceph Storage cluster (the placement is '*').

Note
Depending on the target nodes and the number of deployed or active daemons, you can either relocate the existing containers to the target nodes, or select a subset of nodes that host the monitoring stack daemons. High availability (HA) is not supported. Reducing the placement with count: 1 allows you to migrate the existing daemons in a Hyperconverged Infrastructure, or hardware-limited, scenario without impacting other services.

The Red Hat Ceph Storage Dashboard module adds web-based monitoring and administration to the Ceph Manager. With director-deployed Red Hat Ceph Storage, the Red Hat Ceph Storage Dashboard is enabled as part of the overcloud deploy and is composed of the following components:

  • Ceph Manager module
  • Grafana
  • Prometheus
  • Alertmanager
  • Node exporter

The Red Hat Ceph Storage Dashboard containers are included through tripleo-container-image-prepare parameters, and high availability (HA) relies on HAProxy and Pacemaker to be deployed on the Red Hat OpenStack Platform (RHOSP) environment. For an external Red Hat Ceph Storage cluster, HA is not supported.

Prerequisites
Copy link

You migrate and relocate the Ceph Monitoring components to free Controller nodes. Before you begin the migration, complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see "Red Hat Ceph Storage prerequisites" in the "Adoption overview" chapter.

Migrate the existing daemons to the target nodes
Copy link

This procedure is an example of an environment with 3 Red Hat Ceph Storage nodes or ComputeHCI nodes. This example extends the monitoring labels to all the Red Hat Ceph Storage or ComputeHCI nodes that are part of the cluster, which maintains 3 placements for the target nodes.

Before you begin
Copy link

  • Confirm that the firewall rules are in place and the ports are open for a given monitoring stack service.

Procedure
Copy link

  1. Add the monitoring label to all the Red Hat Ceph Storage or ComputeHCI nodes in the cluster: 
    for item in $(sudo cephadm shell --  ceph orch host ls --format json | jq -r '.[].hostname'); do
    sudo cephadm shell -- ceph orch host label add  $item monitoring;
done
  2. Verify that all the hosts on the target nodes have the monitoring label: 
    [tripleo-admin@controller-0 ~]$ sudo cephadm shell -- ceph orch host ls

HOST                        ADDR           LABELS
cephstorage-0.redhat.local  192.168.24.11  osd monitoring
cephstorage-1.redhat.local  192.168.24.12  osd monitoring
cephstorage-2.redhat.local  192.168.24.47  osd monitoring
controller-0.redhat.local   192.168.24.35  _admin mon mgr monitoring
controller-1.redhat.local   192.168.24.53  mon _admin mgr monitoring
controller-2.redhat.local   192.168.24.10  mon _admin mgr monitoring
  3. Remove the labels from the Controller nodes: 
    $ for i in 0 1 2; do sudo cephadm shell -- ceph orch host label rm "controller-$i.redhat.local" monitoring; done

Removed label monitoring from host controller-0.redhat.local
Removed label monitoring from host controller-1.redhat.local
Removed label monitoring from host controller-2.redhat.local
  4. Dump the current monitoring stack spec: 
    function export_spec {
    local component="$1"
    local target_dir="$2"
    sudo cephadm shell -- ceph orch ls --export "$component" > "$target_dir/$component"
}

SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
mkdir -p ${SPEC_DIR}
for m in grafana prometheus alertmanager; do
    export_spec "$m" "$SPEC_DIR"
done
  5. For each daemon, edit the current spec and replace the placement.hosts: section with the placement.label: section, for example: 
    service_type: grafana
service_name: grafana
placement:
  label: monitoring
networks:
- 172.17.3.0/24
spec:
  port: 3100

    This step also applies to Prometheus and Alertmanager specs.

  6. Apply the new monitoring spec to relocate the monitoring stack daemons: 
    SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
function migrate_daemon {
    local component="$1"
    local target_dir="$2"
    sudo cephadm shell -m "$target_dir" -- ceph orch apply -i /mnt/ceph_specs/$component
}
for m in grafana prometheus alertmanager; do
    migrate_daemon  "$m" "$SPEC_DIR"
done
  7. Verify that the daemons are deployed on the expected nodes: 
    [ceph: root@controller-0 /]# ceph orch ps | grep -iE "(prome|alert|grafa)"
alertmanager.cephstorage-2  cephstorage-2.redhat.local  172.17.3.144:9093,9094
grafana.cephstorage-0       cephstorage-0.redhat.local  172.17.3.83:3100
prometheus.cephstorage-1    cephstorage-1.redhat.local  172.17.3.53:9092
    Note
    After you migrate the monitoring stack, you lose high availability. The monitoring stack daemons no longer have a Virtual IP address and HAProxy anymore. Node exporters are still running on all the nodes.
  8. Review the Red Hat Ceph Storage configuration to ensure that it aligns with the configuration on the target nodes. In particular, focus on the following configuration entries: 
    [ceph: root@controller-0 /]# ceph config dump | grep -i dashboard
...
mgr  advanced  mgr/dashboard/ALERTMANAGER_API_HOST  http://172.17.3.83:9093
mgr  advanced  mgr/dashboard/GRAFANA_API_URL        https://172.17.3.144:3100
mgr  advanced  mgr/dashboard/PROMETHEUS_API_HOST    http://172.17.3.83:9092
mgr  advanced  mgr/dashboard/controller-0.ycokob/server_addr  172.17.3.33
mgr  advanced  mgr/dashboard/controller-1.lmzpuc/server_addr  172.17.3.147
mgr  advanced  mgr/dashboard/controller-2.xpdgfl/server_addr  172.17.3.138
  9. Verify that the API_HOST/URL of the grafana, alertmanager and prometheus services points to the IP addresses on the storage network of the node where each daemon is relocated: 
    [ceph: root@controller-0 /]# ceph orch ps | grep -iE "(prome|alert|grafa)"
alertmanager.cephstorage-0  cephstorage-0.redhat.local  172.17.3.83:9093,9094
alertmanager.cephstorage-1  cephstorage-1.redhat.local  172.17.3.53:9093,9094
alertmanager.cephstorage-2  cephstorage-2.redhat.local  172.17.3.144:9093,9094
grafana.cephstorage-0       cephstorage-0.redhat.local  172.17.3.83:3100
grafana.cephstorage-1       cephstorage-1.redhat.local  172.17.3.53:3100
grafana.cephstorage-2       cephstorage-2.redhat.local  172.17.3.144:3100
prometheus.cephstorage-0    cephstorage-0.redhat.local  172.17.3.83:9092
prometheus.cephstorage-1    cephstorage-1.redhat.local  172.17.3.53:9092
prometheus.cephstorage-2    cephstorage-2.redhat.local  172.17.3.144:9092
     
    [ceph: root@controller-0 /]# ceph config dump
...
...
mgr  advanced  mgr/dashboard/ALERTMANAGER_API_HOST   http://172.17.3.83:9093
mgr  advanced  mgr/dashboard/PROMETHEUS_API_HOST     http://172.17.3.83:9092
mgr  advanced  mgr/dashboard/GRAFANA_API_URL         https://172.17.3.144:3100
    Note
    The Ceph Dashboard, as the service provided by the Ceph mgr, is not impacted by the relocation. You might experience an impact when the active mgr daemon is migrated or is force-failed. However, you can define 3 replicas in the Ceph Manager configuration to redirect requests to a different instance.

Migrate Ceph MDS to new nodes in the cluster
Copy link

Migrate the MDS daemon from a hosts-based approach to a label-based approach. The Shared File Systems service (manila) must be deployed with either a CephFS-native or ceph-nfs back end. The MDS migration is performed by cephadm.

Before you begin
Copy link

About this task
Copy link

The label-based approach ensures that you can visualize the status of the cluster and where daemons are placed by using the ceph orch host command. You can also have a general view of how the daemons are co-located within a given host, as described in the Red Hat Knowledgebase article Red Hat Ceph Storage: Supported configurations.

Procedure
Copy link

  1. Verify that the Red Hat Ceph Storage cluster is healthy and check the MDS status: 
    $ sudo cephadm shell -- ceph fs ls
name: cephfs, metadata pool: manila_metadata, data pools: [manila_data ]

$ sudo cephadm shell -- ceph mds stat
cephfs:1 {0=mds.controller-2.oebubl=up:active} 2 up:standby

$ sudo cephadm shell -- ceph fs status cephfs

cephfs - 0 clients
======
RANK  STATE         	MDS           	ACTIVITY 	DNS	INOS   DIRS   CAPS
 0	active  mds.controller-2.oebubl  Reqs:	0 /s   696	196	173  	0
  	POOL     	TYPE 	USED  AVAIL
manila_metadata  metadata   152M   141G
  manila_data  	data	3072M   141G
  	STANDBY MDS
mds.controller-0.anwiwd
mds.controller-1.cwzhog
  2. Retrieve more detailed information on the Ceph File System (CephFS) MDS status: 
    $ sudo cephadm shell -- ceph fs dump

e8
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1

Filesystem 'cephfs' (1)
fs_name cephfs
epoch   5
flags   12 joinable allow_snaps allow_multimds_snaps
created 2024-01-18T19:04:01.633820+0000
modified    	2024-01-18T19:04:05.393046+0000
tableserver 	0
root	0
session_timeout 60
session_autoclose   	300
max_file_size   1099511627776
required_client_features    	{}
last_failure	0
last_failure_osd_epoch  0
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in  	0
up  	{0=24553}
failed
damaged
stopped
data_pools  	[7]
metadata_pool   9
inline_data 	disabled
balancer
standby_count_wanted	1
[mds.mds.controller-2.oebubl{0:24553} state up:active seq 2 addr [v2:172.17.3.114:6800/680266012,v1:172.17.3.114:6801/680266012] compat {c=[1],r=[1],i=[7ff]}]


Standby daemons:

[mds.mds.controller-0.anwiwd{-1:14715} state up:standby seq 1 addr [v2:172.17.3.20:6802/3969145800,v1:172.17.3.20:6803/3969145800] compat {c=[1],r=[1],i=[7ff]}]
[mds.mds.controller-1.cwzhog{-1:24566} state up:standby seq 1 addr [v2:172.17.3.43:6800/2227381308,v1:172.17.3.43:6801/2227381308] compat {c=[1],r=[1],i=[7ff]}]
dumped fsmap epoch 8
  3. Check the OSD blocklist and clean up the client list: 
    $ sudo cephadm shell -- ceph osd blocklist ls
$ for item in $(sudo cephadm shell -- ceph osd blocklist ls | awk '{print $1}'); do
>     sudo cephadm shell -- ceph osd blocklist rm $item;
> done
    Note

    When a file system client is unresponsive or misbehaving, the access to the file system might be forcibly terminated. This process is called eviction. Evicting a CephFS client prevents it from communicating further with MDS daemons and OSD daemons.

    Ordinarily, a blocklisted client cannot reconnect to the servers; you must unmount and then remount the client. However, permitting a client that was evicted to attempt to reconnect can be useful. Because CephFS uses the RADOS OSD blocklist to control client eviction, you can permit CephFS clients to reconnect by removing them from the blocklist.
  4. Get the hosts that are currently part of the Red Hat Ceph Storage cluster: 
    [ceph: root@controller-0 /]# ceph orch host ls
HOST                        ADDR           LABELS          STATUS
cephstorage-0.redhat.local  192.168.24.25  osd
cephstorage-1.redhat.local  192.168.24.50  osd
cephstorage-2.redhat.local  192.168.24.47  osd
controller-0.redhat.local   192.168.24.24  _admin mgr mon
controller-1.redhat.local   192.168.24.42  mgr _admin mon
controller-2.redhat.local   192.168.24.37  mgr _admin mon
6 hosts in cluster
  5. Apply the MDS labels to the target nodes: 
    for item in $(sudo cephadm shell --  ceph orch host ls --format json | jq -r '.[].hostname'); do
    sudo cephadm shell -- ceph orch host label add  $item mds;
done
  6. Verify that all the hosts have the MDS label: 
    $ sudo cephadm shell -- ceph orch host ls

HOST                    	ADDR       	   LABELS
cephstorage-0.redhat.local  192.168.24.11  osd mds
cephstorage-1.redhat.local  192.168.24.12  osd mds
cephstorage-2.redhat.local  192.168.24.47  osd mds
controller-0.redhat.local   192.168.24.35  _admin mon mgr mds
controller-1.redhat.local   192.168.24.53  mon _admin mgr mds
controller-2.redhat.local   192.168.24.10  mon _admin mgr mds
  7. Dump the current MDS spec: 
    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
$ mkdir -p ${SPEC_DIR}
$ sudo cephadm shell -- ceph orch ls --export mds > ${SPEC_DIR}/mds
  8. Edit the retrieved spec and replace the placement.hosts section with placement.label: 
    service_type: mds
service_id: mds
service_name: mds.mds
placement:
  label: mds
  9. Use the ceph orchestrator to apply the new MDS spec: 
    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
$ sudo cephadm shell -m ${SPEC_DIR}/mds -- ceph orch apply -i /mnt/mds

Scheduling new mds deployment ...

    This results in an increased number of MDS daemons.

  10. Check the new standby daemons that are temporarily added to the CephFS: 
    $ sudo cephadm shell -- ceph fs dump

Active

standby_count_wanted    1
[mds.mds.controller-0.awzplm{0:463158} state up:active seq 307 join_fscid=1 addr [v2:172.17.3.20:6802/51565420,v1:172.17.3.20:6803/51565420] compat {c=[1],r=[1],i=[7ff]}]


Standby daemons:

[mds.mds.cephstorage-1.jkvomp{-1:463800} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.135:6820/2075903648,v1:172.17.3.135:6821/2075903648] compat {c=[1],r=[1],i=[7ff]}]
[mds.mds.controller-2.gfrqvc{-1:475945} state up:standby seq 1 addr [v2:172.17.3.114:6800/2452517189,v1:172.17.3.114:6801/2452517189] compat {c=[1],r=[1],i=[7ff]}]
[mds.mds.cephstorage-0.fqcshx{-1:476503} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.92:6820/4120523799,v1:172.17.3.92:6821/4120523799] compat {c=[1],r=[1],i=[7ff]}]
[mds.mds.cephstorage-2.gnfhfe{-1:499067} state up:standby seq 1 addr [v2:172.17.3.79:6820/2448613348,v1:172.17.3.79:6821/2448613348] compat {c=[1],r=[1],i=[7ff]}]
[mds.mds.controller-1.tyiziq{-1:499136} state up:standby seq 1 addr [v2:172.17.3.43:6800/3615018301,v1:172.17.3.43:6801/3615018301] compat {c=[1],r=[1],i=[7ff]}]
  11. To migrate MDS to the target nodes, set the MDS affinity that manages the MDS failover:
    Note
    It is possible to elect a dedicated MDS as "active" for a particular file system. To configure this preference, CephFS provides a configuration option for MDS called mds_join_fs, which enforces this affinity. When failing over MDS daemons, cluster monitors prefer standby daemons with mds_join_fs equal to the file system name with the failed rank. If no standby exists with mds_join_fs equal to the file system name, it chooses an unqualified standby as a replacement.
     
    $ sudo cephadm shell -- ceph config set mds.mds.cephstorage-0.fqcshx mds_join_fs cephfs
    • Replace mds.mds.cephstorage-0.fqcshx with the daemon deployed on cephstorage-0 that was retrieved from the previous step.
  12. Remove the labels from the Controller nodes and force the MDS failover to the target node: 
    $ for i in 0 1 2; do sudo cephadm shell -- ceph orch host label rm "controller-$i.redhat.local" mds; done

Removed label mds from host controller-0.redhat.local
Removed label mds from host controller-1.redhat.local
Removed label mds from host controller-2.redhat.local

    The switch to the target node happens in the background. The new active MDS is the one that you set by using the mds_join_fs command.

  13. Check the result of the failover and the new deployed daemons: 
    $ sudo cephadm shell -- ceph fs dump
…
…
standby_count_wanted    1
[mds.mds.cephstorage-0.fqcshx{0:476503} state up:active seq 168 join_fscid=1 addr [v2:172.17.3.92:6820/4120523799,v1:172.17.3.92:6821/4120523799] compat {c=[1],r=[1],i=[7ff]}]


Standby daemons:

[mds.mds.cephstorage-2.gnfhfe{-1:499067} state up:standby seq 1 addr [v2:172.17.3.79:6820/2448613348,v1:172.17.3.79:6821/2448613348] compat {c=[1],r=[1],i=[7ff]}]
[mds.mds.cephstorage-1.jkvomp{-1:499760} state up:standby seq 1 join_fscid=1 addr [v2:172.17.3.135:6820/452139733,v1:172.17.3.135:6821/452139733] compat {c=[1],r=[1],i=[7ff]}]


$ sudo cephadm shell -- ceph orch ls

NAME                     PORTS   RUNNING  REFRESHED  AGE  PLACEMENT
crash                                6/6  10m ago    10d  *
mds.mds                          3/3  10m ago    32m  label:mds


$ sudo cephadm shell -- ceph orch ps | grep mds


mds.mds.cephstorage-0.fqcshx  cephstorage-0.redhat.local                     running (79m)     3m ago  79m    27.2M        -  17.2.6-100.el9cp  1af7b794f353  2a2dc5ba6d57
mds.mds.cephstorage-1.jkvomp  cephstorage-1.redhat.local                     running (79m)     3m ago  79m    21.5M        -  17.2.6-100.el9cp  1af7b794f353  7198b87104c8
mds.mds.cephstorage-2.gnfhfe  cephstorage-2.redhat.local                     running (79m)     3m ago  79m    24.2M        -  17.2.6-100.el9cp  1af7b794f353  f3cb859e2a15

Restore Ceph Dashboard failover settings after daemon migration
Copy link

If the Ceph Dashboard is part of the enabled Ceph Manager modules, you need to reconfigure the failover settings.

Procedure
Copy link

  1. Regenerate the following Red Hat Ceph Storage configuration keys to point to the right mgr container: 
    mgr    advanced  mgr/dashboard/controller-0.ycokob/server_addr  172.17.3.33
mgr    advanced  mgr/dashboard/controller-1.lmzpuc/server_addr  172.17.3.147
mgr    advanced  mgr/dashboard/controller-2.xpdgfl/server_addr  172.17.3.138
     
    $ sudo cephadm shell
$ ceph orch ps | awk '/mgr./ {print $1}'
  2. For each retrieved mgr daemon, update the corresponding entry in the Red Hat Ceph Storage configuration: 
    $ ceph config set mgr mgr/dashboard/<>/server_addr/<ip addr>