Migrate Red Hat Ceph Storage RBD to external RHEL nodes

For Hyperconverged Infrastructure (HCI) or dedicated Storage nodes that are running Red Hat Ceph Storage 7 or later, migrate the daemons from the Red Hat OpenStack Platform control plane into the existing external Red Hat Enterprise Linux (RHEL) nodes.

The external RHEL nodes typically include the Compute nodes for an HCI environment or dedicated storage nodes.

Prerequisites
Copy link

Before you begin the migration, complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see the "Red Hat Ceph Storage prerequisites" in the "Adoption overview" chapter.

Migrate Ceph Manager daemons to Red Hat Ceph Storage nodes
Copy link

Migrate Ceph Manager daemons from Red Hat OpenStack Platform Controller nodes to a set of existing Red Hat Ceph Storage nodes, or Compute nodes if Red Hat Ceph Storage is deployed by director with a Hyperconverged Infrastructure (HCI) topology.

About this task
Copy link

Note
The following procedure uses cephadm and the Ceph Orchestrator to drive the Ceph Manager migration, and the Ceph spec to modify the placement and reschedule the Ceph Manager daemons. Ceph Manager is run in an active/passive state. It also provides many modules, including the Ceph Orchestrator. Every potential module, such as the Ceph Dashboard, that is provided by ceph-mgr is implicitly migrated with Ceph Manager.

Procedure
Copy link

  1. SSH into the target node and enable the firewall rules that are required to reach a Ceph Manager service: 
    dports="6800:7300"
ssh heat-admin@<target_node> sudo iptables -I INPUT \
    -p tcp --match multiport --dports $dports -j ACCEPT;
    • Replace <target_node> with the hostname of the hosts that are listed in the Red Hat Ceph Storage environment. Run ceph orch host ls to see the list of the hosts.

      Repeat this step for each target node.

  2. Check that the rules are properly applied to the target node and persist them: 
    $ sudo iptables-save
$ sudo systemctl restart iptables
    Note
    The default dashboard port for ceph-mgr in a greenfield deployment is 8443. With director-deployed Red Hat Ceph Storage, the default port is 8444 because the service ran on the Controller node, and it was necessary to use this port to avoid a conflict. For adoption, update the dashboard port to 8443 in the ceph-mgr configuration and firewall rules.
  3. Log in to controller-0 and update the dashboard port in the ceph-mgr configuration to 8443: 
    $ sudo cephadm shell
$ ceph config set mgr mgr/dashboard/server_port 8443
$ ceph config set mgr mgr/dashboard/ssl_server_port 8443
$ ceph mgr module disable dashboard
$ ceph mgr module enable dashboard
  4. If nftables is used in the existing deployment, edit /etc/nftables/tripleo-rules.nft and add the following content: 
    # 113 ceph_mgr {'dport': ['6800-7300', 8443]}
add rule inet filter TRIPLEO_INPUT tcp dport { 6800-7300,8443 } ct state new counter accept comment "113 ceph_mgr"
  5. Save the file.
  6. Restart the nftables service: 
    $ sudo systemctl restart nftables
  7. Verify that the rules are applied: 
    $ sudo nft list ruleset | grep ceph_mgr
  8. Prepare the target node to host the new Ceph Manager daemon, and add the mgr label to the target node: 
    $ sudo cephadm shell -- ceph orch host label add <target_node> mgr
  9. Repeat steps 1-7 for each target node that hosts a Ceph Manager daemon.
  10. Get the Ceph Manager spec: 
    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
$ mkdir -p ${SPEC_DIR}
$ sudo cephadm shell -- ceph orch ls --export mgr > ${SPEC_DIR}/mgr
  11. Edit the retrieved spec and add the label: mgr section to the placement section: 
    service_type: mgr
service_id: mgr
placement:
  label: mgr
  12. Save the spec.
  13. Apply the spec with cephadm by using the Ceph Orchestrator: 
    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
$ sudo cephadm shell -m ${SPEC_DIR}/mgr -- ceph orch apply -i /mnt/mgr

Results
Copy link

  1. Verify that the new Ceph Manager daemons are created in the target nodes: 
    $ sudo cephadm shell -- ceph orch ps | grep -i mgr
$ sudo cephadm shell -- ceph -s

    The Ceph Manager daemon count should match the number of hosts where the mgr label is added.

    Note
    The migration does not shrink the Ceph Manager daemons. The count grows by the number of target nodes, and migrating Ceph Monitor daemons to Red Hat Ceph Storage nodes decommissions the stand-by Ceph Manager instances. For more information, see Migrating Ceph Monitor daemons to Red Hat Ceph Storage nodes.