Migrate Red Hat Ceph Storage RGW to external RHEL nodes

For Hyperconverged Infrastructure (HCI) or dedicated Storage nodes, migrate the Ceph Object Gateway (RGW) daemons from the Red Hat OpenStack Platform Controller nodes into the existing external Red Hat Enterprise Linux (RHEL) nodes.

The RHEL nodes include the Compute nodes for an HCI environment or Red Hat Ceph Storage nodes. Your environment must have Red Hat Ceph Storage 7 or later and be managed by cephadm or Ceph Orchestrator.

Prerequisites
Copy link

Before you begin the migration, complete the tasks in your Red Hat OpenStack Platform 17.1 environment. For more information, see the "Red Hat Ceph Storage prerequisites" in the "Adoption overview" chapter.

Migrate the Red Hat Ceph Storage RGW back ends
Copy link

Migrate Ceph Object Gateway (RGW) back ends from Controller nodes to Red Hat Ceph Storage nodes. To distribute the correct amount of services to your available nodes, use cephadm labels to refer to a group of nodes where a given daemon type is deployed.

About this task
Copy link

For more information about the cardinality diagram, see Red Hat Ceph Storage daemon cardinality. The following procedure assumes that you have three target nodes, cephstorage-0, cephstorage-1, cephstorage-2.

Procedure
Copy link

  1. Add the RGW label to the Red Hat Ceph Storage nodes that you want to migrate your RGW back ends to: 
    $ sudo cephadm shell -- ceph orch host label add cephstorage-0 rgw;
$ sudo cephadm shell -- ceph orch host label add cephstorage-1 rgw;
$ sudo cephadm shell -- ceph orch host label add cephstorage-2 rgw;

Added label rgw to host cephstorage-0
Added label rgw to host cephstorage-1
Added label rgw to host cephstorage-2

$ sudo cephadm shell -- ceph orch host ls

HOST       	ADDR       	LABELS      	STATUS
cephstorage-0  192.168.24.54  osd rgw
cephstorage-1  192.168.24.44  osd rgw
cephstorage-2  192.168.24.30  osd rgw
controller-0   192.168.24.45  _admin mon mgr
controller-1   192.168.24.11  _admin mon mgr
controller-2   192.168.24.38  _admin mon mgr

6 hosts in cluster
  2. Locate the RGW spec and dump in the spec directory: 
    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
$ mkdir -p ${SPEC_DIR}
$ sudo cephadm shell -- ceph orch ls --export rgw > ${SPEC_DIR}/rgw
$ cat ${SPEC_DIR}/rgw
     
    networks:
- 172.17.3.0/24
placement:
  hosts:
  - controller-0
  - controller-1
  - controller-2
service_id: rgw
service_name: rgw.rgw
service_type: rgw
spec:
  rgw_frontend_port: 8080
  rgw_realm: default
  rgw_zone: default

    This example assumes that 172.17.3.0/24 is the storage network.

  3. In the placement section, ensure that the label and rgw_frontend_port values are set: 
    ---
networks:
- 172.17.3.0/24
placement:
  label: rgw
service_id: rgw
service_name: rgw.rgw
service_type: rgw
spec:
  rgw_frontend_port: 8090
  rgw_realm: default
  rgw_zone: default
  rgw_frontend_ssl_certificate: ...
  ssl: true
    • networks defines the storage network where the RGW back ends are deployed.
    • placement.label: rgw replaces the Controller nodes with the rgw label.
    • spec.rgw_frontend_port specifies the value as 8090 to avoid conflicts with the Ceph ingress daemon.
    • spec.rgw_frontend_ssl_certificate defines the SSL certificate and key concatenation if TLS is enabled as described in Configure RGW with TLS for an external Ceph cluster.
  4. Apply the new RGW spec by using the orchestrator CLI: 
    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
$ sudo cephadm shell -m ${SPEC_DIR}/rgw -- ceph orch apply -i /mnt/rgw

    This command triggers the redeploy, for example:

    ...
osd.9                     	cephstorage-2
rgw.rgw.cephstorage-0.wsjlgx  cephstorage-0  172.17.3.23:8090   starting
rgw.rgw.cephstorage-1.qynkan  cephstorage-1  172.17.3.26:8090   starting
rgw.rgw.cephstorage-2.krycit  cephstorage-2  172.17.3.81:8090   starting
rgw.rgw.controller-1.eyvrzw   controller-1   172.17.3.146:8080  running (5h)
rgw.rgw.controller-2.navbxa   controller-2   172.17.3.66:8080   running (5h)

...
osd.9                     	cephstorage-2
rgw.rgw.cephstorage-0.wsjlgx  cephstorage-0  172.17.3.23:8090  running (19s)
rgw.rgw.cephstorage-1.qynkan  cephstorage-1  172.17.3.26:8090  running (16s)
rgw.rgw.cephstorage-2.krycit  cephstorage-2  172.17.3.81:8090  running (13s)
  5. Ensure that the new RGW back ends are reachable on the new ports, so you can enable an ingress daemon on port 8080 later. Log in to each Red Hat Ceph Storage node that includes RGW and add the iptables rule to allow connections to both 8080 and 8090 ports in the Red Hat Ceph Storage nodes: 
    $ iptables -I INPUT -p tcp -m tcp --dport 8080 -m conntrack --ctstate NEW -m comment --comment "ceph rgw ingress" -j ACCEPT
$ iptables -I INPUT -p tcp -m tcp --dport 8090 -m conntrack --ctstate NEW -m comment --comment "ceph rgw backends" -j ACCEPT
$ sudo iptables-save
$ sudo systemctl restart iptables
  6. If nftables is used in the existing deployment, edit /etc/nftables/tripleo-rules.nft and add the following content: 
    # 100 ceph_rgw {'dport': ['8080','8090']}
add rule inet filter TRIPLEO_INPUT tcp dport { 8080,8090 } ct state new counter accept comment "100 ceph_rgw"
  7. Save the file.
  8. Restart the nftables service: 
    $ sudo systemctl restart nftables
  9. Verify that the rules are applied: 
    $ sudo nft list ruleset | grep ceph_rgw
  10. From a Controller node, such as controller-0, try to reach the RGW back ends: 
    $ curl http://cephstorage-0.storage:8090;

    You should observe the following output:

    <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

    Repeat the verification for each node where a RGW daemon is deployed.

  11. If you migrated RGW back ends to the Red Hat Ceph Storage nodes, there is no internalAPI network, except in the case of HCI nodes. You must reconfigure the RGW keystone endpoint to point to the external network that you propagated: 
    [ceph: root@controller-0 /]# ceph config dump | grep keystone
global   basic rgw_keystone_url  http://172.16.1.111:5000

[ceph: root@controller-0 /]# ceph config set global rgw_keystone_url http://<keystone_endpoint>:5000
    • Replace <keystone_endpoint> with the Identity service (keystone) internal endpoint of the service that is deployed in the OpenStackControlPlane CR when you adopt the Identity service. For more information, see Adopting the Identity service.

Restore Object Storage access with a Ceph ingress daemon
Copy link

To deploy the Ceph ingress daemon, you remove the existing ceph_rgw configuration, clean up the configuration created by director, and redeploy the Object Storage service (swift).

About this task
Copy link

When you deploy the ingress daemon, two new containers are created:

  • HAProxy, which you use to reach the back ends.
  • Keepalived, which you use to own the virtual IP address.

You use the rgw label to distribute the ingress daemon to only the number of nodes that host Ceph Object Gateway (RGW) daemons. For more information about distributing daemons among your nodes, see Red Hat Ceph Storage daemon cardinality.

After you complete this procedure, you can reach the RGW back end from the ingress daemon and use RGW through the Object Storage service CLI.

Procedure
Copy link

  1. Log in to each Controller node and remove the following configuration from the /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg file: 
    listen ceph_rgw
  bind 10.0.0.103:8080 transparent
  mode http
  balance leastconn
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  option httpchk GET /swift/healthcheck
  option httplog
  option forwardfor
   server controller-0.storage.redhat.local 172.17.3.73:8080 check fall 5 inter 2000 rise 2
  server controller-1.storage.redhat.local 172.17.3.146:8080 check fall 5 inter 2000 rise 2
  server controller-2.storage.redhat.local 172.17.3.156:8080 check fall 5 inter 2000 rise 2
  2. Restart haproxy-bundle and confirm that it is started: 
    [root@controller-0 ~]# sudo pcs resource restart haproxy-bundle
haproxy-bundle successfully restarted


[root@controller-0 ~]# sudo pcs status | grep haproxy

  * Container bundle set: haproxy-bundle [undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-haproxy:pcmklatest]:
    * haproxy-bundle-podman-0   (ocf:heartbeat:podman):  Started controller-0
    * haproxy-bundle-podman-1   (ocf:heartbeat:podman):  Started controller-1
    * haproxy-bundle-podman-2   (ocf:heartbeat:podman):  Started controller-2
  3. Confirm that no process is connected to port 8080: 
    [root@controller-0 ~]# ss -antop | grep 8080
[root@controller-0 ~]#

    You can expect the Object Storage service (swift) CLI to fail to establish the connection:

    (overcloud) [root@cephstorage-0 ~]# swift list

HTTPConnectionPool(host='10.0.0.103', port=8080): Max retries exceeded with url: /swift/v1/AUTH_852f24425bb54fa896476af48cbe35d3?format=json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc41beb0430>: Failed to establish a new connection: [Errno 111] Connection refused'))
  4. Set the required images for both HAProxy and Keepalived: 
    [ceph: root@controller-0 /]# ceph config set mgr mgr/cephadm/container_image_haproxy registry.redhat.io/rhceph/rhceph-haproxy-rhel9:latest
[ceph: root@controller-0 /]# ceph config set mgr mgr/cephadm/container_image_keepalived registry.redhat.io/rhceph/keepalived-rhel9:latest
  5. Create a file called rgw_ingress in controller-0: 
    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
$ vim ${SPEC_DIR}/rgw_ingress
  6. Paste the following content into the rgw_ingress file: 
    ---
service_type: ingress
service_id: rgw.rgw
placement:
  label: rgw
spec:
  backend_service: rgw.rgw
  virtual_ip: 10.0.0.89/24
  frontend_port: 8080
  monitor_port: 8898
  virtual_interface_networks:
    - <external_network>
  ssl_cert: ...
  7. Apply the rgw_ingress spec by using the Ceph orchestrator CLI: 
    $ SPEC_DIR=${SPEC_DIR:-"$PWD/ceph_specs"}
$ cephadm shell -m ${SPEC_DIR}/rgw_ingress -- ceph orch apply -i /mnt/rgw_ingress
  8. Wait until the ingress is deployed and query the resulting endpoint: 
    $ sudo cephadm shell -- ceph orch ls

NAME                 	PORTS            	RUNNING  REFRESHED  AGE  PLACEMENT
crash                                         	6/6  6m ago 	3d   *
ingress.rgw.rgw      	10.0.0.89:8080,8898  	6/6  37s ago	60s  label:rgw
mds.mds                   3/3  6m ago 	3d   controller-0;controller-1;controller-2
mgr                       3/3  6m ago 	3d   controller-0;controller-1;controller-2
mon                       3/3  6m ago 	3d   controller-0;controller-1;controller-2
osd.default_drive_group   15  37s ago	3d   cephstorage-0;cephstorage-1;cephstorage-2
rgw.rgw   ?:8090          3/3  37s ago	4m   label:rgw
     
    $ curl 10.0.0.89:8080

---
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[ceph: root@controller-0 /]#
—

Configure Object Storage service endpoints for the new RGW ingress VIP
Copy link

Create or update the Object Storage service (swift) endpoints to configure the new virtual IP address (VIP) that you reserved on the same network that you used to deploy RGW ingress.

Procedure
Copy link

  1. List the current swift endpoints and service: 
     $ oc rsh openstackclient openstack endpoint list | grep "swift.*object"
     
     $ oc rsh openstackclient openstack service list | grep "swift.*object"
  2. If the service and endpoints do not exist, create the missing swift resources: 
    $ oc rsh openstackclient openstack service create --name swift --description 'OpenStack Object Storage' object-store
$ oc rsh openstackclient openstack role add --user swift --project service member
$ oc rsh openstackclient openstack role add --user swift --project service admin
> for i in public internal; do
>     oc rsh openstackclient endpoint create --region regionOne object-store $i http://<RGW_VIP>:8080/swift/v1/AUTH_%\(tenant_id\)s
> done
$ oc rsh openstackclient openstack role add --project admin --user admin swiftoperator
    • Replace <RGW_VIP> with the Ceph RGW ingress VIP.
  3. If the endpoints exist, update the endpoints to point to the right RGW ingress VIP: 
    $ oc rsh openstackclient openstack endpoint set --url http://<RGW_VIP>:8080/swift/v1/AUTH_%\(tenant_id\)s <swift_public_endpoint_uuid>
$ oc rsh openstackclient openstack endpoint set --url http://<RGW_VIP>:8080/swift/v1/AUTH_%\(tenant_id\)s <swift_internal_endpoint_uuid>
$ oc rsh openstackclient openstack endpoint list | grep object
| 0d682ad71b564cf386f974f90f80de0d | regionOne | swift        | object-store | True    | public    | http://172.18.0.100:8080/swift/v1/AUTH_%(tenant_id)s    |
| b311349c305346f39d005feefe464fb1 | regionOne | swift        | object-store | True    | internal  | http://172.18.0.100:8080/swift/v1/AUTH_%(tenant_id)s    |
    • Replace <swift_public_endpoint_uuid> with the UUID of the swift public endpoint.
    • Replace <swift_internal_endpoint_uuid> with the UUID of the swift internal endpoint.
  4. Test the migrated service: 
    $ oc rsh openstackclient openstack container list --debug

...
...
...
REQ: curl -g -i -X GET http://keystone-public-openstack.apps.ocp.openstack.lab -H "Accept: application/json" -H "User-Agent: openstacksdk/1.0.2 keystoneauth1/5.1.3 python-requests/2.25.1 CPython/3.9.23"
Starting new HTTP connection (1): keystone-public-openstack.apps.ocp.openstack.lab:80
http://keystone-public-openstack.apps.ocp.openstack.lab:80 "GET / HTTP/1.1" 300 298
RESP: [300] content-length: 298 content-type: application/json date: Mon, 14 Jul 2025 17:41:29 GMT location: http://keystone-public-openstack.apps.ocp.openstack.lab/v3/ server: Apache set-cookie: b5697f82cf3c19ece8be533395142512=d5c6a9ee2
267c4b63e9f656ad7565270; path=/; HttpOnly vary: X-Auth-Token x-openstack-request-id: req-452e42c5-e60f-440f-a6e8-fe1b9ea89055
RESP BODY: {"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://keystone-public-openstack.apps.ocp.openstack.lab/v3/"}], "media-types": [{"base": "applic
ation/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
GET call to http://keystone-public-openstack.apps.ocp.openstack.lab/ used request id req-452e42c5-e60f-440f-a6e8-fe1b9ea89055

...

REQ: curl -g -i -X GET "http://172.18.0.100:8080/swift/v1/AUTH_44477474b0dc4b5b8911ceec23a22246?format=json" -H "User-Agent: openstacksdk/1.0.2 keystoneauth1/5.1.3 python-requests/2.25.1 CPython/3.9.23" -H "X-Auth-Token: {SHA256}ec5deca0be37bd8bfe659f132b9cdf396b8f409db5dc16972d50cbf3f28474d4"
Starting new HTTP connection (1): 172.18.0.100:8080
http://172.18.0.100:8080 "GET /swift/v1/AUTH_44477474b0dc4b5b8911ceec23a22246?format=json HTTP/1.1" 200 2
RESP: [200] accept-ranges: bytes content-length: 2 content-type: application/json; charset=utf-8 date: Mon, 14 Jul 2025 17:41:31 GMT x-account-bytes-used: 0 x-account-bytes-used-actual: 0 x-account-container-count: 0 x-account-object-count: 0 x-account-storage-policy-default-placement-bytes-used: 0 x-account-storage-policy-default-placement-bytes-used-actual: 0 x-account-storage-policy-default-placement-container-count: 0 x-account-storage-policy-default-placement-object-count: 0 x-openstack-request-id: tx000001e95361131ccf694-006875414a-7753-default x-timestamp: 1752514891.25991 x-trans-id: tx000001e95361131ccf694-006875414a-7753-default
RESP BODY: []
GET call to http://172.18.0.100:8080/swift/v1/AUTH_44477474b0dc4b5b8911ceec23a22246?format=json used request id tx000001e95361131ccf694-006875414a-7753-default

clean_up ListContainer:
END return value: 0