• Take note of the Block Storage service back ends that you use.
• Determine all the transport protocols that the Block Storage service back ends use, such as RBD, iSCSI, FC, NFS, NVMe-TCP, and so on. You must consider them when you place the Block Storage services and when the right storage transport-related binaries are running on the Red Hat OpenShift Container Platform (RHOCP) nodes. For more information about each storage transport protocol, see RHOCP preparation for Block Storage service adoption.
• Use a Block Storage service volume service to deploy each Block Storage service volume back end.

  For example, you have an LVM back end, a Ceph back end, and two entries in cinderVolumes, and you cannot set global defaults for all volume services. You must define a service for each of them:

  apiVersion: core.openstack.org/v1beta1
  kind: OpenStackControlPlane
  metadata:
    name: openstack
  spec:
    cinder:
      enabled: true
      template:
        cinderVolumes:
          lvm:
            customServiceConfig: |
              [DEFAULT]
              debug = True
              [lvm]
  < . . . >
          ceph:
            customServiceConfig: |
              [DEFAULT]
              debug = True
              [ceph]
  < . . . >

  Warning
  Check that all configuration options are still valid for RHOSO 18.0 version. Configuration options might be deprecated, removed, or added. This applies to both back-end driver-specific configuration options and other generic options.

Node selection

You might need to restrict the RHOCP nodes where the Block Storage service volume and backup services run.

An example of when you need to restrict nodes for a specific Block Storage service is when you deploy the Block Storage service with the LVM driver. In that scenario, the LVM data where the volumes are stored only exists in a specific host, so you need to pin the Block Storage-volume service to that specific RHOCP node. Running the service on any other RHOCP node does not work. You cannot use the RHOCP host node name to restrict the LVM back end. You need to identify the LVM back end by using a unique label, an existing label, or a new label:

$ oc label nodes worker0 lvm=cinder-volumes

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: openstack
spec:
  secret: osp-secret
  storageClass: local-storage
  cinder:
    enabled: true
    template:
      cinderVolumes:
        lvm-iscsi:
          nodeSelector:
            lvm: cinder-volumes
< . . . >

For more information about node selection, see About node selectors.

Note

If your nodes do not have enough local disk space for temporary images, you can use a remote NFS location by setting the extra volumes feature, extraMounts.

Transport protocols

Some changes to the storage transport protocols might be required for RHOCP:

• If you use a MachineConfig to make changes to RHOCP nodes, the nodes reboot.
• Check the back-end sections that are listed in the enabled_backends configuration option in your cinder.conf file to determine the enabled storage back-end sections.
• Depending on the back end, you can find the transport protocol by viewing the volume_driver or target_protocol configuration options.
• The iscsid service, multipathd service, and NVMe-TCP kernel modules start automatically on data plane nodes.

  NFS

  • RHOCP connects to NFS back ends without additional changes.

  Rados Block Device and Red Hat Ceph Storage

  • RHOCP connects to Red Hat Ceph Storage back ends without additional changes. You must provide credentials and configuration files to the services.

  iSCSI

  • To connect to iSCSI volumes, the iSCSI initiator must run on the RHOCP hosts where the volume and backup services run. The Linux Open iSCSI initiator does not support network namespaces, so you must only run one instance of the service for the normal RHOCP usage, and the RHOCP CSI plugins and the RHOSP services.
  • If you are not already running iscsid on the RHOCP nodes, then you must apply a MachineConfig. For example:

    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
        service: cinder
      name: 99-master-cinder-enable-iscsid
    spec:
      config:
        ignition:
          version: 3.2.0
        systemd:
          units:
          - enabled: true
            name: iscsid.service

  • If you use labels to restrict the nodes where the Block Storage services run, you must use a MachineConfigPool to limit the effects of the MachineConfig to the nodes where your services might run. For more information, see About node selectors.
  • If you are using a single node deployment to test the process, replace worker with master in the MachineConfig.
  • For production deployments that use iSCSI volumes, configure multipathing for better I/O.

  FC

  • The Block Storage service volume and Block Storage service backup services must run in an RHOCP host that has host bus adapters (HBAs). If some nodes do not have HBAs, then use labels to restrict where these services run. For more information, see About node selectors.
  • If the Image service is configured to use Block Storage service as a back end with FC, the Image service must also run on an RHOCP host that has HBAs and follow the same node selection requirements as the Block Storage service.
  • If you have virtualized RHOCP clusters that use FC you need to expose the host HBAs inside the virtual machine.
  • For production deployments that use FC volumes, configure multipathing for better I/O.

  NVMe-TCP

  • To connect to NVMe-TCP volumes, load NVMe-TCP kernel modules on the RHOCP hosts.
  • If you do not already load the nvme-fabrics module on the RHOCP nodes where the volume and backup services are going to run, then you must apply a MachineConfig. For example:

    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
        service: cinder
      name: 99-master-cinder-load-nvme-fabrics
    spec:
      config:
        ignition:
          version: 3.2.0
        storage:
          files:
            - path: /etc/modules-load.d/nvme_fabrics.conf
              overwrite: false
              # Mode must be decimal, this is 0644
              mode: 420
              user:
                name: root
              group:
                name: root
              contents:
                # Source can be a http, https, tftp, s3, gs, or data as defined in rfc2397.
                # This is the rfc2397 text/plain string format
                source: data:,nvme-fabrics

  • If you use labels to restrict the nodes where Block Storage services run, use a MachineConfigPool to limit the effects of the MachineConfig to the nodes where your services run. For more information, see About node selectors.
  • If you use a single node deployment to test the process, replace worker with master in the MachineConfig.
  • Only load the nvme-fabrics module because it loads the transport-specific modules, such as TCP, RDMA, or FC, as needed.
  • For production deployments that use NVMe-TCP volumes, use multipathing for better I/O. For NVMe-TCP volumes, RHOCP uses native multipathing, called ANA.
  • After the RHOCP nodes reboot and load the nvme-fabrics module, you can confirm that the operating system is configured and that it supports ANA by checking the host:

    $ cat /sys/module/nvme_core/parameters/multipath

    Important
    ANA does not use the Linux Multipathing Device Mapper, but RHOCP requires multipathd to run on Compute nodes for the Compute service (nova) to be able to use multipathing. Multipathing is automatically configured on data plane nodes when they are provisioned.

  Multipathing

  • Use multipathing for iSCSI and FC protocols. To configure multipathing on these protocols, you perform the following tasks:
    • Prepare the RHOCP hosts
    • Configure the Block Storage services
    • Prepare the Compute service nodes
    • Configure the Compute service
  • To prepare the RHOCP hosts, ensure that the Linux Multipath Device Mapper is configured and running on the RHOCP hosts by using MachineConfig. For example:

    # Includes the /etc/multipathd.conf contents and the systemd unit changes
    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
        service: cinder
      name: 99-master-cinder-enable-multipathd
    spec:
      config:
        ignition:
          version: 3.2.0
        storage:
          files:
            - path: /etc/multipath.conf
              overwrite: false
              # Mode must be decimal, this is 0600
              mode: 384
              user:
                name: root
              group:
                name: root
              contents:
                # Source can be a http, https, tftp, s3, gs, or data as defined in rfc2397.
                # This is the rfc2397 text/plain string format
                source: data:,defaults%20%7B%0A%20%20user_friendly_names%20no%0A%20%20recheck_wwid%20yes%0A%20%20skip_kpartx%20yes%0A%20%20find_multipaths%20yes%0A%7D%0A%0Ablacklist%20%7B%0A%7D
        systemd:
          units:
          - enabled: true
            name: multipathd.service

  • If you use labels to restrict the nodes where Block Storage services run, you need to use a MachineConfigPool to limit the effects of the MachineConfig to only the nodes where your services run. For more information, see About node selectors.
  • If you are using a single node deployment to test the process, replace worker with master in the MachineConfig.
  • Cinder volume and backup are configured by default to use multipathing.

Review the following information to guide you in converting your previous configuration:

• Determine what part of the configuration is generic for all the Block Storage services and remove anything that would change when deployed in Red Hat OpenShift Container Platform (RHOCP), such as the connection in the [database] section, the transport_url and log_dir in the [DEFAULT] sections, the whole [coordination] and [barbican] sections. The remaining generic configuration goes into the customServiceConfig option, or a Secret custom resource (CR) and is then used in the customServiceConfigSecrets section, at the cinder: template: level.
• Determine if there is a scheduler-specific configuration and add it to the customServiceConfig option in cinder: template: cinderScheduler.
• Determine if there is an API-specific configuration and add it to the customServiceConfig option in cinder: template: cinderAPI.
• If the Block Storage service backup is deployed, add the Block Storage service backup configuration options to customServiceConfig option, or to a Secret CR that you can add to customServiceConfigSecrets section at the cinder: template: cinderBackup: level. Remove the host configuration in the [DEFAULT] section to support multiple replicas later.
• Determine the individual volume back-end configuration for each of the drivers. The configuration is in the specific driver section, and it includes the [backend_defaults] section and FC zoning sections if you use them. The Block Storage service operator does not support a global customServiceConfig option for all volume services. Each back end has its own section under cinder: template: cinderVolumes, and the configuration goes in the customServiceConfig option or in a Secret CR and is then used in the customServiceConfigSecrets section.
• If any of the Block Storage service volume drivers require a custom vendor image, find the location of the image in the Red Hat Ecosystem Catalog, and create or modify an OpenStackVersion CR to specify the custom image by using the key from the cinderVolumes section.

  For example, if you have the following configuration:

  spec:
    cinder:
      enabled: true
      template:
        cinderVolume:
          pure:
            customServiceConfigSecrets:
              - openstack-cinder-pure-cfg
  < . . . >

  Then the OpenStackVersion CR that describes the container image for that back end looks like the following example:

  apiVersion: core.openstack.org/v1beta1
  kind: OpenStackVersion
  metadata:
    name: openstack
  spec:
    customContainerImages:
      cinderVolumeImages:
        pure: registry.connect.redhat.com/purestorage/openstack-cinder-volume-pure-rhosp-18-0'

  Note
  The name of the OpenStackVersion must match the name of your OpenStackControlPlane CR.
• If your Block Storage services use external files, for example, for a custom policy, or to store credentials or SSL certificate authority bundles to connect to a storage array, make those files available to the right containers. Use Secrets or ConfigMap to store the information in RHOCP and then in the extraMounts key. For example, for Red Hat Ceph Storage credentials that are stored in a Secret called ceph-conf-files, you patch the top-level extraMounts key in the OpenstackControlPlane CR:

  spec:
    extraMounts:
    - extraVol:
      - extraVolType: Ceph
        mounts:
        - mountPath: /etc/ceph
          name: ceph
          readOnly: true
        propagation:
        - CinderVolume
        - CinderBackup
        - Glance
        volumes:
        - name: ceph
          projected:
            sources:
            - secret:
                name: ceph-conf-files

• For a service-specific file, such as the API policy, you add the configuration on the service itself. In the following example, you include the CinderAPI configuration that references the policy you are adding from a ConfigMap called my-cinder-conf that has a policy key with the contents of the policy:

  spec:
    cinder:
      enabled: true
      template:
        cinderAPI:
          customServiceConfig: |
             [oslo_policy]
             policy_file=/etc/cinder/api/policy.yaml
        extraMounts:
        - extraVol:
          - extraVolType: Ceph
            mounts:
            - mountPath: /etc/cinder/api
              name: policy
              readOnly: true
            propagation:
            - CinderAPI
            volumes:
            - name: policy
              projected:
                sources:
                - configMap:
                    name: my-cinder-conf
                    items:
                      - key: policy
                        path: policy.yaml