RHOCP storage sizing

You can configure the Image service (glance) and Object Storage service (swift) to allocate space in the RHOCP backing storage. In this scenario, estimate the RHOCP storage sizing based on the expected use of these services.

Image service storage requirements

The Image service (glance) requires persistent storage for staging and importing data, which you can provide by using Persistent Volume Claims (PVCs) or an external model with custom storage mounts.

External model

If you choose external, no PVCs are created and the Image service acts like a stateless instance with no persistence provided. In this instance, persistence must be provided using extraMounts. NFS is often used to provide persistence. It can be mapped to /var/lib/glance:

...
default:
  storage:
    external: true
...
...
extraMounts:
- extraVol:
  - extraVolType: NFS
    mounts:
    - mountPath: /var/lib/glance/os_glance_staging_store
      name: nfs
    volumes:
    - name: nfs
      nfs:
        path: <nfs_export_path>
        server: <nfs_ip_address>
  • Replace <nfs_export_path> with the export path of your NFS share.
  • Replace <nfs_ip_address> with the IP address of your NFS share. This IP address must be part of the overlay network that is reachable by the Image service.

It should be noted that the configuration sample conflicts with the distributed image import feature. Distributed image import requires RWO storage plugged into a particular instance; it owns the data and receives requests in case its staged data is required for an upload operation. When the External model is adopted, if Red Hat Ceph Storage is used as a backend, and an image conversion operation is run in one of the existing replicas, the glance-operator does not have to make any assumption about the underlying storage that is tied to the staging area, and the conversion operation that uses the os_glance_staging_store directory (within the Pod) interacts with the RWX NFS backend provided via extraMounts. With this scenario, no image-cache PVC can be requested and mounted to a subPath, because it should be the administrator’s responsibility to plan for persistence by using extraMounts.

PVC model

The PVC model is the default. When a GlanceAPI instance is deployed, a PVC is created and bound to /var/lib/glance according to the storageClass and storageRequest passed as input.

...
default:
  replicas: 3
  storage:
    storageRequest: 10G
...

In this model, if Red Hat Ceph Storage is set as a back end, no dedicated image conversion PVC is created. The administrator must think about the PVC sizing in advance; the size of the PVC should be at least up to the largest converted image size. Concurrent conversions within the same Pod might be problematic in terms of PVC size; a conversion will fail or cannot take place if the PVC is full and there’s not enough space. The upload should be retried after the previous conversion is over and the staging area space is released. However, concurrent conversion operations might happen in different Pods. You should deploy at least 3 replicas for a particular glanceAPI. This helps to handle heavy operations such as image conversion.

For a PVC-based layout, the scale out of a glanceAPI in terms of replicas is limited by the available storage provided by the storageClass, and depends on the storageRequest. The storageRequest is a critical parameter, it can be globally defined for all the glanceAPI, or defined with a different value for each API. It will influence the scale out operations for each of them. Other than a local PVC required for the staging area, it is possible to enable image cache, which is translated into an additional PVC bound to each glanceAPI instance. A glance-cache PVC is bound to /var/lib/glance/image-cache. The glance-operator configures the glanceAPI instance accordingly, setting both image_cache_max_size and the image_cache_dir parameters. The number of image cache PVCs follows the same rules described for the local PVC, and the number of requested PVCs is proportional to the number of replicas.

Object Storage service storage requirements

The Object Storage service (swift) requires storage devices for data. These devices must be accessible using the same hostname or IP address during their lifetime. You achieve this by configuring a StatefulSet with a Headless Service.

StatefulSets
If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution. Although individual pods in a StatefulSet are susceptible to failure, the persistent pod identifiers make it easier to match existing volumes to the new pods that replace any that have failed.
Service architecture
The Object Storage service requires services to access Persistent Volumes (PVs), and all of these services run in a single pod.
Volume deletion protection
Volumes are not deleted if the StatefulSet is deleted. An unwanted removal of the StatefulSet (or the whole deployment) does not result in data loss because the volumes can be recovered with administrator interaction.
Headless Service DNS resolution
The Headless Service makes it possible to access the storage pod directly by using a DNS name. For example, if the pod name is swift-storage-0 and the SwiftStorage instance is named swift-storage, it becomes accessible using swift-storage-0.swift-storage. This makes it easily usable within the Object Storage service rings, and IP changes are now transparent and do not require an update of the rings.
Parallel pod management

Parallel pod management tells the StatefulSet controller to launch or end all pods in parallel, and not wait for pods to become Running and Ready or completely terminated before launching or terminating another pod. This option only affects the behavior for scaling operations. Updates are not affected.

This is required to scale by more than one, including new deployments with more than one replica. All pods must be created at the same time. Otherwise, some PVCs will not be bound, the Object Storage service rings cannot be created, and pod startup will be blocked.

Pod distribution
Distribute storage pods to different nodes to avoid single points of failure. You can use a podAntiAffinity rule with preferredDuringSchedulingIgnoredDuringExecution to distribute pods to different nodes if possible. You can use a separate storageClass and PersistentVolumes that are located on different nodes to enforce further distribution.
Network access control
Object Storage service back-end services must only be accessible by other back-end services and the Object Storage service proxy. To limit access, a NetworkPolicy is added to allow only traffic between these pods. The NetworkPolicy itself depends on labels, and these labels must match to allow traffic. Therefore labels must not be unique; instead all pods must use the same label to allow access. This is also the reason why the swift-operator does not use labels from lib-common.
Ring creation and disk information

Object Storage service rings require information about the disks to use, and this includes sizes and hostnames or IPs. Sizes are not known when starting the StatefulSet by using PVCs. The size requirement is a lower limit, but the actual PVs might be much bigger.

The ring creation process works as follows:

  • StatefulSets create PVCs before the ConfigMaps are available.
  • The SwiftRing reconciler watches the SwiftStorage instances and iterates over PVCs to get actual information about the used disks.
  • Once the PVCs are bound, the size is known and the swift-ring-rebalance job creates the Swift rings and the ConfigMap.
  • StatefulSets wait for the ConfigMap to become available, then start the service pods.
Ring storage and updates
Rings are stored in a ConfigMap mounted by the SwiftProxy and SwiftStorage instances by using projected volumes. This makes it possible to mount all required files at the same place, without merging these from other places. Updated ConfigMaps will update these files, and these changes are detected by the Swift services eventually reloading these.
Configuration customization
Some operators use the customServiceConfig option to customize settings. However, the SwiftRing instance deploys multiple backend services, and each of these requires specific files to be customized. Therefore only defaultConfigOverwrite using specific keys as filenames is supported when using the swift-operator.