RHOCP storage sizing
You can configure the Image service (glance) and Object Storage service (swift) to allocate space in the RHOCP backing storage. In this scenario, estimate the RHOCP storage sizing based on the expected use of these services.
Image service storage requirements Copy linkLink copied!
The Image service (glance) requires persistent storage for staging and importing data, which you can provide by using Persistent Volume Claims (PVCs) or an external model with custom storage mounts.
- External model
-
If you choose external, no PVCs are created and the Image service acts like a stateless instance with no persistence provided. In this instance, persistence must be provided using
extraMounts. NFS is often used to provide persistence. It can be mapped to/var/lib/glance:... default: storage: external: true ... ... extraMounts: - extraVol: - extraVolType: NFS mounts: - mountPath: /var/lib/glance/os_glance_staging_store name: nfs volumes: - name: nfs nfs: path: <nfs_export_path> server: <nfs_ip_address>- Replace
<nfs_export_path>with the export path of your NFS share. - Replace
<nfs_ip_address>with the IP address of your NFS share. This IP address must be part of the overlay network that is reachable by the Image service.
It should be noted that the configuration sample conflicts with the distributed image import feature. Distributed image import requires RWO storage plugged into a particular instance; it owns the data and receives requests in case its staged data is required for an upload operation. When the External model is adopted, if Red Hat Ceph Storage is used as a backend, and an image conversion operation is run in one of the existing replicas, the
glance-operatordoes not have to make any assumption about the underlying storage that is tied to the staging area, and the conversion operation that uses theos_glance_staging_store directory(within the Pod) interacts with the RWX NFS backend provided viaextraMounts. With this scenario, no image-cache PVC can be requested and mounted to a subPath, because it should be the administrator’s responsibility to plan for persistence by usingextraMounts. - Replace
- PVC model
-
The PVC model is the default. When a GlanceAPI instance is deployed, a PVC is created and bound to
/var/lib/glanceaccording to thestorageClassandstorageRequestpassed as input.... default: replicas: 3 storage: storageRequest: 10G ...In this model, if Red Hat Ceph Storage is set as a back end, no dedicated image conversion PVC is created. The administrator must think about the PVC sizing in advance; the size of the PVC should be at least up to the largest converted image size. Concurrent conversions within the same Pod might be problematic in terms of PVC size; a conversion will fail or cannot take place if the PVC is full and there’s not enough space. The upload should be retried after the previous conversion is over and the staging area space is released. However, concurrent conversion operations might happen in different Pods. You should deploy at least 3 replicas for a particular
glanceAPI. This helps to handle heavy operations such as image conversion.For a PVC-based layout, the scale out of a
glanceAPIin terms of replicas is limited by the available storage provided by thestorageClass, and depends on thestorageRequest. ThestorageRequestis a critical parameter, it can be globally defined for all theglanceAPI, or defined with a different value for each API. It will influence the scale out operations for each of them. Other than a local PVC required for the staging area, it is possible to enable image cache, which is translated into an additional PVC bound to eachglanceAPIinstance. Aglance-cachePVC is bound to/var/lib/glance/image-cache. The glance-operator configures the glanceAPI instance accordingly, setting bothimage_cache_max_sizeand theimage_cache_dirparameters. The number of image cache PVCs follows the same rules described for the local PVC, and the number of requested PVCs is proportional to the number of replicas.
Object Storage service storage requirements Copy linkLink copied!
The Object Storage service (swift) requires storage devices for data. These devices must be accessible using the same hostname or IP address during their lifetime. You achieve this by configuring a StatefulSet with a Headless Service.
- StatefulSets
- If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution. Although individual pods in a StatefulSet are susceptible to failure, the persistent pod identifiers make it easier to match existing volumes to the new pods that replace any that have failed.
- Service architecture
- The Object Storage service requires services to access Persistent Volumes (PVs), and all of these services run in a single pod.
- Volume deletion protection
- Volumes are not deleted if the StatefulSet is deleted. An unwanted removal of the StatefulSet (or the whole deployment) does not result in data loss because the volumes can be recovered with administrator interaction.
- Headless Service DNS resolution
- The Headless Service makes it possible to access the storage pod directly by using a DNS name. For example, if the pod name is
swift-storage-0and theSwiftStorageinstance is namedswift-storage, it becomes accessible usingswift-storage-0.swift-storage. This makes it easily usable within the Object Storage service rings, and IP changes are now transparent and do not require an update of the rings. - Parallel pod management
-
Parallel pod management tells the StatefulSet controller to launch or end all pods in parallel, and not wait for pods to become
RunningandReadyor completely terminated before launching or terminating another pod. This option only affects the behavior for scaling operations. Updates are not affected.This is required to scale by more than one, including new deployments with more than one replica. All pods must be created at the same time. Otherwise, some PVCs will not be bound, the Object Storage service rings cannot be created, and pod startup will be blocked.
- Pod distribution
- Distribute storage pods to different nodes to avoid single points of failure. You can use a
podAntiAffinityrule withpreferredDuringSchedulingIgnoredDuringExecutionto distribute pods to different nodes if possible. You can use a separatestorageClassand PersistentVolumes that are located on different nodes to enforce further distribution. - Network access control
- Object Storage service back-end services must only be accessible by other back-end services and the Object Storage service proxy. To limit access, a
NetworkPolicyis added to allow only traffic between these pods. TheNetworkPolicyitself depends on labels, and these labels must match to allow traffic. Therefore labels must not be unique; instead all pods must use the same label to allow access. This is also the reason why theswift-operatordoes not use labels fromlib-common. - Ring creation and disk information
-
Object Storage service rings require information about the disks to use, and this includes sizes and hostnames or IPs. Sizes are not known when starting the StatefulSet by using PVCs. The size requirement is a lower limit, but the actual PVs might be much bigger.
The ring creation process works as follows:
- StatefulSets create PVCs before the
ConfigMapsare available. - The
SwiftRingreconciler watches theSwiftStorageinstances and iterates over PVCs to get actual information about the used disks. - Once the PVCs are bound, the size is known and the
swift-ring-rebalancejob creates the Swift rings and theConfigMap. - StatefulSets wait for the
ConfigMapto become available, then start the service pods.
- StatefulSets create PVCs before the
- Ring storage and updates
- Rings are stored in a
ConfigMapmounted by theSwiftProxyandSwiftStorageinstances by using projected volumes. This makes it possible to mount all required files at the same place, without merging these from other places. UpdatedConfigMapswill update these files, and these changes are detected by the Swift services eventually reloading these. - Configuration customization
- Some operators use the
customServiceConfigoption to customize settings. However, theSwiftRinginstance deploys multiple backend services, and each of these requires specific files to be customized. Therefore onlydefaultConfigOverwriteusing specific keys as filenames is supported when using theswift-operator.