$ oc adm new-project openshift-local-storage
OpenShift Container Platform can be provisioned with persistent storage by using local volumes. Local persistent volumes allow you to access local storage devices, such as a disk or partition, by using the standard persistent volume claim interface.
Local volumes can be used without manually scheduling pods to nodes because the system is aware of the volume node constraints. However, local volumes are still subject to the availability of the underlying node and are not suitable for all applications.
Local volumes can only be used as a statically created persistent volume. |
The Local Storage Operator is not installed in OpenShift Container Platform by default. Use the following procedure to install and configure this Operator to enable local volumes in your cluster.
Access to the OpenShift Container Platform web console or command-line interface (CLI).
Create the openshift-local-storage
project:
$ oc adm new-project openshift-local-storage
Optional: Allow local storage creation on infrastructure nodes.
You might want to use the Local Storage Operator to create volumes on infrastructure nodes in support of components such as logging and monitoring.
You must adjust the default node selector so that the Local Storage Operator includes the infrastructure nodes, and not just worker nodes.
To block the Local Storage Operator from inheriting the cluster-wide default selector, enter the following command:
$ oc annotate namespace openshift-local-storage openshift.io/node-selector=''
Optional: Allow local storage to run on the management pool of CPUs in single-node deployment.
Use the Local Storage Operator in single-node deployments and allow the use of CPUs that belong to the management
pool. Perform this step on single-node installations that use management workload partitioning.
To allow Local Storage Operator to run on the management CPU pool, run following commands:
$ oc annotate namespace openshift-local-storage workload.openshift.io/allowed='management'
To install the Local Storage Operator from the web console, follow these steps:
Log in to the OpenShift Container Platform web console.
Navigate to Operators → OperatorHub.
Type Local Storage into the filter box to locate the Local Storage Operator.
Click Install.
On the Install Operator page, select A specific namespace on the cluster. Select openshift-local-storage from the drop-down menu.
Adjust the values for Update Channel and Approval Strategy to the values that you want.
Click Install.
Once finished, the Local Storage Operator will be listed in the Installed Operators section of the web console.
Install the Local Storage Operator from the CLI.
Create an object YAML file to define an Operator group and subscription for the Local Storage Operator,
such as openshift-local-storage.yaml
:
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: local-operator-group
namespace: openshift-local-storage
spec:
targetNamespaces:
- openshift-local-storage
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: local-storage-operator
namespace: openshift-local-storage
spec:
channel: stable
installPlanApproval: Automatic (1)
name: local-storage-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
1 | The user approval policy for an install plan. |
Create the Local Storage Operator object by entering the following command:
$ oc apply -f openshift-local-storage.yaml
At this point, the Operator Lifecycle Manager (OLM) is now aware of the Local Storage Operator. A ClusterServiceVersion (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.
Verify local storage installation by checking that all pods and the Local Storage Operator have been created:
Check that all the required pods have been created:
$ oc -n openshift-local-storage get pods
NAME READY STATUS RESTARTS AGE
local-storage-operator-746bf599c9-vlt5t 1/1 Running 0 19m
Check the ClusterServiceVersion (CSV) YAML manifest to see that the Local Storage Operator is available in the openshift-local-storage
project:
$ oc get csvs -n openshift-local-storage
NAME DISPLAY VERSION REPLACES PHASE
local-storage-operator.4.2.26-202003230335 Local Storage 4.2.26-202003230335 Succeeded
After all checks have passed, the Local Storage Operator is installed successfully.
Local volumes cannot be created by dynamic provisioning. Instead, persistent volumes can be created by the Local Storage Operator. The local volume provisioner looks for any file system or block volume devices at the paths specified in the defined resource.
The Local Storage Operator is installed.
You have a local disk that meets the following conditions:
It is attached to a node.
It is not mounted.
It does not contain partitions.
Create the local volume resource. This resource must define the nodes and paths to the local volumes.
Do not use different storage class names for the same device. Doing so will create multiple persistent volumes (PVs). |
apiVersion: "local.storage.openshift.io/v1"
kind: "LocalVolume"
metadata:
name: "local-disks"
namespace: "openshift-local-storage" (1)
spec:
nodeSelector: (2)
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ip-10-0-140-183
- ip-10-0-158-139
- ip-10-0-164-33
storageClassDevices:
- storageClassName: "local-sc" (3)
forceWipeDevicesAndDestroyAllData: false (4)
volumeMode: Filesystem (5)
fsType: xfs (6)
devicePaths: (7)
- /path/to/device (8)
1 | The namespace where the Local Storage Operator is installed. | ||
2 | Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node . If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes. |
||
3 | The name of the storage class to use when creating persistent volume objects. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes. | ||
4 | This setting defines whether or not to call wipefs , which removes partition table signatures (magic strings) making the disk ready to use for Local Storage Operator (LSO) provisioning. No other data besides signatures is erased. The default is "false" (wipefs is not invoked). Setting forceWipeDevicesAndDestroyAllData to "true" can be useful in scenarios where previous data can remain on disks that need to be re-used. In these scenarios, setting this field to true eliminates the need for administrators to erase the disks manually. Such cases can include single-node OpenShift (SNO) cluster environments where a node can be redeployed multiple times or when using OpenShift Data Foundation (ODF), where previous data can remain on the disks planned to be consumed as object storage devices (OSDs). |
||
5 | The volume mode, either Filesystem or Block , that defines the type of local volumes.
|
||
6 | The file system that is created when the local volume is mounted for the first time. | ||
7 | The path containing a list of local storage devices to choose from. | ||
8 | Replace this value with your actual local disks filepath to the LocalVolume resource by-id , such as /dev/disk/by-id/wwn . PVs are created for these local disks when the provisioner is deployed successfully.
|
apiVersion: "local.storage.openshift.io/v1"
kind: "LocalVolume"
metadata:
name: "local-disks"
namespace: "openshift-local-storage" (1)
spec:
nodeSelector: (2)
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ip-10-0-136-143
- ip-10-0-140-255
- ip-10-0-144-180
storageClassDevices:
- storageClassName: "local-sc" (3)
forceWipeDevicesAndDestroyAllData: false (4)
volumeMode: Block (5)
devicePaths: (6)
- /path/to/device (7)
1 | The namespace where the Local Storage Operator is installed. |
2 | Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node . If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes. |
3 | The name of the storage class to use when creating persistent volume objects. |
4 | This setting defines whether or not to call wipefs , which removes partition table signatures (magic strings) making the disk ready to use for Local Storage Operator (LSO) provisioning. No other data besides signatures is erased. The default is "false" (wipefs is not invoked). Setting forceWipeDevicesAndDestroyAllData to "true" can be useful in scenarios where previous data can remain on disks that need to be re-used. In these scenarios, setting this field to true eliminates the need for administrators to erase the disks manually. Such cases can include single-node OpenShift (SNO) cluster environments where a node can be redeployed multiple times or when using OpenShift Data Foundation (ODF), where previous data can remain on the disks planned to be consumed as object storage devices (OSDs). |
5 | The volume mode, either Filesystem or Block , that defines the type of local volumes. |
6 | The path containing a list of local storage devices to choose from. |
7 | Replace this value with your actual local disks filepath to the LocalVolume resource by-id , such as dev/disk/by-id/wwn . PVs are created for these local disks when the provisioner is deployed successfully. |
If you are running OpenShift Container Platform with RHEL KVM, you must assign a serial number to your VM disk. Otherwise, the VM disk can not be identified after reboot. You can use the |
Create the local volume resource in your OpenShift Container Platform cluster. Specify the file you just created:
$ oc create -f <local-volume>.yaml
Verify that the provisioner was created and that the corresponding daemon sets were created:
$ oc get all -n openshift-local-storage
NAME READY STATUS RESTARTS AGE
pod/diskmaker-manager-9wzms 1/1 Running 0 5m43s
pod/diskmaker-manager-jgvjp 1/1 Running 0 5m43s
pod/diskmaker-manager-tbdsj 1/1 Running 0 5m43s
pod/local-storage-operator-7db4bd9f79-t6k87 1/1 Running 0 14m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/local-storage-operator-metrics ClusterIP 172.30.135.36 <none> 8383/TCP,8686/TCP 14m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/diskmaker-manager 3 3 3 3 3 <none> 5m43s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/local-storage-operator 1/1 1 1 14m
NAME DESIRED CURRENT READY AGE
replicaset.apps/local-storage-operator-7db4bd9f79 1 1 1 14m
Note the desired and current number of daemon set processes. A desired count of 0
indicates that the label selectors were invalid.
Verify that the persistent volumes were created:
$ oc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
local-pv-1cec77cf 100Gi RWO Delete Available local-sc 88m
local-pv-2ef7cd2a 100Gi RWO Delete Available local-sc 82m
local-pv-3fa1c73 100Gi RWO Delete Available local-sc 48m
Editing the |
Local volumes cannot be created by dynamic provisioning. Instead, persistent volumes can be created by defining the persistent volume (PV) in an object definition. The local volume provisioner looks for any file system or block volume devices at the paths specified in the defined resource.
Manual provisioning of PVs includes the risk of potential data leaks across PV reuse when PVCs are deleted. The Local Storage Operator is recommended for automating the life cycle of devices when provisioning local PVs. |
Local disks are attached to the OpenShift Container Platform nodes.
Define the PV. Create a file, such as example-pv-filesystem.yaml
or example-pv-block.yaml
, with the PersistentVolume
object definition. This resource must define the nodes and paths to the local volumes.
Do not use different storage class names for the same device. Doing so will create multiple PVs. |
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-pv-filesystem
spec:
capacity:
storage: 100Gi
volumeMode: Filesystem (1)
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-sc (2)
local:
path: /dev/xvdf (3)
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- example-node
1 | The volume mode, either Filesystem or Block , that defines the type of PVs. |
2 | The name of the storage class to use when creating PV resources. Use a storage class that uniquely identifies this set of PVs. |
3 | The path containing a list of local storage devices to choose from, or a directory. You can only specify a directory with Filesystem volumeMode . |
A raw block volume ( |
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-pv-block
spec:
capacity:
storage: 100Gi
volumeMode: Block (1)
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-sc (2)
local:
path: /dev/xvdf (3)
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- example-node
1 | The volume mode, either Filesystem or Block , that defines the type of PVs. |
2 | The name of the storage class to use when creating PV resources. Be sure to use a storage class that uniquely identifies this set of PVs. |
3 | The path containing a list of local storage devices to choose from. |
Create the PV resource in your OpenShift Container Platform cluster. Specify the file you just created:
$ oc create -f <example-pv>.yaml
Verify that the local PV was created:
$ oc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
example-pv-filesystem 100Gi RWO Delete Available local-sc 3m47s
example-pv1 1Gi RWO Delete Bound local-storage/pvc1 local-sc 12h
example-pv2 1Gi RWO Delete Bound local-storage/pvc2 local-sc 12h
example-pv3 1Gi RWO Delete Bound local-storage/pvc3 local-sc 12h
Local volumes must be statically created as a persistent volume claim (PVC) to be accessed by the pod.
Persistent volumes have been created using the local volume provisioner.
Create the PVC using the corresponding storage class:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: local-pvc-name (1)
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem (2)
resources:
requests:
storage: 100Gi (3)
storageClassName: local-sc (4)
1 | Name of the PVC. |
2 | The type of the PVC. Defaults to Filesystem . |
3 | The amount of storage available to the PVC. |
4 | Name of the storage class required by the claim. |
Create the PVC in the OpenShift Container Platform cluster, specifying the file you just created:
$ oc create -f <local-pvc>.yaml
After a local volume has been mapped to a persistent volume claim it can be specified inside of a resource.
A persistent volume claim exists in the same namespace.
Include the defined claim in the resource spec. The following example declares the persistent volume claim inside a pod:
apiVersion: v1
kind: Pod
spec:
# ...
containers:
volumeMounts:
- name: local-disks (1)
mountPath: /data (2)
volumes:
- name: local-disks
persistentVolumeClaim:
claimName: local-pvc-name (3)
# ...
1 | The name of the volume to mount. |
2 | The path inside the pod where the volume is mounted. Do not mount to the container root, / , or any path that is the same in the host and the container. This can corrupt your host system if the container is sufficiently privileged, such as the host /dev/pts files. It is safe to mount the host by using /host . |
3 | The name of the existing persistent volume claim to use. |
Create the resource in the OpenShift Container Platform cluster, specifying the file you just created:
$ oc create -f <local-pod>.yaml
The Local Storage Operator automates local storage discovery and provisioning. With this feature, you can simplify installation when dynamic provisioning is not available during deployment, such as with bare metal, VMware, or AWS store instances with attached devices.
Automatic discovery and provisioning is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
Automatic discovery and provisioning is fully supported when used to deploy Red Hat OpenShift Data Foundation on-premise or with platform-agnostic deployment. |
Use the following procedure to automatically discover local devices, and to automatically provision local volumes for selected devices.
Use the |
You have cluster administrator permissions.
You have installed the Local Storage Operator.
You have attached local disks to OpenShift Container Platform nodes.
You have access to the OpenShift Container Platform web console and the oc
command-line interface (CLI).
To enable automatic discovery of local devices from the web console:
Click Operators → Installed Operators.
In the openshift-local-storage
namespace, click Local Storage.
Click the Local Volume Discovery tab.
Click Create Local Volume Discovery and then select either Form view or YAML view.
Configure the LocalVolumeDiscovery
object parameters.
Click Create.
The Local Storage Operator creates a local volume discovery instance named auto-discover-devices
.
To display a continuous list of available devices on a node:
Log in to the OpenShift Container Platform web console.
Navigate to Compute → Nodes.
Click the node name that you want to open. The "Node Details" page is displayed.
Select the Disks tab to display the list of the selected devices.
The device list updates continuously as local disks are added or removed. You can filter the devices by name, status, type, model, capacity, and mode.
To automatically provision local volumes for the discovered devices from the web console:
Navigate to Operators → Installed Operators and select Local Storage from the list of Operators.
Select Local Volume Set → Create Local Volume Set.
Enter a volume set name and a storage class name.
Choose All nodes or Select nodes to apply filters accordingly.
Only worker nodes are available, regardless of whether you filter using All nodes or Select nodes. |
Select the disk type, mode, size, and limit you want to apply to the local volume set, and click Create.
A message displays after several minutes, indicating that the "Operator reconciled successfully."
Alternatively, to provision local volumes for the discovered devices from the CLI:
Create an object YAML file to define the local volume set, such as local-volume-set.yaml
, as shown in the following example:
apiVersion: local.storage.openshift.io/v1alpha1
kind: LocalVolumeSet
metadata:
name: example-autodetect
spec:
nodeSelector:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- worker-0
- worker-1
storageClassName: local-sc (1)
volumeMode: Filesystem
fsType: ext4
maxDeviceCount: 10
deviceInclusionSpec:
deviceTypes: (2)
- disk
- part
deviceMechanicalProperties:
- NonRotational
minSize: 10G
maxSize: 100G
models:
- SAMSUNG
- Crucial_CT525MX3
vendors:
- ATA
- ST2000LM
1 | Determines the storage class that is created for persistent volumes that are provisioned from discovered devices. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes. |
2 | When using the local volume set feature, the Local Storage Operator does not support the use of logical volume management (LVM) devices. |
Create the local volume set object:
$ oc apply -f local-volume-set.yaml
Verify that the local persistent volumes were dynamically provisioned based on the storage class:
$ oc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
local-pv-1cec77cf 100Gi RWO Delete Available local-sc 88m
local-pv-2ef7cd2a 100Gi RWO Delete Available local-sc 82m
local-pv-3fa1c73 100Gi RWO Delete Available local-sc 48m
Results are deleted after they are removed from the node. Symlinks must be manually removed. |
Taints can be applied to nodes to prevent them from running general workloads. To allow the Local Storage Operator to use tainted nodes, you must add tolerations to the Pod
or DaemonSet
definition. This allows the created resources to run on these tainted nodes.
You apply tolerations to the Local Storage Operator pod through the LocalVolume
resource
and apply taints to a node through the node specification. A taint on a node instructs the node to repel all pods that do not tolerate the taint. Using a specific taint that is not on other pods ensures that the Local Storage Operator pod can also run on that node.
Taints and tolerations consist of a key, value, and effect. As an argument, it is expressed as |
The Local Storage Operator is installed.
Local disks are attached to OpenShift Container Platform nodes with a taint.
Tainted nodes are expected to provision local storage.
To configure local volumes for scheduling on tainted nodes:
Modify the YAML file that defines the Pod
and add the LocalVolume
spec, as shown in the following example:
apiVersion: "local.storage.openshift.io/v1"
kind: "LocalVolume"
metadata:
name: "local-disks"
namespace: "openshift-local-storage"
spec:
tolerations:
- key: localstorage (1)
operator: Equal (2)
value: "localstorage" (3)
storageClassDevices:
- storageClassName: "local-sc"
volumeMode: Block (4)
devicePaths: (5)
- /dev/xvdg
1 | Specify the key that you added to the node. |
2 | Specify the Equal operator to require the key /value parameters to match. If operator is Exists , the system checks that the key exists and ignores the value. If operator is Equal , then the key and value must match. |
3 | Specify the value local of the tainted node. |
4 | The volume mode, either Filesystem or Block , defining the type of the local volumes. |
5 | The path containing a list of local storage devices to choose from. |
Optional: To create local persistent volumes on only tainted nodes, modify the YAML file and add the LocalVolume
spec, as shown in the following example:
spec:
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
The defined tolerations will be passed to the resulting daemon sets, allowing the diskmaker and provisioner pods to be created for nodes that contain the specified taints.
OpenShift Container Platform provides the following metrics for the Local Storage Operator:
lso_discovery_disk_count
: total number of discovered devices on each node
lso_lvset_provisioned_PV_count
: total number of PVs created by LocalVolumeSet
objects
lso_lvset_unmatched_disk_count
: total number of disks that Local Storage Operator did not select for provisioning because of mismatching criteria
lso_lvset_orphaned_symlink_count
: number of devices with PVs that no longer match LocalVolumeSet
object criteria
lso_lv_orphaned_symlink_count
: number of devices with PVs that no longer match LocalVolume
object criteria
lso_lv_provisioned_PV_count
: total number of provisioned PVs for LocalVolume
To use these metrics, be sure to:
Enable support for monitoring when installing the Local Storage Operator.
When upgrading to OpenShift Container Platform 4.9 or later, enable metric support manually by adding the operator-metering=true
label to the namespace.
For more information about metrics, see Managing metrics.
Occasionally, local volumes and local volume sets must be deleted. While removing the entry in the resource and deleting the persistent volume is typically enough, if you want to reuse the same device path or have it managed by a different storage class, then additional steps are needed.
The following procedure outlines an example for removing a local volume. The same procedure can also be used to remove symlinks for a local volume set custom resource. |
The persistent volume must be in a Released
or Available
state.
Deleting a persistent volume that is still in use can result in data loss or corruption. |
Edit the previously created local volume to remove any unwanted disks.
Edit the cluster resource:
$ oc edit localvolume <name> -n openshift-local-storage
Navigate to the lines under devicePaths
, and delete any representing unwanted disks.
Delete any persistent volumes created.
$ oc delete pv <pv-name>
Delete directory and included symlinks on the node.
The following step involves accessing a node as the root user. Modifying the state of the node beyond the steps in this procedure could result in cluster instability. |
$ oc debug node/<node-name> -- chroot /host rm -rf /mnt/local-storage/<sc-name> (1)
1 | The name of the storage class used to create the local volumes. |
To uninstall the Local Storage Operator, you must remove the Operator and all created resources in the openshift-local-storage
project.
Uninstalling the Local Storage Operator while local storage PVs are still in use is not recommended. While the PVs will remain after the Operator’s removal, there might be indeterminate behavior if the Operator is uninstalled and reinstalled without removing the PVs and local storage resources. |
Access to the OpenShift Container Platform web console.
Delete any local volume resources installed in the project, such as localvolume
, localvolumeset
, and localvolumediscovery
:
$ oc delete localvolume --all --all-namespaces
$ oc delete localvolumeset --all --all-namespaces
$ oc delete localvolumediscovery --all --all-namespaces
Uninstall the Local Storage Operator from the web console.
Log in to the OpenShift Container Platform web console.
Navigate to Operators → Installed Operators.
Type Local Storage into the filter box to locate the Local Storage Operator.
Click the Options menu at the end of the Local Storage Operator.
Click Uninstall Operator.
Click Remove in the window that appears.
The PVs created by the Local Storage Operator will remain in the cluster until deleted. After these volumes are no longer in use, delete them by running the following command:
$ oc delete pv <pv-name>
Delete the openshift-local-storage
project:
$ oc delete project openshift-local-storage