Soon, all Red Hat OpenShift documentation will only be available on docs.redhat.com, the official location of all Red Hat product documentation. Get familiar with the updated experience today!
  1. Documentation
    2. history
×

Dynamic provisioning and creating storage classes

Overview

The StorageClass resource object describes and classifies storage that can be requested, as well as provides a means for passing parameters for dynamically provisioned storage on demand. StorageClass objects can also serve as a management mechanism for controlling different levels of storage and access to the storage. Cluster Administrators (cluster-admin) or Storage Administrators (storage-admin) define and create the StorageClass objects that users can request without needing any intimate knowledge about the underlying storage volume sources.

The OpenShift Container Platform persistent volume framework enables this functionality and allows administrators to provision a cluster with persistent storage. The framework also gives users a way to request those resources without having any knowledge of the underlying infrastructure.

Many storage types are available for use as persistent volumes in OpenShift Container Platform. While all of them can be statically provisioned by an administrator, some types of storage are created dynamically using the built-in provider and plug-in APIs.

To enable dynamic provisioning, add the openshift_master_dynamic_provisioning_enabled variable to the [OSEv3:vars] section of the Ansible inventory file and set its value to True.

[OSEv3:vars]

openshift_master_dynamic_provisioning_enabled=True

Available dynamically provisioned plug-ins

OpenShift Container Platform provides the following provisioner plug-ins, which have generic implementations for dynamic provisioning that use the cluster’s configured provider’s API to create new storage resources:

Storage Type Provisioner Plug-in Name Required Configuration Notes

OpenStack Cinder

kubernetes.io/cinder

Configuring for OpenStack

AWS Elastic Block Store (EBS)

kubernetes.io/aws-ebs

Configuring for AWS

For dynamic provisioning when using multiple clusters in different zones, tag each node with Key=kubernetes.io/cluster/xxxx,Value=clusterid where xxxx and clusterid are unique per cluster. In versions prior to 3.6, this was Key=KubernetesCluster,Value=clusterid.

GCE Persistent Disk (gcePD)

kubernetes.io/gce-pd

Configuring for GCE

In multi-zone configurations, it is advisable to run one Openshift cluster per GCE project to avoid PVs from getting created in zones where no node from current cluster exists.

GlusterFS

kubernetes.io/glusterfs

Container Native Storage with GlusterFS

Container Native Storage (CNS) utilizes Heketi to manage Gluster Storage.

Ceph RBD

kubernetes.io/rbd

Configuring Ceph RBD

Trident from NetApp

netapp.io/trident

Configuring for Trident

Storage orchestrator for NetApp ONTAP, SolidFire, and E-Series storage.

VMWare vSphere

kubernetes.io/vsphere-volume

Getting Started with vSphere and Kubernetes

Azure Disk

kubernetes.io/azure-disk

Configuring for Azure

Any chosen provisioner plug-in also requires configuration for the relevant cloud, host, or third-party provider as per the relevant documentation.

Defining a StorageClass

StorageClass objects are currently a globally scoped object and need to be created by cluster-admin or storage-admin users.

For GCE and AWS, a default StorageClass is created during OpenShift Container Platform installation. You can change the default StorageClass or delete it.

There are currently six plug-ins that are supported. The following sections describe the basic object definition for a StorageClass and specific examples for each of the supported plug-in types.

Basic StorageClass object definition

StorageClass Basic object definition
kind: StorageClass (1)
apiVersion: storage.k8s.io/v1 (2)
metadata:
  name: foo (3)
  annotations: (4)
     ...
provisioner: kubernetes.io/plug-in-type (5)
parameters: (6)
  param1: value
  ...
  paramN: value
1 (required) The API object type.
2 (required) The current apiVersion.
3 (required) The name of the StorageClass.
4 (optional) Annotations for the StorageClass
5 (required) The type of provisioner associated with this storage class.
6 (optional) The parameters required for the specific provisioner, this will change from plug-in to plug-in.

StorageClass annotations

To set a StorageClass as the cluster-wide default:

   storageclass.kubernetes.io/is-default-class: "true"

This enables any Persistent Volume Claim (PVC) that does not specify a specific volume to automatically be provisioned through the default StorageClass

Beta annotation storageclass.beta.kubernetes.io/is-default-class is still working. However it will be removed in a future release.

To set a StorageClass description:

   kubernetes.io/description: My StorageClass Description

OpenStack Cinder object definition

cinder-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gold
provisioner: kubernetes.io/cinder
parameters:
  type: fast  (1)
  availability: nova (2)
  fsType: ext4 (3)
1 Volume type created in Cinder. Default is empty.
2 Availability Zone. If not specified, volumes are generally round-robined across all active zones where the OpenShift Container Platform cluster has a node.
3 File system that is created on dynamically provisioned volumes. This value is copied to the fsType field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is ext4.

AWS ElasticBlockStore (EBS) object definition

aws-ebs-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1 (1)
  zone: us-east-1d (2)
  iopsPerGB: "10" (3)
  encrypted: "true" (4)
  kmsKeyId: keyvalue (5)
  fsType: ext4 (6)
1 Select from io1, gp2, sc1, st1. The default is gp2. See AWS documentation for valid Amazon Resource Name (ARN) values.
2 AWS zone. If no zone is specified, volumes are generally round-robined across all active zones where the OpenShift Container Platform cluster has a node. Zone and zones parameters must not be used at the same time.
3 Only for io1 volumes. I/O operations per second per GiB. The AWS volume plug-in multiplies this with the size of the requested volume to compute IOPS of the volume. The value cap is 20,000 IOPS, which is the maximum supported by AWS. See AWS documentation for further details.
4 Denotes whether to encrypt the EBS volume. Valid values are true or false.
5 Optional. The full ARN of the key to use when encrypting the volume. If none is supplied, but encypted is set to true, then AWS generates a key. See AWS documentation for a valid ARN value.
6 File system that is created on dynamically provisioned volumes. This value is copied to the fsType field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is ext4.

GCE PersistentDisk (gcePD) object definition

gce-pd-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard  (1)
  zone: us-central1-a  (2)
  zones: us-central1-a, us-central1-b, us-east1-b  (3)
  fsType: ext4 (4)
1 Select either pd-standard or pd-ssd. The default is pd-ssd.
2 GCE zone. If no zone is specified, volumes are generally round-robined across all active zones where the OpenShift Container Platform cluster has a node. Zone and zones parameters must not be used at the same time.
3 A comma-separated list of GCE zone(s). If no zone is specified, volumes are generally round-robined across all active zones where the OpenShift Container Platform cluster has a node. Zone and zones parameters must not be used at the same time.
4 File system that is created on dynamically provisioned volumes. This value is copied to the fsType field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is ext4.

GlusterFS object definition

glusterfs-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://127.0.0.1:8081" (1)
  restuser: "admin" (2)
  secretName: "heketi-secret" (3)
  secretNamespace: "default" (4)
  gidMin: "40000" (5)
  gidMax: "50000" (6)
1 Gluster REST service/Heketi service URL that provisions Gluster volumes on demand. The general format should be {http/https}://{IPaddress}:{Port}. This is a mandatory parameter for the GlusterFS dynamic provisioner. If the Heketi service is exposed as a routable service in the OpenShift Container Platform, it will have a resolvable fully qualified domain name and Heketi service URL. For additional information and configuration, See Container-Native Storage for OpenShift Container Platform.
2 Gluster REST service/Heketi user who has access to create volumes in the Gluster Trusted Pool.
3 Identification of a Secret instance that contains a user password to use when talking to the Gluster REST service. Optional; an empty password will be used when both secretNamespace and secretName are omitted. The provided secret must be of type "kubernetes.io/glusterfs".
4 The namespace of mentioned secretName. Optional; an empty password will be used when both secretNamespace and secretName are omitted. The provided secret must be of type "kubernetes.io/glusterfs".
5 Optional. The minimum value of GID range for the storage class.
6 Optional. The maximum value of GID range for the storage class.

When the gidMin and gidMax values are not specified, the volume is provisioned with a value between 2000 and 2147483647, which are defaults for gidMin and gidMax respectively. If specified, a unique value (GID) in this range (gidMin-gidMax) is used for dynamically provisioned volumes. The GID of the provisioned volume will be set to this value. It is required to run Heketi version 3 or later to make use of this feature. This GID is released from the pool when the subjected volume is deleted. The GID pool is per storage class, if 2 or more storage classes have GID ranges that overlap there will be duplicate GIDs dispatched by the provisioner.

When the persistent volumes are dynamically provisioned, the Gluster plug-in automatically creates an endpoint and a headless service of the name gluster-dynamic-<claimname>. When the persistent volume claim is deleted, this dynamic endpoint and service is deleted automatically.

Example of a Secret
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  # base64 encoded password. E.g.: echo -n "mypassword" | base64
  key: bXlwYXNzd29yZA==
type: kubernetes.io/glusterfs

Ceph RBD object definition

ceph-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/rbd
parameters:
  monitors: 10.16.153.105:6789  (1)
  adminId: admin  (2)
  adminSecretName: ceph-secret  (3)
  adminSecretNamespace: kube-system  (4)
  pool: kube  (5)
  userId: kube  (6)
  userSecretName: ceph-secret-user  (7)
  fsType: ext4 (8)
1 Ceph monitors, comma-delimited. It is required.
2 Ceph client ID that is capable of creating images in the pool. Default is "admin".
3 Secret Name for adminId. It is required. The provided secret must have type "kubernetes.io/rbd".
4 The namespace for adminSecret. Default is "default".
5 Ceph RBD pool. Default is "rbd".
6 Ceph client ID that is used to map the Ceph RBD image. Default is the same as adminId.
7 The name of Ceph Secret for userId to map Ceph RBD image. It must exist in the same namespace as PVCs. It is required.
8 File system that is created on dynamically provisioned volumes. This value is copied to the fsType field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is ext4.

Trident object definition

trident.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gold
provisioner: netapp.io/trident (1)
parameters: (2)
  media: "ssd"
  provisioningType: "thin"
  snapshots: "true"

Trident uses the parameters as selection criteria for the different pools of storage that are registered with it. Trident itself is configured separately.

1 For more information about installing Trident with OpenShift Container Platform, see the Trident documentation.
2 For more information about supported parameters, see the storage attributes section of the Trident documentation.

VMware vSphere object definition

vsphere-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: slow
provisioner: kubernetes.io/vsphere-volume (1)
parameters:
  diskformat: thin (2)
1 For more information about using VMWare vSphere with OpenShift Container Platform, see the VMWare vSphere documentation.
2 diskformat: thin, zeroedthick and eagerzeroedthick. See vSphere docs for details. Default: thin

Azure Disk object definition

azure-advanced-disk-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/azure-disk
parameters:
  storageAccount: azure_storage_account_name  (1)
  storageaccounttype: Standard_LRS  (2)
  kind: Dedicated  (3)
1 Azure storage account name. This must reside in the same resource group as the cluster. If a storage account is specified, the location is ignored. If a storage account is not specified, a new storage account gets created in the same resource group as the cluster. If you are specifying a storageAccount, the value for kind must be Dedicated.
2 Azure storage account SKU tier. Default is empty. Note: Premium VM can attach both Standard_LRS and Premium_LRS disks, Standard VM can only attach Standard_LRS disks, Managed VM can only attach managed disks, and unmanaged VM can only attach unmanaged disks.
3 Possible values are Shared (default), Dedicated, and Managed.

  1. If kind is set to Shared, Azure creates all unmanaged disks in a few shared storage accounts in the same resource group as the cluster.

  2. If kind is set to Managed, Azure creates new managed disks.

  3. If kind is set to Dedicated and a storageAccount is specified, Azure uses the specified storage account for the new unmanaged disk in the same resource group as the cluster. For this to work:

    • The specified storage account must be in the same region.

    • Azure Cloud Provider must have a write access to the storage account.

  4. If kind is set to Dedicated and a storageAccount is not specified, Azure creates a new dedicated storage account for the new unmanaged disk in the same resource group as the cluster.

Azure StorageClass is revised in OpenShift Container Platform version 3.7. If you upgraded from a previous version, either:

  • specify the property kind: dedicated to continue using the Azure StorageClass created before the upgrade. Or,

  • add the location parameter (for example, "location": "southcentralus",) in the azure.conf file to use the default property kind: shared. Doing this creates new storage accounts for future use.

Changing the default StorageClass

If you are using GCE and AWS, use the following process to change the default StorageClass:

  1. List the StorageClass:

    $ oc get storageclass

NAME                 TYPE
gp2 (default)        kubernetes.io/aws-ebs (1)
standard             kubernetes.io/gce-pd
    1 (default) denotes the default StorageClass.

  2. Change the value of the annotation storageclass.kubernetes.io/is-default-class to false for the default StorageClass:

    $ oc patch storageclass gp2 -p '{"metadata": {"annotations": \
    {"storageclass.kubernetes.io/is-default-class": "false"}}}'

  3. Make another StorageClass the default by adding or modifying the annotation as storageclass.kubernetes.io/is-default-class=true.

    $ oc patch storageclass standard -p '{"metadata": {"annotations": \
    {"storageclass.kubernetes.io/is-default-class": "true"}}}'

  4. Verify the changes:

    $ oc get storageclass

NAME                 TYPE
gp2                  kubernetes.io/aws-ebs
standard (default)   kubernetes.io/gce-pd

Additional information and examples