×

Overview

OpenShift Container Platform is capable of provisioning persistent volumes (PVs) using the Container Storage Interface (CSI) driver for AWS Elastic File Service (EFS).

Familiarity with persistent storage and configuring CSI volumes is recommended when working with a CSI Operator and driver.

After installing the AWS EFS CSI Driver Operator, OpenShift Container Platform installs the AWS EFS CSI Operator and the AWS EFS CSI driver by default in the openshift-cluster-csi-drivers namespace. This allows the AWS EFS CSI Driver Operator to create CSI-provisioned PVs that mount to AWS EFS assets.

  • The AWS EFS CSI Driver Operator, after being installed, does not create a storage class by default to use to create persistent volume claims (PVCs). However, you can manually create the AWS EFS StorageClass. The AWS EFS CSI Driver Operator supports dynamic volume provisioning by allowing storage volumes to be created on-demand. This eliminates the need for cluster administrators to pre-provision storage.

  • The AWS EFS CSI driver enables you to create and mount AWS EFS PVs.

AWS EFS only supports regional volumes, not zonal volumes.

About CSI

Storage vendors have traditionally provided storage drivers as part of Kubernetes. With the implementation of the Container Storage Interface (CSI), third-party providers can instead deliver storage plug-ins using a standard interface without ever having to change the core Kubernetes code.

CSI Operators give OpenShift Container Platform users storage options, such as volume snapshots, that are not possible with in-tree volume plug-ins.

Installing the AWS EFS CSI Driver Operator

The AWS EFS CSI Driver Operator is not installed in OpenShift Container Platform by default. Use the following procedure to install and configure the AWS EFS CSI Driver Operator in your cluster.

Prerequisites
  • Access to the OpenShift Container Platform web console.

Procedure

To install the AWS EFS CSI Driver Operator from the web console:

  1. Log in to the web console.

  2. Install the AWS EFS CSI Operator:

    1. Click OperatorsOperatorHub.

    2. Locate the AWS EFS CSI Operator by typing AWS EFS CSI in the filter box.

    3. Click the AWS EFS CSI Driver Operator button.

      Be sure to select the AWS EFS CSI Driver Operator and not the AWS EFS Operator. The AWS EFS Operator is a community Operator and is not supported by Red Hat.

    4. On the AWS EFS CSI Driver Operator page, click Install.

    5. On the Install Operator page, ensure that:

      • All namespaces on the cluster (default) is selected.

      • Installed Namespace is set to openshift-cluster-csi-drivers.

    6. Click Install.

      After the installation finishes, the AWS EFS CSI Operator is listed in the Installed Operators section of the web console.

  3. If you are using AWS EFS with AWS Secure Token Service (STS), you must configure the AWS EFS CSI Driver with STS. For more information, see "Configuring AWS EFS CSI Driver with STS".

  4. Install the AWS EFS CSI Driver:

    1. Click administrationCustomResourceDefinitionsClusterCSIDriver.

    2. On the Instances tab, click Create ClusterCSIDriver.

    3. Use the following YAML file:

      apiVersion: operator.openshift.io/v1
      kind: ClusterCSIDriver
      metadata:
          name: efs.csi.aws.com
      spec:
        managementState: Managed
    4. Click Create.

    5. Wait for the following Conditions to change to a "true" status:

      • AWSEFSDriverCredentialsRequestControllerAvailable

      • AWSEFSDriverNodeServiceControllerAvailable

      • AWSEFSDriverControllerServiceControllerAvailable

Configuring AWS EFS CSI Driver Operator with Secure Token Service

This procedure explains how to configure the AWS EFS CSI Driver Operator with OpenShift Container Platform on AWS Secure Token Service (STS).

Perform this procedure after installing the AWS EFS CSI Operator, but before installing the AWS EFS CSI driver as part of Installing the AWS EFS CSI Driver Operator procedure. If you perform this procedure after installing the driver and creating volumes, your volumes will fail to mount into pods.

Prerequisites
  • AWS account credentials

Procedure

To configure the AWS EFS CSI Driver Operator with STS:

  1. Extract the CCO utility (ccoctl) binary from the OpenShift Container Platform release image, which you used to install the cluster with STS. For more information, see "Configuring the Cloud Credential Operator utility".

  2. Create and save an EFS CredentialsRequest YAML file, such as shown in the following example, and then place it in the credrequests directory:

    Example
    apiVersion: cloudcredential.openshift.io/v1
    kind: CredentialsRequest
    metadata:
      name: openshift-aws-efs-csi-driver
      namespace: openshift-cloud-credential-operator
    spec:
      providerSpec:
        apiVersion: cloudcredential.openshift.io/v1
        kind: AWSProviderSpec
        statementEntries:
        - action:
          - elasticfilesystem:*
          effect: Allow
          resource: '*'
      secretRef:
        name: aws-efs-cloud-credentials
        namespace: openshift-cluster-csi-drivers
      serviceAccountNames:
      - aws-efs-csi-driver-operator
      - aws-efs-csi-driver-controller-sa
  3. Run the ccoctl tool to generate a new IAM role in AWS, and create a YAML file for it in the local file system (<path_to_ccoctl_output_dir>/manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml).

    $ ccoctl aws create-iam-roles --name=<name> --region=<aws_region> --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests --identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com
    • name=<name> is the name used to tag any cloud resources that are created for tracking.

    • region=<aws_region> is the AWS region where cloud resources are created.

    • dir=<path_to_directory_with_list_of_credentials_requests>/credrequests is the directory containing the EFS CredentialsRequest file in previous step.

    • <aws_account_id> is the AWS account ID.

      Example
      $ ccoctl aws create-iam-roles --name my-aws-efs --credentials-requests-dir credrequests --identity-provider-arn arn:aws:iam::123456789012:oidc-provider/my-aws-efs-oidc.s3.us-east-2.amazonaws.com
      Example output
      2022/03/21 06:24:44 Role arn:aws:iam::123456789012:role/my-aws-efs -openshift-cluster-csi-drivers-aws-efs-cloud- created
      2022/03/21 06:24:44 Saved credentials configuration to: /manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml
      2022/03/21 06:24:45 Updated Role policy for Role my-aws-efs-openshift-cluster-csi-drivers-aws-efs-cloud-
  4. Create the AWS EFS cloud credentials and secret:

    $ oc create -f <path_to_ccoctl_output_dir>/manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml
    Example
    $ oc create -f /manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml
    Example output
    secret/aws-efs-cloud-credentials created

Creating the AWS EFS storage class

Storage classes are used to differentiate and delineate storage levels and usages. By defining a storage class, users can obtain dynamically provisioned persistent volumes.

The AWS EFS CSI Driver Operator, after being installed, does not create a storage class by default. However, you can manually create the AWS EFS storage class.

Creating the AWS EFS storage class using the console

Procedure
  1. In the OpenShift Container Platform console, click StorageStorageClasses.

  2. On the StorageClasses page, click Create StorageClass.

  3. On the StorageClass page, perform the following steps:

    1. Enter a name to reference the storage class.

    2. Optional: Enter the description.

    3. Select the reclaim policy.

    4. Select efs.csi.aws.com from the Provisioner drop-down list.

    5. Optional: Set the configuration parameters for the selected provisioner.

  4. Click Create.

Creating the AWS EFS storage class using the CLI

Procedure
  • Create a StorageClass object:

    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: efs-sc
    provisioner: efs.csi.aws.com
    parameters:
      provisioningMode: efs-ap (1)
      fileSystemId: fs-a5324911 (2)
      directoryPerms: "700" (3)
      gidRangeStart: "1000" (4)
      gidRangeEnd: "2000" (4)
      basePath: "/dynamic_provisioning" (5)
    1 provisioningMode must be efs-ap to enable dynamic provisioning.
    2 fileSystemId must be the ID of the EFS volume created manually.
    3 directoryPerms is the default permission of the root directory of the volume. In this example, the volume is accessible only by the owner.
    4 gidRangeStart and gidRangeEnd set the range of POSIX Group IDs (GIDs) that are used to set the GID of the AWS access point. If not specified, the default range is 50000-7000000. Each provisioned volume, and thus AWS access point, is assigned a unique GID from this range.
    5 basePath is the directory on the EFS volume that is used to create dynamically provisioned volumes. In this case, a PV is provisioned as “/dynamic_provisioning/<random uuid>” on the EFS volume. Only the subdirectory is mounted to pods that use the PV.

    A cluster admin can create several StorageClass objects, each using a different EFS volume.

Creating and configuring access to EFS volumes in AWS

This procedure explains how to create and configure EFS volumes in AWS so that you can use them in OpenShift Container Platform.

Prerequisites
  • AWS account credentials

Procedure

To create and configure access to an EFS volume in AWS:

  1. On the AWS console, open https://console.aws.amazon.com/efs.

  2. Click Create file system:

    • Enter a name for the file system.

    • For Virtual Private Cloud (VPC), select your OpenShift Container Platform’s' virtual private cloud (VPC).

    • Accept default settings for all other selections.

  3. Wait for the volume and mount targets to finish being fully created:

    1. Go to https://console.aws.amazon.com/efs#/file-systems.

    2. Click your volume, and on the Network tab wait for all mount targets to become available (~1-2 minutes).

  4. On the Network tab, copy the Security Group ID (you will need this in the next step).

  5. Go to https://console.aws.amazon.com/ec2/v2/home#SecurityGroups, and find the Security Group used by the EFS volume.

  6. On the Inbound rules tab, click Edit inbound rules, and then add a new rule with the following settings to allow OpenShift Container Platform nodes to access EFS volumes :

    • Type: NFS

    • Protocol: TCP

    • Port range: 2049

    • Source: Custom/IP address range of your nodes (for example: “10.0.0.0/16”)

      This step allows OpenShift Container Platform to use NFS ports from the cluster.

  7. Save the rule.

Dynamic provisioning for AWS EFS

The AWS EFS CSI Driver supports a different form of dynamic provisioning than other CSI drivers. It provisions new PVs as subdirectories of a pre-existing EFS volume. The PVs are independent of each other. However, they all share the same EFS volume. When the volume is deleted, all PVs provisioned out of it are deleted too. The EFS CSI driver creates an AWS Access Point for each such subdirectory. Due to AWS AccessPoint limits, you can only dynamically provision 120 PVs from a single StorageClass/EFS volume.

Note that PVC.spec.resources is not enforced by EFS.

In the example below, you request 5 GiB of space. However, the created PV is limitless and can store any amount of data (like petabytes). A broken application, or even a rogue application, can cause significant expenses when it stores too much data on the volume.

Using monitoring of EFS volume sizes in AWS is strongly recommended.

Prerequisites
  • You have created AWS EFS volumes.

  • You have created the AWS EFS storage class.

Procedure

To enable dynamic provisioning:

  • Create a PVC (or StatefulSet or Template) as usual, referring to the StorageClass created above.

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: test
    spec:
      storageClassName: efs-sc
      accessModes: