OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history
×

Persistent Storage Using Red Hat Gluster Storage

Overview

Red Hat Gluster Storage can be configured to provide persistent storage and dynamic provisioning for OpenShift Container Platform. It can be used both containerized within OpenShift Container Platform (Container-Native Storage) and non-containerized on its own nodes (Container-Ready Storage).

Container-Native Storage

With Container-Native Storage, Red Hat Gluster Storage runs containerized directly on OpenShift Container Platform nodes. This allows for compute and storage instances to be scheduled and run from the same set of hardware.

Architecture - Container-Native Storage
Figure 1. Architecture - Container-Native Storage

Container-Native Storage is available starting with Red Hat Gluster Storage 3.1 update 3. See Container-Native Storage for OpenShift Container Platform for additional documentation.

Container-Ready Storage

With Container-Ready Storage, Red Hat Gluster Storage runs on its own dedicated nodes and is managed by an instance of heketi, the GlusterFS volume management REST service. This heketi service can run either standalone or containerized. Containerization allows for an easy mechanism to provide high-availability to the service. This documentation will focus on the configuration where heketi is containerized.

Standalone Red Hat Gluster Storage

If you have a standalone Red Hat Gluster Storage cluster available in your environment, you can make use of volumes on that cluster using OpenShift Container Platform’s GlusterFS volume plug-in. This solution is a conventional deployment where applications run on dedicated compute nodes, an OpenShift Container Platform cluster, and storage is provided from its own dedicated nodes.

Architecture - Standalone Red Hat Gluster Storage Cluster Using OpenShift Container Platform’s GlusterFS Volume Plug-in
Figure 2. Architecture - Standalone Red Hat Gluster Storage Cluster Using OpenShift Container Platform’s GlusterFS Volume Plug-in

See the Red Hat Gluster Storage Installation Guide and the Red Hat Gluster Storage Administration Guide for more on Red Hat Gluster Storage.

High availability of storage in the infrastructure is left to the underlying storage provider.

GlusterFS Volumes

GlusterFS volumes present a POSIX-compliant filesystem and are comprised of one or more "bricks" across one or more nodes in their cluster. A brick is just a directory on a given storage node and is typically the mount point for a block storage device. GlusterFS handles distribution and replication of files across a given volume’s bricks per that volume’s configuration.

It is recommended to use heketi for most common volume management operations such as create, delete, and resize. OpenShift Container Platform expects heketi to be present when using the GlusterFS provisioner. heketi by default will create volumes that are three-ray replica, that is volumes where each file has three copies across three different nodes. As such it is recommended that any Red Hat Gluster Storage clusters which will be used by heketi have at least three nodes available.

There are many features available for GlusterFS volumes, but they are beyond the scope of this documentation.

gluster-block Volumes

gluster-block volumes are volumes that can be mounted over iSCSI. This is done by creating a file on an existing GlusterFS volume and then presenting that file as a block device via an iSCSI target. Such GlusterFS volumes are called block-hosting volumes.

gluster-block volumes present a sort of trade-off. Being consumed as iSCSI targets, gluster-block volumes can only be mounted by one node/client at a time which is in contrast to GlusterFS volumes which can be mounted by multiple nodes/clients. Being files on the backend, however, allows for operations which are typically costly on GlusterFS volumes (e.g. metadata lookups) to be converted to ones which are typically much faster on GlusterFS volumes (e.g. reads and writes). This leads to potentially substantial performance improvements for certain workloads.

At this time, it is recommended to only use gluster-block volumes for OpenShift Logging and OpenShift Metrics storage.

Gluster S3 Storage

The Gluster S3 service allows user applications to access GlusterFS storage via an S3 interface. The service binds to two GlusterFS volumes, one for object data and one for object metadata, and translates incoming S3 REST requests into filesystem operations on the volumes. It is recommended to run the service as a pod inside OpenShift Container Platform.

At this time, use and installation of the Gluster S3 service is in tech preview.

Considerations

This section covers a few topics that should be taken into consideration when using Red Hat Gluster Storage with OpenShift Container Platform.

Software Prerequisites

To access GlusterFS volumes, the mount.glusterfs command must be available on all schedulable nodes. For RPM-based systems, the glusterfs-fuse package must be installed:

# yum install glusterfs-fuse

This package comes installed on every RHEL system. However, it is recommended to update to the latest available version from Red Hat Gluster Storage. To do this, the following RPM repository must be enabled:

# subscription-manager repos --enable=rh-gluster-3-client-for-rhel-7-server-rpms

If glusterfs-fuse is already installed on the nodes, ensure that the latest version is installed:

# yum update glusterfs-fuse

Hardware Requirements

Any nodes used in a Container-Native Storage or Container-Ready Storage cluster are considered storage nodes. Storage nodes can be grouped into distinct cluster groups, though a single node can not be in multiple groups. For each group of storage nodes:

  • A minimum of three storage nodes per group is required.

  • Each storage node must have a minimum of 8 GB of RAM. This is to allow running the Red Hat Gluster Storage pods, as well as other applications and the underlying operating system.

    • Each GlusterFS volume also consumes memory on every storage node in its storage cluster, which is about 30 MB. The total amount of RAM should be determined based on how many concurrent volumes are desired or anticipated.

  • Each storage node must have at least one raw block device with no present data or metadata. These block devices will be used in their entirety for GlusterFS storage. Make sure the following are not present:

    • Partition tables (GPT or MSDOS)

    • Filesystems or residual filesystem signatures

    • LVM2 signatures of former Volume Groups and Logical Volumes

    • LVM2 metadata of LVM2 physical volumes

    If in doubt, wipefs -a <device> should clear any of the above.

It is recommended to plan for two clusters: one dedicated to storage for infrastructure applications (such as an OpenShift Container Registry) and one dedicated to storage for general applications. This would require a total of six storage nodes. This recommendation is made to avoid potential impacts on performance in I/O and volume creation.

Storage Sizing

Every GlusterFS cluster must be sized based on the needs of the anticipated applications that will use its storage. For example, there are sizing guides available for both OpenShift Logging and OpenShift Metrics.

Some additional things to consider are:

  • For each Container-Native Storage or Container-Ready Storage cluster, the default behavior is to create GlusterFS volumes with three-way replication. As such, the total storage to plan for should be the desired capacity times three.

    • As an example, each heketi instance creates a heketidbstorage volume that is 2 GB in size, requiring a total of 6 GB of raw storage across three nodes in the storage cluster. This capacity is always required and should be taken into consideration for sizing calculations.

    • Applications like an integrated OpenShift Container Registry share a single GlusterFS volume across multiple instances of the application.

  • gluster-block volumes require the presence of a GlusterFS block-hosting volume with enough capacity to hold the full size of any given block volume’s capacity.

    • By default, if no such block-hosting volume exists, one will be automatically created at a set size. The default for this size is 100 GB. If there is not enough space in the cluster to create the new block-hosting volume, the creation of the block volume will fail. Both the auto-create behavior and the auto-created volume size are configurable.

    • Applications with multiple instances that use gluster-block volumes, such as OpenShift Logging and OpenShift Metrics, will use one volume per instance.

  • The Gluster S3 service binds to two GlusterFS volumes. In a default installation via the Advanced Installer, these volumes are 1 GB each, consuming a total of 6 GB of raw storage.

Volume Operation Behaviors

Volume operations, such as create and delete, can be impacted by a variety of environmental circumstances and can in turn affect applications as well.

  • If the application pod requests a dynamically provisioned GlusterFS persistent volume claim (PVC), then extra time might have to be considered for the volume to be created and bound to the corresponding PVC. This effects the startup time for an application pod.

    Creation time of GlusterFS volumes scales linearly depending on the number of volumes. As an example, given 100 volumes in a cluster using recommended hardware specifications, each volume took approximately 6 seconds to be created, allocated, and bound to a pod.

  • When a PVC is deleted, that action will trigger the deletion of the underlying GlusterFS volume. While PVCs will disappear immediately from the oc get pvc output, this does not mean the volume has been fully deleted. A GlusterFS volume can only be considered deleted when it does not show up in the command-line outputs for heketi-cli volume list and gluster volume list.

    The time to delete the GlusterFS volume and recycle its storage depends on and scales linearly with the number of active GlusterFS volumes. While pending volume deletes do not affect running applications, storage administrators should be aware of and be able to estimate how long they will take, especially when tuning resource consumption at scale.

Volume Security

This section covers Red Hat Gluster Storage volume security, including Portable Operating System Interface [for Unix] (POSIX) permissions and SELinux considerations. Understanding the basics of Volume Security, POSIX permissions, and SELinux is presumed.

POSIX Permissions

Red Hat Gluster Storage volumes present POSIX-compliant file systems. As such, access permissions can be managed using standard command-line tools such as chmod and chown.

For Container-Native Storage and Container-Ready Storage, it is also possible to specify a group ID that will own the root of the volume at volume creation time. For static provisioning, this is specified as part of the heketi-cli volume creation command:

$ heketi-cli volume create --size=100 --gid=10001000

The PersistentVolume that will be associated with this volume must be annotated with the group ID so that pods consuming the PersistentVolume can have access to the file system. This annotation takes the form of:

pv.beta.kubernetes.io/gid: "<GID>" ---

For dynamic provisioning, the provisioner automatically generates and applies a group ID. It is possible to control the range from which this group ID is selected using the gidMin and gidMax StorageClass parameters (see Dynamic Provisioning). The provisioner also takes care of annotating the generated PersistentVolume with the group ID.

SELinux

By default, SELinux does not allow writing from a pod to a remote Red Hat Gluster Storage server. To enable writing to Red Hat Gluster Storage volumes with SELinux on, run the following on each node running GlusterFS:

$ sudo setsebool -P virt_sandbox_use_fusefs on (1)
$ sudo setsebool -P virt_use_fusefs on
1 The -P option makes the boolean persistent between reboots.

The virt_sandbox_use_fusefs boolean is defined by the docker-selinux package. If you get an error saying it is not defined, ensure that this package is installed.

If you use Atomic Host, the SELinux booleans are cleared when you upgrade Atomic Host. When you upgrade Atomic Host, you must set these boolean values again.

Support Requirements

The following requirements must be met to create a supported integration of Red Hat Gluster Storage and OpenShift Container Platform.

For Container-Ready Storage or standalone Red Hat Gluster Storage:

  • Minimum version: Red Hat Gluster Storage 3.1.3

  • All Red Hat Gluster Storage nodes must have valid subscriptions to Red Hat Network channels and Subscription Manager repositories.

  • Red Hat Gluster Storage nodes must adhere to the requirements specified in the Planning Red Hat Gluster Storage Installation.

  • Red Hat Gluster Storage nodes must be completely up to date with the latest patches and upgrades. Refer to the Red Hat Gluster Storage Installation Guide to upgrade to the latest version.

  • A fully-qualified domain name (FQDN) must be set for each Red Hat Gluster Storage node. Ensure that correct DNS records exist, and that the FQDN is resolvable via both forward and reverse DNS lookup.

Installation

For standalone Red Hat Gluster Storage, there is no component installation required to use it with OpenShift Container Platform. OpenShift Container Platform comes with a built-in GlusterFS volume driver, allowing it to make use of existing volumes on existing clusters. See provisioning for more on how to make use of existing volumes.

For Container-Native Storage and Container-Ready Storage, it is recommended to use the Advanced Installer to install the required components.

Container-Ready Storage: Installing Red Hat Gluster Storage Nodes

For Container-Ready Storage, each Red Hat Gluster Storage node must have the appropriate system configurations (e.g. firewall ports, kernel modules) and the Red Hat Gluster Storage services must be running. The services should not be further configured, and should not have formed a Trusted Storage Pool.

The installation of Red Hat Gluster Storage nodes is beyond the scope of this documentation. For more information, see Setting Up Container-Ready Storage.

Using the Advanced Installer

The Advanced Installer can be used to install one or both of two GlusterFS node groups:

  • glusterfs: A general storage cluster for use by user applications.

  • glusterfs_registry: A dedicated storage cluster for use by infrastructure applications such as an integrated OpenShift Container Registry.

It is recommended to deploy both groups to avoid potential impacts on performance in I/O and volume creation. Both of these are defined in the inventory hosts file.

The definition of the clusters is done by including the relevant names in the [OSEv3:children] group, creating similarly named groups, and then populating the groups with the node information. The clusters can then be configured through a variety of variables in the [OSEv3:vars] group. glusterfs variables begin with openshift_storage_glusterfs_ and glusterfs_registry variables begin with openshift_storage_glusterfs_registry_. A few other variables, such as openshift_hosted_registry_storage_kind, interact with the GlusterFS clusters.

It is recommended to specify version tags for all containerized components. This is primarily to prevent components, particularly the Red Hat Gluster Storage pods, from upgrading after an outage which may lead to a cluster of widely disparate software versions. The relevant variables are:

  • openshift_storage_glusterfs_version

  • openshift_storage_glusterfs_block_version

  • openshift_storage_glusterfs_s3_version

  • openshift_storage_glusterfs_heketi_version

  • openshift_storage_glusterfs_registry_version

  • openshift_storage_glusterfs_registry_block_version

  • openshift_storage_glusterfs_registry_s3_version

  • openshift_storage_glusterfs_registry_heketi_version

For a complete list of variables, see the GlusterFS role README on GitHub.

Once the variables are configured, there are several playbooks available depending on the circumstances of the installation:

  • The main playbook of the Advanced Installer can be used to deploy the GlusterFS clusters in tandem with an initial installation of OpenShift Container Platform.

  • playbooks/openshift-glusterfs/config.yml can be used to deploy the clusters onto an existing OpenShift Container Platform installation.

  • playbooks/openshift-glusterfs/registry.yml can be used to deploy the clusters onto an existing OpenShift Container Platform installation. In addition, this will deploy an integrated OpenShift Container Registry which uses GlusterFS storage.

    There must not be a pre-existing registry in the OpenShift Container Platform cluster.

  • playbooks/openshift-glusterfs/uninstall.yml can be used to remove existing clusters matching the configuration in the inventory hosts file. This is useful for cleaning up the OpenShift Container Platform environment in the case of a failed deployment due to configuration errors.

    The GlusterFS playbooks are not guaranteed to be idempotent.

    Running the playbooks more than once for a given installation is currently not supported without deleting the entire GlusterFS installation (including disk data) and starting over.

Example: Basic Container-Native Storage Installation

  1. In your inventory file, add glusterfs in the [OSEv3:children] section to enable the [glusterfs] group:

    [OSEv3:children]
masters
nodes
glusterfs

  2. Add a [glusterfs] section with entries for each storage node that will host the GlusterFS storage. For each node, set glusterfs_devices to a list of raw block devices that will be completely managed as part of a GlusterFS cluster. There must be at least one device listed. Each device must be bare, with no partitions or LVM PVs. Specifying the variable takes the form:

    <hostname_or_ip> glusterfs_devices='[ "</path/to/device1/>", "</path/to/device2>", ... ]'

    For example:

    [glusterfs]
node11.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node12.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node13.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'

  3. Add the hosts listed under [glusterfs] to the [nodes] group:

    [nodes]
...
node11.example.com openshift_schedulable=True
node12.example.com openshift_schedulable=True
node13.example.com openshift_schedulable=True

Example: Basic Container-Ready Storage Installation

  1. In your inventory file, add glusterfs in the [OSEv3:children] section to enable the [glusterfs] group:

    [OSEv3:children]
masters
nodes
glusterfs

  2. Include the following variables in the [OSEv3:vars] section, adjusting them as needed for your configuration:

    [OSEv3:vars]
...
openshift_storage_glusterfs_is_native=false
openshift_storage_glusterfs_storageclass=true
openshift_storage_glusterfs_heketi_is_native=true
openshift_storage_glusterfs_heketi_executor=ssh
openshift_storage_glusterfs_heketi_ssh_port=22
openshift_storage_glusterfs_heketi_ssh_user=root
openshift_storage_glusterfs_heketi_ssh_sudo=false
openshift_storage_glusterfs_heketi_ssh_keyfile="/root/.ssh/id_rsa"

  3. Add a [glusterfs] section with entries for each storage node that will host the GlusterFS storage. For each node, set glusterfs_devices to a list of raw block devices that will be completely managed as part of a GlusterFS cluster. There must be at least one device listed. Each device must be bare, with no partitions or LVM PVs. Also, set glusterfs_ip to the IP address of the node. Specifying the variable takes the form:

    <hostname_or_ip> glusterfs_ip=<ip_address> glusterfs_devices='[ "</path/to/device1/>", "</path/to/device2>", ... ]'

    For example:

    [glusterfs]
gluster1.example.com glusterfs_ip=192.168.10.11 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
gluster2.example.com glusterfs_ip=192.168.10.12 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
gluster3.example.com glusterfs_ip=192.168.10.13 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'

Example: Container-Native Storage with an Integrated OpenShift Container Registry

  1. In your inventory file, set the following variable under [OSEv3:vars]:

    [OSEv3:vars]
...
openshift_hosted_registry_storage_kind=glusterfs

  2. Add glusterfs_registry in the [OSEv3:children] section to enable the [glusterfs_registry] group:

    [OSEv3:children]
masters
nodes
glusterfs_registry

  3. Add a [glusterfs_registry] section with entries for each storage node that will host the GlusterFS storage. For each node, set glusterfs_devices to a list of raw block devices that will be completely managed as part of a GlusterFS cluster. There must be at least one device listed. Each device must be bare, with no partitions or LVM PVs. Specifying the variable takes the form:

    <hostname_or_ip> glusterfs_devices='[ "</path/to/device1/>", "</path/to/device2>", ... ]'

    For example:

    [glusterfs_registry]
node11.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node12.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node13.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'

  4. Add the hosts listed under [glusterfs_registry] to the [nodes] group:

    [nodes]
...
node11.example.com openshift_schedulable=True
node12.example.com openshift_schedulable=True
node13.example.com openshift_schedulable=True

Example: Container-Native Storage for OpenShift Logging and Metrics

  1. In your inventory file, set the following variables under [OSEv3:vars]:

    [OSEv3:vars]
...
openshift_metrics_hawkular_nodeselector={"role":"infra"}  (1)
openshift_metrics_cassandra_nodeselector={"role":"infra"} (1)
openshift_metrics_heapster_nodeselector={"role":"infra"}  (1)
openshift_metrics_storage_kind=dynamic
openshift_metrics_cassanda_pvc_storage_class_name="glusterfs-registry-block" (2)

openshift_logging_es_nodeselector={"role":"infra"}        (1)
openshift_logging_kibana_nodeselector={"role":"infra"}    (1)
openshift_logging_curator_nodeselector={"role":"infra"}   (1)
openshift_logging_storage_kind=dynamic
openshift_logging_es_pvc_size=10Gi                        (3)
openshift_logging_elasticsearch_storage_type=pvc             (4)
openshift_logging_es_pvc_storage_class_name="glusterfs-registry-block"       (2)

openshift_storage_glusterfs_registry_block_deploy=true
openshift_storage_glusterfs_registry_block_host_vol_size=50
openshift_storage_glusterfs_registry_block_storageclass=true
openshift_storage_glusterfs_registry_block_storageclass_default=true

openshift_storageclass_default=false
    1 It is recommended to run the integrated OpenShift Container Registry, Logging, and Metrics on nodes dedicated to "infrastructure" applications, that is applications deployed by administrators to provide services for the OpenShift Container Platform cluster.
    2 Specify the StorageClass to be used for Logging and Metrics. This name is generated from the name of the target GlusterFS cluster (e.g., glusterfs-<name>-block). In this example, this defaults to registry.
    3 OpenShift Logging requires that a PVC size be specified. The supplied value is only an example, not a recommendation.
    4 If using Persistent Elasticsearch Storage, set the storage type to pvc.

    See the GlusterFS role README for details on these and other variables.

  2. Add glusterfs_registry in the [OSEv3:children] section to enable the [glusterfs_registry] group:

    [OSEv3:children]
masters
nodes
glusterfs_registry

  3. Add a [glusterfs_registry] section with entries for each storage node that will host the GlusterFS storage. For each node, set glusterfs_devices to a list of raw block devices that will be completely managed as part of a GlusterFS cluster. There must be at least one device listed. Each device must be bare, with no partitions or LVM PVs. Specifying the variable takes the form:

    <hostname_or_ip> glusterfs_devices='[ "</path/to/device1/>", "</path/to/device2>", ... ]'

    For example:

    [glusterfs_registry]
node11.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node12.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node13.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'

  4. Add the hosts listed under [glusterfs_registry] to the [nodes] group:

    [nodes]
...
node11.example.com openshift_schedulable=True
node12.example.com openshift_schedulable=True
node13.example.com openshift_schedulable=True

  5. Run the Advanced Installer. This can be either as part of an initial OpenShift Container Platform installation:

    ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

    or brownfield:

    ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/config.yml

Example: Container-Native Storage for Applications, Registry, Logging, and Metrics

  1. In your inventory file, set the following variables under [OSEv3:vars]:

    [OSEv3:vars]
...
openshift_registry_selector="role=infra"                  (1)
openshift_hosted_registry_storage_kind=glusterfs

openshift_metrics_hawkular_nodeselector={"role":"infra"}  (1)
openshift_metrics_cassandra_nodeselector={"role":"infra"} (1)
openshift_metrics_heapster_nodeselector={"role":"infra"}  (1)
openshift_metrics_storage_kind=dynamic
openshift_metrics_cassanda_pvc_storage_class_name="glusterfs-registry-block" (2)

openshift_logging_es_nodeselector={"role":"infra"}        (1)
openshift_logging_kibana_nodeselector={"role":"infra"}    (1)
openshift_logging_curator_nodeselector={"role":"infra"}   (1)
openshift_logging_storage_kind=dynamic
openshift_logging_es_pvc_size=10Gi                        (3)
openshift_logging_elasticsearch_storage_type=pvc              (4)
openshift_logging_es_pvc_storage_class_name="glusterfs-registry-block"       (2)

openshift_storage_glusterfs_block_deploy=false

openshift_storage_glusterfs_registry_block_deploy=true
openshift_storage_glusterfs_registry_block_storageclass=true
openshift_storage_glusterfs_registry_block_storageclass_default=false
    1 Running the integrated OpenShift Container Registry, Logging, and Metrics on infrastructure nodes is recommended. Infrastructure node are nodes dedicated to running applications deployed by administrators to provide services for the OpenShift Container Platform cluster.
    2 Specify the StorageClass to be used for Logging and Metrics. This name is generated from the name of the target GlusterFS cluster, for example glusterfs-<name>-block. In this example, <name> defaults to registry.
    3 Specifying a PVC size is required for OpenShift Logging. The supplied value is only an example, not a recommendation.
    4 If using Persistent Elasticsearch Storage, set the storage type to pvc.

  2. Add glusterfs and glusterfs_registry in the [OSEv3:children] section to enable the [glusterfs] and [glusterfs_registry] groups:

    [OSEv3:children]
...
glusterfs
glusterfs_registry

  3. Add [glusterfs] and [glusterfs_registry] sections with entries for each storage node that will host the GlusterFS storage. For each node, set glusterfs_devices to a list of raw block devices that will be completely managed as part of a GlusterFS cluster. There must be at least one device listed. Each device must be bare, with no partitions or LVM PVs. Specifying the variable takes the form:

    <hostname_or_ip> glusterfs_devices='[ "</path/to/device1/>", "</path/to/device2>", ... ]'

    For example:

    [glusterfs]
node11.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node12.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node13.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'

[glusterfs_registry]
node14.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node15.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
node16.example.com glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'

  4. Add the hosts listed under [glusterfs] and [glusterfs_registry] to the [nodes] group:

    [nodes]
...
node11.example.com openshift_schedulable=True openshift_node_labels="{'role': 'app'}"   (1)
node12.example.com openshift_schedulable=True openshift_node_labels="{'role': 'app'}"   (1)
node13.example.com openshift_schedulable=True openshift_node_labels="{'role': 'app'}"   (1)
node14.example.com openshift_schedulable=True openshift_node_labels="{'role': 'infra'}" (1)
node15.example.com openshift_schedulable=True openshift_node_labels="{'role': 'infra'}" (1)
node16.example.com openshift_schedulable=True openshift_node_labels="{'role': 'infra'}" (1)
    1 The nodes are marked to denote whether they will allow general applications or infrastructure applications to be scheduled on them. It is up to the administrator to configure how applications will be constrained.

  5. Run the Advanced Installer.

    • For an initial OpenShift Container Platform installation:

      ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

    • For a standalone installation onto an existing OpenShift Container Platform cluster:

      ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/config.yml

Example: Container-Ready Storage for Applications, Registry, Logging, and Metrics

  1. In your inventory file, set the following variables under [OSEv3:vars]:

    [OSEv3:vars]
...
openshift_registry_selector="role=infra"                  (1)
openshift_hosted_registry_storage_kind=glusterfs

openshift_metrics_hawkular_nodeselector={"role":"infra"}  (1)
openshift_metrics_cassandra_nodeselector={"role":"infra"} (1)
openshift_metrics_heapster_nodeselector={"role":"infra"}  (1)
openshift_metrics_storage_kind=dynamic
openshift_metrics_cassanda_pvc_storage_class_name="glusterfs-registry-block" (2)

openshift_logging_es_nodeselector={"role":"infra"}        (1)
openshift_logging_kibana_nodeselector={"role":"infra"}    (1)
openshift_logging_curator_nodeselector={"role":"infra"}   (1)
openshift_logging_storage_kind=dynamic
openshift_logging_es_pvc_size=10Gi                        (3)
openshift_logging_elasticsearch_storage_type              (4)
openshift_logging_es_pvc_storage_class_name="glusterfs-registry-block"       (2)

openshift_storage_glusterfs_is_native=false
openshift_storage_glusterfs_block_deploy=false
openshift_storage_glusterfs_storageclass=true
openshift_storage_glusterfs_heketi_is_native=true
openshift_storage_glusterfs_heketi_executor=ssh
openshift_storage_glusterfs_heketi_ssh_port=22
openshift_storage_glusterfs_heketi_ssh_user=root
openshift_storage_glusterfs_heketi_ssh_sudo=false
openshift_storage_glusterfs_heketi_ssh_keyfile="/root/.ssh/id_rsa"

openshift_storage_glusterfs_is_native=false
openshift_storage_glusterfs_registry_block_deploy=true
openshift_storage_glusterfs_registry_block_storageclass=true
openshift_storage_glusterfs_registry_block_storageclass_default=true
openshift_storage_glusterfs_registry_heketi_is_native=true
openshift_storage_glusterfs_registry_heketi_executor=ssh
openshift_storage_glusterfs_registry_heketi_ssh_port=22
openshift_storage_glusterfs_registry_heketi_ssh_user=root
openshift_storage_glusterfs_registry_heketi_ssh_sudo=false
openshift_storage_glusterfs_registry_heketi_ssh_keyfile="/root/.ssh/id_rsa"
    1 It is recommended to run the integrated OpenShift Container Registry on nodes dedicated to "infrastructure" applications, that is applications deployed by administrators to provide services for the OpenShift Container Platform cluster. It is up to the administrator to select and label nodes for infrastructure applications.
    2 Specify the StorageClass to be used for Logging and Metrics. This name is generated from the name of the target GlusterFS cluster (e.g., glusterfs-<name>-block). In this example, this defaults to registry.
    3 OpenShift Logging requires that a PVC size be specified. The supplied value is only an example, not a recommendation.
    4 If using Persistent Elasticsearch Storage, set the storage type to pvc.

  2. Add glusterfs and glusterfs_registry in the [OSEv3:children] section to enable the [glusterfs] and [glusterfs_registry] groups:

    [OSEv3:children]
...
glusterfs
glusterfs_registry

  3. Add [glusterfs] and [glusterfs_registry] sections with entries for each storage node that will host the GlusterFS storage. For each node, set glusterfs_devices to a list of raw block devices that will be completely managed as part of a GlusterFS cluster. There must be at least one device listed. Each device must be bare, with no partitions or LVM PVs. Also, set glusterfs_ip to the IP address of the node. Specifying the variable takes the form:

    <hostname_or_ip> glusterfs_ip=<ip_address> glusterfs_devices='[ "</path/to/device1/>", "</path/to/device2>", ... ]'

    For example:

    [glusterfs]
gluster1.example.com glusterfs_ip=192.168.10.11 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
gluster2.example.com glusterfs_ip=192.168.10.12 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
gluster3.example.com glusterfs_ip=192.168.10.13 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'

[glusterfs_registry]
gluster4.example.com glusterfs_ip=192.168.10.14 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
gluster5.example.com glusterfs_ip=192.168.10.15 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'
gluster6.example.com glusterfs_ip=192.168.10.16 glusterfs_devices='[ "/dev/xvdc", "/dev/xvdd" ]'

  4. Run the Advanced Installer. This can be either as part of an initial OpenShift Container Platform installation:

    ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

    or as a standalone operation onto an existing OpenShift Container Platform cluster:

    ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/config.yml

Uninstall Container-Native Storage

For Container-Native Storage, an OpenShift Container Platform install comes with a playbook to uninstall all resources and artifacts from the cluster. To use the playbook, provide the original inventory file that was used to install the target instance of Container-Native Storage and run the following playbook:

# ansible-playbook -i <path_to_inventory_file> /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/uninstall.yml

In addition, the playbook supports the use of a variable called openshift_storage_glusterfs_wipe which, when enabled, destroys any data on the block devices that were used for Red Hat Gluster Storage backend storage. To use the openshift_storage_glusterfs_wipe variable:

# ansible-playbook -i <path_to_inventory_file> -e
"openshift_storage_glusterfs_wipe=true"
/usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/uninstall.yml

This procedure destroys data. Proceed with caution.

Provisioning

GlusterFS volumes can be provisioned either statically or dynamically. Static provisioning is available with all configurations. Only Container-Native Storage and Container-Ready Storage support dynamic provisioning.

Static Provisioning

  1. To enable static provisioning, first create a GlusterFS volume. See the Red Hat Gluster Storage Administration Guide for information on how to do this using the gluster command-line interface or the heketi project site for information on how to do this using heketi-cli. For this example, the volume will be named myVol1.

  2. Define the following Service and Endpoints in gluster-endpoints.yaml:

    ---
apiVersion: v1
kind: Service
metadata:
  name: glusterfs-cluster (1)
spec:
  ports:
  - port: 1
---
apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster (1)
subsets:
  - addresses:
      - ip: 192.168.122.221 (2)
    ports:
      - port: 1 (3)
  - addresses:
      - ip: 192.168.122.222 (2)
    ports:
      - port: 1 (3)
  - addresses:
      - ip: 192.168.122.223 (2)
    ports:
      - port: 1 (3)
    1 These names must match.
    2 The ip values must be the actual IP addresses of a Red Hat Gluster Storage server, not hostnames.
    3 The port number is ignored.

  3. From the OpenShift Container Platform master host, create the Service and Endpoints:

    $ oc create -f gluster-endpoints.yaml
service "glusterfs-cluster" created
endpoints "glusterfs-cluster" created

  4. Verify that the Service and Endpoints were created:

    $ oc get services
NAME                       CLUSTER_IP       EXTERNAL_IP   PORT(S)    SELECTOR        AGE
glusterfs-cluster          172.30.205.34    <none>        1/TCP      <none>          44s

$ oc get endpoints
NAME                ENDPOINTS                                               AGE
docker-registry     10.1.0.3:5000                                           4h
glusterfs-cluster   192.168.122.221:1,192.168.122.222:1,192.168.122.223:1   11s
kubernetes          172.16.35.3:8443                                        4d

    Endpoints are unique per project. Each project accessing the GlusterFS volume needs its own Endpoints.

  5. In order to access the volume, the container must run with either a user ID (UID) or group ID (GID) that has access to the file system on the volume. This information can be discovered in the following manner:

    $ mkdir -p /mnt/glusterfs/myVol1

$ mount -t glusterfs 192.168.122.221:/myVol1 /mnt/glusterfs/myVol1

$ ls -lnZ /mnt/glusterfs/
drwxrwx---. 592 590 system_u:object_r:fusefs_t:s0    myVol1  (1) (2)
    1 The UID is 592.
    2 The GID is 590.

  6. Define the following PersistentVolume (PV) in gluster-pv.yaml:

    apiVersion: v1
kind: PersistentVolume
metadata:
  name: gluster-default-volume (1)
  annotations:
    pv.beta.kubernetes.io/gid: "590" (2)
spec:
  capacity:
    storage: 2Gi (3)
  accessModes: (4)
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster (5)
    path: myVol1 (6)
    readOnly: false
  persistentVolumeReclaimPolicy: Retain
    1 The name of the volume.
    2 The GID on the root of the GlusterFS volume.
    3 The amount of storage allocated to this volume.
    4 accessModes are used as labels to match a PV and a PVC. They currently do not define any form of access control.
    5 The Endpoints resource previously created.
    6 The GlusterFS volume that will be accessed.

  7. From the OpenShift Container Platform master host, create the PV:

    $ oc create -f gluster-pv.yaml

  8. Verify that the PV was created:

    $ oc get pv
NAME                     LABELS    CAPACITY     ACCESSMODES   STATUS      CLAIM     REASON    AGE
gluster-default-volume   <none>    2147483648   RWX           Available                       2s

  9. Create a PersistentVolumeClaim (PVC) that will bind to the new PV in gluster-claim.yaml:

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gluster-claim  (1)
spec:
  accessModes:
  - ReadWriteMany      (2)
  resources:
     requests:
       storage: 1Gi    (3)
    1 The claim name is referenced by the pod under its volumes section.
    2 Must match the accessModes of the PV.
    3 This claim will look for PVs offering 1Gi or greater capacity.

  10. From the OpenShift Container Platform master host, create the PVC:

    $ oc create -f gluster-claim.yaml

  11. Verify that the PV and PVC are bound:

    $ oc get pv
NAME         LABELS    CAPACITY   ACCESSMODES   STATUS      CLAIM          REASON    AGE
gluster-pv   <none>    1Gi        RWX           Available   gluster-claim            37s

$ oc get pvc
NAME            LABELS    STATUS    VOLUME       CAPACITY   ACCESSMODES   AGE
gluster-claim   <none>    Bound     gluster-pv   1Gi        RWX           24s

PVCs are unique per project. Each project accessing the GlusterFS volume needs its own PVC. PVs are not bound to a single project, so PVCs across multiple projects may refer to the same PV.

Dynamic Provisioning

  1. To enable dynamic provisioning, first create a StorageClass object definition. The definition below is based on the minimum requirements needed for this example to work with OpenShift Container Platform. See Dynamic Provisioning and Creating Storage Classes for additional parameters and specification definitions.

    kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: glusterfs
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://10.42.0.0:8080" (1)
  restauthenabled: "false" (2)
    1 The heketi server URL.
    2 Since authentication is not turned on in this example, set to false.

  2. From the OpenShift Container Platform master host, create the StorageClass:

    # oc create -f gluster-storage-class.yaml
storageclass "glusterfs" created

  3. Create a PVC using the newly-created StorageClass. For example:

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gluster1
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 30Gi
  storageClassName: glusterfs

  4. From the OpenShift Container Platform master host, create the PVC:

    # oc create -f glusterfs-dyn-pvc.yaml
persistentvolumeclaim "gluster1" created

  5. View the PVC to see that the volume was dynamically created and bound to the PVC:

    # oc get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
gluster1   Bound    pvc-78852230-d8e2-11e6-a3fa-0800279cf26f   30Gi       RWX           glusterfs      42s