Docker stores images and containers in a graph driver (a pluggable storage technology), such as DeviceMapper, Overlay, and Btrfs.
Each has advantages and disadvantages. For example, Overlay is faster than DeviceMapper at starting and stopping containers,
but is not Portable Operating System Interface for Unix (POSIX) compliant because of the architectural limitations of a union file system, and does not yet support SELinux.
In production environments, using a LVM thin pool on top of regular block
devices (not loop devices) for container images and container root file systems
storage is recommended.
Using a loop device can affect performance issues. While you can still
continue to use it, Docker logs the following warning message:
devmapper: Usage of loopback devices is strongly discouraged for production use.
Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to
To ease Docker backend storage configuration, use the
utility, which automates much of the configuration details:
If you had a separate disk drive dedicated to Docker storage (for example,
/dev/xvdb), add the following to the /etc/sysconfig/docker-storage-setup
# systemctl restart docker-storage-setup
After the restart,
docker-storage-setup sets up a volume group named
docker_vg and creates a thin-pool logical volume. Documentation for thin
provisioning on RHEL is available in the
Administrator Guide. View the newly created volumes with the
# lsblk /dev/xvdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvdb 202:16 0 20G 0 disk
└─xvdb1 202:17 0 10G 0 part
├─docker_vg-docker--pool_tmeta 253:0 0 12M 0 lvm
│ └─docker_vg-docker--pool 253:2 0 6.9G 0 lvm
└─docker_vg-docker--pool_tdata 253:1 0 6.9G 0 lvm
└─docker_vg-docker--pool 253:2 0 6.9G 0 lvm
Thin-provisioned volumes are not mounted and have no file system (individual
containers do have an XFS file system), thus they do not show up in
To verify that Docker is using an LVM thin pool, and to monitor disk space
utilization, use the
docker info command. The
Pool Name corresponds with
VG you specified in /etc/sysconfig/docker-storage-setup:
# docker info | egrep -i 'storage|pool|space|filesystem'
Storage Driver: devicemapper
Pool Name: docker_vg-docker--pool
Pool Blocksize: 524.3 kB
Backing Filesystem: xfs
Data Space Used: 62.39 MB
Data Space Total: 6.434 GB
Data Space Available: 6.372 GB
Metadata Space Used: 40.96 kB
Metadata Space Total: 16.78 MB
Metadata Space Available: 16.74 MB
By default, a thin pool is configured to use 40% of the underlying block device.
As you use the storage, LVM automatically extends the thin pool up to 100%. This
is why the
Data Space Total value does not match the full size of the
underlying LVM device. This auto-extend technique was used to unify the storage
approach taken in both Red Hat Enterprise Linux and Red Hat Atomic Host, which
only uses a single partition.
In development, Docker in Red Hat distributions defaults to a
loopback mounted sparse file. To see if your system is using the loopback mode:
# docker info|grep loop0
Data file: /dev/loop0
Red Hat strongly recommends using the DeviceMapper storage driver in thin-pool mode for production workloads.
OverlayFS is also supported for container runtimes use cases as of Red Hat Enterprise Linux
7.2, and provides faster start up time and page cache sharing, which can
potentially improve density by reducing overall memory utilization.
Benefits of Using the OverlayFS Versus DeviceMapper with SELinux
The main advantage of the OverlayFS graph is Linux page cache sharing among
containers that share an image on the same node. This attribute of OverlayFS leads to
reduced input/output (I/O) during container startup (and, thus, faster container
startup time by several hundred milliseconds), as well as reduced memory usage
when similar images are running on a node. Both of these results are beneficial
in many environments, especially those with the goal of optimizing for density
and have high container churn rate (such as a build farm), or those that have
significant overlap in image content.
Page cache sharing is not possible with DeviceMapper because thin-provisioned
devices are allocated on a per-container basis.
DeviceMapper is the default Docker storage configuration on Red Hat Enterprise Linux.
The use of OverlayFS as the container storage
technology is under evaluation and moving Red Hat Enterprise Linux to OverlayFS as
the default in future releases is under consideration.
Comparing the Overlay Versus overlay2 Graph Drivers
OverlayFS is a type of union file system. It allows you to overlay one file system on top of another.
Changes are recorded in the upper file system, while the lower file system remains unmodified.
This allows multiple users to share a file-system image, such as a container or a DVD-ROM, where the base image is on read-only media.
OverlayFS layers two directories on a single Linux host and presents them as a single directory. These directories are called layers, and the unification process is referred to as a union mount.
OverlayFS uses one of two graph drivers, overlay or overlay2. As of Red Hat Enterprise
Linux 7.2, overlay became a supported graph driver.
As of Red Hat Enterprise Linux 7.4, overlay2 became supported. SELinux on the docker daemon became supported in
Red Hat Enterprise Linux 7.4. See the Red Hat Enterprise Linux release notes
for information on using OverlayFS with your version of RHEL, including supportability and usage caveats.
The overlay2 driver natively supports up to 128 lower OverlayFS layers but,
the overlay driver works only with a single lower OverlayFS layer. Because of this capability, the overlay2 driver provides better performance
for layer-related Docker commands, such as
docker build, and consumes fewer inodes on the backing filesystem.
Because the overlay driver works with a single lower OverlayFS layer, you cannot implement multi-layered images as multiple OverlayFS layers.
Instead, each image layer is implemented as its own directory under /var/lib/docker/overlay.
Hard links are then used as a space-efficient way to reference data shared with lower layers.
Docker recommends using the overlay2 driver with OverlayFS rather than
the overlay driver, because it is more efficient in terms of inode utilization.
To use overlay2 with RHEL or CentOS you need version 3.10.0-693 or higher of the kernel.