OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history bug_report picture_as_pdf
×

Preparing your cluster for OpenShift Virtualization

edit
edit

Supported platforms

You can use the following platforms with OpenShift Virtualization:

  • Amazon Web Services bare metal instances.

edit

OpenShift Virtualization on Red Hat OpenShift Service on AWS

You can run OpenShift Virtualization on a Red Hat OpenShift Service on AWS (ROSA) Classic cluster.

Before you set up your cluster, review the following summary of supported features and limitations:

Installing

  • You can install the cluster by using installer-provisioned infrastructure, ensuring that you specify bare-metal instance types for the worker nodes. For example, you can use the c5n.metal type value for a machine based on x86_64 architecture.

    For more information, see the Red Hat OpenShift Service on AWS documentation about installing on AWS.

Accessing virtual machines (VMs)

  • There is no change to how you access VMs by using the virtctl CLI tool or the Red Hat OpenShift Service on AWS web console.

  • You can expose VMs by using a NodePort or LoadBalancer service.

    • The load balancer approach is preferable because Red Hat OpenShift Service on AWS automatically creates the load balancer in AWS and manages its lifecycle. A security group is also created for the load balancer, and you can use annotations to attach existing security groups. When you remove the service, Red Hat OpenShift Service on AWS removes the load balancer and its associated resources.

Networking

  • If your application requires a flat layer 2 network or control over the IP pool, consider using OVN-Kubernetes secondary overlay networks.

Storage

  • You can use any storage solution that is certified by the storage vendor to work with the underlying platform.

    AWS bare-metal and ROSA clusters might have different supported storage solutions. Ensure that you confirm support with your storage vendor.

  • Using Amazon Elastic File System (EFS) or Amazon Elastic Block Store (EBS) with OpenShift Virtualization might cause performance and functionality limitations as shown in the following table:

    Table 1. EFS and EBS performance and functionality limitations
    Feature EBS volume EFS volume Shared storage solutions

    gp2

    gp3

    io2

    VM live migration

    Not available

    Not available

    Available

    Available

    Available

    Fast VM creation by using cloning

    Available

    Not available

    Available

    VM backup and restore by using snapshots

    Available

    Not available

    Available

    Consider using CSI storage, which supports ReadWriteMany (RWX), cloning, and snapshots to enable live migration, fast VM creation, and VM snapshots capabilities.

Additional resources

Hardware and operating system requirements

Review the following hardware and operating system requirements for OpenShift Virtualization.

CPU requirements

  • Supported by Red Hat Enterprise Linux (RHEL) 9.

    See Red Hat Ecosystem Catalog for supported CPUs.

    If your worker nodes have different CPUs, live migration failures might occur because different CPUs have different capabilities. You can mitigate this issue by ensuring that your worker nodes have CPUs with the appropriate capacity and by configuring node affinity rules for your virtual machines.

    See Configuring a required node affinity rule for details.

  • Support for AMD and Intel 64-bit architectures (x86-64-v2).

  • Support for Intel 64 or AMD64 CPU extensions.

  • Intel VT or AMD-V hardware virtualization extensions enabled.

  • NX (no execute) flag enabled.

Operating system requirements

  • Red Hat Enterprise Linux CoreOS (RHCOS) installed on worker nodes.

Storage requirements

  • Supported by Red Hat OpenShift Service on AWS.

  • If the storage provisioner supports snapshots, you must associate a VolumeSnapshotClass object with the default storage class.

edit

About volume and access modes for virtual machine disks

If you use the storage API with known storage providers, the volume and access modes are selected automatically. However, if you use a storage class that does not have a storage profile, you must configure the volume and access mode.

For best results, use the ReadWriteMany (RWX) access mode and the Block volume mode. This is important for the following reasons:

  • ReadWriteMany (RWX) access mode is required for live migration.

  • The Block volume mode performs significantly better than the Filesystem volume mode. This is because the Filesystem volume mode uses more storage layers, including a file system layer and a disk image file. These layers are not necessary for VM disk storage.

You cannot live migrate virtual machines with the following configurations:

  • Storage volume with ReadWriteOnce (RWO) access mode

  • Passthrough features such as GPUs

Set the evictionStrategy field to None for these virtual machines. The None strategy powers down VMs during node reboots.

Live migration requirements

  • Shared storage with ReadWriteMany (RWX) access mode.

  • Sufficient RAM and network bandwidth.

    You must ensure that there is enough memory request capacity in the cluster to support node drains that result in live migrations. You can determine the approximate required spare memory by using the following calculation:

    Product of (Maximum number of nodes that can drain in parallel) and (Highest total VM memory request allocations across nodes)

    The default number of migrations that can run in parallel in the cluster is 5.

  • If the virtual machine uses a host model CPU, the nodes must support the virtual machine’s host model CPU.

  • A dedicated Multus network for live migration is highly recommended. A dedicated network minimizes the effects of network saturation on tenant workloads during migration.

edit

Physical resource overhead requirements

OpenShift Virtualization is an add-on to Red Hat OpenShift Service on AWS and imposes additional overhead that you must account for when planning a cluster. Each cluster machine must accommodate the following overhead requirements in addition to the Red Hat OpenShift Service on AWS requirements. Oversubscribing the physical resources in a cluster can affect performance.

The numbers noted in this documentation are based on Red Hat’s test methodology and setup. These numbers can vary based on your own individual setup and environments.

Memory overhead

Calculate the memory overhead values for OpenShift Virtualization by using the equations below.

Cluster memory overhead
Memory overhead per infrastructure node ≈ 150 MiB
Memory overhead per worker node ≈ 360 MiB

Additionally, OpenShift Virtualization environment resources require a total of 2179 MiB of RAM that is spread across all infrastructure nodes.

Virtual machine memory overhead
Memory overhead per virtual machine ≈ (1.002 × requested memory) \
              + 218 MiB \ (1)
              + 8 MiB × (number of vCPUs) \ (2)
              + 16 MiB × (number of graphics devices) \ (3)
              + (additional memory overhead) (4)
1 Required for the processes that run in the virt-launcher pod.
2 Number of virtual CPUs requested by the virtual machine.
3 Number of virtual graphics cards requested by the virtual machine.
4 Additional memory overhead:

  • If your environment includes a Single Root I/O Virtualization (SR-IOV) network device or a Graphics Processing Unit (GPU), allocate 1 GiB additional memory overhead for each device.

  • If Secure Encrypted Virtualization (SEV) is enabled, add 256 MiB.

  • If Trusted Platform Module (TPM) is enabled, add 53 MiB.

CPU overhead

Calculate the cluster processor overhead requirements for OpenShift Virtualization by using the equation below. The CPU overhead per virtual machine depends on your individual setup.

Cluster CPU overhead
CPU overhead for infrastructure nodes ≈ 4 cores

OpenShift Virtualization increases the overall utilization of cluster level services such as logging, routing, and monitoring. To account for this workload, ensure that nodes that host infrastructure components have capacity allocated for 4 additional cores (4000 millicores) distributed across those nodes.

CPU overhead for worker nodes ≈ 2 cores + CPU overhead per virtual machine

Each worker node that hosts virtual machines must have capacity for 2 additional cores (2000 millicores) for OpenShift Virtualization management workloads in addition to the CPUs required for virtual machine workloads.

Virtual machine CPU overhead

If dedicated CPUs are requested, there is a 1:1 impact on the cluster CPU overhead requirement. Otherwise, there are no specific rules about how many CPUs a virtual machine requires.

Storage overhead

Use the guidelines below to estimate storage overhead requirements for your OpenShift Virtualization environment.

Cluster storage overhead
Aggregated storage overhead per node ≈ 10 GiB

10 GiB is the estimated on-disk storage impact for each node in the cluster when you install OpenShift Virtualization.

Virtual machine storage overhead

Storage overhead per virtual machine depends on specific requests for resource allocation within the virtual machine. The request could be for ephemeral storage on the node or storage resources hosted elsewhere in the cluster. OpenShift Virtualization does not currently allocate any additional ephemeral storage for the running container itself.

Example

As a cluster administrator, if you plan to host 10 virtual machines in the cluster, each with 1 GiB of RAM and 2 vCPUs, the memory impact across the cluster is 11.68 GiB. The estimated on-disk storage impact for each node in the cluster is 10 GiB and the CPU impact for worker nodes that host virtual machine workloads is a minimum of 2 cores.

Additional resources