vCPU
In OpenShift Container Platform version 4.10, you can install a cluster on Microsoft Azure by using infrastructure that you provide.
Several Azure Resource Manager (ARM) templates are provided to assist in completing these steps or to help model your own.
The steps for performing a user-provisioned infrastructure installation are provided as an example only. Installing a cluster with infrastructure you provide requires knowledge of the cloud provider and the installation process of OpenShift Container Platform. Several ARM templates are provided to assist in completing these steps or to help model your own. You are also free to create the required resources through other methods; the templates are just an example. |
You reviewed details about the OpenShift Container Platform installation and update processes.
You read the documentation on selecting a cluster installation method and preparing it for users.
You configured an Azure account to host the cluster.
You downloaded the Azure CLI and installed it on your computer. See Install the Azure CLI in the Azure documentation. The documentation below was last tested using version 2.2.0
of the Azure CLI. Azure CLI commands might perform differently based on the version you use.
If you use a firewall and plan to use the Telemetry service, you configured the firewall to allow the sites that your cluster requires access to.
If the cloud identity and access management (IAM) APIs are not accessible in your environment, or if you do not want to store an administrator-level credential secret in the kube-system
namespace, you can manually create and maintain IAM credentials.
Be sure to also review this site list if you are configuring a proxy. |
In OpenShift Container Platform 4.10, you require access to the internet to install your cluster.
You must have internet access to:
Access OpenShift Cluster Manager to download the installation program and perform subscription management. If the cluster has internet access and you do not disable Telemetry, that service automatically entitles your cluster.
Access Quay.io to obtain the packages that are required to install your cluster.
Obtain the packages that are required to perform cluster updates.
If your cluster cannot have direct internet access, you can perform a restricted network installation on some types of infrastructure that you provision. During that process, you download the required content and use it to populate a mirror registry with the installation packages. With some installation types, the environment that you install your cluster in will not require internet access. Before you update the cluster, you update the content of the mirror registry. |
Before you can install OpenShift Container Platform, you must configure an Azure project to host it.
All Azure resources that are available through public endpoints are subject to resource name restrictions, and you cannot create resources that use certain terms. For a list of terms that Azure restricts, see Resolve reserved resource name errors in the Azure documentation. |
The OpenShift Container Platform cluster uses a number of Microsoft Azure components, and the default Azure subscription and service limits, quotas, and constraints affect your ability to install OpenShift Container Platform clusters.
Default limits vary by offer category types, such as Free Trial and Pay-As-You-Go, and by series, such as Dv2, F, and G. For example, the default for Enterprise Agreement subscriptions is 350 cores. Check the limits for your subscription type and if necessary, increase quota limits for your account before you install a default cluster on Azure. |
The following table summarizes the Azure components whose limits can impact your ability to install and run OpenShift Container Platform clusters.
Component | Number of components required by default | Default Azure limit | Description | ||||||
---|---|---|---|---|---|---|---|---|---|
vCPU |
40 |
20 per region |
A default cluster requires 40 vCPUs, so you must increase the account limit. By default, each cluster creates the following instances:
Because the bootstrap machine uses To deploy more worker nodes, enable autoscaling, deploy large workloads, or use a different instance type, you must further increase the vCPU limit for your account to ensure that your cluster can deploy the machines that you require. By default, the installation program distributes control plane and compute machines across all availability zones within a region. To ensure high availability for your cluster, select a region with at least three availability zones. If your region contains fewer than three availability zones, the installation program places more than one control plane machine in the available zones. |
||||||
OS Disk |
7 |
VM OS disk must be able to sustain a tested and recommended minimum throughput of 5000 IOPS / 200MBps. This throughput can be provided by having a minimum of 1 TiB Premium SSD (P30). In Azure, disk performance is directly dependent on SSD disk sizes, so to achieve the throughput supported by Host caching must be set to |
|||||||
VNet |
1 |
1000 per region |
Each default cluster requires one Virtual Network (VNet), which contains two subnets. |
||||||
Network interfaces |
7 |
65,536 per region |
Each default cluster requires seven network interfaces. If you create more machines or your deployed workloads create load balancers, your cluster uses more network interfaces. |
||||||
Network security groups |
2 |
5000 |
Each cluster creates network security groups for each subnet in the VNet. The default cluster creates network security groups for the control plane and for the compute node subnets:
|
||||||
Network load balancers |
3 |
1000 per region |
Each cluster creates the following load balancers:
If your applications create more Kubernetes |
||||||
Public IP addresses |
3 |
Each of the two public load balancers uses a public IP address. The bootstrap machine also uses a public IP address so that you can SSH into the machine to troubleshoot issues during installation. The IP address for the bootstrap node is used only during installation. |
|||||||
Private IP addresses |
7 |
The internal load balancer, each of the three control plane machines, and each of the three worker machines each use a private IP address. |
|||||||
Spot VM vCPUs (optional) |
0 If you configure spot VMs, your cluster must have two spot VM vCPUs for every compute node. |
20 per region |
This is an optional component. To use spot VMs, you must increase the Azure default limit to at least twice the number of compute nodes in your cluster.
|
To install OpenShift Container Platform, the Microsoft Azure account you use must have a dedicated public hosted DNS zone in your account. This zone must be authoritative for the domain. This service provides cluster DNS resolution and name lookup for external connections to the cluster.
Identify your domain, or subdomain, and registrar. You can transfer an existing domain and registrar or obtain a new one through Azure or another source.
For more information about purchasing domains through Azure, see Buy a custom domain name for Azure App Service in the Azure documentation. |
If you are using an existing domain and registrar, migrate its DNS to Azure. See Migrate an active DNS name to Azure App Service in the Azure documentation.
Configure DNS for your domain. Follow the steps in the Tutorial: Host your domain in Azure DNS in the Azure documentation to create a public hosted zone for your domain or subdomain, extract the new authoritative name servers, and update the registrar records for the name servers that your domain uses.
Use an appropriate root domain, such as openshiftcorp.com
, or subdomain,
such as clusters.openshiftcorp.com
.
If you use a subdomain, follow your company’s procedures to add its delegation records to the parent domain.
You can view Azure’s DNS solution by visiting this example for creating DNS zones.
To increase an account limit, file a support request on the Azure portal.
You can increase only one type of quota per support request. |
From the Azure portal, click Help + support in the lower left corner.
Click New support request and then select the required values:
From the Issue type list, select Service and subscription limits (quotas).
From the Subscription list, select the subscription to modify.
From the Quota type list, select the quota to increase. For example, select Compute-VM (cores-vCPUs) subscription limit increases to increase the number of vCPUs, which is required to install a cluster.
Click Next: Solutions.
On the Problem Details page, provide the required information for your quota increase:
Click Provide details and provide the required details in the Quota details window.
In the SUPPORT METHOD and CONTACT INFO sections, provide the issue severity and your contact details.
Click Next: Review + create and then click Create.
Because your cluster has limited access to automatic machine management when you use infrastructure that you provision, you must provide a mechanism for approving cluster certificate signing requests (CSRs) after installation. The kube-controller-manager
only approves the kubelet client CSRs. The machine-approver
cannot guarantee the validity of a serving certificate that is requested by using kubelet credentials because it cannot confirm that the correct machine issued the request. You must determine and implement a method of verifying the validity of the kubelet serving certificate requests and approving them.
OpenShift Container Platform needs a service principal so it can manage Microsoft Azure resources. Before you can create a service principal, your Azure account subscription must have the following roles:
User Access Administrator
Owner
To set roles on the Azure portal, see the Manage access to Azure resources using RBAC and the Azure portal in the Azure documentation.
Because OpenShift Container Platform and its installation program create Microsoft Azure resources by using the Azure Resource Manager, you must create a service principal to represent it.
Install or update the Azure CLI.
Your Azure account has the required roles for the subscription that you use.
Log in to the Azure CLI:
$ az login
If your Azure account uses subscriptions, ensure that you are using the right subscription:
View the list of available accounts and record the tenantId
value for the
subscription you want to use for your cluster:
$ az account list --refresh
[
{
"cloudName": "AzureCloud",
"id": "9bab1460-96d5-40b3-a78e-17b15e978a80",
"isDefault": true,
"name": "Subscription Name",
"state": "Enabled",
"tenantId": "6057c7e9-b3ae-489d-a54e-de3f6bf6a8ee",
"user": {
"name": "you@example.com",
"type": "user"
}
}
]
View your active account details and confirm that the tenantId
value matches
the subscription you want to use:
$ az account show
{
"environmentName": "AzureCloud",
"id": "9bab1460-96d5-40b3-a78e-17b15e978a80",
"isDefault": true,
"name": "Subscription Name",
"state": "Enabled",
"tenantId": "6057c7e9-b3ae-489d-a54e-de3f6bf6a8ee", (1)
"user": {
"name": "you@example.com",
"type": "user"
}
}
1 | Ensure that the value of the tenantId parameter is the correct subscription ID. |
If you are not using the right subscription, change the active subscription:
$ az account set -s <subscription_id> (1)
1 | Specify the subscription ID. |
Verify the subscription ID update:
$ az account show
{
"environmentName": "AzureCloud",
"id": "33212d16-bdf6-45cb-b038-f6565b61edda",
"isDefault": true,
"name": "Subscription Name",
"state": "Enabled",
"tenantId": "8049c7e9-c3de-762d-a54e-dc3f6be6a7ee",
"user": {
"name": "you@example.com",
"type": "user"
}
}
Record the tenantId
and id
parameter values from the output. You need these values during the OpenShift Container Platform installation.
Create the service principal for your account:
$ az ad sp create-for-rbac --role Contributor --name <service_principal> \ (1)
--scopes /subscriptions/<subscription_id> (2)
1 | Specify the service principal name. |
2 | Specify the subscription ID. |
Creating 'Contributor' role assignment under scope '/subscriptions/<subscription_id>'
The output includes credentials that you must protect. Be sure that you do not
include these credentials in your code or check the credentials into your source
control. For more information, see https://aka.ms/azadsp-cli
{
"appId": "ac461d78-bf4b-4387-ad16-7e32e328aec6",
"displayName": <service_principal>",
"password": "00000000-0000-0000-0000-000000000000",
"tenantId": "8049c7e9-c3de-762d-a54e-dc3f6be6a7ee"
}
Record the values of the appId
and password
parameters from the previous
output. You need these values during OpenShift Container Platform installation.
Assign the User Access Administrator
role by running the following command:
$ az role assignment create --role "User Access Administrator" \
--assignee-object-id $(az ad sp show --id <appId> --query id -o tsv) (1)
1 | Specify the appId parameter value for your service principal. |
For more information about CCO modes, see About the Cloud Credential Operator.
The installation program dynamically generates the list of available Microsoft Azure regions based on your subscription.
australiacentral
(Australia Central)
australiaeast
(Australia East)
australiasoutheast
(Australia South East)
brazilsouth
(Brazil South)
canadacentral
(Canada Central)
canadaeast
(Canada East)
centralindia
(Central India)
centralus
(Central US)
eastasia
(East Asia)
eastus
(East US)
eastus2
(East US 2)
francecentral
(France Central)
germanywestcentral
(Germany West Central)
japaneast
(Japan East)
japanwest
(Japan West)
koreacentral
(Korea Central)
koreasouth
(Korea South)
northcentralus
(North Central US)
northeurope
(North Europe)
norwayeast
(Norway East)
southafricanorth
(South Africa North)
southcentralus
(South Central US)
southeastasia
(Southeast Asia)
southindia
(South India)
swedencentral
(Sweden Central)
switzerlandnorth
(Switzerland North)
uaenorth
(UAE North)
uksouth
(UK South)
ukwest
(UK West)
westcentralus
(West Central US)
westeurope
(West Europe)
westindia
(West India)
westus
(West US)
westus2
(West US 2)
westus3
(West US 3)
Support for the following Microsoft Azure Government (MAG) regions was added in OpenShift Container Platform version 4.6:
usgovtexas
(US Gov Texas)
usgovvirginia
(US Gov Virginia)
You can reference all available MAG regions in the Azure documentation. Other provided MAG regions are expected to work with OpenShift Container Platform, but have not been tested.
For a cluster that contains user-provisioned infrastructure, you must deploy all of the required machines.
This section describes the requirements for deploying OpenShift Container Platform on user-provisioned infrastructure.
The smallest OpenShift Container Platform clusters require the following hosts:
Hosts | Description |
---|---|
One temporary bootstrap machine |
The cluster requires the bootstrap machine to deploy the OpenShift Container Platform cluster on the three control plane machines. You can remove the bootstrap machine after you install the cluster. |
Three control plane machines |
The control plane machines run the Kubernetes and OpenShift Container Platform services that form the control plane. |
At least two compute machines, which are also known as worker machines. |
The workloads requested by OpenShift Container Platform users run on the compute machines. |
To maintain high availability of your cluster, use separate physical hosts for these cluster machines. |
The bootstrap and control plane machines must use Red Hat Enterprise Linux CoreOS (RHCOS) as the operating system. However, the compute machines can choose between Red Hat Enterprise Linux CoreOS (RHCOS), Red Hat Enterprise Linux (RHEL) 8.4, or RHEL 8.5.
Note that RHCOS is based on Red Hat Enterprise Linux (RHEL) 8 and inherits all of its hardware certifications and requirements. See Red Hat Enterprise Linux technology capabilities and limits.
Each cluster machine must meet the following minimum requirements:
Machine | Operating System | vCPU [1] | Virtual RAM | Storage | IOPS [2] |
---|---|---|---|---|---|
Bootstrap |
RHCOS |
4 |
16 GB |
100 GB |
300 |
Control plane |
RHCOS |
4 |
16 GB |
100 GB |
300 |
Compute |
RHCOS, RHEL 8.4, or RHEL 8.5 [3] |
2 |
8 GB |
100 GB |
300 |
One vCPU is equivalent to one physical core when simultaneous multithreading (SMT), or hyperthreading, is not enabled. When enabled, use the following formula to calculate the corresponding ratio: (threads per core × cores) × sockets = vCPUs.
OpenShift Container Platform and Kubernetes are sensitive to disk performance, and faster storage is recommended, particularly for etcd on the control plane nodes which require a 10 ms p99 fsync duration. Note that on many cloud platforms, storage size and IOPS scale together, so you might need to over-allocate storage volume to obtain sufficient performance.
As with all user-provisioned installations, if you choose to use RHEL compute machines in your cluster, you take responsibility for all operating system life cycle management and maintenance, including performing system updates, applying patches, and completing all other required tasks. Use of RHEL 7 compute machines is deprecated and has been removed in OpenShift Container Platform 4.10 and later.
You are required to use Azure virtual machines with |
If an instance type for your platform meets the minimum requirements for cluster machines, it is supported to use in OpenShift Container Platform.
The following Microsoft Azure instance types have been tested with OpenShift Container Platform.
standardBSFamily
standardDADSv5Family
standardDASv4Family
standardDASv5Family
standardDCSv3Family
standardDCSFamily
standardDCSv2Family
standardDDCSv3Family
standardDDSv4Family
standardDDSv5Family
standardDSFamily
standardDSv2Family
standardDSv2PromoFamily
standardDSv3Family
standardDSv4Family
standardDSv5Family
standardEADSv5Family
standardEASv4Family
standardEASv5Family
standardEBDSv5Family
standardEBSv5Family
standardEDSv4Family
standardEDSv5Family
standardEIADSv5Family
standardEIASv4Family
standardEIASv5Family
standardEIDSv5Family
standardEISv3Family
standardEISv5Family
standardESv3Family
standardESv4Family
standardESv5Family
standardFXMDVSFamily
standardFSFamily
standardFSv2Family
standardGSFamily
standardHBrsv2Family
standardHBSFamily
standardHCSFamily
standardLASv3Family
standardLSFamily
standardLSv2Family
standardLSv3Family
standardMDSMediumMemoryv2Family
standardMIDSMediumMemoryv2Family
standardMISMediumMemoryv2Family
standardMSFamily
standardMSMediumMemoryv2Family
StandardNCADSA100v4Family
Standard NCASv3_T4 Family
standardNCSv2Family
standardNCSv3Family
standardNDSv2Family
standardNPSFamily
StandardNVADSA10v5Family
standardNVSv3Family
standardXEISv4Family
If you are deploying an OpenShift Container Platform cluster using the Azure Marketplace offering, you must first obtain the Azure Marketplace image. The installation program uses this image to deploy worker nodes. When obtaining your image, consider the following:
While the images are the same, the Azure Marketplace publisher is different depending on your region. If you are located in North America, specify redhat
as the publisher. If you are located in EMEA, specify redhat-limited
as the publisher.
The offer includes a rh-ocp-worker
SKU and a rh-ocp-worker-gen1
SKU. The rh-ocp-worker
SKU represents a Hyper-V generation version 2 VM image. The default instance types used in OpenShift Container Platform are version 2 compatible. If you are going to use an instance type that is only version 1 compatible, use the image associated with the rh-ocp-worker-gen1
SKU. The rh-ocp-worker-gen1
SKU represents a Hyper-V version 1 VM image.