$ sudo openstack quota set --secgroups 250 --secgroup-rules 1000 --ports 1500 --subnets 250 --networks 250 <project>install-config.yaml file for RHOSP with KuryrIn OpenShift Container Platform version 4.10, you can install a cluster on Red Hat OpenStack Platform (RHOSP) that runs on user-provisioned infrastructure.
Using your own infrastructure allows you to integrate your cluster with existing infrastructure and modifications. The process requires more labor on your part than installer-provisioned installations, because you must create all RHOSP resources, like Nova servers, Neutron ports, and security groups. However, Red Hat provides Ansible playbooks to help you in the deployment process.
You reviewed details about the OpenShift Container Platform installation and update processes.
You read the documentation on selecting a cluster installation method and preparing it for users.
You verified that OpenShift Container Platform 4.10 is compatible with your RHOSP version by using the Supported platforms for OpenShift clusters section. You can also compare platform support across different versions by viewing the OpenShift Container Platform on RHOSP support matrix.
You have an RHOSP account where you want to install OpenShift Container Platform.
On the machine from which you run the installation program, you have:
A single directory in which you can keep the files you create during the installation process
Python 3
Kuryr is a container network interface (CNI) plug-in solution that uses the Neutron and Octavia Red Hat OpenStack Platform (RHOSP) services to provide networking for pods and Services.
Kuryr and OpenShift Container Platform integration is primarily designed for OpenShift Container Platform clusters running on RHOSP VMs. Kuryr improves the network performance by plugging OpenShift Container Platform pods into RHOSP SDN. In addition, it provides interconnectivity between pods and RHOSP virtual instances.
Kuryr components are installed as pods in OpenShift Container Platform using the
openshift-kuryr namespace:
kuryr-controller - a single service instance installed on a master node.
This is modeled in OpenShift Container Platform as a Deployment object.
kuryr-cni - a container installing and configuring Kuryr as a CNI driver on
each OpenShift Container Platform node. This is modeled in OpenShift Container Platform as a DaemonSet object.
The Kuryr controller watches the OpenShift Container Platform API server for pod, service, and namespace create, update, and delete events. It maps the OpenShift Container Platform API calls to corresponding objects in Neutron and Octavia. This means that every network solution that implements the Neutron trunk port functionality can be used to back OpenShift Container Platform via Kuryr. This includes open source solutions such as Open vSwitch (OVS) and Open Virtual Network (OVN) as well as Neutron-compatible commercial SDNs.
Kuryr is recommended for OpenShift Container Platform deployments on encapsulated RHOSP tenant networks to avoid double encapsulation, such as running an encapsulated OpenShift Container Platform SDN over an RHOSP network.
If you use provider networks or tenant VLANs, you do not need to use Kuryr to avoid double encapsulation. The performance benefit is negligible. Depending on your configuration, though, using Kuryr to avoid having two overlays might still be beneficial.
Kuryr is not recommended in deployments where all of the following criteria are true:
The RHOSP version is less than 16.
The deployment uses UDP services, or a large number of TCP services on few hypervisors.
or
The ovn-octavia Octavia driver is disabled.
The deployment uses a large number of TCP services on few hypervisors.
When using Kuryr SDN, the pods, services, namespaces, and network policies are using resources from the RHOSP quota; this increases the minimum requirements. Kuryr also has some additional requirements on top of what a default install requires.
Use the following quota to satisfy a default cluster’s minimum requirements:
| Resource | Value |
|---|---|
Floating IP addresses |
3 - plus the expected number of Services of LoadBalancer type |
Ports |
1500 - 1 needed per Pod |
Routers |
1 |
Subnets |
250 - 1 needed per Namespace/Project |
Networks |
250 - 1 needed per Namespace/Project |
RAM |
112 GB |
vCPUs |
28 |
Volume storage |
275 GB |
Instances |
7 |
Security groups |
250 - 1 needed per Service and per NetworkPolicy |
Security group rules |
1000 |
Server groups |
2 - plus 1 for each additional availability zone in each machine pool |
Load balancers |
100 - 1 needed per Service |
Load balancer listeners |
500 - 1 needed per Service-exposed port |
Load balancer pools |
500 - 1 needed per Service-exposed port |
A cluster might function with fewer than recommended resources, but its performance is not guaranteed.
|
If RHOSP object storage (Swift) is available and operated by a user account with the |
|
If you are using Red Hat OpenStack Platform (RHOSP) version 16 with the Amphora driver rather than the OVN Octavia driver, security groups are associated with service accounts instead of user projects. |
Take the following notes into consideration when setting resources:
The number of ports that are required is larger than the number of pods. Kuryr uses ports pools to have pre-created ports ready to be used by pods and speed up the pods' booting time.
Each network policy is mapped into an RHOSP security group, and
depending on the NetworkPolicy spec, one or more rules are added to the
security group.
Each service is mapped to an RHOSP load balancer. Consider this requirement when estimating the number of security groups required for the quota.
If you are using
RHOSP version 15 or earlier, or the ovn-octavia driver, each load balancer
has a security group with the user project.
The quota does not account for load balancer resources (such as VM resources), but you must consider these resources when you decide the RHOSP deployment’s size. The default installation will have more than 50 load balancers; the clusters must be able to accommodate them.
If you are using RHOSP version 16 with the OVN Octavia driver enabled, only one load balancer VM is generated; services are load balanced through OVN flows.
An OpenShift Container Platform deployment comprises control plane machines, compute machines, and a bootstrap machine.
To enable Kuryr SDN, your environment must meet the following requirements:
Run RHOSP 13+.
Have Overcloud with Octavia.
Use Neutron Trunk ports extension.
Use openvswitch firewall driver if ML2/OVS Neutron driver is used instead
of ovs-hybrid.
When using Kuryr SDN, you must increase quotas to satisfy the Red Hat OpenStack Platform (RHOSP) resources used by pods, services, namespaces, and network policies.
Increase the quotas for a project by running the following command:
$ sudo openstack quota set --secgroups 250 --secgroup-rules 1000 --ports 1500 --subnets 250 --networks 250 <project>Kuryr CNI leverages the Neutron Trunks extension to plug containers into the
Red Hat OpenStack Platform (RHOSP) SDN, so you must use the trunks extension for Kuryr to properly work.
In addition, if you leverage the default ML2/OVS Neutron driver, the firewall
must be set to openvswitch instead of ovs_hybrid so that security groups are
enforced on trunk subports and Kuryr can properly handle network policies.
Kuryr SDN uses Red Hat OpenStack Platform (RHOSP)'s Octavia LBaaS to implement OpenShift Container Platform services. Thus, you must install and configure Octavia components in RHOSP to use Kuryr SDN.
To enable Octavia, you must include the Octavia service during the installation of the RHOSP Overcloud, or upgrade the Octavia service if the Overcloud already exists. The following steps for enabling Octavia apply to both a clean install of the Overcloud or an Overcloud update.
|
The following steps only capture the key pieces required during the deployment of RHOSP when dealing with Octavia. It is also important to note that registry methods vary. This example uses the local registry method. |
If you are using the local registry, create a template to upload the images to the registry. For example:
(undercloud) $ openstack overcloud container image prepare \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
--namespace=registry.access.redhat.com/rhosp13 \
--push-destination=<local-ip-from-undercloud.conf>:8787 \
--prefix=openstack- \
--tag-from-label {version}-{product-version} \
--output-env-file=/home/stack/templates/overcloud_images.yaml \
--output-images-file /home/stack/local_registry_images.yamlVerify that the local_registry_images.yaml file contains the Octavia images.
For example:
...
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-43
push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-health-manager:13.0-45
push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-housekeeping:13.0-45
push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-worker:13.0-44
push_destination: <local-ip-from-undercloud.conf>:8787|
The Octavia container versions vary depending upon the specific RHOSP release installed. |
Pull the container images from registry.redhat.io to the Undercloud node:
(undercloud) $ sudo openstack overcloud container image upload \
--config-file /home/stack/local_registry_images.yaml \
--verboseThis may take some time depending on the speed of your network and Undercloud disk.
Since an Octavia load balancer is used to access the OpenShift Container Platform API, you must increase their listeners' default timeouts for the connections. The default timeout is 50 seconds. Increase the timeout to 20 minutes by passing the following file to the Overcloud deploy command:
(undercloud) $ cat octavia_timeouts.yaml
parameter_defaults:
OctaviaTimeoutClientData: 1200000
OctaviaTimeoutMemberData: 1200000|
This is not needed for RHOSP 13.0.13+. |
Install or update your Overcloud environment with Octavia:
$ openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
-e octavia_timeouts.yaml|
This command only includes the files associated with Octavia; it varies based on your specific installation of RHOSP. See the RHOSP documentation for further information. For more information on customizing your Octavia installation, see installation of Octavia using Director. |
|
When leveraging Kuryr SDN, the Overcloud installation requires the Neutron |
In RHOSP versions earlier than 13.0.13, add the project ID
to the octavia.conf configuration file after you create the project.
To enforce network policies across services, like when traffic goes through the Octavia load balancer, you must ensure Octavia creates the Amphora VM security groups on the user project.
This change ensures that required load balancer security groups belong to that project, and that they can be updated to enforce services isolation.
|
This task is unnecessary in RHOSP version 13.0.13 or later. Octavia implements a new ACL API that restricts access to the load balancers VIP. |
Get the project ID
$ openstack project show <project>+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | |
| domain_id | default |
| enabled | True |
| id | PROJECT_ID |
| is_domain | False |
| name | *<project>* |
| parent_id | default |
| tags | [] |
+-------------+----------------------------------+Add the project ID to octavia.conf for the controllers.
Source the stackrc file:
$ source stackrc # Undercloud credentialsList the Overcloud controllers:
$ openstack server list+--------------------------------------+--------------+--------+-----------------------+----------------+------------+
│
| ID | Name | Status | Networks
| Image | Flavor |
│
+--------------------------------------+--------------+--------+-----------------------+----------------+------------+
│
| 6bef8e73-2ba5-4860-a0b1-3937f8ca7e01 | controller-0 | ACTIVE |
ctlplane=192.168.24.8 | overcloud-full | controller |
│
| dda3173a-ab26-47f8-a2dc-8473b4a67ab9 | compute-0 | ACTIVE |
ctlplane=192.168.24.6 | overcloud-full | compute |
│
+--------------------------------------+--------------+--------+-----------------------+----------------+------------+SSH into the controller(s).
$ ssh heat-admin@192.168.24.8Edit the octavia.conf file to add the project into the list of projects where
Amphora security groups are on the user’s account.
# List of project IDs that are allowed to have Load balancer security groups # belonging to them. amp_secgroup_allowed_projects = PROJECT_ID
Restart the Octavia worker so the new configuration loads.
controller-0$ sudo docker restart octavia_worker|
Depending on your RHOSP environment, Octavia might not support UDP listeners. If you use Kuryr SDN on RHOSP version 13.0.13 or earlier, UDP services are not supported. RHOSP version 16 or later support UDP. |
Octavia supports multiple provider drivers through the Octavia API.
To see all available Octavia provider drivers, on a command line, enter:
$ openstack loadbalancer provider list+---------+-------------------------------------------------+
| name | description |
+---------+-------------------------------------------------+
| amphora | The Octavia Amphora driver. |
| octavia | Deprecated alias of the Octavia Amphora driver. |
| ovn | Octavia OVN driver. |
+---------+-------------------------------------------------+Beginning with RHOSP version 16, the Octavia OVN provider driver (ovn) is supported on
OpenShift Container Platform on RHOSP deployments.
ovn is an integration driver for the load balancing
that Octavia and OVN provide. It supports basic load balancing capabilities,
and is based on OpenFlow rules. The driver is automatically enabled
in Octavia by Director on deployments that use OVN Neutron ML2.
The Amphora provider driver is the default driver. If ovn is enabled, however, Kuryr uses it.
If Kuryr uses ovn instead of Amphora, it offers the following benefits:
Decreased resource requirements. Kuryr does not require a load balancer VM for each service.
Reduced network latency.
Increased service creation speed by using OpenFlow rules instead of a VM for each service.
Distributed load balancing actions across all nodes instead of centralized on Amphora VMs.
Using OpenShift Container Platform with Kuryr SDN has several known limitations.
Using OpenShift Container Platform with Kuryr SDN has several limitations that apply to all versions and environments:
Service objects with the NodePort type are not supported.
Clusters that use the OVN Octavia provider driver support Service objects for which the .spec.selector property is unspecified only if the .subsets.addresses property of the Endpoints object includes the subnet of the nodes or pods.
If the subnet on which machines are created is not connected to a router, or if the subnet is connected, but the router has no external gateway set, Kuryr cannot create floating IPs for Service objects with type LoadBalancer.
Configuring the sessionAffinity=ClientIP property on Service objects does not have an effect. Kuryr does not support this setting.
Using OpenShift Container Platform with Kuryr SDN has several limitations that depend on the RHOSP version.
RHOSP versions before 16 use the default Octavia load balancer driver (Amphora). This driver requires that one Amphora load balancer VM is deployed per OpenShift Container Platform service. Creating too many services can cause you to run out of resources.
Deployments of later versions of RHOSP that have the OVN Octavia driver disabled also use the Amphora driver. They are subject to the same resource concerns as earlier versions of RHOSP.
Octavia RHOSP versions before 13.0.13 do not support UDP listeners. Therefore, OpenShift Container Platform UDP services are not supported.
Octavia RHOSP versions before 13.0.13 cannot listen to multiple protocols on the same port. Services that expose the same port to different protocols, like TCP and UDP, are not supported.
Kuryr SDN does not support automatic unidling by a service.
There are limitations when using Kuryr SDN that depend on your deployment environment.
Because of Octavia’s lack of support for the UDP protocol and multiple listeners, if the RHOSP version is earlier than 13.0.13, Kuryr forces pods to use TCP for DNS resolution.
In Go versions 1.12 and earlier, applications that are compiled with CGO support disabled use UDP only. In this case,
the native Go resolver does not recognize the use-vc option in resolv.conf, which controls whether TCP is forced for DNS resolution.
As a result, UDP is still used for DNS resolution, which fails.
To ensure that TCP forcing is allowed, compile applications either with the environment variable CGO_ENABLED set to 1, i.e. CGO_ENABLED=1, or ensure that the variable is absent.
In Go versions 1.13 and later, TCP is used automatically if DNS resolution using UDP fails.
|
musl-based containers, including Alpine-based containers, do not support the |
As a result of the RHOSP upgrade process, the Octavia API might be changed, and upgrades to the Amphora images that are used for load balancers might be required.
You can address API changes on an individual basis.
If the Amphora image is upgraded, the RHOSP operator can handle existing load balancer VMs in two ways:
Upgrade each VM by triggering a load balancer failover.
Leave responsibility for upgrading the VMs to users.
If the operator takes the first option, there might be short downtimes during failovers.
If the operator takes the second option, the existing load balancers will not support upgraded Octavia API features, like UDP listeners. In this case, users must recreate their Services to use these features.
|
If OpenShift Container Platform detects a new Octavia version that supports UDP load balancing, it recreates the DNS service automatically. The service recreation ensures that the service default supports UDP load balancing. The recreation causes the DNS service approximately one minute of downtime. |
By default, the OpenShift Container Platform installation process stands up three control plane and three compute machines.
Each machine requires:
An instance from the RHOSP quota
A port from the RHOSP quota
A flavor with at least 16 GB memory, 4 vCPUs, and 25 GB storage space
|
Compute machines host the applications that you run on OpenShift Container Platform; aim to run as many as you can. |
During installation, a bootstrap machine is temporarily provisioned to stand up the control plane. After the production control plane is ready, the bootstrap machine is deprovisioned.
The bootstrap machine requires:
An instance from the RHOSP quota
A port from the RHOSP quota
A flavor with at least 16 GB memory, 4 vCPUs, and 25 GB storage space
In OpenShift Container Platform 4.10, you require access to the internet to install your cluster.
You must have internet access to:
Access OpenShift Cluster Manager to download the installation program and perform subscription management. If the cluster has internet access and you do not disable Telemetry, that service automatically entitles your cluster.
Access Quay.io to obtain the packages that are required to install your cluster.
Obtain the packages that are required to perform cluster updates.
|
If your cluster cannot have direct internet access, you can perform a restricted network installation on some types of infrastructure that you provision. During that process, you download the required content and use it to populate a mirror registry with the installation packages. With some installation types, the environment that you install your cluster in will not require internet access. Before you update the cluster, you update the content of the mirror registry. |
The Ansible playbooks that simplify the installation process on user-provisioned infrastructure require several Python modules. On the machine where you will run the installer, add the modules' repositories and then download them.
| These instructions assume that you are using Red Hat Enterprise Linux (RHEL) 8. |
Python 3 is installed on your machine.
On a command line, add the repositories:
Register with Red Hat Subscription Manager:
$ sudo subscription-manager register # If not done alreadyPull the latest subscription data:
$ sudo subscription-manager attach --pool=$YOUR_POOLID # If not done alreadyDisable the current repositories:
$ sudo subscription-manager repos --disable=* # If not done alreadyAdd the required repositories:
$ sudo subscription-manager repos \
--enable=rhel-8-for-x86_64-baseos-rpms \
--enable=openstack-16-tools-for-rhel-8-x86_64-rpms \
--enable=ansible-2.9-for-rhel-8-x86_64-rpms \
--enable=rhel-8-for-x86_64-appstream-rpmsInstall the modules:
$ sudo yum install python3-openstackclient ansible python3-openstacksdk python3-netaddrEnsure that the python command points to python3:
$ sudo alternatives --set python /usr/bin/python3Download Ansible playbooks that you can use to install OpenShift Container Platform on your own Red Hat OpenStack Platform (RHOSP) infrastructure.
The curl command-line tool is available on your machine.
To download the playbooks to your working directory, run the following script from a command line:
$ xargs -n 1 curl -O <<< '
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/bootstrap.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/common.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/compute-nodes.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/control-plane.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/inventory.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/network.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/security-groups.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-bootstrap.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-compute-nodes.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-control-plane.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-load-balancers.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-network.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-security-groups.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-containers.yaml'The playbooks are downloaded to your machine.
|
During the installation process, you can modify the playbooks to configure your deployment. Retain all playbooks for the life of your cluster. You must have the playbooks to remove your OpenShift Container Platform cluster from RHOSP. |
|
You must match any edits you make in the |
Before you install OpenShift Container Platform, download the installation file on a local computer.
You have a computer that runs Linux or macOS, with 500 MB of local disk space
Access the Infrastructure Provider page on the OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.
Select your infrastructure provider.
Navigate to the page for your installation type, download the installation program that corresponds with your host operating system and architecture, and place the file in the directory where you will store the installation configuration files.
|
The installation program creates several files on the computer that you use to install your cluster. You must keep the installation program and the files that the installation program creates after you finish installing the cluster. Both files are required to delete the cluster. |
|
Deleting the files created by the installation program does not remove your cluster, even if the cluster failed during installation. To remove your cluster, complete the OpenShift Container Platform uninstallation procedures for your specific cloud provider. |
Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:
$ tar -xvf openshift-install-linux.tar.gzDownload your installation pull secret from the Red Hat OpenShift Cluster Manager. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.
During an OpenShift Container Platform installation, you can provide an SSH public key to the installation program. The key is passed to the Red Hat Enterprise Linux CoreOS (RHCOS) nodes through their Ignition config files and is used to authenticate SSH access to the nodes. The key is added to the ~/.ssh/authorized_keys list for the core user on each node, which enables password-less authentication.
After the key is passed to the nodes, you can use the key pair to SSH in to the RHCOS nodes as the user core. To access the nodes through SSH, the private key identity must be managed by SSH for your local user.
If you want to SSH in to your cluster nodes to perform installation debugging or disaster recovery, you must provide the SSH public key during the installation process. The ./openshift-install gather command also requires the SSH public key to be in place on the cluster nodes.
|
Do not skip this procedure in production environments, where disaster recovery and debugging is required. |
|
You must use a local key, not one that you configured with platform-specific approaches such as AWS key pairs. |
If you do not have an existing SSH key pair on your local machine to use for authentication onto your cluster nodes, create one. For example, on a computer that uses a Linux operating system, run the following command:
$ ssh-keygen -t ed25519 -N '' -f <path>/<file_name> (1)| 1 | Specify the path and file name, such as ~/.ssh/id_ed25519, of the new SSH key. If you have an existing key pair, ensure your public key is in the your ~/.ssh directory. |
|
If you plan to install an OpenShift Container Platform cluster that uses FIPS Validated / Modules in Process cryptographic libraries on the |
View the public SSH key:
$ cat <path>/<file_name>.pubFor example, run the following to view the ~/.ssh/id_ed25519.pub public key:
$ cat ~/.ssh/id_ed25519.pubAdd the SSH private key identity to the SSH agent for your local user, if it has not already been added. SSH agent management of the key is required for password-less SSH authentication onto your cluster nodes, or if you want to use the ./openshift-install gather command.
|
On some distributions, default SSH private key identities such as |
If the ssh-agent process is not already running for your local user, start it as a background task:
$ eval "$(ssh-agent -s)"Agent pid 31874|
If your cluster is in FIPS mode, only use FIPS-compliant algorithms to generate the SSH key. The key must be either RSA or ECDSA. |
Add your SSH private key to the ssh-agent:
$ ssh-add <path>/<file_name> (1)| 1 | Specify the path and file name for your SSH private key, such as ~/.ssh/id_ed25519 |
Identity added: /home/<you>/<path>/<file_name> (<computer_name>)When you install OpenShift Container Platform, provide the SSH public key to the installation program.
The OpenShift Container Platform installation program requires that a Red Hat Enterprise Linux CoreOS (RHCOS) image be present in the Red Hat OpenStack Platform (RHOSP) cluster. Retrieve the latest RHCOS image, then upload it using the RHOSP CLI.
The RHOSP CLI is installed.
Log in to the Red Hat Customer Portal’s Product Downloads page.
Under Version, select the most recent release of OpenShift Container Platform 4.10 for Red Hat Enterprise Linux (RHEL) 8.
|
The RHCOS images might not change with every release of OpenShift Container Platform. You must download images with the highest version that is less than or equal to the OpenShift Container Platform version that you install. Use the image versions that match your OpenShift Container Platform version if they are available. |