$ sudo openstack quota set --secgroups 250 --secgroup-rules 1000 --ports 1500 --subnets 250 --networks 250 <project>
install-config.yaml
file for RHOSP with KuryrIn OpenShift Container Platform version 4.10, you can install a cluster on Red Hat OpenStack Platform (RHOSP) that runs on user-provisioned infrastructure.
Using your own infrastructure allows you to integrate your cluster with existing infrastructure and modifications. The process requires more labor on your part than installer-provisioned installations, because you must create all RHOSP resources, like Nova servers, Neutron ports, and security groups. However, Red Hat provides Ansible playbooks to help you in the deployment process.
You reviewed details about the OpenShift Container Platform installation and update processes.
You read the documentation on selecting a cluster installation method and preparing it for users.
You verified that OpenShift Container Platform 4.10 is compatible with your RHOSP version by using the Supported platforms for OpenShift clusters section. You can also compare platform support across different versions by viewing the OpenShift Container Platform on RHOSP support matrix.
You have an RHOSP account where you want to install OpenShift Container Platform.
On the machine from which you run the installation program, you have:
A single directory in which you can keep the files you create during the installation process
Python 3
Kuryr is a container network interface (CNI) plugin solution that uses the Neutron and Octavia Red Hat OpenStack Platform (RHOSP) services to provide networking for pods and Services.
Kuryr and OpenShift Container Platform integration is primarily designed for OpenShift Container Platform clusters running on RHOSP VMs. Kuryr improves the network performance by plugging OpenShift Container Platform pods into RHOSP SDN. In addition, it provides interconnectivity between pods and RHOSP virtual instances.
Kuryr components are installed as pods in OpenShift Container Platform using the
openshift-kuryr
namespace:
kuryr-controller
- a single service instance installed on a master
node.
This is modeled in OpenShift Container Platform as a Deployment
object.
kuryr-cni
- a container installing and configuring Kuryr as a CNI driver on
each OpenShift Container Platform node. This is modeled in OpenShift Container Platform as a DaemonSet
object.
The Kuryr controller watches the OpenShift Container Platform API server for pod, service, and namespace create, update, and delete events. It maps the OpenShift Container Platform API calls to corresponding objects in Neutron and Octavia. This means that every network solution that implements the Neutron trunk port functionality can be used to back OpenShift Container Platform via Kuryr. This includes open source solutions such as Open vSwitch (OVS) and Open Virtual Network (OVN) as well as Neutron-compatible commercial SDNs.
Kuryr is recommended for OpenShift Container Platform deployments on encapsulated RHOSP tenant networks to avoid double encapsulation, such as running an encapsulated OpenShift Container Platform SDN over an RHOSP network.
If you use provider networks or tenant VLANs, you do not need to use Kuryr to avoid double encapsulation. The performance benefit is negligible. Depending on your configuration, though, using Kuryr to avoid having two overlays might still be beneficial.
Kuryr is not recommended in deployments where all of the following criteria are true:
The RHOSP version is less than 16.
The deployment uses UDP services, or a large number of TCP services on few hypervisors.
or
The ovn-octavia
Octavia driver is disabled.
The deployment uses a large number of TCP services on few hypervisors.
When using Kuryr SDN, the pods, services, namespaces, and network policies are using resources from the RHOSP quota; this increases the minimum requirements. Kuryr also has some additional requirements on top of what a default install requires.
Use the following quota to satisfy a default cluster’s minimum requirements:
Resource | Value |
---|---|
Floating IP addresses |
3 - plus the expected number of Services of LoadBalancer type |
Ports |
1500 - 1 needed per Pod |
Routers |
1 |
Subnets |
250 - 1 needed per Namespace/Project |
Networks |
250 - 1 needed per Namespace/Project |
RAM |
112 GB |
vCPUs |
28 |
Volume storage |
275 GB |
Instances |
7 |
Security groups |
250 - 1 needed per Service and per NetworkPolicy |
Security group rules |
1000 |
Server groups |
2 - plus 1 for each additional availability zone in each machine pool |
Load balancers |
100 - 1 needed per Service |
Load balancer listeners |
500 - 1 needed per Service-exposed port |
Load balancer pools |
500 - 1 needed per Service-exposed port |
A cluster might function with fewer than recommended resources, but its performance is not guaranteed.
If RHOSP object storage (Swift) is available and operated by a user account with the |
If you are using Red Hat OpenStack Platform (RHOSP) version 16 with the Amphora driver rather than the OVN Octavia driver, security groups are associated with service accounts instead of user projects. |
Take the following notes into consideration when setting resources:
The number of ports that are required is larger than the number of pods. Kuryr uses ports pools to have pre-created ports ready to be used by pods and speed up the pods' booting time.
Each network policy is mapped into an RHOSP security group, and
depending on the NetworkPolicy
spec, one or more rules are added to the
security group.
Each service is mapped to an RHOSP load balancer. Consider this requirement when estimating the number of security groups required for the quota.
If you are using
RHOSP version 15 or earlier, or the ovn-octavia driver
, each load balancer
has a security group with the user project.
The quota does not account for load balancer resources (such as VM resources), but you must consider these resources when you decide the RHOSP deployment’s size. The default installation will have more than 50 load balancers; the clusters must be able to accommodate them.
If you are using RHOSP version 16 with the OVN Octavia driver enabled, only one load balancer VM is generated; services are load balanced through OVN flows.
An OpenShift Container Platform deployment comprises control plane machines, compute machines, and a bootstrap machine.
To enable Kuryr SDN, your environment must meet the following requirements:
Run RHOSP 13+.
Have Overcloud with Octavia.
Use Neutron Trunk ports extension.
Use openvswitch
firewall driver if ML2/OVS Neutron driver is used instead
of ovs-hybrid
.
When using Kuryr SDN, you must increase quotas to satisfy the Red Hat OpenStack Platform (RHOSP) resources used by pods, services, namespaces, and network policies.
Increase the quotas for a project by running the following command:
$ sudo openstack quota set --secgroups 250 --secgroup-rules 1000 --ports 1500 --subnets 250 --networks 250 <project>
Kuryr CNI leverages the Neutron Trunks extension to plug containers into the
Red Hat OpenStack Platform (RHOSP) SDN, so you must use the trunks
extension for Kuryr to properly work.
In addition, if you leverage the default ML2/OVS Neutron driver, the firewall
must be set to openvswitch
instead of ovs_hybrid
so that security groups are
enforced on trunk subports and Kuryr can properly handle network policies.
Kuryr SDN uses Red Hat OpenStack Platform (RHOSP)'s Octavia LBaaS to implement OpenShift Container Platform services. Thus, you must install and configure Octavia components in RHOSP to use Kuryr SDN.
To enable Octavia, you must include the Octavia service during the installation of the RHOSP Overcloud, or upgrade the Octavia service if the Overcloud already exists. The following steps for enabling Octavia apply to both a clean install of the Overcloud or an Overcloud update.
The following steps only capture the key pieces required during the deployment of RHOSP when dealing with Octavia. It is also important to note that registry methods vary. This example uses the local registry method. |
If you are using the local registry, create a template to upload the images to the registry. For example:
(undercloud) $ openstack overcloud container image prepare \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
--namespace=registry.access.redhat.com/rhosp13 \
--push-destination=<local-ip-from-undercloud.conf>:8787 \
--prefix=openstack- \
--tag-from-label {version}-{product-version} \
--output-env-file=/home/stack/templates/overcloud_images.yaml \
--output-images-file /home/stack/local_registry_images.yaml
Verify that the local_registry_images.yaml
file contains the Octavia images.
For example:
...
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-43
push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-health-manager:13.0-45
push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-housekeeping:13.0-45
push_destination: <local-ip-from-undercloud.conf>:8787
- imagename: registry.access.redhat.com/rhosp13/openstack-octavia-worker:13.0-44
push_destination: <local-ip-from-undercloud.conf>:8787
The Octavia container versions vary depending upon the specific RHOSP release installed. |
Pull the container images from registry.redhat.io
to the Undercloud node:
(undercloud) $ sudo openstack overcloud container image upload \
--config-file /home/stack/local_registry_images.yaml \
--verbose
This may take some time depending on the speed of your network and Undercloud disk.
Since an Octavia load balancer is used to access the OpenShift Container Platform API, you must increase their listeners' default timeouts for the connections. The default timeout is 50 seconds. Increase the timeout to 20 minutes by passing the following file to the Overcloud deploy command:
(undercloud) $ cat octavia_timeouts.yaml
parameter_defaults:
OctaviaTimeoutClientData: 1200000
OctaviaTimeoutMemberData: 1200000
This is not needed for RHOSP 13.0.13+. |
Install or update your Overcloud environment with Octavia:
$ openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
-e octavia_timeouts.yaml
This command only includes the files associated with Octavia; it varies based on your specific installation of RHOSP. See the RHOSP documentation for further information. For more information on customizing your Octavia installation, see installation of Octavia using Director. |
When leveraging Kuryr SDN, the Overcloud installation requires the Neutron |
In RHOSP versions earlier than 13.0.13, add the project ID
to the octavia.conf
configuration file after you create the project.
To enforce network policies across services, like when traffic goes through the Octavia load balancer, you must ensure Octavia creates the Amphora VM security groups on the user project.
This change ensures that required load balancer security groups belong to that project, and that they can be updated to enforce services isolation.
This task is unnecessary in RHOSP version 13.0.13 or later. Octavia implements a new ACL API that restricts access to the load balancers VIP. |
Get the project ID
$ openstack project show <project>
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | |
| domain_id | default |
| enabled | True |
| id | PROJECT_ID |
| is_domain | False |
| name | *<project>* |
| parent_id | default |
| tags | [] |
+-------------+----------------------------------+
Add the project ID to octavia.conf
for the controllers.
Source the stackrc
file:
$ source stackrc # Undercloud credentials
List the Overcloud controllers:
$ openstack server list
+--------------------------------------+--------------+--------+-----------------------+----------------+------------+
│
| ID | Name | Status | Networks
| Image | Flavor |
│
+--------------------------------------+--------------+--------+-----------------------+----------------+------------+
│
| 6bef8e73-2ba5-4860-a0b1-3937f8ca7e01 | controller-0 | ACTIVE |
ctlplane=192.168.24.8 | overcloud-full | controller |
│
| dda3173a-ab26-47f8-a2dc-8473b4a67ab9 | compute-0 | ACTIVE |
ctlplane=192.168.24.6 | overcloud-full | compute |
│
+--------------------------------------+--------------+--------+-----------------------+----------------+------------+
SSH into the controller(s).
$ ssh heat-admin@192.168.24.8
Edit the octavia.conf
file to add the project into the list of projects where
Amphora security groups are on the user’s account.
# List of project IDs that are allowed to have Load balancer security groups # belonging to them. amp_secgroup_allowed_projects = PROJECT_ID
Restart the Octavia worker so the new configuration loads.
controller-0$ sudo docker restart octavia_worker
Depending on your RHOSP environment, Octavia might not support UDP listeners. If you use Kuryr SDN on RHOSP version 13.0.13 or earlier, UDP services are not supported. RHOSP version 16 or later support UDP. |
Octavia supports multiple provider drivers through the Octavia API.
To see all available Octavia provider drivers, on a command line, enter:
$ openstack loadbalancer provider list
+---------+-------------------------------------------------+
| name | description |
+---------+-------------------------------------------------+
| amphora | The Octavia Amphora driver. |
| octavia | Deprecated alias of the Octavia Amphora driver. |
| ovn | Octavia OVN driver. |
+---------+-------------------------------------------------+
Beginning with RHOSP version 16, the Octavia OVN provider driver (ovn
) is supported on
OpenShift Container Platform on RHOSP deployments.
ovn
is an integration driver for the load balancing
that Octavia and OVN provide. It supports basic load balancing capabilities,
and is based on OpenFlow rules. The driver is automatically enabled
in Octavia by Director on deployments that use OVN Neutron ML2.
The Amphora provider driver is the default driver. If ovn
is enabled, however, Kuryr uses it.
If Kuryr uses ovn
instead of Amphora, it offers the following benefits:
Decreased resource requirements. Kuryr does not require a load balancer VM for each service.
Reduced network latency.
Increased service creation speed by using OpenFlow rules instead of a VM for each service.
Distributed load balancing actions across all nodes instead of centralized on Amphora VMs.
Using OpenShift Container Platform with Kuryr SDN has several known limitations.
Using OpenShift Container Platform with Kuryr SDN has several limitations that apply to all versions and environments:
Service
objects with the NodePort
type are not supported.
Clusters that use the OVN Octavia provider driver support Service
objects for which the .spec.selector
property is unspecified only if the .subsets.addresses
property of the Endpoints
object includes the subnet of the nodes or pods.
If the subnet on which machines are created is not connected to a router, or if the subnet is connected, but the router has no external gateway set, Kuryr cannot create floating IPs for Service
objects with type LoadBalancer
.
Configuring the sessionAffinity=ClientIP
property on Service
objects does not have an effect. Kuryr does not support this setting.
Using OpenShift Container Platform with Kuryr SDN has several limitations that depend on the RHOSP version.
RHOSP versions before 16 use the default Octavia load balancer driver (Amphora). This driver requires that one Amphora load balancer VM is deployed per OpenShift Container Platform service. Creating too many services can cause you to run out of resources.
Deployments of later versions of RHOSP that have the OVN Octavia driver disabled also use the Amphora driver. They are subject to the same resource concerns as earlier versions of RHOSP.
Octavia RHOSP versions before 13.0.13 do not support UDP listeners. Therefore, OpenShift Container Platform UDP services are not supported.
Octavia RHOSP versions before 13.0.13 cannot listen to multiple protocols on the same port. Services that expose the same port to different protocols, like TCP and UDP, are not supported.
Kuryr SDN does not support automatic unidling by a service.
There are limitations when using Kuryr SDN that depend on your deployment environment.
Because of Octavia’s lack of support for the UDP protocol and multiple listeners, if the RHOSP version is earlier than 13.0.13, Kuryr forces pods to use TCP for DNS resolution.
In Go versions 1.12 and earlier, applications that are compiled with CGO support disabled use UDP only. In this case,
the native Go resolver does not recognize the use-vc
option in resolv.conf
, which controls whether TCP is forced for DNS resolution.
As a result, UDP is still used for DNS resolution, which fails.
To ensure that TCP forcing is allowed, compile applications either with the environment variable CGO_ENABLED
set to 1
, i.e. CGO_ENABLED=1
, or ensure that the variable is absent.
In Go versions 1.13 and later, TCP is used automatically if DNS resolution using UDP fails.
musl-based containers, including Alpine-based containers, do not support the |
As a result of the RHOSP upgrade process, the Octavia API might be changed, and upgrades to the Amphora images that are used for load balancers might be required.
You can address API changes on an individual basis.
If the Amphora image is upgraded, the RHOSP operator can handle existing load balancer VMs in two ways:
Upgrade each VM by triggering a load balancer failover.
Leave responsibility for upgrading the VMs to users.
If the operator takes the first option, there might be short downtimes during failovers.
If the operator takes the second option, the existing load balancers will not support upgraded Octavia API features, like UDP listeners. In this case, users must recreate their Services to use these features.
If OpenShift Container Platform detects a new Octavia version that supports UDP load balancing, it recreates the DNS service automatically. The service recreation ensures that the service default supports UDP load balancing. The recreation causes the DNS service approximately one minute of downtime. |
By default, the OpenShift Container Platform installation process creates three control plane machines.
Each machine requires:
An instance from the RHOSP quota
A port from the RHOSP quota
A flavor with at least 16 GB memory and 4 vCPUs
At least 100 GB storage space from the RHOSP quota
By default, the OpenShift Container Platform installation process creates three compute machines.
Each machine requires:
An instance from the RHOSP quota
A port from the RHOSP quota
A flavor with at least 8 GB memory and 2 vCPUs
At least 100 GB storage space from the RHOSP quota
Compute machines host the applications that you run on OpenShift Container Platform; aim to run as many as you can. |
During installation, a bootstrap machine is temporarily provisioned to stand up the control plane. After the production control plane is ready, the bootstrap machine is deprovisioned.
The bootstrap machine requires:
An instance from the RHOSP quota
A port from the RHOSP quota
A flavor with at least 16 GB memory and 4 vCPUs
At least 100 GB storage space from the RHOSP quota
In OpenShift Container Platform 4.10, you require access to the internet to install your cluster.
You must have internet access to:
Access OpenShift Cluster Manager to download the installation program and perform subscription management. If the cluster has internet access and you do not disable Telemetry, that service automatically entitles your cluster.
Access Quay.io to obtain the packages that are required to install your cluster.
Obtain the packages that are required to perform cluster updates.
If your cluster cannot have direct internet access, you can perform a restricted network installation on some types of infrastructure that you provision. During that process, you download the required content and use it to populate a mirror registry with the installation packages. With some installation types, the environment that you install your cluster in will not require internet access. Before you update the cluster, you update the content of the mirror registry. |
The Ansible playbooks that simplify the installation process on user-provisioned infrastructure require several Python modules. On the machine where you will run the installer, add the modules' repositories and then download them.
These instructions assume that you are using Red Hat Enterprise Linux (RHEL) 8. |
Python 3 is installed on your machine.
On a command line, add the repositories:
Register with Red Hat Subscription Manager:
$ sudo subscription-manager register # If not done already
Pull the latest subscription data:
$ sudo subscription-manager attach --pool=$YOUR_POOLID # If not done already
Disable the current repositories:
$ sudo subscription-manager repos --disable=* # If not done already
Add the required repositories:
$ sudo subscription-manager repos \
--enable=rhel-8-for-x86_64-baseos-rpms \
--enable=openstack-16-tools-for-rhel-8-x86_64-rpms \
--enable=ansible-2.9-for-rhel-8-x86_64-rpms \
--enable=rhel-8-for-x86_64-appstream-rpms
Install the modules:
$ sudo yum install python3-openstackclient ansible python3-openstacksdk python3-netaddr
Ensure that the python
command points to python3
:
$ sudo alternatives --set python /usr/bin/python3
Download Ansible playbooks that you can use to install OpenShift Container Platform on your own Red Hat OpenStack Platform (RHOSP) infrastructure.
The curl command-line tool is available on your machine.
To download the playbooks to your working directory, run the following script from a command line:
$ xargs -n 1 curl -O <<< '
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/bootstrap.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/common.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/compute-nodes.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/control-plane.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/inventory.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/network.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/security-groups.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-bootstrap.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-compute-nodes.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-control-plane.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-load-balancers.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-network.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-security-groups.yaml
https://raw.githubusercontent.com/openshift/installer/release-4.10/upi/openstack/down-containers.yaml'
The playbooks are downloaded to your machine.
During the installation process, you can modify the playbooks to configure your deployment. Retain all playbooks for the life of your cluster. You must have the playbooks to remove your OpenShift Container Platform cluster from RHOSP. |
You must match any edits you make in the |