apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
name: cluster
spec:
trustedCA:
name: ""
status:
After installing OpenShift Container Platform, you can further expand and customize your network to your requirements.
The configuration for the cluster network is specified as part of the Cluster Network Operator (CNO) configuration and stored in a custom resource (CR) object that is named cluster
. The CR specifies the fields for the Network
API in the operator.openshift.io
API group.
The CNO configuration inherits the following fields during cluster installation from the Network
API in the Network.config.openshift.io
API group and these fields cannot be changed:
clusterNetwork
IP address pools from which pod IP addresses are allocated.
serviceNetwork
IP address pool for services.
defaultNetwork.type
Cluster network provider, such as OpenShift SDN or OVN-Kubernetes.
After cluster installation, you cannot modify the fields listed in the previous section. |
The Proxy object is used to manage the cluster-wide egress proxy. When a cluster is
installed or upgraded without the proxy configured, a Proxy object is still
generated but it will have a nil spec
. For example:
apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
name: cluster
spec:
trustedCA:
name: ""
status:
A cluster administrator can configure the proxy for OpenShift Container Platform by modifying
this cluster
Proxy object.
Only the Proxy object named |
Cluster administrator permissions
OpenShift Container Platform oc
CLI tool installed
Create a ConfigMap that contains any additional CA certificates required for proxying HTTPS connections.
You can skip this step if the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle. |
Create a file called user-ca-bundle.yaml
with the following contents, and provide the values of your PEM-encoded certificates:
apiVersion: v1
data:
ca-bundle.crt: | (1)
<MY_PEM_ENCODED_CERTS> (2)
kind: ConfigMap
metadata:
name: user-ca-bundle (3)
namespace: openshift-config (4)
1 | This data key must be named ca-bundle.crt . |
2 | One or more PEM-encoded X.509 certificates used to sign the proxy’s identity certificate. |
3 | The ConfigMap name that will be referenced from the Proxy object. |
4 | The ConfigMap must be in the openshift-config namespace. |
Create the ConfigMap from this file:
$ oc create -f user-ca-bundle.yaml
Use the oc edit
command to modify the Proxy object:
$ oc edit proxy/cluster
Configure the necessary fields for the proxy:
apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
name: cluster
spec:
httpProxy: http://<username>:<pswd>@<ip>:<port> (1)
httpsProxy: http://<username>:<pswd>@<ip>:<port> (2)
noProxy: example.com (3)
readinessEndpoints:
- http://www.google.com (4)
- https://www.google.com
trustedCA:
name: user-ca-bundle (5)
1 | A proxy URL to use for creating HTTP connections outside the cluster. The
URL scheme must be http . |
2 | A proxy URL to use for creating HTTPS connections outside the cluster. If
this is not specified, then httpProxy is used for both HTTP and HTTPS
connections. |
3 | A comma-separated list of destination domain names, domains, IP addresses or
other network CIDRs to exclude proxying.
Preface a domain with This field is ignored if neither the |
4 | One or more URLs external to the cluster to use to perform a readiness check
before writing the httpProxy and httpsProxy values to status. |
5 | A reference to the ConfigMap in the openshift-config namespace that
contains additional CA certificates required for proxying HTTPS connections.
Note that the ConfigMap must already exist before referencing it here. This
field is required unless the proxy’s identity certificate is signed by an
authority from the RHCOS trust bundle. |
Save the file to apply the changes.
The URL scheme must be |
After you deploy a cluster, you can modify its DNS to use only a private zone.
Review the DNS
custom resource for your cluster:
$ oc get dnses.config.openshift.io/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: DNS
metadata:
creationTimestamp: "2019-10-25T18:27:09Z"
generation: 2
name: cluster
resourceVersion: "37966"
selfLink: /apis/config.openshift.io/v1/dnses/cluster
uid: 0e714746-f755-11f9-9cb1-02ff55d8f976
spec:
baseDomain: <base_domain>
privateZone:
tags:
Name: <infrastructureID>-int
kubernetes.io/cluster/<infrastructureID>: owned
publicZone:
id: Z2XXXXXXXXXXA4
status: {}
Note that the spec
section contains both a private and a public zone.
Patch the DNS
custom resource to remove the public zone:
$ oc patch dnses.config.openshift.io/cluster --type=merge --patch='{"spec": {"publicZone": null}}'
dns.config.openshift.io/cluster patched
Because the Ingress Controller consults the DNS
definition when it creates Ingress
objects, when you create or modify Ingress
objects, only private records are created.
DNS records for the existing Ingress objects are not modified when you remove the public zone. |
Optional: Review the DNS
custom resource for your cluster and confirm that the public zone was removed:
$ oc get dnses.config.openshift.io/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: DNS
metadata:
creationTimestamp: "2019-10-25T18:27:09Z"
generation: 2
name: cluster
resourceVersion: "37966"
selfLink: /apis/config.openshift.io/v1/dnses/cluster
uid: 0e714746-f755-11f9-9cb1-02ff55d8f976
spec:
baseDomain: <base_domain>
privateZone:
tags:
Name: <infrastructureID>-int
kubernetes.io/cluster/<infrastructureID>-wfpg4: owned
status: {}
OpenShift Container Platform provides the following methods for communicating from outside the cluster with services running in the cluster:
If you have HTTP/HTTPS, use an Ingress Controller.
If you have a TLS-encrypted protocol other than HTTPS, such as TLS with the SNI header, use an Ingress Controller.
Otherwise, use a load balancer, an external IP, or a node port.
Method | Purpose |
---|---|
Allows access to HTTP/HTTPS traffic and TLS-encrypted protocols other than HTTPS, such as TLS with the SNI header. |
|
Automatically assign an external IP by using a load balancer service |
Allows traffic to non-standard ports through an IP address assigned from a pool. |
Allows traffic to non-standard ports through a specific IP address. |
|
Expose a service on all nodes in the cluster. |
As a cluster administrator, you can expand the available node port range. If your cluster uses of a large number of node ports, you might need to increase the number of available ports.
The default port range is 30000-32767
. You can never reduce the port range, even if you first expand it beyond the default range.
Your cluster infrastructure must allow access to the ports that you specify within the expanded range. For example, if you expand the node port range to 30000-32900
, the inclusive port range of 32768-32900
must be allowed by your firewall or packet filtering configuration.
You can expand the node port range for the cluster.
Install the OpenShift CLI (oc
).
Log in to the cluster with a user with cluster-admin
privileges.
To expand the node port range, enter the following command. Replace <port>
with the largest port number in the new range.
$ oc patch network.config.openshift.io cluster --type=merge -p \
'{
"spec":
{ "serviceNodePortRange": "30000-<port>" }
}'
network.config.openshift.io/cluster patched
To confirm that the configuration is active, enter the following command. It can take several minutes for the update to apply.
$ oc get configmaps -n openshift-kube-apiserver config \
-o jsonpath="{.data['config\.yaml']}" | \
grep -Eo '"service-node-port-range":["[[:digit:]]+-[[:digit:]]+"]'
"service-node-port-range":["30000-33000"]
As a cluster administrator or project administrator, you can configure network policies for a project.
In a cluster using a Kubernetes Container Network Interface (CNI) plug-in that supports Kubernetes network policy, network isolation is controlled entirely by NetworkPolicy
objects.
In OpenShift Container Platform 4.7, OpenShift SDN supports using network policy in its default network isolation mode.
IPBlock is supported by network policy with limitations for OpenShift SDN; it supports IPBlock without except clauses. If you create a policy with an IPBlock section that includes an except clause, the SDN pods log warnings and the entire IPBlock section of that policy is ignored. |
Network policy does not apply to the host network namespace. Pods with host networking enabled are unaffected by network policy rules. |
By default, all pods in a project are accessible from other pods and network endpoints. To isolate one or more pods in a project, you can create NetworkPolicy
objects in that project to indicate the allowed incoming connections. Project administrators can create and delete NetworkPolicy
objects within their own project.
If a pod is matched by selectors in one or more NetworkPolicy
objects, then the pod will accept only connections that are allowed by at least one of those NetworkPolicy
objects. A pod that is not selected by any NetworkPolicy
objects is fully accessible.
The following example NetworkPolicy
objects demonstrate supporting different scenarios:
Deny all traffic:
To make a project deny by default, add a NetworkPolicy
object that matches all pods but accepts no traffic:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: deny-by-default
spec:
podSelector:
ingress: []
Only allow connections from the OpenShift Container Platform Ingress Controller:
To make a project allow only connections from the OpenShift Container Platform Ingress Controller, add the following NetworkPolicy
object.
For the OVN-Kubernetes network provider plug-in, when the Ingress Controller is configured to use the |
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-openshift-ingress
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
network.openshift.io/policy-group: ingress
podSelector: {}
policyTypes:
- Ingress
If the Ingress Controller is configured with endpointPublishingStrategy: HostNetwork
, then the Ingress Controller pod runs on the host network.
When running on the host network, the traffic from the Ingress Controller is assigned the netid:0
Virtual Network ID (VNID).
The netid
for the namespace that is associated with the Ingress Operator is different, so the matchLabel
in the allow-from-openshift-ingress
network policy does not match traffic from the default
Ingress Controller.
With OpenShift SDN, the default
namespace is assigned the netid:0
VNID and you can allow traffic from the default
Ingress Controller by labeling your default
namespace with network.openshift.io/policy-group: ingress
.
Only accept connections from pods within a project:
To make pods accept connections from other pods in the same project, but reject all other connections from pods in other projects, add the following NetworkPolicy
object:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-same-namespace
spec:
podSelector:
ingress:
- from:
- podSelector: {}
Only allow HTTP and HTTPS traffic based on pod labels:
To enable only HTTP and HTTPS access to the pods with a specific label (role=frontend
in following example), add a NetworkPolicy
object similar to the following:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-http-and-https
spec:
podSelector:
matchLabels:
role: frontend
ingress:
- ports:
- protocol: TCP
port: 80
- protocol: TCP
port: 443
Accept connections by using both namespace and pod selectors:
To match network traffic by combining namespace and pod selectors, you can use a NetworkPolicy
object similar to the following:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-pod-and-namespace-both
spec:
podSelector:
matchLabels:
name: test-pods
ingress:
- from:
- namespaceSelector:
matchLabels:
project: project_name
podSelector:
matchLabels:
name: test-pods
NetworkPolicy
objects are additive, which means you can combine multiple NetworkPolicy
objects together to satisfy complex network requirements.
For example, for the NetworkPolicy
objects defined in previous samples, you can define both allow-same-namespace
and allow-http-and-https
policies within the same project. Thus allowing the pods with the label role=frontend
, to accept any connection allowed by each policy. That is, connections on any port from pods in the same namespace, and connections on ports 80
and 443
from pods in any namespace.
The following annotates an example NetworkPolicy object:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-27107 (1)
spec:
podSelector: (2)
matchLabels:
app: mongodb
ingress:
- from:
- podSelector: (3)
matchLabels:
app: app
ports: (4)
- protocol: TCP
port: 27017
1 | The name of the NetworkPolicy object. |
2 | A selector describing the pods the policy applies to. The policy object can only select pods in the project that the NetworkPolicy object is defined. |
3 | A selector matching the pods that the policy object allows ingress traffic from. The selector will match pods in any project. |
4 | A list of one or more destination ports to accept traffic on. |
To define granular rules describing ingress network traffic allowed for projects in your cluster, you can create a network policy.
Your cluster is using a default CNI network provider that supports NetworkPolicy
objects, such as the OpenShift SDN network provider with mode: NetworkPolicy
set. This mode is the default for OpenShift SDN.
You installed the OpenShift CLI (oc
).
You are logged in to the cluster with a user with cluster-admin
privileges.
Create a policy rule:
Create a <policy-name>.yaml
file where <policy-name>
describes the policy
rule.
In the file you just created define a policy object, such as in the following example:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: <policy-name> (1)
spec:
podSelector:
ingress: []
1 | Specify a name for the policy object. |
Run the following command to create the policy object:
$ oc create -f <policy-name>.yaml -n <project>
In the following example, a new NetworkPolicy
object is created in a project
named project1
:
$ oc create -f default-deny.yaml -n project1
networkpolicy "default-deny" created
You can configure your project to isolate it from pods and services in other project namespaces.
Your cluster is using a cluster network provider that supports NetworkPolicy
objects, such as
the OVN-Kubernetes network provider or the OpenShift SDN network provider with mode: NetworkPolicy
set.
This mode is the default for OpenShift SDN.
You installed the OpenShift CLI (oc
).
You are logged in to the cluster with a user with cluster-admin
privileges.
Create the following NetworkPolicy
objects:
A policy named allow-from-openshift-ingress
.
For the OVN-Kubernetes network provider plug-in, when the Ingress Controller is configured to use the |
$ cat << EOF| oc create -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-openshift-ingress
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
network.openshift.io/policy-group: ingress
podSelector: {}
policyTypes:
- Ingress
EOF
A policy named allow-from-openshift-monitoring
:
$ cat << EOF| oc create -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-openshift-monitoring
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
network.openshift.io/policy-group: monitoring
podSelector: {}
policyTypes:
- Ingress
EOF
A policy named allow-same-namespace
:
$ cat << EOF| oc create -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-same-namespace
spec:
podSelector:
ingress:
- from:
- podSelector: {}
EOF
If the default
Ingress Controller configuration has the spec.endpointPublishingStrategy: HostNetwork
value set, you must apply a label to the default
OpenShift Container Platform namespace to allow network traffic between the Ingress Controller and the project:
Determine if your default
Ingress Controller uses the HostNetwork
endpoint publishing strategy:
$ oc get --namespace openshift-ingress-operator ingresscontrollers/default \
--output jsonpath='{.status.endpointPublishingStrategy.type}'
If the previous command reports the endpoint publishing strategy as HostNetwork
, set a label on the default
namespace:
$ oc label namespace default 'network.openshift.io/policy-group=ingress'
Confirm that the NetworkPolicy
object exists in your current project
by running the following command:
$ oc get networkpolicy <policy-name> -o yaml
In the following example, the allow-from-openshift-ingress
NetworkPolicy
object is displayed:
$ oc get -n project1 networkpolicy allow-from-openshift-ingress -o yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-openshift-ingress
namespace: project1
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
network.openshift.io/policy-group: ingress
podSelector: {}
policyTypes:
- Ingress
As a cluster administrator, you can modify the new project template to
automatically include NetworkPolicy
objects when you create a new project.
As a cluster administrator, you can modify the default project template so that new projects are created using your custom requirements.
To create your own custom project template:
Log in as a user with cluster-admin
privileges.
Generate the default project template:
$ oc adm create-bootstrap-project-template -o yaml > template.yaml
Use a text editor to modify the generated template.yaml
file by adding
objects or modifying existing objects.
The project template must be created in the openshift-config
namespace. Load
your modified template:
$ oc create -f template.yaml -n openshift-config
Edit the project configuration resource using the web console or CLI.
Using the web console:
Navigate to the Administration → Cluster Settings page.
Click Global Configuration to view all configuration resources.
Find the entry for Project and click Edit YAML.
Using the CLI:
Edit the project.config.openshift.io/cluster
resource:
$ oc edit project.config.openshift.io/cluster
Update the spec
section to include the projectRequestTemplate
and name
parameters, and set the name of your uploaded project template. The default name
is project-request
.
apiVersion: config.openshift.io/v1
kind: Project
metadata:
...
spec:
projectRequestTemplate:
name: <template_name>
After you save your changes, create a new project to verify that your changes were successfully applied.
As a cluster administrator, you can add network policies to the default template for new projects.
OpenShift Container Platform will automatically create all the NetworkPolicy
objects specified in the template in the project.
Your cluster is using a default CNI network provider that supports NetworkPolicy
objects, such as the OpenShift SDN network provider with mode: NetworkPolicy
set. This mode is the default for OpenShift SDN.
You installed the OpenShift CLI (oc
).
You must log in to the cluster with a user with cluster-admin
privileges.
You must have created a custom default project template for new projects.
Edit the default template for a new project by running the following command:
$ oc edit template <project_template> -n openshift-config
Replace <project_template>
with the name of the default template that you
configured for your cluster. The default template name is project-request
.
In the template, add each NetworkPolicy
object as an element to the objects
parameter. The objects
parameter accepts a collection of one or more objects.
In the following example, the objects
parameter collection includes several NetworkPolicy
objects.
For the OVN-Kubernetes network provider plug-in, when the Ingress Controller is configured to use the |
objects:
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-same-namespace
spec:
podSelector:
ingress:
- from:
- podSelector: {}
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-openshift-ingress
spec:
ingress:
- from:
- namespaceSelector:
matchLabels:
network.openshift.io/policy-group: ingress
podSelector: {}
policyTypes:
- Ingress
...
Optional: Create a new project to confirm that your network policy objects are created successfully by running the following commands:
Create a new project:
$ oc new-project <project> (1)
1 | Replace <project> with the name for the project you are creating. |
Confirm that the network policy objects in the new project template exist in the new project:
$ oc get networkpolicy
NAME POD-SELECTOR AGE
allow-from-openshift-ingress <none> 7s
allow-from-same-namespace <none> 7s
The following are the only supported configurations for the Red Hat OpenShift Service Mesh:
Red Hat OpenShift Container Platform version 4.x.
OpenShift Online and OpenShift Dedicated are not supported for Red Hat OpenShift Service Mesh. |
The deployment must be contained to a single OpenShift Container Platform cluster that is not federated.
This release of Red Hat OpenShift Service Mesh is only available on OpenShift Container Platform x86_64, IBM Z, and IBM Power Systems.
IBM Z is only supported on OpenShift Container Platform 4.6 and later.
IBM Power Systems is only supported on OpenShift Container Platform 4.6 and later.
This release only supports configurations where all Service Mesh components are contained in the OpenShift cluster in which it operates. It does not support management of microservices that reside outside of the cluster, or in a multi-cluster scenario.
This release only supports configurations that do not integrate external services such as virtual machines.
For additional information about Red Hat OpenShift Service Mesh lifecycle and supported configurations, refer to the Support Policy.
Red Hat OpenShift Service Mesh supports the following network configurations.
Openshift-SDN
OVN-Kubernetes is supported as a technology preview in OpenShift Container Platform version 4.7.
The Kiali observability console is only supported on the two most recent releases of the Chrome, Edge, Firefox, or Safari browsers.
This release only supports the following Mixer adapter:
3scale Istio Adapter
To install the Red Hat OpenShift Service Mesh Operator, you must first install these Operators:
Elasticsearch - Based on the open source Elasticsearch project that enables you to configure and manage an Elasticsearch cluster for tracing and logging with Jaeger.
Jaeger - based on the open source Jaeger project, lets you perform tracing to monitor and troubleshoot transactions in complex distributed systems.
Kiali - based on the open source Kiali project, provides observability for your service mesh. By using Kiali you can view configurations, monitor traffic, and view and analyze traces in a single console.
After you install the OpenShift Elasticsearch, Jaeger, and Kiali Operators, then you install the Red Hat OpenShift Service Mesh Operator. The Service Mesh Operator defines and monitors the ServiceMeshControlPlane
resources that manage the deployment, updating, and deletion of the Service Mesh components.
Red Hat OpenShift Service Mesh - based on the open source Istio project, lets you connect, secure, control, and observe the microservices that make up your applications.
Install Red Hat OpenShift Service Mesh in your OpenShift Container Platform environment.
The OpenShift Container Platform HAProxy router scales to optimize performance.
The OpenShift Container Platform Ingress Controller, or router, is the Ingress point for all external traffic destined for OpenShift Container Platform services.
When evaluating a single HAProxy router performance in terms of HTTP requests handled per second, the performance varies depending on many factors. In particular:
HTTP keep-alive/close mode
Route type
TLS session resumption client support
Number of concurrent connections per target route
Number of target routes
Back end server page size
Underlying infrastructure (network/SDN solution, CPU, and so on)
While performance in your specific environment will vary, Red Hat lab tests on a public cloud instance of size 4 vCPU/16GB RAM. A single HAProxy router handling 100 routes terminated by backends serving 1kB static pages is able to handle the following number of transactions per second.
In HTTP keep-alive mode scenarios:
Encryption | LoadBalancerService | HostNetwork |
---|---|---|
none |
21515 |
29622 |
edge |
16743 |
22913 |
passthrough |
36786 |
53295 |
re-encrypt |
21583 |
25198 |
In HTTP close (no keep-alive) scenarios:
Encryption | LoadBalancerService | HostNetwork |
---|---|---|
none |
5719 |
8273 |
edge |
2729 |
4069 |
passthrough |
4121 |
5344 |
re-encrypt |
2320 |
2941 |
Default Ingress Controller configuration with ROUTER_THREADS=4
was used and two different endpoint publishing strategies (LoadBalancerService/HostNetwork) were tested.
TLS session resumption was used for encrypted routes. With HTTP keep-alive, a single HAProxy router is capable of saturating 1 Gbit NIC at page sizes as small as 8 kB.
When running on bare metal with modern processors, you can expect roughly twice the performance of the public cloud instance above. This overhead is introduced by the virtualization layer in place on public clouds and holds mostly true for private cloud-based virtualization as well. The following table is a guide to how many applications to use behind the router:
Number of applications | Application type |
---|---|
5-10 |
static file/web server or caching proxy |
100-1000 |
applications generating dynamic content |
In general, HAProxy can support routes for 5 to 1000 applications, depending on the technology in use. Ingress Controller performance might be limited by the capabilities and performance of the applications behind it, such as language or static versus dynamic content.
Ingress, or router, sharding should be used to serve more routes towards applications and help horizontally scale the routing tier.
OpenShift Container Platform no longer supports modifying Ingress Controller deployments by setting environment variables such as ROUTER_THREADS
, ROUTER_DEFAULT_TUNNEL_TIMEOUT
, ROUTER_DEFAULT_CLIENT_TIMEOUT
, ROUTER_DEFAULT_SERVER_TIMEOUT
, and RELOAD_INTERVAL
.
You can modify the Ingress Controller deployment, but if the Ingress Operator is enabled, the configuration is overwritten.
You can configure some aspects of a OpenShift Container Platform on Red Hat OpenStack Platform (RHOSP) cluster after installation.
After you install OpenShift Container Platform, configure Red Hat OpenStack Platform (RHOSP) to allow application network traffic.
You do not need to perform this procedure if you provided values for |
OpenShift Container Platform cluster must be installed
Floating IP addresses are enabled as described in the OpenShift Container Platform on RHOSP installation documentation.
After you install the OpenShift Container Platform cluster, attach a floating IP address to the ingress port:
Show the port:
$ openstack port show <cluster_name>-<cluster_ID>-ingress-port
Attach the port to the IP address:
$ openstack floating ip set --port <ingress_port_ID> <apps_FIP>
Add a wildcard A
record for *apps.
to your DNS file:
*.apps.<cluster_name>.<base_domain> IN A <apps_FIP>
If you do not control the DNS server but want to enable application access for non-production purposes, you can add these hostnames to
|