OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history
×

Getting Traffic into the Cluster

Overview

There are many ways to access the cluster. This section describes some commonly used approaches.

The recommendation is:

  • If you have HTTP/HTTPS, use the router.

  • If you have a TLS-encrypted protocol other than HTTPS (for example, TLS with the SNI header), use the router.

  • Otherwise, use Load Balancer, ExternalIP, or NodePort.

TCP or UDP offers several approaches:

  • Use the non-cloud Load Balancer. This limits you to a single edge router IP (which can be a virtual IP (VIP), but still is a single machine for initial load balancing). It simplifies the administrator’s job, but uses one IP per service.

  • Manually assign ExternalIPs to the service. You can assign a set of IPs, so you can have multiple machines for the incoming load balancing. However, this requires elevated permissions to assign, and manual tracking of what IP:ports that are used.

  • Use NodePorts to expose the service on all nodes in the cluster. This is more wasteful of scarce port resources. However, it is slightly easier to set up multiple. Again, this requires more privileges.

The router is the most common way to access the cluster. This is limited to HTTP/HTTPS(SNI)/TLS(SNI), which covers web applications.

ExternalIP or NodePort is useful when the HTTP protocol is not being used or non-standard ports are in use. There is more manual setup and monitoring involved.

The administrator must set up the external port to the cluster networking environment so that requests can reach the cluster. For example, names can be configured into DNS to point to specific nodes or other IP addresses in the cluster. The DNS wildcard feature can be used to configure a subset of names to an IP address in the cluster. This is convenient when using routers because it allows the users to set up routes within the cluster without further administrator attention.

The administrator must ensure that the local firewall on each node permits the request to reach the IP address.

Using a Router

This is the most common way to access the cluster. A router is configured to accept external requests and proxy them based on the configured routes. This is limited to HTTP/HTTPS(SNI)/TLS(SNI), which covers web applications.

An administrator can create a wildcard DNS entry, and then set up a router. Afterward, the users can self-service the edge router without having to contact the administrators. The router has controls to allow the administrator to specify whether the users can self-provision host names, or if they must fit a pattern the administrator defines. The other solutions require the administrator to do the provisioning, or they require that the administrator delegates a lot of privilege.

A set of routes can be created in the various projects. The overall set of routes is available to the set of routers. Each router selects from the set of routes. All routers see all routes unless restricted by labels on the router, which is called router sharding.

Using a Load Balancer Service

Load balancers are available on AWS and GCE clouds, and non-cloud options are also available.

The non-cloud load balancer allocates a unique IP from a configured pool. This limits you to a single edge router IP, which can be a VIP, but still will be a single machine for initial load balancing. The non-cloud load balancer simplifies the administrator’s job by providing the needed IP address, but uses one IP per service.

Using a Service ExternalIP

Administrators can assign a list of externalIPs, for which nodes in the cluster will also accept traffic for the service. These IPs are not managed by OpenShift Container Platform and administrators are responsible for ensuring that traffic arrives at a node with this IP. A common example of this is configuring a highly available service.

The supplied list of IP addresses is used for load balancing incoming requests. The service port is opened on the externalIPs on all nodes running kube-proxy.

ExternalIPs require elevated permissions to assign, and manual tracking of the IP:ports that are in use.

Traffic from hosts that are external to the cluster that is going to the external IP address ends up on a node in the cluster. When it arrives, the service that set up the external IP redirects it to a service endpoint that it is managing. The service load balances among its endpoints.

An externally visible IP for the service can be configured in several ways:

  • Manually configure the ExternalIP with a known external IP address.

  • Configure ExternalIP to a VIP that is generated from (VRRP).

  • In a cloud (AWS or GCE) use type=LoadBalancer

  • In a non-cloud environment, configure an ingressIP range (IngressIPNetworkCIDR), service.type=LoadBalancer and service.port.ingressIP.

The administrator must ensure the external IPs are routed to the nodes and local firewall rules on all nodes allow access to the open port.

Example External IP Configured within a Service Definition
$ oc create -f - <<INGRESS
apiVersion: v1
kind: Service
metadata:
  name: postgresql-ingress
spec:
  externalIPs:
  - 10.9.54.100 (1)
  ports:
  - port: 5432
    protocol: TCP
  selector:
    name: postgresql
  type: LoadBalancer
INGRESS
1 The IP address assigned to the external IP.

The external IP address is not managed by the underlying Kubernetes infrastructure and must be maintained and provided by a cluster administrator. While external IPs provide a solution for accessing services on the OpenShift Container Platform cluster, there are several shortcomings:

  • The cluster administrator user must ensure the external IP is not in any range that the cluster is configured to use. The user needs to work with the administrator that controls the network external to the cluster to assign the external IP address and ensure traffic to the IP can get to the nodes in the cluster. The cluster administrator must ensure firewall rules permit the packets to get into the nodes for the ports they want to expose.

  • There are no protections in place to restrict the usage of a particular external address configured within the cluster. This allows the potential for a single address to be used by multiple services targeting the same port. When this situation occurs, the service which requested the port first is given use of the port.

Ingress IP Self-Service provides a solution.

Using Ingress IP Self-Service

Ingress IP Self-Service streamlines the allocation of External IPs for accessing services in the cluster. Cluster administrators can designate a range of addresses using a CIDR notation, which allows an application user to make a request against the cluster for an external IP address. When a service is configured with the type LoadBalancer, an External IP address will be automatically assigned from the designated range.

Ingress IP Self-Service is only applicable in non-cloud based environments.

A common use case for Ingress IP Self-Service is the ability to provide database services, such as PostgreSQL, to clients outside of the OpenShift Container Platform cluster.

Defining the Edge Router IP Range

The ability for cluster administrators to automatically allocate External IP addresses using the edge router is enabled by default within OpenShift Container Platform and configured to use the 172.46.0.0/16 range. An alternate range can be specified by configuring the ingressIPNetworkCIDR parameter in the /etc/origin/master-config.yaml file:

networkConfig:
  ingressIPNetworkCIDR: 10.9.54.0/25

Restart the OpenShift Container Platform master service to apply the changes.

Deploy a Sample Application

To expose a PostgreSQL as a service for external consumption, the application must be first deployed into the cluster. Create a new project or use and existing project and instantiate one of the PostgreSQL templates.

The postgresql-ephemeral template does not make use of persistent storage. Once the application is scaled down or destroyed, any existing data will be lost. To use persistent storage, specify the postgresql-persistent template instead.

After instantiating the template, a ClusterIP-based service and DeploymentConfig is created and a new pod containing PostgreSQL will be started.

Configuring an IP Address for a Service

To allow the cluster to automatically assign an IP address for a service, create a service definition similar to the following that will create a new Ingress service:

$ oc create -f - <<INGRESS
apiVersion: v1
kind: Service
metadata:
  name: postgresql-ingress
spec:
  ports:
  - name: postgresql
    port: 5432
  type: LoadBalancer (1)
  selector:
    name: postgresql
INGRESS
1 The LoadBalancer type of service will make the request for an external service on behalf of the application user.

Alternatively, the oc expose command can be used to create the service:

$ oc expose dc postgresql --type=LoadBalancer --name=postgresql-ingress

Once the service is created, the external IP address is automatically allocated by the cluster and can be confirmed by running:

$ oc get svc postgresql-ingress
Example oc get Output
NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
postgresql-ingress    172.30.74.106   10.9.54.100,10.9.54.100    5432/TCP    30s

Specifying the type LoadBalancer also configures the service with a nodePort value. nodePort exposes the service port on all nodes in the cluster. Any packet that arrives on any node in the cluster targeting the nodePort ends up in the service. Then, it is load balanced to the service’s endpoints.

To discover the node port automatically assigned, run:

$ oc export svc postgresql-ingress
Example oc export Output
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: postgresql-persistent
    template: postgresql-persistent-template
  name: postgresql-ingress
spec:
  ports:
  - nodePort: 32439 (1)
    port: 5432
    protocol: TCP
    targetPort: 5432
  selector:
    name: postgresql
  sessionAffinity: None
  type: LoadBalancer
1 Automatically assigned port.

A PostgreSQL client can now be configured to connect directly to any node using the value of the assigned nodePort. A nodePort works with any IP address that allows traffic to terminate at any node in the cluster.

Configuring the Service to be Highly Available

Instead of connecting directly to individual nodes, you can use one of OpenShift Container Platform’s highly availability strategies by deploying the IP failover router to provide access services configured with external IP addresses. This allows cluster administrators the flexibility of defining the edge router points within a cluster, and making the service highly available.

Nodes that have IP failover routers deployed to them must be in the same Layer 2 switching domain for ARP broadcasts to communicate to switches what appropriate port the destination should flow to.

High availability is limited to a maximum of 255 VIPs. This is a limitation of the Virtual Router Redundancy Protocol (VRRP). The VIPs do not have to be sequential.

Learn more about IP failover.

Using a Service NodePort

Use NodePorts to expose the service nodePort on all nodes in the cluster. NodePorts are in the 30-60k range by default, which means a NodePort is unlikely to match a service’s intended port (for example, 8080 may be exposed as 31020). This use of ports is wasteful of scarce port resources. However, it is slightly easier to set up. Again, this requires more privileges.

The administrator must ensure the external IPs are routed to the nodes and local firewall rules on all nodes allow access to the open port.

NodePorts and externalIP are independent and both can be used concurrently.

Using Virtual IPs

High availability improves the chances that an IP address will remain active, by assigning a virtual IP address to the host in a configured pool of hosts. If the host goes down, the virtual IP address is automatically transferred to another host in the pool.

Non-Cloud Edge Router Load Balancer

In a non-cloud environment, cluster administrators can assign a unique external IP address to a service (as described here). When routed correctly, external traffic can reach the service endpoints via any TCP/UDP port the service exposes. This is simpler than having to manage the port space of a limited number of shared IP addresses, when manually assigning external IPs to services.

Edge Load Balancer

An edge load balancer can be used to accept traffic from outside networks and proxy the traffic to pods inside the cluster.

In this configuration, the internal pod network is visible to the outside.