Overview

This section describes some commonly used approaches to access your pods in a cluster.

The recommendation is:

TCP or UDP offers several approaches:

  • Use the non-cloud Load Balancer. This limits you to a single edge router IP (which can be a virtual IP (VIP), but is still a single machine for initial load balancing). It simplifies the administrator’s job, but uses one IP per service.

  • Manually assign ExternalIPs to the service. You can assign a set of IPs, so you can have multiple machines for the incoming load balancing. However, this requires elevated permissions to assign, and manual tracking of what IP:ports that are used.

  • Use NodePorts to expose a port for the service on all nodes in the cluster. This is more wasteful of scarce port resources. However, it is slightly easier to set up multiple. Again, this requires more privileges.

The router is the most common way to access the cluster. This is limited to HTTP/HTTPS(SNI)/TLS(SNI), which covers web applications.

ExternalIP, IngressIP, or NodePort is useful when the HTTP protocol is not being used or non-standard ports are in use.

By setting the ExternalIP or IngressIP on the service, Kubernetes sets up iptables rules to allow traffic arriving at any cluster node that is targeting that IP address to be sent to one of the internal pods. This is similar to the internal service IP addresses, but the externalIP tells Kubernetes that this one should also be exposed externally at the given IP. The administrator must assign the IP address to a host (node) interface on one of the nodes in the cluster. Alternatively, the address can be used as a virtual IP (VIP).

The administrator can also set up DNS entries for the IP addresses. For example, names can be configured into DNS to point to specific nodes or other IP addresses in the cluster. Use the DNS wildcard feature to configure a subset of names to an IP address in the cluster. DNS wildcard is convenient when using routers, because it allows the users to set up routes within the cluster without further administrator attention.

The administrator must ensure that the local firewall on each node permits the request to reach the IP address.

Using a Router and Routes

Using a router and routes is the most common way to access the cluster. A router is configured to accept external requests and proxy them based on the configured routes. This is limited to HTTP/HTTPS(SNI)/TLS(SNI), which covers web applications.

An administrator can create a wildcard DNS entry, and then set up a router. Afterward, the users can self-service the edge router without having to contact the administrators. The router has controls to allow the administrator to specify whether the users can self-provision host names, or if they must fit a pattern the administrator defines. The other solutions require the administrator to do the provisioning, or they require that the administrator delegates a lot of privilege.

When a set of routes is created in various projects, the overall set of routes is available to the set of routers. Each router admits (or selects) routes from the set of routes. By default, all routers admit all routes.

Routers that have permission to view all of the labels in all namespaces can select routes to admit based on the labels. This is called router sharding. This is useful when balancing incoming traffic load among a set of routers and when isolating traffic to a specific router. For example, company A goes to one router and company B to another.

Since a router runs on a specific node, when it or the node fails traffic ingress stops. The impact of this can be reduced by creating redundant routers on different nodes and using high availability to switch the router IP address when a node fails.

Using a Load Balancer Service

Load balancers are available on AWS and GCE clouds, and non-cloud options are also available.

The non-cloud load balancer allocates a unique IP from a configured pool. This limits you to a single edge router IP, which you can assign to an interface on one of the nodes in the cluster or use it as a virtual IP (VIP) in a highly available configuration. The non-cloud load balancer simplifies the administrator’s job by providing the needed IP address, but uses one IP per service.

Using a Service ExternalIP

Administrators can assign a list of externalIPs, for which nodes in the cluster will also accept traffic for the service. These IPs are not managed by OpenShift Container Platform and administrators are responsible for ensuring that traffic arrives at a node with this IP. A common example of this is configuring a highly available service.

The supplied list of IP addresses is used for load balancing incoming requests. The service port is opened on the externalIPs on all nodes running kube-proxy.

ExternalIPs require elevated permissions to assign, and manual tracking of the IP:ports that are in use.

Traffic from hosts that are external to the cluster that is going to the external IP address ends up on a node in the cluster. When it arrives, the service that set up the external IP redirects it to a service endpoint that it is managing. The service load balances among its endpoints.

An externally visible IP for the service can be configured in several ways:

  • Manually configure the ExternalIP with a known external IP address.

  • Configure ExternalIP to a VIP that is generated from (VRRP).

  • In a cloud (AWS or GCE) use type=LoadBalancer

  • In a non-cloud environment, configure an ingressIP range (IngressIPNetworkCIDR), service.type=LoadBalancer and service.port.ingressIP.

The administrator must ensure the external IPs are routed to the nodes and local firewall rules on all nodes allow access to the open port.

Example External IP Configured within a Service Definition
$ oc create -f - <<INGRESS
apiVersion: v1
kind: Service
metadata:
  name: postgresql-ingress
spec:
  externalIPs:
  - 10.9.54.100 (1)
  ports:
  - port: 5432
    protocol: TCP
  selector:
    name: postgresql
  type: LoadBalancer
INGRESS
1 The IP address assigned to the external IP.

The external IP address is not managed by the underlying Kubernetes infrastructure and must be maintained and provided by a cluster administrator. While external IPs provide a solution for accessing services on the OpenShift Container Platform cluster, there are several shortcomings:

  • The cluster administrator user must ensure the external IP is not in any range that the cluster is configured to use. The user needs to work with the administrator that controls the network external to the cluster to assign the external IP address and ensure traffic to the IP can get to the nodes in the cluster. The cluster administrator must ensure firewall rules permit the packets to get into the nodes for the ports they want to expose.

  • There are no protections in place to restrict the usage of a particular external address configured within the cluster. This allows the potential for a single address to be used by multiple services targeting the same port. When this situation occurs, the service which requested the port first is given use of the port.

Ingress IP Self-Service provides a solution.

Using Ingress IP Self-Service

Ingress IP Self-Service streamlines the allocation of External IPs for accessing services in the cluster. Cluster administrators can designate a range of addresses using a CIDR notation, which allows an application user to make a request against the cluster for an external IP address. This range should not conflict with the range reserved for ExternalIP addresses. When a service is configured with the type LoadBalancer, an External IP address is automatically assigned from the designated range.

Ingress IP Self-Service is only applicable in non-cloud based environments.

A common use case for Ingress IP Self-Service is the ability to provide database services, such as PostgreSQL, to clients outside of the OpenShift Container Platform cluster.

Defining the Edge Router IP Range

The ability for cluster administrators to automatically allocate External IP addresses using the edge router is enabled by default within OpenShift Container Platform and configured to use the 172.46.0.0/16 range. An alternate range can be specified by configuring the IngressIPNetworkCIDR parameter in the /etc/origin/master-config.yaml file:

networkConfig:
  IngressIPNetworkCIDR: 10.9.54.0/25

Restart the OpenShift Container Platform master service to apply the changes.

Deploy a Sample Application

To expose a PostgreSQL as a service for external consumption, the application must be first deployed into the cluster. Create a new project or use and existing project and instantiate one of the PostgreSQL templates.

The postgresql-ephemeral template does not make use of persistent storage. Once the application is scaled down or destroyed, any existing data will be lost. To use persistent storage, specify the postgresql-persistent template instead.

After instantiating the template, a ClusterIP-based service and DeploymentConfig is created and a new pod containing PostgreSQL will be started.

Configuring an IP Address for a Service

To allow the cluster to automatically assign an IP address for a service, create a service definition similar to the following that will create a new Ingress service:

$ oc create -f - <<INGRESS
apiVersion: v1
kind: Service
metadata:
  name: postgresql-ingress
spec:
  ports:
  - name: postgresql
    port: 5432
  type: LoadBalancer (1)
  selector:
    name: postgresql
INGRESS
1 The LoadBalancer type of service will make the request for an external service on behalf of the application user.

Alternatively, the oc expose command can be used to create the service:

$ oc expose dc postgresql --type=LoadBalancer --name=postgresql-ingress

Once the service is created, the external IP address is automatically allocated by the cluster and can be confirmed by running:

$ oc get svc postgresql-ingress
Example oc get Output
NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
postgresql-ingress    172.30.74.106   10.9.54.100,10.9.54.100    5432/TCP    30s

Specifying the type LoadBalancer also configures the service with a nodePort value. nodePort exposes the service port on all nodes in the cluster. Any packet that arrives on any node in the cluster targeting the nodePort ends up in the service. Then, it is load balanced to the service’s endpoints.

To discover the node port automatically assigned, run:

$ oc export svc postgresql-ingress
Example oc export Output
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: postgresql-persistent
    template: postgresql-persistent-template
  name: postgresql-ingress
spec:
  ports:
  - nodePort: 32439 (1)
    port: 5432
    protocol: TCP
    targetPort: 5432
  selector:
    name: postgresql
  sessionAffinity: None
  type: LoadBalancer
1 Automatically assigned port.

A PostgreSQL client can now be configured to connect directly to any node using the value of the assigned nodePort. A nodePort works with any IP address that allows traffic to terminate at any node in the cluster.

Configuring the Service to be Highly Available

Instead of connecting directly to individual nodes, you can use one of OpenShift Container Platform’s highly availability strategies by deploying the IP failover router to provide access services configured with external IP addresses. This allows cluster administrators the flexibility of defining the edge router points within a cluster, and making the service highly available.

Nodes that have IP failover routers deployed to them must be in the same Layer 2 switching domain for ARP broadcasts to communicate to switches what appropriate port the destination should flow to.

High availability is limited to a maximum of 255 VIPs. This is a limitation of the Virtual Router Redundancy Protocol (VRRP). The VIPs do not have to be sequential.

Using a Service NodePort

Use NodePorts to expose the service nodePort on all nodes in the cluster. NodePorts are in the 30-60k range by default, which means a NodePort is unlikely to match a service’s intended port (for example, 8080 may be exposed as 31020). This use of ports is wasteful of scarce port resources. However, it is slightly easier to set up. Again, this requires more privileges.

The administrator must ensure the external IPs are routed to the nodes and local firewall rules on all nodes allow access to the open port.

NodePorts and externalIP are independent and both can be used concurrently.

Using Virtual IPs

High availability improves the chances that an IP address will remain active, by assigning a virtual IP address to the host in a configured pool of hosts. If the host goes down, the virtual IP address is automatically transferred to another host in the pool.

Non-Cloud Edge Router Load Balancer

In a non-cloud environment, cluster administrators can assign a unique external IP address to a service (as described here). When routed correctly, external traffic can reach the service endpoints via any TCP/UDP port the service exposes. This is simpler than having to manage the port space of a limited number of shared IP addresses, when manually assigning external IPs to services.

Edge Load Balancer

An edge load balancer can be used to accept traffic from outside networks and proxy the traffic to pods inside the cluster.

In this configuration, the internal pod network is visible to the outside.