You can use Ingress sharding, also known as router sharding, to distribute a set of routes across multiple routers by adding labels to routes, namespaces, or both. The Ingress Controller uses a corresponding set of selectors to admit only the routes that have a specified label. Each Ingress shard comprises the routes that are filtered using a given selection expression.
As the primary mechanism for traffic to enter the cluster, the demands on the Ingress Controller can be significant. As a cluster administrator, you can shard the routes to:
-
Balance Ingress Controllers, or routers, with several routes to speed up responses to changes.
-
Allocate certain routes to have different reliability guarantees than other routes.
-
Allow certain Ingress Controllers to have different policies defined.
-
Allow only specific routes to use additional features.
-
Expose different routes on different addresses so that internal and external users can see different routes, for example.
-
Transfer traffic from one version of an application to another during a blue green deployment.
When Ingress Controllers are sharded, a given route is admitted to zero or more Ingress Controllers in the group. A route’s status describes whether an Ingress Controller has admitted it or not. An Ingress Controller will only admit a route if it is unique to its shard.
An Ingress Controller can use three sharding methods:
-
Adding only a namespace selector to the Ingress Controller, so that all routes in a namespace with labels that match the namespace selector are in the Ingress shard.
-
Adding only a route selector to the Ingress Controller, so that all routes with labels that match the route selector are in the Ingress shard.
-
Adding both a namespace selector and route selector to the Ingress Controller, so that routes with labels that match the route selector in a namespace with labels that match the namespace selector are in the Ingress shard.
With sharding, you can distribute subsets of routes over multiple Ingress Controllers. These subsets can be non-overlapping, also called traditional sharding, or overlapping, otherwise known as overlapped sharding.
Traditional sharding example
An Ingress Controller finops-router
is configured with the label selector spec.namespaceSelector.matchLabels.name
set to finance
and ops
:
Example YAML definition for finops-router
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: finops-router
namespace: openshift-ingress-operator
spec:
namespaceSelector:
matchLabels:
name:
- finance
- ops
A second Ingress Controller dev-router
is configured with the label selector spec.namespaceSelector.matchLabels.name
set to dev
:
Example YAML definition for dev-router
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: dev-router
namespace: openshift-ingress-operator
spec:
namespaceSelector:
matchLabels:
name: dev
If all application routes are in separate namespaces, each labeled with name:finance
, name:ops
, and name:dev
respectively, this configuration effectively distributes your routes between the two Ingress Controllers. OpenShift Container Platform routes for console, authentication, and other purposes should not be handled.
In the above scenario, sharding becomes a special case of partitioning, with no overlapping subsets. Routes are divided between router shards.
|
The default Ingress Controller continues to serve all routes unless the namespaceSelector or routeSelector fields contain routes that are meant for exclusion. See this Red Hat Knowledgebase solution and the section "Sharding the default Ingress Controller" for more information on how to exclude routes from the default Ingress Controller.
|
Overlapped sharding example
In addition to finops-router
and dev-router
in the example above, you also have devops-router
, which is configured with the label selector spec.namespaceSelector.matchLabels.name
set to dev
and ops
:
Example YAML definition for devops-router
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: devops-router
namespace: openshift-ingress-operator
spec:
namespaceSelector:
matchLabels:
name:
- dev
- ops
The routes in the namespaces labeled name:dev
and name:ops
are now serviced by two different Ingress Controllers. With this configuration, you have overlapping subsets of routes.
With overlapping subsets of routes you can create more complex routing rules. For example, you can divert higher priority traffic to the dedicated finops-router
while sending lower priority traffic to devops-router
.
Sharding the default Ingress Controller
After creating a new Ingress shard, there might be routes that are admitted to your new Ingress shard that are also admitted by the default Ingress Controller. This is because the default Ingress Controller has no selectors and admits all routes by default.
You can restrict an Ingress Controller from servicing routes with specific labels using either namespace selectors or route selectors. The following procedure restricts the default Ingress Controller from servicing your newly sharded finance
, ops
, and dev
, routes using a namespace selector. This adds further isolation to Ingress shards.
|
You must keep all of OpenShift Container Platform’s administration routes on the same Ingress Controller. Therefore, avoid adding additional selectors to the default Ingress Controller that exclude these essential routes.
|
Procedure
-
Modify the default Ingress Controller by running the following command:
$ oc edit ingresscontroller -n openshift-ingress-operator default
-
Edit the Ingress Controller to contain a namespaceSelector
that excludes the routes with any of the finance
, ops
, and dev
labels:
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
namespaceSelector:
matchExpressions:
- key: type
operator: NotIn
values:
- finance
- ops
- dev
The default Ingress Controller will no longer serve the namespaces labeled name:finance
, name:ops
, and name:dev
.
Ingress sharding and DNS
The cluster administrator is responsible for making a separate DNS entry for each router in a project. A router will not forward unknown routes to another router.
Consider the following example:
Separate DNS entries must resolve *.foo.com
to the node hosting Router A and *.example.com
to the node hosting Router B:
Configuring Ingress Controller sharding by using route labels
Ingress Controller sharding by using route labels means that the Ingress
Controller serves any route in any namespace that is selected by the route
selector.
Figure 1. Ingress sharding using route labels
Ingress Controller sharding is useful when balancing incoming traffic load among
a set of Ingress Controllers and when isolating traffic to a specific Ingress
Controller. For example, company A goes to one Ingress Controller and company B
to another.
Procedure
-
Edit the router-internal.yaml
file:
# cat router-internal.yaml
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: sharded
namespace: openshift-ingress-operator
spec:
domain: <apps-sharded.basedomain.example.net> (1)
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
routeSelector:
matchLabels:
type: sharded
status: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""
1 |
Specify a domain to be used by the Ingress Controller. This domain must be different from the default Ingress Controller domain. |
-
Apply the Ingress Controller router-internal.yaml
file:
# oc apply -f router-internal.yaml
The Ingress Controller selects routes in any namespace that have the label
type: sharded
.
-
Create a new route using the domain configured in the router-internal.yaml
:
$ oc expose svc <service-name> --hostname <route-name>.apps-sharded.basedomain.example.net
Configuring Ingress Controller sharding by using namespace labels
Ingress Controller sharding by using namespace labels means that the Ingress
Controller serves any route in any namespace that is selected by the namespace
selector.
Figure 2. Ingress sharding using namespace labels
Ingress Controller sharding is useful when balancing incoming traffic load among
a set of Ingress Controllers and when isolating traffic to a specific Ingress
Controller. For example, company A goes to one Ingress Controller and company B
to another.
Procedure
-
Edit the router-internal.yaml
file:
# cat router-internal.yaml
Example output
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: sharded
namespace: openshift-ingress-operator
spec:
domain: <apps-sharded.basedomain.example.net> (1)
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
namespaceSelector:
matchLabels:
type: sharded
status: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""
1 |
Specify a domain to be used by the Ingress Controller. This domain must be different from the default Ingress Controller domain. |
-
Apply the Ingress Controller router-internal.yaml
file:
# oc apply -f router-internal.yaml
The Ingress Controller selects routes in any namespace that is selected by the
namespace selector that have the label type: sharded
.
-
Create a new route using the domain configured in the router-internal.yaml
:
$ oc expose svc <service-name> --hostname <route-name>.apps-sharded.basedomain.example.net