Before the implementation of user-defined networks (UDNs) in the default the OVN-Kubernetes CNI plugin for OpenShift Container Platform, the Kubernetes Layer 3 topology was supported as the primary network, or main network, to where all pods attach. The Kubernetes design principle requires that all pods communicate with each other by their IP addresses, and Kubernetes restricts inter-pod traffic according to the Kubernetes network policy. While the Kubernetes design is useful for simple deployments, the Layer 3 topology restricts customization of primary network segment configurations, especially for modern multi-tenant deployments.
UDN improves the flexibility and segmentation capabilities of the default Layer 3 topology for a Kubernetes pod network by enabling custom Layer 2, Layer 3, and localnet network segments, where all these segments are isolated by default. These segments act as either primary or secondary networks for container pods and virtual machines that use the default OVN-Kubernetes CNI plugin. UDNs enable a wide range of network architectures and topologies, enhancing network flexibility, security, and performance. You can build a UDN by using a Virtual Router Function (VRF).
The following diagram shows four cluster namespaces, where each namespace has a single assigned UDN, and each UDN has an assigned custom subnet for its pod IP allocations. The OVN-Kubernetes handles any overlapping UDN subnets. Without using the Kubernetes network policy, a pod attached to a UDN can communicate with other pods in that UDN. By default, these pods are isolated from communicating with pods that exist in other UDNs. For microsegmentation, you can apply the Kubernetes network policy within a UDN. You can assign one or more UDNs to a namespace, with a limitation of only one primary UDN to a namespace, and one or more namespaces to a UDN.
Nodes that use |
A cluster administrator can use a user-defined network to create and define additional networks that span multiple namespaces at the cluster level by leveraging the ClusterUserDefinedNetwork
custom resource (CR). Additionally, a cluster administrator or a cluster user can use a user-defined network to define additional networks at the namespace level with the UserDefinedNetwork
CR.
The following diagram shows tenant isolation that a cluster administrator created by defining a ClusterUserDefinedNetwork
CR for each tenant. This network configuration allows a network to span across many namespaces. In the diagram, the udn-1
disconnected network selects namespace-1
and namespace-2
, while the udn-2
disconnected network selects namespace-3
and namespace-4
. A tenant acts as a disconnected network that is isolated from other tenants' networks. Pods from a namespace can communicate with pods in another namespace only if those namespaces exist in the same tenant network.
The following sections further emphasize the benefits and limitations of user-defined networks, the best practices when creating a ClusterUserDefinedNetwork
or UserDefinedNetwork
CR, how to create the CR, and additional configuration details that might be relevant to your deployment.
User-defined networks provide the following benefits:
Enhanced network isolation for security
Tenant isolation: Namespaces can have their own isolated primary network, similar to how tenants are isolated in Red Hat OpenStack Platform (RHOSP). This improves security by reducing the risk of cross-tenant traffic.
Network flexibility
Layer 2 and layer 3 support: Cluster administrators can configure primary networks as layer 2 or layer 3 network types. This provides the flexibility of a secondary network to the primary network.
Simplified network management
Reduced network configuration complexity: With user-defined networks, the need for complex network policies are eliminated because isolation can be achieved by grouping workloads in different networks.
Advanced capabilities
Consistent and selectable IP addressing: Users can specify and reuse IP subnets across different namespaces and clusters, providing a consistent networking environment.
Support for multiple networks: The user-defined networking feature allows administrators to connect multiple namespaces to a single network, or to create distinct networks for different sets of namespaces.
Simplification of application migration from Red Hat OpenStack Platform (RHOSP)
Network parity: With user-defined networking, the migration of applications from OpenStack to OpenShift Container Platform is simplified by providing similar network isolation and configuration options.
Developers and administrators can create a user-defined network that is namespace scoped using the custom resource. An overview of the process is as follows:
An administrator creates a namespace for a user-defined network with the k8s.ovn.org/primary-user-defined-network
label.
The UserDefinedNetwork
CR is created by either the cluster administrator or the user.
The user creates pods in the namespace.
While user-defined networks (UDN) offer highly customizable network configuration options, there are limitations that cluster administrators and developers should be aware of when implementing and managing these networks. Consider the following limitations before implementing a UDN.
DNS limitations:
DNS lookups for pods resolve to the pod’s IP address on the cluster default network. Even if a pod is part of a user-defined network, DNS lookups will not resolve to the pod’s IP address on that user-defined network. However, DNS lookups for services and external entities will function as expected.
When a pod is assigned to a primary UDN, it can access the Kubernetes API (KAPI) and DNS services on the cluster’s default network.
Initial network assignment: You must create the namespace and network before creating pods. Assigning a namespace with pods to a new network or creating a UDN in an existing namespace will not be accepted by OVN-Kubernetes.
Health check limitations: Kubelet health checks are performed by the cluster default network, which does not confirm the network connectivity of the primary interface on the pod. Consequently, scenarios where a pod appears healthy by the default network, but has broken connectivity on the primary interface, are possible with user-defined networks.
Network policy limitations: Network policies that enable traffic between namespaces connected to different user-defined primary networks are not effective. These traffic policies do not take effect because there is no connectivity between these isolated networks.
Before setting up a ClusterUserDefinedNetwork
custom resource (CR), users should consider the following information:
A ClusterUserDefinedNetwork
CR is intended for use by cluster administrators and should not be used by non-administrators. If used incorrectly, it might result in security issues with your deployment, cause disruptions, or break the cluster network.
ClusterUserDefinedNetwork
CRs should not be created in the default namespace. This can result in no isolation and, as a result, could introduce security risks to the cluster.
ClusterUserDefinedNetwork
CRs should not target openshift-*
namespaces.
OpenShift Container Platform administrators should be aware that all namespaces of a cluster are selected when one of the following conditions are met:
The matchLabels
selector is left empty.
The matchExpressions
selector is left empty.
The namespaceSelector
is initialized, but does not specify matchExpressions
or matchLabel
. For example: namespaceSelector: {}
.
For primary networks, the namespace used for the ClusterUserDefinedNetwork
CR must include the k8s.ovn.org/primary-user-defined-network
label. This label cannot be updated, and can only be added when the namespace is created. The following conditions apply with the k8s.ovn.org/primary-user-defined-network
namespace label:
If the namespace is missing the k8s.ovn.org/primary-user-defined-network
label and a pod is created, the pod attaches itself to the default network.
If the namespace is missing the k8s.ovn.org/primary-user-defined-network
label and a primary ClusterUserDefinedNetwork
CR is created that matches the namespace, an error is reported and the network is not created.
If the namespace is missing the k8s.ovn.org/primary-user-defined-network
label and a primary ClusterUserDefinedNetwork
CR already exists, a pod in the namespace is created and attached to the default network.
If the namespace has the label, and a primary ClusterUserDefinedNetwork
CR does not exist, a pod in the namespace is not created until the ClusterUserDefinedNetwork
CR is created.
The following procedure creates a ClusterUserDefinedNetwork
custom resource (CR). Based upon your use case, create your request using either the cluster-layer-two-udn.yaml
example for a Layer2
topology type or the cluster-layer-three-udn.yaml
example for a Layer3
topology type.
|
You have logged in as a user with cluster-admin
privileges.
Optional: For a ClusterUserDefinedNetwork
CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network
label by entering the following command:
$ cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
name: <cudn_namespace_name>
labels:
k8s.ovn.org/primary-user-defined-network: ""
EOF
Create a request for either a Layer2
or Layer3
topology type cluster-wide user-defined network:
Create a YAML file, such as cluster-layer-two-udn.yaml
, to define your request for a Layer2
topology as in the following example:
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
name: <cudn_name> (1)
spec:
namespaceSelector: (2)
matchLabels: (3)
- "<example_namespace_one>":"" (4)
- "<example_namespace_two>":"" (4)
network: (5)
topology: Layer2 (6)
layer2: (7)
role: Primary (8)
subnets:
- "2001:db8::/64"
- "10.100.0.0/16" (9)
1 | Name of your ClusterUserDefinedNetwork CR. |
2 | A label query over the set of namespaces that the cluster UDN CR applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces. |
3 | Uses the matchLabels selector type, where terms are evaluated with an AND relationship. |
4 | Because the matchLabels selector type is used, provisions namespaces matching both <example_namespace_one> and <example_namespace_two> . |
5 | Describes the network configuration. |
6 | The topology field describes the network configuration; accepted values are Layer2 and Layer3 . Specifying a Layer2 topology type creates one logical switch that is shared by all nodes. |
7 | This field specifies the topology configuration. It can be layer2 or layer3 . |
8 | Specifies Primary or Secondary . Primary is the only role specification supported in 4.18. |
9 | For Layer2 topology types the following specifies config details for the subnet field:
|
Create a YAML file, such as cluster-layer-three-udn.yaml
, to define your request for a Layer3
topology as in the following example:
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
name: <cudn_name> (1)
spec:
namespaceSelector: (2)
matchExpressions: (3)
- key: kubernetes.io/metadata.name (4)
operator: In (5)
values: ["<example_namespace_one>, <example_namespace_two>"] (6)
network: (7)
topology: Layer3 (8)
layer3: (9)
role: Primary (10)
subnets: (11)
- cidr: 10.100.0.0/16
hostSubnet: 64
1 | Name of your ClusterUserDefinedNetwork CR. |
2 | A label query over the set of namespaces that the cluster UDN applies to. Uses the standard Kubernetes MatchLabel selector. Must not point to default or openshift-* namespaces. |
3 | Uses the matchExpressions selector type, where terms are evaluated with an OR relationship. |
4 | Specifies the label key to match. |
5 | Specifies the operator. Valid values include: In , NotIn , Exists , and DoesNotExist . |
6 | Because the matchExpressions type is used, provisions namespaces matching either <example_namespace_one> or <example_namespace_two> . |
7 | Describes the network configuration. |
8 | The topology field describes the network configuration; accepted values are Layer2 and Layer3 . Specifying a Layer3 topology type creates a layer 2 segment per node, each with a different subnet. Layer 3 routing is used to interconnect node subnets. |
9 | This field specifies the topology configuration. Valid values are layer2 or layer3 . |
10 | Specifies a Primary or Secondary role. Primary is the only role specification supported in 4.18. |
11 | For Layer3 topology types the following specifies config details for the subnet field:
|
Apply your request by running the following command:
$ oc create --validate=true -f <example_cluster_udn>.yaml
Where <example_cluster_udn>.yaml
is the name of your Layer2
or Layer3
configuration file.
Verify that your request is successful by running the following command:
$ oc get clusteruserdefinednetwork <cudn_name> -o yaml
Where <cudn_name>
is the name you created of your cluster-wide user-defined network.
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
creationTimestamp: "2024-12-05T15:53:00Z"
finalizers:
- k8s.ovn.org/user-defined-network-protection
generation: 1
name: my-cudn
resourceVersion: "47985"
uid: 16ee0fcf-74d1-4826-a6b7-25c737c1a634
spec:
namespaceSelector:
matchExpressions:
- key: custom.network.selector
operator: In
values:
- example-namespace-1
- example-namespace-2
- example-namespace-3
network:
layer3:
role: Primary
subnets:
- cidr: 10.100.0.0/16
topology: Layer3
status:
conditions:
- lastTransitionTime: "2024-11-19T16:46:34Z"
message: 'NetworkAttachmentDefinition has been created in following namespaces:
[example-namespace-1, example-namespace-2, example-namespace-3]'
reason: NetworkAttachmentDefinitionReady
status: "True"
type: NetworkCreated
Before setting up a UserDefinedNetwork
custom resource (CR), you should consider the following information:
openshift-*
namespaces should not be used to set up a UserDefinedNetwork
CR.
UserDefinedNetwork
CRs should not be created in the default namespace. This can result in no isolation and, as a result, could introduce security risks to the cluster.
For primary networks, the namespace used for the UserDefinedNetwork
CR must include the k8s.ovn.org/primary-user-defined-network
label. This label cannot be updated, and can only be added when the namespace is created. The following conditions apply with the k8s.ovn.org/primary-user-defined-network
namespace label:
If the namespace is missing the k8s.ovn.org/primary-user-defined-network
label and a pod is created, the pod attaches itself to the default network.
If the namespace is missing the k8s.ovn.org/primary-user-defined-network
label and a primary UserDefinedNetwork
CR is created that matches the namespace, a status error is reported and the network is not created.
If the namespace is missing the k8s.ovn.org/primary-user-defined-network
label and a primary UserDefinedNetwork
CR already exists, a pod in the namespace is created and attached to the default network.
If the namespace has the label, and a primary UserDefinedNetwork
CR does not exist, a pod in the namespace is not created until the UserDefinedNetwork
CR is created.
2 masquerade IP addresses are required for user defined networks. You must reconfigure your masquerade subnet to be large enough to hold the required number of networks.
|
Ensure tenants are using the UserDefinedNetwork
resource and not the NetworkAttachmentDefinition
(NAD) CR. This can create security risks between tenants.
When creating network segmentation, you should only use the NetworkAttachmentDefinition
CR if user-defined network segmentation cannot be completed using the UserDefinedNetwork
CR.
The cluster subnet and services CIDR for a UserDefinedNetwork
CR cannot overlap with the default cluster subnet CIDR. OVN-Kubernetes network plugin uses 100.64.0.0/16
as the default join subnet for the network. You must not use that value to configure a UserDefinedNetwork
CR’s joinSubnets
field. If the default address values are used anywhere in the network for the cluster you must override the default values by setting the joinSubnets
field. For more information, see "Additional configuration details for user-defined networks".
A layer 2 topology creates a virtual switch that is distributed across all nodes in a cluster. Virtual machines and pods connect to this virtual switch so that all these components can communicate with each other within the same subnet. If you decide not to specify a layer 2 subnet, then you must manually configure IP addresses for each pod in your cluster. When not specifying a layer 2 subnet, port security is limited to preventing Media Access Control (MAC) spoofing only, and does not include IP spoofing. A layer 2 topology creates a single broadcast domain that can be challenging in large network environments, whereby the topology might cause a broadcast storm that can degrade network performance.
A layer 3 topology creates a unique layer 2 segment for each node in a cluster. The layer 3 routing mechanism interconnects these segments so that virtual machines and pods that are hosted on different nodes can communicate with each other. A layer 3 topology can effectively manage large broadcast domains by assigning each domain to a specific node, so that broadcast traffic has a reduced scope. To configure a layer 3 topology, you must configure cidr
and hostSubnet
parameters.
The following procedure creates a UserDefinedNetwork
CR that is namespace scoped. Based upon your use case, create your request by using either the my-layer-two-udn.yaml
example for a Layer2
topology type or the my-layer-three-udn.yaml
example for a Layer3
topology type.
Optional: For a UserDefinedNetwork
CR that uses a primary network, create a namespace with the k8s.ovn.org/primary-user-defined-network
label by entering the following command:
$ cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
name: <udn_namespace_name>
labels:
k8s.ovn.org/primary-user-defined-network: ""
EOF
Create a request for either a Layer2
or Layer3
topology type user-defined network:
Create a YAML file, such as my-layer-two-udn.yaml
, to define your request for a Layer2
topology as in the following example:
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
name: udn-1 (1)
namespace: <some_custom_namespace>
spec:
topology: Layer2 (2)
layer2: (3)
role: Primary (4)
subnets:
- "10.0.0.0/24"
- "2001:db8::/60" (5)
1 | Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO). |
2 | The topology field describes the network configuration; accepted values are Layer2 and Layer3 . Specifying a Layer2 topology type creates one logical switch that is shared by all nodes. |
3 | This field specifies the topology configuration. It can be layer2 or layer3 . |
4 | Specifies a Primary or Secondary role. |
5 | For Layer2 topology types the following specifies config details for the subnet field:
|
Create a YAML file, such as my-layer-three-udn.yaml
, to define your request for a Layer3
topology as in the following example:
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
name: udn-2-primary (1)
namespace: <some_custom_namespace>
spec:
topology: Layer3 (2)
layer3: (3)
role: Primary (4)
subnets: (5)
- cidr: 10.150.0.0/16
hostSubnet: 24
- cidr: 2001:db8::/60
hostSubnet: 64
# ...
1 | Name of your UserDefinedNetwork resource. This should not be default or duplicate any global namespaces created by the Cluster Network Operator (CNO). |
2 | The topology field describes the network configuration; accepted values are Layer2 and Layer3 . Specifying a Layer3 topology type creates a layer 2 segment per node, each with a different subnet. Layer 3 routing is used to interconnect node subnets. |
3 | This field specifies the topology configuration. Valid values are layer2 or layer3 . |
4 | Specifies a Primary or Secondary role. |
5 | For Layer3 topology types the following specifies config details for the subnet field:
|
Apply your request by running the following command:
$ oc apply -f <my_layer_two_udn>.yaml
Where <my_layer_two_udn>.yaml
is the name of your Layer2
or Layer3
configuration file.
Verify that your request is successful by running the following command:
$ oc get userdefinednetworks udn-1 -n <some_custom_namespace> -o yaml
Where some_custom_namespace
is the namespace you created for your user-defined network.
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
creationTimestamp: "2024-08-28T17:18:47Z"
finalizers:
- k8s.ovn.org/user-defined-network-protection
generation: 1
name: udn-1
namespace: some-custom-namespace
resourceVersion: "53313"
uid: f483626d-6846-48a1-b88e-6bbeb8bcde8c
spec:
layer2:
role: Primary
subnets:
- 10.0.0.0/24
- 2001:db8::/60
topology: Layer2
status:
conditions:
- lastTransitionTime: "2024-08-28T17:18:47Z"
message: NetworkAttachmentDefinition has been created
reason: NetworkAttachmentDefinitionReady
status: "True"
type: NetworkCreated
The following table explains additional configurations for ClusterUserDefinedNetwork
and UserDefinedNetwork
custom resources (CRs) that are optional. It is not recommended to set these fields without explicit need and understanding of OVN-Kubernetes network topology.
Field | Type | Description |
---|---|---|
|
object |
When omitted, the platform sets default values for the The |
|
object |
The Setting a value of |
|
object |
The Enabled: Disabled: |
|
integer |
The maximum transmission units (MTU). The default value is |
The following tables explain the status condition types returned for ClusterUserDefinedNetwork
and UserDefinedNetwork
CRs when describing the resource. These conditions can be used to troubleshoot your deployment.
Condition type | Status | Reason and Message | |
---|---|---|---|
|
|
When |
|
Reason |
Message |
||
|
'NetworkAttachmentDefinition has been created in following namespaces: [example-namespace-1, example-namespace-2, example-namespace-3]'` |
||
|
|
When |
|
Reason |
Message |
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
Condition type | Status | Reason and Message | |
---|---|---|---|
|
|
When |
|
Reason |
Message |
||
|
|
||
|
|
When |
|
Reason |
Message |
||
|
|
By default, pods on a user-defined network (UDN) are isolated from the default network. This means that default network pods, such as those running monitoring services (Prometheus or Alertmanager) or the OpenShift Container Platform image registry, cannot initiate connections to UDN pods.
To allow default network pods to connect to a user-defined network pod, you can use the k8s.ovn.org/open-default-ports
annotation. This annotation opens specific ports on the user-defined network pod for access from the default network.
The following pod specification allows incoming TCP connections on port 80
and UDP traffic on port 53
from the default network:
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.ovn.org/open-default-ports: |
- protocol: tcp
port: 80
- protocol: udp
port: 53
# ...
Open ports are accessible on the pod’s default network IP, not its UDN network IP. |