$ oc get nns
Node network state is the network configuration for all nodes in the cluster. A NodeNetworkState
object exists on every node in the cluster. This object is periodically updated and captures the state of the network for that node.
List all the NodeNetworkState
objects in the cluster:
$ oc get nns
Inspect a NodeNetworkState
object to view the network on that node. The output in this example has been redacted for clarity:
$ oc get nns node01 -o yaml
apiVersion: nmstate.io/v1
kind: NodeNetworkState
metadata:
name: node01 (1)
status:
currentState: (2)
dns-resolver:
# ...
interfaces:
# ...
route-rules:
# ...
routes:
# ...
lastSuccessfulUpdateTime: "2020-01-31T12:14:00Z" (3)
1 | The name of the NodeNetworkState object is taken from the node. |
2 | The currentState contains the complete network configuration for the node, including DNS, interfaces, and routes. |
3 | Timestamp of the last successful update. This is updated periodically as long as the node is reachable and can be used to evalute the freshness of the report. |
Create an interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy
manifest to the cluster. The manifest details the requested configuration for the interface.
By default, the manifest applies to all nodes in the cluster. To add the interface to specific nodes, add the spec: nodeSelector
parameter and the appropriate <key>:<value>
for your node selector.
You can configure multiple nmstate-enabled nodes concurrently. The configuration applies to 50% of the nodes in parallel. This strategy prevents the entire cluster from being unavailable if the network connection fails. To apply the policy configuration in parallel to a specific portion of the cluster, use the maxUnavailable
field.
Create the NodeNetworkConfigurationPolicy
manifest. The following example configures a Linux bridge on all worker nodes and configures the DNS resolver:
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br1-eth1-policy (1)
spec:
nodeSelector: (2)
node-role.kubernetes.io/worker: "" (3)
maxUnavailable: 3 (4)
desiredState:
interfaces:
- name: br1
description: Linux bridge with eth1 as a port (5)
type: linux-bridge
state: up
ipv4:
dhcp: true
enabled: true
auto-dns: false
bridge:
options:
stp:
enabled: false
port:
- name: eth1
dns-resolver: (6)
config:
search:
- example.com
- example.org
server:
- 8.8.8.8
1 | Name of the policy. |
2 | Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster. |
3 | This example uses the node-role.kubernetes.io/worker: "" node selector to select all worker nodes in the cluster. |
4 | Optional: Specifies the maximum number of nmstate-enabled nodes that the policy configuration can be applied to concurrently. This parameter can be set to either a percentage value (string), for example, "10%" , or an absolute value (number), such as 3 . |
5 | Optional: Human-readable description for the interface. |
6 | Optional: Specifies the search and server settings for the DNS server. |
Create the node network policy:
$ oc apply -f br1-eth1-policy.yaml (1)
1 | File name of the node network configuration policy manifest. |
A NodeNetworkConfigurationPolicy
manifest describes your requested network configuration for nodes in the cluster.
The node network policy includes your requested network configuration and the status of execution of the policy on the cluster as a whole.
When you apply a node network policy, a NodeNetworkConfigurationEnactment
object is created for every node in the cluster. The node network configuration enactment is a read-only object that represents the status of execution of the policy on that node.
If the policy fails to be applied on the node, the enactment for that node includes a traceback for troubleshooting.
To confirm that a policy has been applied to the cluster, list the policies and their status:
$ oc get nncp
Optional: If a policy is taking longer than expected to successfully configure, you can inspect the requested state and status conditions of a particular policy:
$ oc get nncp <policy> -o yaml
Optional: If a policy is taking longer than expected to successfully configure on all nodes, you can list the status of the enactments on the cluster:
$ oc get nnce
Optional: To view the configuration of a particular enactment, including any error reporting for a failed configuration:
$ oc get nnce <node>.<policy> -o yaml
You can remove an interface from one or more nodes in the cluster by editing the NodeNetworkConfigurationPolicy
object and setting the state
of the interface to absent
.
Removing an interface from a node does not automatically restore the node network configuration to a previous state. If you want to restore the previous state, you will need to define that node network configuration in the policy.
If you remove a bridge or bonding interface, any node NICs in the cluster that were previously attached or subordinate to that bridge or bonding interface are placed in a down
state and become unreachable. To avoid losing connectivity, configure the node NIC in the same policy so that it has a status of up
and either DHCP or a static IP address.
Deleting the node network policy that added an interface does not change the configuration of the policy on the node.
Although a |
Update the NodeNetworkConfigurationPolicy
manifest used to create the interface. The following example removes a Linux bridge and configures the eth1
NIC with DHCP to avoid losing connectivity:
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: <br1-eth1-policy> (1)
spec:
nodeSelector: (2)
node-role.kubernetes.io/worker: "" (3)
desiredState:
interfaces:
- name: br1
type: linux-bridge
state: absent (4)
- name: eth1 (5)
type: ethernet (6)
state: up (7)
ipv4:
dhcp: true (8)
enabled: true (9)
1 | Name of the policy. |
2 | Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster. |
3 | This example uses the node-role.kubernetes.io/worker: "" node selector to select all worker nodes in the cluster. |
4 | Changing the state to absent removes the interface. |
5 | The name of the interface that is to be unattached from the bridge interface. |
6 | The type of interface. This example creates an Ethernet networking interface. |
7 | The requested state for the interface. |
8 | Optional: If you do not use dhcp , you can either set a static IP or leave the interface without an IP address. |
9 | Enables ipv4 in this example. |
Update the policy on the node and remove the interface:
$ oc apply -f <br1-eth1-policy.yaml> (1)
1 | File name of the policy manifest. |
Before you read the different example NodeNetworkConfigurationPolicy
(NNCP) manifest configurations, consider the following factors when you apply a policy so that your cluster runs at its best performance conditions:
When you need to apply a policy to more than one node, create a NodeNetworkConfigurationPolicy
manifest for each target node. The Kubernetes NMState Operator applies the policy to each node with an NNCP in an unspecified order. Scoping a policy with this approach reduces the length of time for policy application but risks a cluster-wide outage if an error is in the cluster’s configuration. To avoid this type of error, initially apply NNCP to some nodes, and after you confirm they are configured correctly, proceed with applying the policy to the remaining nodes.
When you need to apply a policy to many nodes but you only want to create a single NNCP for all target nodes, the Kubernetes NMState Operator applies the policy to each node in sequence. You can set the speed and coverage of policy application for target nodes with the maxUnavailable
parameter in the cluster configuration. By setting a lower percentage value for the parameter, you can reduce the risk of a cluster-wide outage if the outage impacts the small percentage of nodes that are receiving the policy application.
Consider specifying all related network configurations in a single policy.
When a node restarts, the Kubernetes NMState Operator cannot control the order that it applies policies to nodes. The Kubernetes NMState Operator might apply interdependent policies in a sequence that results in a degraded network object.
Create a Linux bridge interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy
manifest
to the cluster.
The following YAML file is an example of a manifest for a Linux bridge interface. It includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br1-eth1-policy (1)
spec:
nodeSelector: (2)
kubernetes.io/hostname: <node01> (3)
desiredState:
interfaces:
- name: br1 (4)
description: Linux bridge with eth1 as a port (5)
type: linux-bridge (6)
state: up (7)
ipv4:
dhcp: true (8)
enabled: true (9)
bridge:
options:
stp:
enabled: false (10)
port:
- name: eth1 (11)
1 | Name of the policy. |
2 | Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster. |
3 | This example uses a hostname node selector. |
4 | Name of the interface. |
5 | Optional: Human-readable description of the interface. |
6 | The type of interface. This example creates a bridge. |
7 | The requested state for the interface after creation. |
8 | Optional: If you do not use dhcp , you can either set a static IP or leave the interface without an IP address. |
9 | Enables ipv4 in this example. |
10 | Disables stp in this example. |
11 | The node NIC to which the bridge attaches. |
Create a VLAN interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy
manifest to the cluster.
Define all related configurations for the VLAN interface of a node in a single When a node restarts, the Kubernetes NMState Operator cannot control the order in which policies are applied. Therefore, if you use separate policies for related network configurations, the Kubernetes NMState Operator might apply these policies in a sequence that results in a degraded network object. |
The following YAML file is an example of a manifest for a VLAN interface. It includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: vlan-eth1-policy (1)
spec:
nodeSelector: (2)
kubernetes.io/hostname: <node01> (3)
desiredState:
interfaces:
- name: eth1.102 (4)
description: VLAN using eth1 (5)
type: vlan (6)
state: up (7)
vlan:
base-iface: eth1 (8)
id: 102 (9)
1 | Name of the policy. |
2 | Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster. |
3 | This example uses a hostname node selector. |
4 | Name of the interface. When deploying on bare metal, only the <interface_name>.<vlan_number> VLAN format is supported. |
5 | Optional: Human-readable description of the interface. |
6 | The type of interface. This example creates a VLAN. |
7 | The requested state for the interface after creation. |
8 | The node NIC to which the VLAN is attached. |
9 | The VLAN tag. |
Create a bond interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy
manifest
to the cluster.
OpenShift Container Platform only supports the following bond modes:
|
The following YAML file is an example of a manifest for a bond interface. It includes samples values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: bond0-eth1-eth2-policy (1)
spec:
nodeSelector: (2)
kubernetes.io/hostname: <node01> (3)
desiredState:
interfaces:
- name: bond0 (4)
description: Bond with ports eth1 and eth2 (5)
type: bond (6)
state: up (7)
ipv4:
dhcp: true (8)
enabled: true (9)
link-aggregation:
mode: active-backup (10)
options:
miimon: '140' (11)
port: (12)
- eth1
- eth2
mtu: 1450 (13)
1 | Name of the policy. |
2 | Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster. |
3 | This example uses a hostname node selector. |
4 | Name of the interface. |
5 | Optional: Human-readable description of the interface. |
6 | The type of interface. This example creates a bond. |
7 | The requested state for the interface after creation. |
8 | Optional: If you do not use dhcp , you can either set a static IP or leave the interface without an IP address. |
9 | Enables ipv4 in this example. |
10 | The driver mode for the bond. This example uses an active backup mode. |
11 | Optional: This example uses miimon to inspect the bond link every 140ms. |
12 | The subordinate node NICs in the bond. |
13 | Optional: The maximum transmission unit (MTU) for the bond. If not specified, this value is set to 1500 by default. |
Configure an Ethernet interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy
manifest to the cluster.
The following YAML file is an example of a manifest for an Ethernet interface. It includes sample values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: eth1-policy (1)
spec:
nodeSelector: (2)
kubernetes.io/hostname: <node01> (3)
desiredState:
interfaces:
- name: eth1 (4)
description: Configuring eth1 on node01 (5)
type: ethernet (6)
state: up (7)
ipv4:
dhcp: true (8)
enabled: true (9)
1 | Name of the policy. |
2 | Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster. |
3 | This example uses a hostname node selector. |
4 | Name of the interface. |
5 | Optional: Human-readable description of the interface. |
6 | The type of interface. This example creates an Ethernet networking interface. |
7 | The requested state for the interface after creation. |
8 | Optional: If you do not use dhcp , you can either set a static IP or leave the interface without an IP address. |
9 | Enables ipv4 in this example. |
You can create multiple interfaces in the same node network configuration policy. These interfaces can reference each other, allowing you to build and deploy a network configuration by using a single policy manifest.
The following example YAML file creates a bond that is named bond10
across two NICs and VLAN that is named bond10.103
that connects to the bond.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: bond-vlan (1)
spec:
nodeSelector: (2)
kubernetes.io/hostname: <node01> (3)
desiredState:
interfaces:
- name: bond10 (4)
description: Bonding eth2 and eth3 (5)
type: bond (6)
state: up (7)
link-aggregation:
mode: balance-xor (8)
options:
miimon: '140' (9)
port: (10)
- eth2
- eth3
- name: bond10.103 (4)
description: vlan using bond10 (5)
type: vlan (6)
state: up (7)
vlan:
base-iface: bond10 (11)
id: 103 (12)
ipv4:
dhcp: true (13)
enabled: true (14)
1 | Name of the policy. |
2 | Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster. |
3 | This example uses hostname node selector. |
4 | Name of the interface. |
5 | Optional: Human-readable description of the interface. |
6 | The type of interface. |
7 | The requested state for the interface after creation. |
8 | The driver mode for the bond. |
9 | Optional: This example uses miimon to inspect the bond link every 140ms. |
10 | The subordinate node NICs in the bond. |
11 | The node NIC to which the VLAN is attached. |
12 | The VLAN tag. |
13 | Optional: If you do not use dhcp, you can either set a static IP or leave the interface without an IP address. |
14 | Enables ipv4 in this example. |
Capturing the static IP of a NIC is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
Create a Linux bridge interface on nodes in the cluster and transfer the static IP configuration of the NIC to the bridge by applying a single NodeNetworkConfigurationPolicy
manifest to the cluster.
The following YAML file is an example of a manifest for a Linux bridge interface. It includes sample values that you must replace with your own information.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: br1-eth1-copy-ipv4-policy (1)
spec:
nodeSelector: (2)
node-role.kubernetes.io/worker: ""
capture:
eth1-nic: interfaces.name=="eth1" (3)
eth1-routes: routes.running.next-hop-interface=="eth1"
br1-routes: capture.eth1-routes | routes.running.next-hop-interface := "br1"
desiredState:
interfaces:
- name: br1
description: Linux bridge with eth1 as a port
type: linux-bridge (4)
state: up
ipv4: "{{ capture.eth1-nic.interfaces.0.ipv4 }}" (5)
bridge:
options:
stp:
enabled: false
port:
- name: eth1 (6)
routes:
config: "{{ capture.br1-routes.routes.running }}"
1 | The name of the policy. |
2 | Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster. This example uses the node-role.kubernetes.io/worker: "" node selector to select all worker nodes in the cluster. |
3 | The reference to the node NIC to which the bridge attaches. |
4 | The type of interface. This example creates a bridge. |
5 | The IP address of the bridge interface. This value matches the IP address of the NIC which is referenced by the spec.capture.eth1-nic entry. |
6 | The node NIC to which the bridge attaches. |
The following example configuration snippets show different methods of IP management.
These examples use the ethernet
interface type to simplify the example while showing the related context in the policy configuration. These IP management examples can be used with the other interface types.
The following snippet statically configures an IP address on the Ethernet interface:
# ...
interfaces:
- name: eth1
description: static IP on eth1
type: ethernet
state: up
ipv4:
dhcp: false
address:
- ip: 192.168.122.250 (1)
prefix-length: 24
enabled: true
# ...
1 | Replace this value with the static IP address for the interface. |
The following snippet ensures that the interface has no IP address:
# ...
interfaces:
- name: eth1
description: No IP on eth1
type: ethernet
state: up
ipv4:
enabled: false
# ...
The following snippet configures an Ethernet interface that uses a dynamic IP address, gateway address, and DNS:
# ...
interfaces:
- name: eth1
description: DHCP on eth1
type: ethernet
state: up
ipv4:
dhcp: true
enabled: true
# ...
The following snippet configures an Ethernet interface that uses a dynamic IP address but does not use a dynamic gateway address or DNS:
# ...
interfaces:
- name: eth1
description: DHCP without gateway or DNS on eth1
type: ethernet
state: up
ipv4:
dhcp: true
auto-gateway: false
auto-dns: false
enabled: true
# ...
By default, the nmstate
API stores DNS values globally as against storing them in a network interface. For certain situations, you must configure a network interface to store DNS values. To define a DNS configuration for a network interface, you must initially specify the dns-resolver
section in the network interface’s YAML configuration file.
Setting a DNS configuration is comparable to modifying the |
You cannot use |
The following example shows a default situation that stores DNS values globally:
Configure a static DNS without a network interface. Note that when updating the /etc/resolv.conf
file on a host node, you do not need to specify an interface, IPv4 or IPv6, in the NodeNetworkConfigurationPolicy
(NNCP) manifest.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: worker-0-dns-testing
spec:
nodeSelector:
kubernetes.io/hostname: <target_node>
desiredState:
dns-resolver:
config:
search:
- example.com
- example.org
server:
- 2001:db8:f::1
- 192.0.2.251
# ...
The following examples show situations that require configuring a network interface to store DNS values:
If you want to rank a static DNS name server over a dynamic DNS name server, define the interface that runs either the Dynamic Host Configuration Protocol (DHCP) or the IPv6 Autoconfiguration (autoconf
) mechanism in the network interface YAML configuration file.
192.0.2.1
to DNS name servers retrieved from the DHCPv4 network protocol# ...
dns-resolver:
config:
server:
- 192.0.2.1
interfaces:
- name: eth1
type: ethernet
state: up
ipv4:
enabled: true
dhcp: true
auto-dns: true
# ...
If you need to configure a network interface to store DNS values instead of adopting the default method, which uses the nmstate
API to store DNS values globally, you can set static DNS values and static IP addresses in the network interface YAML file.
Storing DNS values at the network interface level might cause name resolution issues after you attach the interface to network components, such as an Open vSwitch (OVS) bridge, a Linux bridge, or a bond. |
# ...
dns-resolver:
config:
search:
- example.com
- example.org
server:
- 2001:db8:1::d1
- 2001:db8:1::d2
- 192.0.2.1
interfaces:
- name: eth1
type: ethernet
state: up
ipv4:
address:
- ip: 192.0.2.251
prefix-length: 24
dhcp: false
enabled: true
ipv6:
address:
- ip: 2001:db8:1::1
prefix-length: 64
dhcp: false
enabled: true
autoconf: false
# ...
If you want to set static DNS search domains and dynamic DNS name servers for your network interface, define the dynamic interface that runs either the Dynamic Host Configuration Protocol (DHCP) or the IPv6 Autoconfiguration (autoconf
) mechanism in the network interface YAML configuration file.
example.com
and example.org
static DNS search domains along with dynamic DNS name server settings# ...
dns-resolver:
config:
search:
- example.com
- example.org
server: []
interfaces:
- name: eth1
type: ethernet
state: up
ipv4:
enabled: true
dhcp: true
auto-dns: true
ipv6:
enabled: true
dhcp: true
autoconf: true
auto-dns: true
# ...
The following snippet configures a static route and a static IP on interface eth1
.
dns-resolver:
config:
# ...
interfaces:
- name: eth1
description: Static routing on eth1
type: ethernet
state: up
ipv4:
dhcp: false
enabled: true
address:
- ip: 192.0.2.251 (1)
prefix-length: 24
routes:
config:
- destination: 198.51.100.0/24
metric: 150
next-hop-address: 192.0.2.1 (2)
next-hop-interface: eth1
table-id: 254
# ...
1 | The static IP address for the Ethernet interface. |
2 | Next hop address for the node traffic. This must be in the same subnet as the IP address set for the Ethernet interface. |