Working With Network Interfaces in XCP-NG
Networking in XCP-NG is a bit of a departure from some other hypervisors such as VMware. A cluster of servers, referred to as a pool, contains a set of networks that are shared across all contributing hosts. These networks are based on Physical Interfaces (PIFs) on each host. These PIFs are in turn based on a network interface on each host. This can be a physical NIC or a network bond. The caveat being that the PIFs comprising a network are identical across all hosts in the pool. This can require some planning and pre-configuration effort in order to ensure that interfaces align.
As an example, here are the PIFs for a network with a single uplink on a pool with 2 hosts
[10:38 prod-hv-xcpng01 ~]# xe pif-list network-uuid=fc8da461-6c60-bf63-637e-c015195ab82d
uuid ( RO) : b8b0939b-2415-890d-de06-65f6123f6738
device ( RO): eth0
MAC ( RO): 28:ac:9e:22:2c:46
currently-attached ( RO): false
VLAN ( RO): -1
network-uuid ( RO): fc8da461-6c60-bf63-637e-c015195ab82d
host-uuid ( RO): 8082c1c5-8738-49c5-9646-2b09daf1876f
uuid ( RO) : e651aba6-b28e-b755-4d2d-a68568cea3c8
device ( RO): eth0
MAC ( RO): 28:ac:9e:22:2a:2a
currently-attached ( RO): false
VLAN ( RO): -1
network-uuid ( RO): fc8da461-6c60-bf63-637e-c015195ab82d
host-uuid ( RO): 5eb6a2d1-7e04-44b7-90ea-83367d600a8a
There is a good chunk of information in this result so let's break that down.
Each host contains:
- Device the PIF or physical interface this network is attached to. Note that both hosts show eth0
- MAC the MAC address of the associated NIC for eth0
- Currently-Attached the status of this network
- VLAN the VLAN, if any this network is tagged with; -1 being untagged or native
- Network-UUID the unique ID of the pool network this PIF is associated with; this is what we used to narrow down the results
- Host-UUID the unique ID of the host this PIF is associated with
As shown in the output the network fc8da461-6c60-bf63-637e-c015195ab82d uses eth0 across all hosts; which can present issues if the PIF for eth0 is tied to an undesirable NIC on other hosts. This can be resolved using the CLI to "rename" the PIFs "eth" name using the MAC address of the desired NIC.
The basic command to accomplish this is interface-rename and has several options:
-r, --rename rename physical interfaces. It is not safe to rename
interfaces which have traffic passing, or higher level
networking constructs on them (bonds/bridges/etc).
Use at your own risk after boot
-l, --list list current physical device information in a concise
manner as a reference for --update
-u, --update manually update the order of devices. <args> should
be one or more <target eth name>=MAC|PCI|Phys|"SMBios"
--reset-to-install reset configuration to install state
The one we are first interested in is -l or --list to obtain information on the PIFs and their associated NICs.
interface-rename --list
Here are 2 examples from different classes of hardware:
Cisco UCS Server
Name MAC PCI ethN Phys SMBios Driver Version Firmware
eth0 28:ac:9e:22:2c:46 0000:3b:00.0[0] eth0 em1 Onboard LAN1 ixgbe 5.18.6 0x800016f7, 0.386.96
eth1 28:ac:9e:22:2c:47 0000:3b:00.1[0] eth1 em2 Onboard LAN2 ixgbe 5.18.6 0x800016f7, 0.386.96
eth2 28:ac:9e:c4:37:0d 0000:1d:00.0[0] eth2 p5p1 enic 4.5.0.7-939.23 4.5(2b)
eth3 28:ac:9e:c4:37:0e 0000:1d:00.1[0] eth3 p5p2 enic 4.5.0.7-939.23 4.5(2b)
eth4 a0:36:9f:42:a5:9c 0000:d8:00.0[0] eth8 p2p0_1 SlotID:2 ixgbe 5.18.6 0x800006c5, 15.0.27
eth5 a0:36:9f:42:a5:9e 0000:d8:00.1[0] eth9 p2p0_1 SlotID:2 ixgbe 5.18.6 0x800006c5, 15.0.27
eth6 b4:96:91:1a:3c:3c 0000:5e:00.0[0] eth4 p1p1 SlotID:1 igb 5.13.20 1.63, 0x80000e85, 0.385.8
eth7 b4:96:91:1a:3c:3d 0000:5e:00.1[0] eth5 p1p2 SlotID:1 igb 5.13.20 1.63, 0x80000e85, 0.385.8
eth8 b4:96:91:1a:3c:3e 0000:5e:00.2[0] eth6 p1p3 SlotID:1 igb 5.13.20 1.63, 0x80000e85, 0.385.8
eth9 b4:96:91:1a:3c:3f 0000:5e:00.3[0] eth7 p1p4 SlotID:1 igb 5.13.20 1.63, 0x80000e85, 0.385.8
Mini PC
[10:34 maw-void-prod-xcpng03 ~]# interface-rename --list
Name MAC PCI ethN Phys SMBios Driver Version Firmware
eth0 1c:83:41:40:97:26 0000:02:00.0[0] eth0 em1 Onboard LAN Brodcom r8169 rtl8168h-2_0.0.2 02/26/15
eth1 1c:83:41:40:97:27 0000:04:00.0[0] eth1 igc 5.10.200-1 2017:888d
The Cisco UCS server you may notice has already had some interfaces renamed to resolve this exact issue.
For the sake of simplicity here are some example hosts where you may encounter a need to rename interfaces. In this case we want to use our 10gb NICs for storage and VM traffic, this cannot be done currently as eth0 on host 2 is 10 gigabit but host 1 is gigabit. The plan may look something like this:
- 1gb link - management - eth0
- 1gb link - unused - eth1
- 10gb link - VM traffic - eth2
- 10gb link - Storage traffic - eth3
Ideally you could create a bond of both 10gb links which would also work but still requires renaming the interfaces.
First we run interface-rename --list to get the list of interfaces.
Host 1
Name MAC PCI ethN Phys SMBios Driver Version Firmware
eth0 28:ac:9e:22:2c:46 0000:3b:00.0[0] eth0 em1 Onboard LAN1 igb 5.13.20 0x800016f7, 0.386.96
eth1 28:ac:9e:22:2c:47 0000:3b:00.1[0] eth1 em2 Onboard LAN2 igb 5.13.20 0x800016f7, 0.386.96
eth2 a0:36:9f:42:a5:9c 0000:d8:00.0[0] eth2 p2p0_1 SlotID:2 ixgbe 5.18.6 0x800006c5, 15.0.27
eth3 a0:36:9f:42:a5:9e 0000:d8:00.1[0] eth3 p2p0_1 SlotID:2 ixgbe 5.18.6 0x800006c5, 15.0.27
Host 2
Name MAC PCI ethN Phys SMBios Driver Version Firmware
eth0 28:ac:9e:5f:9d:11 0000:3b:00.0[0] eth0 em1 Onboard LAN1 ixgbe 5.18.6 0x800016f7, 0.386.96
eth1 28:ac:9e:5f:9d:12 0000:3b:00.1[0] eth1 em2 Onboard LAN2 ixgbe 5.18.6 0x800016f7, 0.386.96
eth2 b4:96:91:1a:3c:3c 0000:5e:00.0[0] eth2 p1p1 SlotID:1 igb 5.13.20 1.63, 0x80000e85, 0.385.8
eth3 b4:96:91:1a:3c:3d 0000:5e:00.1[0] eth3 p1p2 SlotID:1 igb 5.13.20 1.63, 0x80000e85, 0.385.8
Let's make the changes on host 2. We need to bring the interfaces down to start. In this example I am changing everything at once, this would assume you have console access either via a monitor or over the network using IPMI or another solution as you will not have network access.
ifconfig eth0 down
ifconfig eth1 down
ifconfig eth2 down
ifconfig eth3 down
Now we run the interface-rename command with --update while specifying the PIF and associated MAC address to update.
interface-rename --update eth0=b4:96:91:1a:3c:3c eth2=28:ac:9e:5f:9d:11
interface-rename --update eth1=b4:96:91:1a:3c:3d eth3=28:ac:9e:5f:9d:12
This will set eth0 as MAC b4:96:91:1a:3c:3c and eth2 as MAC 28:ac:9e:5f:9d:11, effectively flipping them. The same is true for eth1 and eth3.
Next we need to get the UUID of the PIFs and have the hosts forget them:
xe pif-list
uuid ( RO) : 439666dd-da4b-72e6-8748-997d702331a8
device ( RO): eth0
MAC ( RO): 28:ac:9e:5f:9d:11
currently-attached ( RO): true
VLAN ( RO): -1
network-uuid ( RO): c1bc9cc9-5fae-3634-36eb-1218fb5583de
host-uuid ( RO): 5eb6a2d1-7e04-44b7-90ea-83367d600a8a
uuid ( RO) : d2b72cf1-15d5-9e29-2d8c-828c2e5fe815
device ( RO): eth1
MAC ( RO): 28:ac:9e:5f:9d:12
currently-attached ( RO): false
VLAN ( RO): -1
network-uuid ( RO): d5473d25-7653-2b46-bc44-e9c387f46060
host-uuid ( RO): 5eb6a2d1-7e04-44b7-90ea-83367d600a8a
uuid ( RO) : b8b0939b-2415-890d-de06-65f6123f6738
device ( RO): eth2
MAC ( RO): b4:96:91:1a:3c:3c
currently-attached ( RO): true
VLAN ( RO): -1
network-uuid ( RO): fc8da461-6c60-bf63-637e-c015195ab82d
host-uuid ( RO): 8082c1c5-8738-49c5-9646-2b09daf1876f
uuid ( RO) : 600d8527-bbab-d994-c733-b6714ae125d2
device ( RO): eth3
MAC ( RO): b4:96:91:1a:3c:3d
currently-attached ( RO): true
VLAN ( RO): -1
network-uuid ( RO): 31b11390-d803-13dd-6a4a-f6bcc744c1b7
host-uuid ( RO): 8082c1c5-8738-49c5-9646-2b09daf1876f
Note the UUID associated with PIFs eth0 and eth2 so we can reference it with the xe pif-forget command:
xe pif-forget uuid=439666dd-da4b-72e6-8748-997d702331a8
xe pif-forget uuid=b8b0939b-2415-890d-de06-65f6123f6738
xe pif-forget uuid=d2b72cf1-15d5-9e29-2d8c-828c2e5fe815
xe pif-forget uuid=600d8527-bbab-d994-c733-b6714ae125d2
Now bring the new interfaces back up:
ifconfig eth0 up
ifconfig eth1 up
ifconfig eth2 up
ifconfig eth3 up
We now need to introduce to new PIFs to the host. You can get the host UUID from either the previous pif-list or by running xe host-list.
xe pif-introduce device=eth0 host-uuid=8082c1c5-8738-49c5-9646-2b09daf1876f mac=b4:96:91:1a:3c:3c
xe pif-introduce device=eth1 host-uuid=8082c1c5-8738-49c5-9646-2b09daf1876f mac=b4:96:91:1a:3c:3d
xe pif-introduce device=eth2 host-uuid=8082c1c5-8738-49c5-9646-2b09daf1876f mac=28:ac:9e:5f:9d:11
xe pif-introduce device=eth3 host-uuid=8082c1c5-8738-49c5-9646-2b09daf1876f mac=28:ac:9e:5f:9d:12
Now if we run interface-rename --list to get the list of interfaces we can see the changes that were made. The PIFs match up correctly between the hosts.
Name MAC PCI ethN Phys SMBios Driver Version Firmware
eth0 28:ac:9e:22:2c:46 0000:3b:00.0[0] eth0 em1 Onboard LAN1 igb 5.13.20 0x800016f7, 0.386.96
eth1 28:ac:9e:22:2c:47 0000:3b:00.1[0] eth1 em2 Onboard LAN2 igb 5.13.20 0x800016f7, 0.386.96
eth2 a0:36:9f:42:a5:9c 0000:d8:00.0[0] eth2 p2p0_1 SlotID:2 ixgbe 5.18.6 0x800006c5, 15.0.27
eth3 a0:36:9f:42:a5:9e 0000:d8:00.1[0] eth3 p2p0_1 SlotID:2 ixgbe 5.18.6 0x800006c5, 15.0.27
Host 2
Name MAC PCI ethN Phys SMBios Driver Version Firmware
eth0 b4:96:91:1a:3c:3c 0000:5e:00.0[0] eth2 p1p1 SlotID:1 igb 5.13.20 1.63, 0x80000e85, 0.385.8
eth1 b4:96:91:1a:3c:3d 0000:5e:00.1[0] eth3 p1p2 SlotID:1 igb 5.13.20 1.63, 0x80000e85, 0.385.8
eth2 28:ac:9e:5f:9d:11 0000:3b:00.0[0] eth0 em1 Onboard LAN1 ixgbe 5.18.6 0x800016f7, 0.386.96
eth3 28:ac:9e:5f:9d:12 0000:3b:00.1[0] eth1 em2 Onboard LAN2 ixgbe 5.18.6 0x800016f7, 0.386.96