Installing Mellanox Connectx-3 / Connectx-3 Pro 56Gb/40Gbe NICs in XCP-ng 8.3 LTS With VDI Firmware defaulting to Infiniband (ib) When You Want Ethernet (en) Instead (But Still Keeping the Infiniband Option Open!)

coolelectricity

How’s that for a title? Oddly Specific, yet IYKYK.

Sometimes, all goes perfectly smooth with XCP-ng and the Connectx-3 NICs. Other times… not so much. This is for the “not to much” times.

Complete Guide: Configuring Mellanox ConnectX-3 NICs in XCP-ng 8.3

Problem Description

Mellanox ConnectX-3 Pro NICs may not be automatically recognized as Ethernet interfaces in XCP-ng 8.3 LTS. The card appears in lspci output but no network interfaces are created. This occurs because:

  1. Since XCP-ng 7.2, ConnectX-3 cards no longer automatically switch to Ethernet mode
  2. The cards default to InfiniBand (IB) mode, loading mlx4_ib modules instead of mlx4_en
  3. Without proper Ethernet interfaces, the card cannot be used for networking in XCP-ng

Initial Symptoms

When experiencing this issue, you’ll see:

# Card visible in PCI devices
lspci | grep -i Mellanox
01:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

# But no eth interfaces for the card
ip a
# Only shows existing interfaces, no new eth1/eth2

# Wrong driver module loaded
lsmod | grep mlx4
mlx4_ib                204800  0
ib_core               270336  1 mlx4_ib
mlx4_core             352256  1 mlx4_ib
# Note: mlx4_en is NOT loaded

# Card in InfiniBand mode
cat /sys/bus/pci/devices/0000:01:00.0/mlx4_port1
auto (ib)
cat /sys/bus/pci/devices/0000:01:00.0/mlx4_port2
auto (ib)

Prerequisites

  1. Identify your card’s PCI address:
lspci | grep -i Mellanox
# Note the address (e.g., 01:00.0)
  1. Install the alternate driver package (recommended for stability):
yum install mlx4-modules-alt

This (supposedly) provides the LTS 4.9 version of MLX4 drivers specifically optimized for ConnectX-3 cards (again… supposedly).

Solution: Configure Card for Ethernet Mode

Step 1: Create modprobe configuration

Create a configuration file to force the driver to load in Ethernet mode:

echo "options mlx4_core port_type_array=2,2" > /etc/modprobe.d/mlx4.conf

This parameter tells the mlx4_core driver to initialize both ports as Ethernet (value 2) instead of auto-detecting or using InfiniBand mode.

Step 2: Rebuild initramfs

Include the configuration in the boot image:

dracut --force

This ensures the configuration is applied early in the boot process before the driver loads.

Step 3: Verify configuration was added

lsinitrd | grep mlx4.conf
# Should show:
# -rw-r--r--   1 root     root           38 [date] etc/modprobe.d/mlx4.conf

Step 4: Reboot the system

reboot

The card will now initialize in Ethernet mode from boot.

Post-Reboot Verification

Step 1: Verify correct modules loaded

lsmod | grep mlx4

Expected output:

mlx4_en               159744  0
mlx4_ib                16384  0
mlx4_core             397312  1 mlx4_en
mlx_compat             49152  3 mlx4_core,mlx4_ib,mlx4_en
devlink                77824  2 mlx4_core,mlx4_en

Important: mlx4_en should now be loaded and shown as using mlx4_core.

Step 2: Check network interfaces

ip link show

You should see new interfaces (typically eth1 and eth2) with proper naming:

3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DORMANT group default qlen 1000
    link/ether ee:ff:aa:33:44:01 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DORMANT group default qlen 1000
    link/ether ee:ff:aa:33:44:02 brd ff:ff:ff:ff:ff:ff

Step 3: Register interfaces with XCP-ng

Scan for new physical interfaces:

xe pif-scan host-uuid=$(xe host-list --minimal)

Step 4: Verify XCP-ng recognition

xe pif-list

Expected output should include:

uuid ( RO)                  : [uuid]
                device ( RO): eth1
                   MAC ( RO): [mac address]
    currently-attached ( RO): false
                  VLAN ( RO): -1
          network-uuid ( RO): [network uuid]
             host-uuid ( RO): [host uuid]

uuid ( RO)                  : [uuid]
                device ( RO): eth2
                   MAC ( RO): [mac address]
    currently-attached ( RO): false
                  VLAN ( RO): -1
          network-uuid ( RO): [network uuid]
             host-uuid ( RO): [host uuid]

Troubleshooting

If interfaces don’t appear after reboot

  1. Check if the modprobe configuration is active:
cat /sys/module/mlx4_core/parameters/port_type_array
2,2
  1. Manually verify port modes:
cat /sys/bus/pci/devices/0000:03:00.0/mlx4_port1
# Should show: eth (not ib)
cat /sys/bus/pci/devices/0000:03:00.0/mlx4_port2
# Should show: eth (not ib)

If you see “Missing UAR, aborting” error

Check dmesg for this specific error:

dmesg | grep -i "missing uar"

If present, you need to add a kernel parameter:

  1. Edit /boot/efi/EFI/xenserver/grub.cfg (UEFI) or /boot/grub/grub.cfg (BIOS)
  2. Add pci=realloc=on to kernel parameters
  3. Reboot

Note: This was NOT required in the documented case above but may be needed on some systems.

Important Notes

  1. This configuration persists across XCP-ng updates – The modprobe.d configuration and initramfs changes survive system updates.
  2. No manual interface renaming required – XCP-ng’s interface-rename service handles naming automatically when interfaces appear at boot.
  3. Alternate driver package – The mlx4-modules-alt package is optional but recommended as it was specifically created by XCP-ng to address ConnectX-3 issues.
  4. Firmware considerations – Get it while you still can. NVIDIA bought up Mellanox awhile back and will likely drop the Connectx-3 from its listed drivers, sooner than later. Try to set your ConnectX-3 firmware with msflint to version 2.42.5000 (or newer?) for best the compatibility. You CAN cross flash OEM variants to the “Mellanox Official” Firmware but its admittedly dangerous, unless you know EXACTLY with variant of your HP/Dell/IBM/OEM etc NIC is. There are typically 2 to 5 different possibilities, based on the model number and then the “Letter Codes” – Similiar to how most OEMs do their NIC and Storage Contoller firmwares, the OEM firmware usually cripples the device in some way, either in performance or options/features/settings. (It’s not ALWAYS a bad thing, but USUALLY. OEM firmware is a lot like training wheels. Easier to used when you are just getting started out, then later, annoying once you have it figured out.)
  5. Firmware Simplifications – You could set your ConnectX-3 to be Ethernet-only, in the firmware itself, removing the added complexity that XCP-ng / Xen can get hung up on with attempting to auto-select Ethernet (en) over Infiniband (ib). Setting the NIC ports individually on dual-port NICs is also an option, too (eg. Port 1 is ethernet only, while Port 2 is VDI (which effectively means “either Ethernet or Infiniband, based on driver and settings used in the OS – oversimplifying it, but you get the concept). If you NEVER want to get that extra performance from Infiniband for any reason (eg, you only have a Brocade/Ruckus, Cisco, HP or Arista Switch with no support for Infinifband on the QSFP Ports) then this might make life easier – but these NICs have some age to them nowadays, and there ARE limits to how many times you can flash ROMs, so there’s that… These are just options worth mentioning; Recommended for beginners or resellers or Service Providers that bill based on annual contracts instead of billable hours for service calls)

Multiple ConnectX-3 Cards

The same port_type_array=2,2 configuration works for multiple ConnectX-3 cards without modification. The parameter applies the “both ports in Ethernet mode” pattern to ALL detected cards.

Adding Additional Cards

Best Practice: Add NICs one at a time and reboot between each addition. This ensures predictable interface numbering (eth1, eth2, eth3, eth4 in order) and makes it easier to identify which physical card corresponds to which interfaces. When removing cards, remove them in reverse order to maintain consistent numbering for remaining cards.

When adding a second (or third) ConnectX-3 card:

  1. Physically install the card
  2. Boot the system – the existing modprobe configuration will automatically apply
  3. Verify both cards initialized:
dmesg | grep -E "mlx4.*Initializing"
# Should show entries for each PCI address (e.g., 03:00.0, 0b:00.0)
  1. Check interfaces:
ip link show
# Will show eth1, eth2 (first card), eth3, eth4 (second card), etc.
  1. Register new interfaces with XCP-ng:
xe pif-scan host-uuid=$(xe host-list --minimal)
xe pif-list

Removing Cards

When removing a ConnectX-3 card:

  1. Shut down the host
  2. Remove the card
  3. Boot normally – the configuration remains valid
  4. Clean up orphaned PIFs if needed:
xe pif-list
# Find any PIFs for the removed card
xe pif-forget uuid=<uuid-of-removed-pif>

The port_type_array=2,2 configuration is resilient – it doesn’t specify a fixed number of ports, just the pattern to apply to each card found.

Summary

The key to getting ConnectX-3 NICs working in XCP-ng 8.3 is forcing them to initialize in Ethernet mode at boot using the port_type_array=2,2 module parameter. This clean, persistent solution requires:

  1. One configuration file (/etc/modprobe.d/mlx4.conf)
  2. One initramfs rebuild (dracut --force)
  3. One reboot
  4. One PIF scan (xe pif-scan)

No hacks, no manual interface manipulation, no startup scripts needed. The interfaces will appear with proper naming and full XCP-ng integration. The same configuration works whether you have one card or multiple cards installed.

That said: If you have more than 1 Connectx-3 NIC (Which is kinda nutty, but, hey, they are cheap AF right now!) then add them one at a time and work out the configuration details. If you try to handle multiple NICs at the same time, you might ruin your whole day. This is a “CoolElectricity Pro Tip Recommendation” – but you do you.

Somethings to keep in mind about 40Gbe:

40Gbe sustained requires (AT LEAST) about 4 CPU cores in a gen 12 or older CPU while using the typical default MTU of 1500 – you can reduce the CPU load SIGNIFICANTLY by using MTU 9000 (HIGHLY RECOMMENDED) or if your switches are limited to ~4092 MTU that’s still a huge improvement (~3 times less CPU) Hardware offloading is AMAZING too, but don’t get lost in the ether of Hardware Offloading and/or MTU settings. It’s easy to set them, but hard to sync them all up to work together, unless you know what you are doing. With network configurations, Less is more. Slow is Smooth, Smooth is Fast.

CE General rule of thumb for the “advanced users and homelabbers”:
– MTU 1500 for 1Gbe or less
– MTU 9000 for 2.5Gbe or more (But this means every switch and router and server NIC / OS / Firmware / SNMP options need to have at least MTU 9000, 4092 is a typical compromise with nearly all the benefits for 99.999999999% of home labs and 99.999% of business.

For scale, 40Gbe network saturation requires disk speeds of ~4GB/s to 5GB/s which equates to a lower end Gen 4 NVMe drive, or 8x to 10x SSDs in a single stripe. That means that a 1TB low end Gen 4 NVMe at 4GB/s will take ~250 seconds to transfer over an internal 40Gbe network link.

When was the last time you needed 1TB of data transferred in 8 to 9 minutes for any reason? Translation: Using a PCie 3.0 x4 slot at its full 32Gbps capabilities is not at all unreasonable of an idea, even with a DUAL 40Gbe/56Gb Port QSFP NIC like most Connectx-3’s.

if you are only getting ~20-28ish Gbps network speeds on your 40Gbe link, that’s still pretty amazing. You can tune it forever and ever, or live with the fact that you might never see more than 50-70% of your networks full I/O potential but live a life, instead.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.