Notes on Mellanox ConnectX-3 Pro CX314 40G Ethernet NIC

I spent some time struggling with Mellanox ConnectX-3 Pro cards I bought from eBay on both Linux and Windows systems. Here are my notes for the forthcomers.

1. Grab the MFT tools from NVIDIA. The latest version (4.28.0) DOES NOT support these kinds of old cards. Use version 4.22.1-406-LTS instead, it works for both Linux and Windows without needing workarounds.

https://network.nvidia.com/products/adapter-software/firmware-tools/

1.1 You may see posts about setting these cards to Ethernet mode instead of IB/VPI mode (such as this). The CX314A is an Ethernet-only card. You do not need to install things like WinOF. There won’t be a Port configuration tab in Windows. There won’t be lines reporting port modes when using MFT to query card info.

With that said, you might be able to flash CX354 firmware onto this CX314 and get full IB/VPI connectivity. We do not include that here.

2. Flash the card with stock firmware. You can check the card firmware version with the command mlxfwmanager (no arguments.) The HP firmware will show up as version 2.43.7028, while the latest stock firmware version is 2.42.5000. Don’t worry about the lower version number. Flashing the firmware can be done with either mlxup (https://network.nvidia.com/support/firmware/mlxup-mft/) or mlxfwmanager. The command line is:

mlxup -f -u
# -f : force (replace the FW with a higher version number)
# -u : update 

# OR 
mlxfwmanager --online -f -u 
# --online: Fetch online firmware catalog 
# -f -u : same as above

3. Some of my hassles came from TrueNAS SCALE not configuring network bridges correctly (perhaps). I have another post on TrueNAS forum here. Note that this might be an occasional occurrence.

If you are configuring the bridge in TrueNAS, don’t forget to check the link status after configuring the Web UI / Console TUI, and add the bridge member manually if needed.

# Check link status
ip link

# Assume your CX314 is mapped to enp166s0 (and enp166s0d1)
# if enp166s0 does not have "master br0" in its status, execute the following.
ip link set enp166s0 master br0
ifconfig enp166s0 up # OR ip link set enp166s0 up

This issue sometimes persists after the server restarts (so you have to manually add bridge members again,) but sometimes it goes away.

4. Without a fan, I was only able to get 11-13Gbps. With a Noctua A4x10 I got 40Gb (iperf3 -c -P 2 -l 1M).