The Fast and the Furious: OpenShift Storage Edition

I was working with a customer recently who was having some storage performance problems with OpenShift. They have two physical NICs bonded together using OVS with source MAC-based load balancing. Unfortunately, all their storage traffic was ending up on the same physical NIC. Not exactly ideal when you paid for two of them.

They had been trying to tune their iSCSI sessions to improve performance. I suggested they fix the underlying network configuration first, either by using independent interfaces with multipathing or changing the bond configuration to something that could actually distribute the traffic. They didn’t want to change the network and wanted to see what they could get out of the existing configuration by tuning iSCSI sessions instead.

This reminded me of the first Fast and the Furious movie. Brian shows up with his green Eclipse, hits the NOS, and proceeds to blow the engine trying to keep up with Dom. He had plenty of horsepower, but wasn’t doing a very good job of using it. The customer’s networking problem was pretty similar.

Granny Shifting

I was basically having the networking version of this conversation. Adding more iSCSI sessions can improve performance, but it doesn’t necessarily distribute those sessions across multiple physical NICs. In this case, OVS was using source MAC-based load balancing, and the storage traffic was originating from the same MAC address. You can add 2, 4, 8, or 16 sessions, but if the bond keeps sending everything out the same physical interface, you’re not solving the problem.

There are several ways to address this. You could use LACP with L3/L4 hashing, which would allow different TCP sessions to use different links. Or you could give the storage initiator separate interfaces and let multipathing manage the paths. I’ve covered these options in previous posts on LACP versus multipathing and routed iSCSI and NVMe/TCP.

Personally, I prefer independent storage paths when possible. I can see which interface each session is using, control the routes, and let multipathing handle failures. The customer still didn’t want to change the OVS configuration, so I started thinking about how to give them separate storage paths without making them change the networking they already had.

You Never Had Me

What if I just created an SR-IOV Virtual Function (VF) on each physical interface? The Physical Functions (PFs) could remain members of the existing OVS bond, and the VFs could be used as independent storage interfaces. Each VF would have its own MAC address, IP address, and routing configuration. The storage initiator would see two separate interfaces even though the physical NICs were still doing their normal OpenShift networking jobs.

Since the storage is routed, I could use separate routing tables to make sure each VF sent traffic through the correct physical interface without modifying the host’s default route. I wasn’t sure whether RHCOS would allow SR-IOV VFs to be created on PFs that were already members of an OVS bond, particularly when those interfaces were in promiscuous mode, but I figured it was worth testing.

I’ve got an OpenShift 4.21 lab with three bare-metal worker nodes running RHCOS 9.6. Each worker has an NVIDIA ConnectX-7 quad-port adapter, with two ports used for the OpenShift machine network through br-ex and the other two used for VM VLAN networking. The machine network uses a Linux active-backup bond underneath OVS, which isn’t quite the same configuration as the customer, but it lets me test both an active and standby interface.

I started by creating a VF on the standby interface using the following command:

echo 1 > /sys/class/net/<pf>/device/sriov_numvfs

The VF appeared, the physical interface stayed up, and the node continued working. I moved the VF into a separate network namespace so I could configure it without changing the host networking. Everything looked fine, so I moved on to the active member of the bond. As soon as I enabled SR-IOV, the node lost network connectivity. The worker went NotReady, several operators started restarting, and I couldn’t reach the gateway. The NIC still had link, the bond hadn’t failed over, and the host was still transmitting traffic, but it wasn’t receiving anything useful. Disabling SR-IOV restored connectivity after several seconds.

Johnny Tran and His Friends

Remember when Brian and Dom get away from the police, only to have Johnny Tran and his buddies show up and shoot up the Eclipse? That’s about how well my first SR-IOV experiment went. I figured the VF was probably interfering with the PF’s MAC filtering, or perhaps NetworkManager was trying to configure the new interface and doing something stupid. Another possibility was that the embedded switch was sending traffic to the VF instead of the PF.

I started by disabling automatic VF driver probing so I could enable SR-IOV without actually loading a VF driver or creating a network interface.

echo 0 > /sys/class/net/<pf>/device/sriov_drivers_autoprobe
echo 1 > /sys/class/net/<pf>/device/sriov_numvfs

There was no IP address, no DHCP, and no traffic coming from the VF, but the node still lost connectivity. Whatever was happening wasn’t caused by anything the VF was transmitting. Simply enabling SR-IOV was enough to break the network, so I started looking at the hardware counters to see what was actually happening.

When I enabled SR-IOV, the active interface’s received traffic dropped to almost zero, while the standby interface started receiving traffic intended for the host. That suggested the upstream switch had stopped forwarding traffic to the active port and was flooding some of it over the other link. I captured traffic on both interfaces and noticed that spanning-tree BPDUs stopped arriving on the active interface when SR-IOV was enabled. When I disabled it, the BPDUs returned, followed by an RSTP proposal/agreement exchange, and then normal traffic resumed. It certainly looked like the switch was putting the interface into a discarding state, but I didn’t have the switch logs to confirm exactly what was happening.

I moved the testing to the standby interface so I could investigate without repeatedly disconnecting the node. While watching the physical NIC counters, I noticed something odd. The NIC was transmitting exactly as many multicast frames as it was receiving, despite the host not sending any multicast traffic on that interface.

rx_multicast_phy: 9
tx_multicast_phy: 9

I checked the transmit packet captures and found nothing. The frames were being transmitted by the NIC hardware rather than the operating system. The embedded switch was receiving multicast traffic from the physical uplink and immediately sending it right back out the same port, including spanning-tree BPDUs. So the switch sends a BPDU to the server, and the server’s NIC sends it right back. I can see why the switch might have a problem with that.

That explained the apparent spanning-tree behavior, but it didn’t explain why the standby interface worked initially while the active interface didn’t. It turned out the difference was promiscuous mode. I’d been running tcpdump on the standby interface, which enabled promiscuous mode. Without that, the reflection didn’t happen. I ran several more tests to confirm the behavior.

The combination of SR-IOV legacy eswitch mode and promiscuous or all-multicast mode caused the NIC to reflect incoming multicast frames back out the uplink. OVS puts the active bond member into promiscuous mode, which explained why enabling SR-IOV on that interface immediately caused problems. The standby interface wasn’t promiscuous unless I enabled it myself.

This also meant the customer’s configuration would likely have the same problem if they used this combination of NIC and driver. Their OVS bond uses both physical members for load balancing, and both would be operating in promiscuous mode. My solution to using both NICs could potentially disable both NICs. That seemed like a minor flaw in the plan.

Apparently NVIDIA Already Fixed This

I started searching NVIDIA’s documentation and looking through the upstream mlx5 driver code. Eventually, I found an issue in the MLNX_OFED 23.10 release notes, reference 3602955, describing a problem where multicast steering rules sent received multicast traffic back out the uplink. The documented issue specifically involved a VF in all-multicast mode, while my testing involved a PF in promiscuous or all-multicast mode, but the behavior was nearly identical.

Looking through the Linux mlx5 embedded-switch implementation, I found that the legacy eswitch creates an all-multicast forwarding rule toward the uplink without restricting the incoming source port in the same way as some of the other rules. That appeared to explain how an incoming multicast frame could get matched by a rule that sent it right back where it came from. NVIDIA had fixed the issue in its OFED driver, but the RHEL 9.6 inbox driver I was using still had the behavior.

I decided to install NVIDIA’s driver and see whether that fixed it. Installing an alternate NIC driver on RHCOS isn’t quite as simple as installing an RPM, so I installed Node Feature Discovery and the NVIDIA Network Operator, then configured a NicClusterPolicy to deploy the DOCA driver on just my test worker.

The driver container built successfully, but the installation failed because it couldn’t unload the existing RDMA modules. After digging through the dependencies, I found that rpcrdma was being held open by the kernel’s NFS server, which had RDMA listeners enabled. I stopped the NFS server, but a few minutes later it was running again. Portworx had restarted it. Of course it did.

I eventually stopped Portworx temporarily on the worker, stopped the NFS server, and let the driver installation complete. This isn’t something I’d recommend doing on a production node without planning for the storage interruption, but this was a lab and I wanted to see if it would work. The new driver loaded successfully:

mlx5_core 2607.0.100
DOCA 26.07

One additional issue showed up after the driver reload. The VM networking bond selected its other member, which turned out not to be receiving the expected VLANs from the switch. That was a separate switch configuration problem, but it broke VM VLAN connectivity until I switched the active member back. Apparently I found another problem to fix while I was there.

With the driver installed and networking restored, I repeated the multicast tests. This time, enabling SR-IOV with promiscuous mode produced 13 received multicast packets and zero transmitted multicast packets. I tested all-multicast mode and got the same result. Then I enabled SR-IOV on the active member of the OpenShift bond, which had previously taken the node off the network within seconds. This time, the host remained connected to its gateway, inbound traffic continued flowing, the bond didn’t fail over, and OpenShift never marked the worker NotReady. The driver really had fixed the problem.

Back to the Original Problem

Now that I could enable SR-IOV without disconnecting the node, I went back to the original experiment. I created one VF on each physical NIC, assigned each VF its own MAC and IP address, and moved them into separate network namespaces. Since the storage traffic needed to be routed through the machine-network gateway, I configured the routes inside each namespace rather than changing the host’s default route.

Both VFs could reach the gateway and the FlashArray storage interfaces. I also checked connectivity to the iSCSI and NVMe/TCP ports to make sure each interface could independently reach the storage targets.

The VFs were using their respective physical interfaces, while the PFs continued participating in the existing OVS-backed bond. That’s exactly what I wanted to test. There are still some things to handle before this is ready for production, including persistent VF creation, NetworkManager configuration, source-based routing, and storage initiator configuration. I also haven’t run a full multipath performance or failover test with actual storage I/O. The connectivity tests show the interfaces work independently, but that’s not the same thing as proving a storage workload distributes evenly across both links.

The original question was whether RHCOS could support a VF on each NIC while those same NICs remained members of an OVS bond. With the NVIDIA DOCA driver on my ConnectX-7 hardware, the answer is yes.

I Owe You a Ten-Second Car

At the end of the movie, Brian hands Dom the keys to the Supra and lets him drive away. That’s probably a good place to end this experiment too. The customer didn’t want to change their existing networking, and I wanted them to use independent interfaces with proper storage multipathing. Using SR-IOV, it looks like we can do both.

Would I still rather fix the bond configuration or give storage dedicated physical interfaces? Absolutely. This adds quite a bit of complexity to work around something that could probably be solved with a different network configuration. But there are situations where changing the existing networking isn’t practical, and at least now I know this is possible.

I just wouldn’t recommend trying it with the inbox mlx5 driver on this particular combination of RHCOS and ConnectX-7 hardware unless you enjoy troubleshooting why your switch suddenly stopped talking to your server.

References

• NVIDIA MLNX_OFED 23.10 — Bug Fix 3602955
• NVIDIA DOCA SR-IOV documentation
• Linux mlx5 eswitch source
• NVIDIA Network Operator