Linux Storage Networking, Part 1 of 4.
So you’ve got two NICs in a Linux host, four iSCSI ports on the array, and eight paths to your storage. multipath -ll says everything is happy. You should be able to pull a cable and keep going.
Should is doing a lot of work there.
If you’re chasing an iSCSI multipath problem with two NICs on the same subnet, the usual suspects are interface binding, ARP and reverse-path filtering. Bind the sessions to their NICs, set arp_ignore, arp_announce and rp_filter, and then go look at which interface the replies actually come back on. NVMe/TCP has the same problem, so I’ll cover its binding too.
For those of us who came from VMware, the idea is familiar. Configure the storage interfaces, bind the initiator to them, check the paths. VMware sure made it easy. At least if you came along after 6.x. If you remember ESX 3.x you will feel my pain. Linux gives you the same pieces. It also gives you a few defaults that can put traffic from both interfaces on the same wire. Eight paths in the output means eight sessions logged in. It doesn’t mean you have two independent ways to reach the array.
I started covering some of this in the Proxmox multipathing plugins post. This time I want to dig into what’s happening underneath the storage sessions. It turned into four posts: interface binding and ARP, bonding, routed storage, and finally OpenShift.
I’m using a FlashArray in the examples. The Linux behavior applies to other IP storage too. This first part is about IPv4, iSCSI and NVMe/TCP. The CSI driver can wait until part four.
Two Linux storage NICs on the same subnet
Here is the starting point:
ens1f0 has 10.10.1.11. ens1f1 has 10.10.1.12. Both are on VLAN 100. The switches carry that VLAN across their inter-switch link, and the array has two ports on each controller. If both NICs connect to all four array ports, that gives us eight paths per volume.
That is the intended topology. Now we need Linux to use it that way.
When you look at ip addr, it is reasonable to think an IP belongs to the interface listed above it. Linux generally treats a local IP as belonging to the host. This is called the weak host model. By default, the host can answer ARP for an address through another interface, and choosing a source IP does not, by itself, choose the outgoing NIC.
Look at the routes:
Both routes reach 10.10.1.0/24. The route through ens1f0 has the lower metric. An ordinary route lookup prefers it, even when the source address is .12:
$ ip route get 10.10.1.100 from 10.10.1.1210.10.1.100 from 10.10.1.12 dev ens1f0 uid 0 cache
Well that’s annoying. The address configured on the second NIC is sending through the first NIC.
You can create two storage sessions with different source IPs and still have both depend on ens1f0. Multipath sees the sessions. It doesn’t inspect the cabling for you.
Bind iSCSI and NVMe/TCP sessions to each NIC
For this same-subnet example, I want the storage initiator explicitly bound to each NIC. With software iSCSI, create an iface record for each interface:
iscsiadm -m iface -I ens1f0 -o newiscsiadm -m iface -I ens1f0 -o update -n iface.net_ifacename -v ens1f0iscsiadm -m iface -I ens1f1 -o newiscsiadm -m iface -I ens1f1 -o update -n iface.net_ifacename -v ens1f1iscsiadm -m discovery -t sendtargets -p 10.10.1.100 -I ens1f0 -I ens1f1iscsiadm -m node -I ens1f0 --loginiscsiadm -m node -I ens1f1 --login
These examples assume a new configuration with the intended array records. Existing sessions need a planned logout and login before changing their bindings; editing the record doesn’t move a running session. The Open-iSCSI documentation covers iface creation and that restriction.
For NVMe/TCP, specify the host interface and address. Replace SUBSYSTEM_NQN with your subsystem’s NQN:
nvme connect -t tcp -a 10.10.1.100 -s 4420 -n 'SUBSYSTEM_NQN' \ --host-iface=ens1f0 --host-traddr=10.10.1.11nvme connect -t tcp -a 10.10.1.100 -s 4420 -n 'SUBSYSTEM_NQN' \ --host-iface=ens1f1 --host-traddr=10.10.1.12
That shows one array portal through both NICs. Repeat for the other intended portals. Your kernel and nvme-cli need to support interface binding; the connect documentation describes the options.
Then inspect what was created:
iscsiadm -m ifaceiscsiadm -m session -P 3nvme list-subsys
For iSCSI, check the iface record’s net_ifacename and the session’s Iface Netdev, source IP and portal. An iface name is just a record name. Calling a record ens1f1 doesn’t bind it unless its settings actually say so.
Now traffic has an explicit outgoing interface. The return traffic still needs to find it.
ARP flux: why both NICs answer for one IP
The array wants to send something to .12. It broadcasts an ARP request asking who has that address. Both host NICs receive it because they share the VLAN.
At the default arp_ignore = 0, both can answer. ens1f1 replies with its MAC, ending in :02. ens1f0 replies with its MAC, ending in :01. Both are answering for 10.10.1.12.
The array can learn the wrong IP-to-MAC mapping. Refresh the entry and it might learn the other one. This is ARP flux. It makes a nice intermittent problem because the configuration stays the same while the neighbor entry changes.
You can catch it with a packet capture:
tcpdump -eni ens1f0 arp
aa:bb:cc:00:00:01 > 24:a9:37:00:00:64, ethertype ARP (0x0806), length 42: Reply 10.10.1.12 is-at aa:bb:cc:00:00:01, length 28
That is the first NIC answering for the second NIC’s storage address. For this design, that is not what we want.
arp_ignore = 1 limits replies to addresses configured on the receiving interface. 2 adds a same-subnet check for the sender. In this example, setting it to 2 leaves ens1f1 as the only interface answering for .12.
There is another setting to deal with, though.
arp_announce: ARP requests can teach the wrong address too
arp_ignore handles replies. arp_announce handles the sender address Linux puts into its own ARP requests.
Suppose an unbound packet sourced from .12 leaves through ens1f0. With the default arp_announce = 0, the ARP request can advertise .12 alongside ens1f0‘s MAC. The array may update its existing neighbor entry from those sender fields. We just taught it the wrong mapping without answering anything.
Set arp_announce = 2 and, with the addressing shown here, the request on ens1f0 uses .11. The request on ens1f1 uses .12. Now each NIC only advertises its own address.
Both settings are defined in the kernel’s IP sysctl documentation. It’s easy to fix the replies and forget about the requests. You need both.
Why multipath can fail when you pull a cable
If both source addresses have been learned against the first NIC’s MAC, the array sends their traffic toward that NIC. Pull its cable and the healthy second NIC doesn’t automatically fix the array’s neighbor table. It can transmit all day. Replies are still going somewhere else.
This is also where troubleshooting gets confusing. A device-bound socket can reject packets arriving on the wrong interface. You may see failed logins or flapping paths before anyone pulls a cable, rather than eight working paths followed by a sudden outage. Linux documents that receive restriction under SO_BINDTODEVICE. So binding the sessions without fixing ARP can turn a hidden problem into a very visible one.
The wrong neighbor entry can break the second path while both cables are still connected. After a link failure, recovery needs both the correct return path and working storage sessions.
Recovery depends on the array relearning the right MAC, the sessions recovering, and the storage timeout settings. If all paths disappear, your multipath policy determines whether I/O queues for a while or fails back to the application. A working spare NIC doesn’t help until the traffic actually reaches it.
Configure arp_ignore, arp_announce and rp_filter
For the two storage interfaces in this example:
# /etc/sysctl.d/99-storage-arp.confnet.ipv4.conf.ens1f0.arp_ignore = 2net.ipv4.conf.ens1f0.arp_announce = 2net.ipv4.conf.ens1f1.arp_ignore = 2net.ipv4.conf.ens1f1.arp_announce = 2net.ipv4.conf.ens1f0.rp_filter = 2net.ipv4.conf.ens1f1.rp_filter = 2
Apply the file after the interfaces exist:
sysctl -p /etc/sysctl.d/99-storage-arp.confsysctl net.ipv4.conf.ens1f0.arp_ignore net.ipv4.conf.ens1f0.arp_announcesysctl net.ipv4.conf.ens1f1.arp_ignore net.ipv4.conf.ens1f1.arp_announcesysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.ens1f1.rp_filter
The rp_filter lines handle reverse-path filtering. Strict mode (1) checks whether the best route back to an incoming packet’s source uses the interface it arrived on. Our two routes prefer ens1f0, so a reply arriving on ens1f1 can fail that check. Loose mode (2) checks that the source is reachable through some interface. That accommodates this topology.
A few details save a lot of head scratching:
- For these three settings, Linux uses the higher value of
conf.alland the interface setting. Check both. A lower per-interface setting cannot undo a higher global one. conf.defaultsupplies defaults for newly created interfaces. It affects future interfaces beyond storage too. I have kept this example scoped to the named NICs; verify that your boot configuration applies it when those NICs or VLANs appear.- A dotted VLAN name needs care in a sysctl key. For
ens1f0.100, usenet.ipv4.conf.ens1f0/100.arp_ignore, or the all-slash formnet/ipv4/conf/ens1f0.100/arp_ignore. Read the value back afterward. - Leave
arp_filteralone for this example. It makes ARP replies depend on routing. Without the corresponding source routing, it can stop the second interface answering at all.
Changing host settings doesn’t clear the array’s neighbor cache. Neither does ip neigh flush on the host. Verify that the array relearns the intended mappings, using its supported neighbor-cache procedure if needed. Coordinate session reconnects with the workloads using those devices.
From another machine on the storage VLAN, use arping against each host address. Each should resolve to its intended MAC. Capture on both host NICs and check the return traffic too. A correct sysctl file is only half the job.
Separate storage subnets make this easier
If I get to choose the design, I would put Fabric A on one subnet and VLAN, and Fabric B on another. For example, 10.10.1.0/24 on VLAN 100 and 10.10.2.0/24 on VLAN 200, with array ports in each fabric.
Now each NIC has a different connected route. Each fabric has its own broadcast domain. An ARP request on Fabric B doesn’t reach the Fabric A NIC, and the straightforward return route uses the expected interface. Putting two prefixes on one shared VLAN doesn’t get you this, by the way. It needs to be two broadcast domains.
I would still make the initiator bindings explicit and check the sessions. But this is a lot easier to explain to the next person who has to work on it.
Finally, run some I/O and test the failure. Pull one storage cable, watch the paths on that interface fail, and verify the workload keeps going. Restore it, let the paths recover, and test the other side. multipath -ll looking good before the test is nice. What it does during the test is the part that matters.
Next up: LACP versus multipath for iSCSI and NVMe/TCP. Can we avoid all this by bonding the NICs?
