NSX-T Packet Captures

There are plenty of posts telling you how to do packet captures on NSX-T. Most of them do a great job explaining where to take captures and what commands to run. If you read them you’ll see the number of commands to run and the number of places to run them are both large. I recently had a customer with a problem which needed packet captures across the NSX-T infrastructure to find where packets were being dropped. So of course I scripted it. The script has some hard coded values that are specific to AVS naming so if you want to use if you’ll have to change some values. I’ll point those out later in the post.

To understand where to capture the packets you need to understand the traffic flow. For ingress packets (sourced from outside NSX) the flow looks something like this:

SRC -> ESXi uplinks (where edge nodes are running) -> ESXi switchports to edge VM uplinks -> T0 SR uplinks inside edge vm -> T0 DR downlink to T1 -> T1 SR uplink to T0 -> T1 DR downlink to segment -> Destination VM switchport on destination ESXi.

There are other points in between but these work nicely for ensuring the traffic is going where you want.

Unfortunately for almost all of the script there are no APIs that provide the functionality needed to capture the traffic. So I’ve used fabric as an ssh library for python to connect to all the devices. This creates problems with decoding the returns but luckily the hard ones to decode come from NSX and they provide a json output option.

To call the script you need a few values. The source IP of the traffic, the destination VM name of the traffic, vCenter IP and credentials, ESXi credentials, and NSX credentials. To call the script run a command like this:

python main.py --srcip 10.11.248.11 --dstVM ubuntu-template --vcenter 10.112.224.2 --vcenter_user administrator@vsphere.local --esx_user root --vcenter_password $VCSA --esx_password $esx --nsx_password $nsx

Here I have put the passwords all as environment variables. If you do not pass the password values the script will prompt for them. First the script will find all the components needed to do the captures.

vcenter_args = vCenterArgs(host=args.vcenter, user=args.vcenter_user, password=args.vcenter_password)
si = connect(vcenter_args)
dstvm = get_obj(si.RetrieveContent(), [vim.VirtualMachine], args.dstVM)
print("Found VM: "+dstvm.name)
print("VM IP: "+dstvm.summary.guest.ipAddress)
dstvmhostip=get_vmk1_ip(dstvm.runtime.host)
print("DstVM Host IP: "+dstvmhostip)
mgtVMFolder = get_obj(si.RetrieveContent(),[vim.Folder],"MGMT-VM")
evms = []
for vm in mgtVMFolder.childEntity:
    if "EVM01" in vm.name:
        evm01=vm
        evms.append(vm)
    if "EVM02" in vm.name:
        evm02=vm
        evms.append(vm)
print("Found EVM01: "+evm01.name)
print("EVM01 IP: "+evm01.summary.guest.ipAddress)
print("EVM01 Host IP: "+ get_vmk1_ip(evm01.runtime.host))
print("Found EVM02: "+evm02.name)
print("EVM02 IP: "+evm02.summary.guest.ipAddress)
print("EVM02 Host IP: "+ get_vmk1_ip(evm02.runtime.host))

Notice the hard coded MGMT-VM, EMV01, and EVM02 names. These are example names for a folder the edge VMs are in and the edge VMs. Change these to match you’re environment. The output should be similar to this:

Found VM: ubuntu-template
VM IP: 10.112.228.10
DstVM Host IP: 10.112.225.3
Found EVM01: TNT24-EVM01
EVM01 IP: 10.112.224.7
EVM01 Host IP: 10.112.225.4
Found EVM02: TNT24-EVM02
EVM02 IP: 10.112.224.8
EVM02 Host IP: 10.112.225.2

Next the host uplink ports on the ESXi hosts that the edges are running on are checked for packets.

print("Starting caputre for host uplinks")
print("-------")
for vm in evms:
    for uplink in ["vmnic0","vmnic3"]:
        print("Edge VM: {vm} Host: {host} Uplink: {uplink}".format(vm=vm.name, host = get_vmk1_ip(vm.runtime.host), uplink=uplink))
        try:
            result = host_uplink_capture(get_vmk1_ip(vm.runtime.host),args.esx_user, args.esx_password,uplink,args.srcip,dstvm.summary.guest.ipAddress)
            print(result.stdout)
        except:
            print("-------")

Here notice the hard coded vmnic names. Change these to match the vlan backed uplinks for your edge vms. These physical interfaces should be the links where the edges are uplinked to your physical switches for north south routing. This will loop through the edge VMs identified earlier, find the host they are running on, then loop through the two vmnics. This will capture all traffic in and out of the environment matching the filter of the srcip and the dstip (the ip of the destination VM). This output should look like this:

Starting capture for host uplinks
-------
Edge VM: TNT24-EVM01 Host: 10.112.225.4 Uplink: vmnic0
Connect to 10.112.225.4
14:21:39.492406 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.3190: Flags [S.], seq 835025739, ack 3969031246, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
14:21:39.520000 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3190: Flags [.], ack 164, win 501, length 0
14:21:39.520692 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.3190: Flags [P.], seq 1:156, ack 164, win 501, length 155: HTTP: HTTP/1.0 200 OK
14:21:39.520694 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.3190: Flags [FP.], seq 156:1247, ack 164, win 501, length 1091: HTTP
14:21:39.566179 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3190: Flags [.], ack 165, win 501, length 0
14:21:39.575260 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.3192: Flags [S.], seq 147415829, ack 2141003205, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
14:21:39.599997 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3192: Flags [.], ack 164, win 501, length 0
14:21:39.600314 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.3192: Flags [P.], seq 1:156, ack 164, win 501, length 155: HTTP: HTTP/1.0 200 OK
14:21:39.600316 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.3192: Flags [FP.], seq 156:1247, ack 164, win 501, length 1091: HTTP
14:21:39.630286 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.3193: Flags [S.], seq 2011525162, ack 568893586, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

-------
Edge VM: TNT24-EVM01 Host: 10.112.225.4 Uplink: vmnic3
Connect to 10.112.225.4
-------
-------
Edge VM: TNT24-EVM02 Host: 10.112.225.2 Uplink: vmnic0
Connect to 10.112.225.2
-------
-------
Edge VM: TNT24-EVM02 Host: 10.112.225.2 Uplink: vmnic3
Connect to 10.112.225.2
14:21:57.141113 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 66: 10.11.248.11.3628 > 10.112.228.10.8080: Flags [S], seq 4278221461, win 64240, options [mss 1418,nop,wscale 8,nop,nop,sackOK], length 0
14:21:57.142994 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 56: 10.11.248.11.3627 > 10.112.228.10.8080: Flags [.], ack 548275647, win 2049, length 0
14:21:57.143190 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 217: 10.11.248.11.3627 > 10.112.228.10.8080: Flags [P.], seq 0:163, ack 1, win 2049, length 163: HTTP: GET / HTTP/1.1
14:21:57.165046 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 56: 10.11.248.11.3628 > 10.112.228.10.8080: Flags [.], ack 2950496090, win 2049, length 0
14:21:57.165199 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 217: 10.11.248.11.3628 > 10.112.228.10.8080: Flags [P.], seq 0:163, ack 1, win 2049, length 163: HTTP: GET / HTTP/1.1
14:21:57.166848 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 56: 10.11.248.11.3627 > 10.112.228.10.8080: Flags [.], ack 1248, win 2044, length 0
14:21:57.187577 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 56: 10.11.248.11.3627 > 10.112.228.10.8080: Flags [F.], seq 163, ack 1248, win 2044, length 0
14:21:57.189124 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 56: 10.11.248.11.3628 > 10.112.228.10.8080: Flags [.], ack 1248, win 2044, length 0
14:21:57.189411 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 56: 10.11.248.11.1691 > 10.112.228.10.22: Flags [.], ack 3678283208, win 4095, length 0
14:21:57.198824 10:b3:d5:84:24:43 > 00:50:56:96:54:f1, ethertype IPv4 (0x0800), length 66: 10.11.248.11.3629 > 10.112.228.10.8080: Flags [S], seq 2463720335, win 64240, options [mss 1418,nop,wscale 8,nop,nop,sackOK], length 0

The cli command called on the host vi the host_uplink_capture function is:

pktcap-uw --uplink {uplink} --ip {srcip} --ip {dstip} -c 10 --dir 2 -o - | tcpdump-uw -enr-

There is a 10 second timeout so if no traffic is captured the command will not continue to block. If the command reaches the timeout the pcktcap-uw process does not stop and must be killed via the command:

kill -9 $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)

The next step is to check the host switch ports that the edge VMs are connected to.

for vm in evms:
    print("Edge VM: {vm} Host: {host}".format(vm=vm.name,host=get_vmk1_ip(vm.runtime.host)))
    try:
        result = host_vm_switchport_capture(get_vmk1_ip(vm.runtime.host),args.esx_user, args.esx_password,vm,args.srcip,dstvm.summary.guest.ipAddress)
        print(result.stdout)
    except:
        print("-------")
    print("--------")

This calls another function that first finds the VM world ID:

esxcli network vm list | grep {vmname} | awk '{{ print $1 }}'

Then finds the switchportIDs for that world:

esxcli network vm port list -w {worldid} | grep \ \ Port\ ID:  | awk '{{ print $3 }}'

Then the script will loop through all the switchports and capture traffic on them. This could be cleaned up by only capturing on the uplink interfaces but that would require API calls to NSX Manager to determine which interfaces and alot more code so I took the lazy way out and just looped through all of them. The capture command here is:

pktcap-uw -K --switchport {switchportid} -c 10 --ip {srcip} --ip {dstip} --dir 2 -o - | tcpdump-uw -enr -

The output should be verify similar to the uplinks and looks something like this:

Check Switch Ports
--------
Edge VM: TNT24-EVM01 Host: 10.112.225.4
Connect to 10.112.225.4
World ID: 15852561
Switchport ID: 33570840
14:22:01.464781 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 182: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 3678295976:3678296104, ack 393270942, win 4014, length 128
14:22:01.470950 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 182: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 128:256, ack 1, win 4014, length 128
14:22:01.527533 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.3743: Flags [S.], seq 2885344877, ack 967503946, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
14:22:01.552089 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3743: Flags [.], ack 164, win 501, length 0
14:22:01.552473 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 182: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 256:384, ack 1, win 4014, length 128
14:22:01.553008 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 182: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 384:512, ack 1, win 4014, length 128
14:22:01.553009 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.3743: Flags [P.], seq 1:156, ack 164, win 501, length 155: HTTP: HTTP/1.0 200 OK
14:22:01.553013 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.3743: Flags [FP.], seq 156:1247, ack 164, win 501, length 1091: HTTP
14:22:01.603509 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3743: Flags [.], ack 165, win 501, length 0
14:22:01.631865 00:50:56:96:16:57 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 182: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 512:640, ack 1, win 4014, length 128

Switchport ID: 33570841
14:22:03.812348 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3797: Flags [.], ack 3143371143, win 501, length 0
14:22:03.813143 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.3797: Flags [P.], seq 0:155, ack 1, win 501, length 155: HTTP: HTTP/1.0 200 OK
14:22:03.813147 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.3797: Flags [FP.], seq 155:1246, ack 1, win 501, length 1091: HTTP
14:22:03.859493 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3797: Flags [.], ack 2, win 501, length 0
14:22:04.173237 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.3805: Flags [S.], seq 1220529125, ack 2180237818, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
14:22:04.197714 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3805: Flags [.], ack 164, win 501, length 0
14:22:04.198713 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.3805: Flags [P.], seq 1:156, ack 164, win 501, length 155: HTTP: HTTP/1.0 200 OK
14:22:04.198716 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.3805: Flags [FP.], seq 156:1247, ack 164, win 501, length 1091: HTTP
14:22:04.242751 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.3805: Flags [.], ack 165, win 501, length 0
14:22:04.250507 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.3807: Flags [S.], seq 3154114750, ack 133703122, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

Switchport ID: 33570842
Timeout
Switchport ID: 50331672
Timeout

The next checks are from inside the Edge VM. This is where the bulk of the work in the script takes place

print("Check Edges")
for vm in evms:
    print("Edge VM: {vm} Host: {host}".format(vm=vm.name,host=get_vmk1_ip(vm.runtime.host)))
    result = edge_interface_capture(vm,args.nsx_password,args.srcip,dstvm.summary.guest.ipAddress)

This starts the work by looping through the edge VMs and passing the information off to the edge_interface_capture function. This started as a single function but ended up being many. Initially we need the logical-router interface ifuuids. The edge commands to get these are

get logical-routers
Logical Router
UUID                                   VRF    LR-ID  Name                              Type                        Ports   Neighbors
736a80e3-23f6-5a2d-81d6-bbefb2786666   0      0                                        TUNNEL                      3       2/5000
f497dcfd-9b09-446b-a7e8-f00d452a9291   1      4      SR-TNT24-T0                       SERVICE_ROUTER_TIER0        6       2/50000
1fd61dce-88da-4f6b-b43b-cb9587c5abd1   3      1      DR-TNT24-T0                       DISTRIBUTED_ROUTER_TIER0    5       1/50000
d43b0584-e9a7-46a8-bc47-ada87340ec59   4      2      DR-TNT24-T1                       DISTRIBUTED_ROUTER_TIER1    6       0/50000
01834ff3-6840-48c0-ac57-abc22b32f98a   6      7      SR-TNT24-T1                       SERVICE_ROUTER_TIER1        5       2/50000

Find the correct UUID and then get the interfaces

get logical-router {vrid} interfaces

For the script these commands are piped to json

get logical-routers | json

This makes parsing the objects much easier. Use json.loads and you get a json object with the data

routers = json.loads(result.stdout)

First i loop through the T0 interfaces and then the T1 interfaces. This calls another funcation with the json object for the particular router interfaces.

edge_interface_capture_process(c, data,srcip, dstip)

This function ensures the order of the captures is correct. First the T0 SR, T0 SR, T1 SR, then T1 DR. Next it calls the fucntion to actuall capture the data.

edge_interface_catpure_process_perrouter(c,data,srdr,srcip,  dstip)

The edge_interface_capture_process_perrouter function check in order the interfaces on each router. First the uplink interfaces then the downlink interfaces. It will loop through all of the uplinks on the T0 SR first. The output looks like:

Check Edges
Edge VM: TNT24-EVM01 Host: 10.112.225.4
Connect to 10.112.224.7
Checking Logical Router: SR-TNT24-T0
Router Type: SERVICE_ROUTER_TIER0
Checking Uplinks
Checking Interface uplink: TNT24-T0-PRIV01 ifuuid: 5bc8e4d5-88bc-4793-9925-a0def647a1f3
14:21:23.395179 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5396: Flags [.], ack 3150827899, win 501, length 0
<base64>AAEAAgADAFBWlltnCABFAAAoO55AAD4GEKEKcOQKCgv4Cx+QFRRqVGMSu83Re1AQAfUt+gAA</base64>

14:21:23.395437 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.5396: Flags [P.], seq 0:155, ack 1, win 501, length 155: HTTP: HTTP/1.0 200 OK
<base64>AAEAAgADAFBWlltnCABFAADDO59AAD4GEAUKcOQKCgv4Cx+QFRRqVGMSu83Re1AYAfWigAAASFRUUC8xLjAgMjAwIE9LDQpTZXJ2ZXI6IFNpbXBsZUhUVFAvMC42IFB5dGhvbi8zLjkuNQ0KRGF0ZTogVGh1LCAyMCBKYW4gMjAyMiAxNDoyMzowNyBHTVQNCkNvbnRlbnQtdHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04DQpDb250ZW50LUxlbmd0aDogMTA5MQ0KDQo=</base64>

14:21:23.395472 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.5396: Flags [FP.], seq 155:1246, ack 1, win 501, length 1091: HTTP
<base64>AAEAAgADAFBWlltnCABFAARrO6BAAD4GDFwKcOQKCgv4Cx+QFRRqVGOtu83Re1AZAfWyCQAAPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvVFIvaHRtbDQvc3RyaWN0LmR0ZCI+CjxodG1sPgo8aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgo8dGl0bGU+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L3RpdGxlPgo8L2hlYWQ+Cjxib2R5Pgo8aDE+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L2gxPgo8aHI+Cjx1bD4KPGxpPjxhIGhyZWY9Ii5iYXNoX2hpc3RvcnkiPi5iYXNoX2hpc3Rvcnk8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5iYXNoX2xvZ291dCI+LmJhc2hfbG9nb3V0PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuYmFzaHJjIj4uYmFzaHJjPC9hPjwvbGk+CjxsaT48YSBocmVmPSIuY2FjaGUvIj4uY2FjaGUvPC9hPjwvbGk+CjxsaT48YSBocmVmPSIucHJvZmlsZSI+LnByb2ZpbGU8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5weXRob25faGlzdG9yeSI+LnB5dGhvbl9oaXN0b3J5PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuc3NoLyI+LnNzaC88L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWwiPi5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWw8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii52aW1pbmZvIj4udmltaW5mbzwvYT48L2xpPgo8bGk+PGEgaHJlZj0iYmFkLnBjYXAiPmJhZC5wY2FwPC9hPjwvbGk+CjxsaT48YSBocmVmPSJiYWQudHh0Ij5iYWQudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGkvIj5lc3hjbGkvPC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyIj5lc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyPC9hPjwvbGk+CjxsaT48YSBocmVmPSJnb29kLnBjYXAiPmdvb2QucGNhcDwvYT48L2xpPgo8bGk+PGEgaHJlZj0iZ29vZC50eHQiPmdvb2QudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJuc3gtaW5ncmVzcy1wYWNrZXQtY2FwdHVyZS8iPm5zeC1pbmdyZXNzLXBhY2tldC1jYXB0dXJlLzwvYT48L2xpPgo8L3VsPgo8aHI+CjwvYm9keT4KPC9odG1sPgo=</base64>

14:21:23.441419 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5396: Flags [.], ack 2, win 501, length 0
<base64>AAEAAgADAFBWlltnCABFAAAoAABAAD4GTD8KcOQKCgv4Cx+QFRRqVGfxu83RfFAQAfUpGgAA</base64>

14:21:23.530543 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.5400: Flags [S.], seq 542455584, ack 4085186343, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
<base64>AAEAAgADAFBWlltnCABFAAA0AABAAD4GTDMKcOQKCgv4Cx+QFRggVTcg837/J4AS+vAEuwAAAgQFtAEBBAIBAwMH</base64>

14:21:23.554286 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5400: Flags [.], ack 164, win 501, length 0
<base64>AAEAAgADAFBWlltnCABFAAAokqxAAD4GuZIKcOQKCgv4Cx+QFRggVTch837/ylAQAfU95gAA</base64>

14:21:23.554763 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.5400: Flags [P.], seq 1:156, ack 164, win 501, length 155: HTTP: HTTP/1.0 200 OK
<base64>AAEAAgADAFBWlltnCABFAADDkq1AAD4GuPYKcOQKCgv4Cx+QFRggVTch837/ylAYAfWybAAASFRUUC8xLjAgMjAwIE9LDQpTZXJ2ZXI6IFNpbXBsZUhUVFAvMC42IFB5dGhvbi8zLjkuNQ0KRGF0ZTogVGh1LCAyMCBKYW4gMjAyMiAxNDoyMzowNyBHTVQNCkNvbnRlbnQtdHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04DQpDb250ZW50LUxlbmd0aDogMTA5MQ0KDQo=</base64>

14:21:23.554781 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.5400: Flags [FP.], seq 156:1247, ack 164, win 501, length 1091: HTTP
<base64>AAEAAgADAFBWlltnCABFAARrkq5AAD4GtU0KcOQKCgv4Cx+QFRggVTe8837/ylAZAfXB9QAAPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvVFIvaHRtbDQvc3RyaWN0LmR0ZCI+CjxodG1sPgo8aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgo8dGl0bGU+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L3RpdGxlPgo8L2hlYWQ+Cjxib2R5Pgo8aDE+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L2gxPgo8aHI+Cjx1bD4KPGxpPjxhIGhyZWY9Ii5iYXNoX2hpc3RvcnkiPi5iYXNoX2hpc3Rvcnk8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5iYXNoX2xvZ291dCI+LmJhc2hfbG9nb3V0PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuYmFzaHJjIj4uYmFzaHJjPC9hPjwvbGk+CjxsaT48YSBocmVmPSIuY2FjaGUvIj4uY2FjaGUvPC9hPjwvbGk+CjxsaT48YSBocmVmPSIucHJvZmlsZSI+LnByb2ZpbGU8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5weXRob25faGlzdG9yeSI+LnB5dGhvbl9oaXN0b3J5PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuc3NoLyI+LnNzaC88L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWwiPi5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWw8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii52aW1pbmZvIj4udmltaW5mbzwvYT48L2xpPgo8bGk+PGEgaHJlZj0iYmFkLnBjYXAiPmJhZC5wY2FwPC9hPjwvbGk+CjxsaT48YSBocmVmPSJiYWQudHh0Ij5iYWQudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGkvIj5lc3hjbGkvPC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyIj5lc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyPC9hPjwvbGk+CjxsaT48YSBocmVmPSJnb29kLnBjYXAiPmdvb2QucGNhcDwvYT48L2xpPgo8bGk+PGEgaHJlZj0iZ29vZC50eHQiPmdvb2QudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJuc3gtaW5ncmVzcy1wYWNrZXQtY2FwdHVyZS8iPm5zeC1pbmdyZXNzLXBhY2tldC1jYXB0dXJlLzwvYT48L2xpPgo8L3VsPgo8aHI+CjwvYm9keT4KPC9odG1sPgo=</base64>

14:21:23.599030 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5400: Flags [.], ack 165, win 501, length 0
<base64>AAEAAgADAFBWlltnCABFAAAoAABAAD4GTD8KcOQKCgv4Cx+QFRggVTwA837/y1AQAfU5BgAA</base64>

14:21:23.608293 00:50:56:96:5b:67 > 00:01:00:02:00:03, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.5402: Flags [S.], seq 2349317346, ack 2430256211, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
<base64>AAEAAgADAFBWlltnCABFAAA0AABAAD4GTDMKcOQKCgv4Cx+QFRqMB7zikNrEU4AS+vCwvAAAAgQFtAEBBAIBAwMH</base64>

10 packets captured
64 packets received by filter
0 packets dropped by kernel

Depening upon where the traffic is coming in some interfaces may not see traffic. The commands run on the edge are

set capture session 0 interface {intid} direction dual
set capture session 0 count 10 expression host {srcip} and host {dstip}

This will timeout after 10 seconds like the previous captures. Next the script checks the T0 DR dowlinks. This is the link between the T0 and the T1. It should see the same traffic as well. If the T1 is not active on the edge node being checked it should not have any traffic.

Checking Downlinks
-------
Router Type: DISTRIBUTED_ROUTER_TIER0
Checking Uplinks
Checking Downlinks
Checking Interface downlink: TNT24-T0-TNT24-T1-t0_lrp ifuuid: debd3dc1-b2e6-4c73-9673-e2b68b45c17f
14:21:35.182955 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5690: Flags [.], ack 3684862599, win 501, length 0
<base64>AlBWVkRSAlBWVkRVCABFAAAoG0BAAD8GL/8KcOQKCgv4Cx+QFjqjvXdp26KKh1AQAfUGMwAA</base64>

14:21:35.183153 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.5690: Flags [P.], seq 0:155, ack 1, win 501, length 155: HTTP: HTTP/1.0 200 OK
<base64>AlBWVkRSAlBWVkRVCABFAADDG0FAAD8GL2MKcOQKCgv4Cx+QFjqjvXdp26KKh1AYAfV4uAAASFRUUC8xLjAgMjAwIE9LDQpTZXJ2ZXI6IFNpbXBsZUhUVFAvMC42IFB5dGhvbi8zLjkuNQ0KRGF0ZTogVGh1LCAyMCBKYW4gMjAyMiAxNDoyMzoxOSBHTVQNCkNvbnRlbnQtdHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04DQpDb250ZW50LUxlbmd0aDogMTA5MQ0KDQo=</base64>

14:21:35.183178 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.5690: Flags [FP.], seq 155:1246, ack 1, win 501, length 1091: HTTP
<base64>AlBWVkRSAlBWVkRVCABFAARrG0JAAD8GK7oKcOQKCgv4Cx+QFjqjvXgE26KKh1AZAfWKQgAAPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvVFIvaHRtbDQvc3RyaWN0LmR0ZCI+CjxodG1sPgo8aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgo8dGl0bGU+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L3RpdGxlPgo8L2hlYWQ+Cjxib2R5Pgo8aDE+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L2gxPgo8aHI+Cjx1bD4KPGxpPjxhIGhyZWY9Ii5iYXNoX2hpc3RvcnkiPi5iYXNoX2hpc3Rvcnk8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5iYXNoX2xvZ291dCI+LmJhc2hfbG9nb3V0PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuYmFzaHJjIj4uYmFzaHJjPC9hPjwvbGk+CjxsaT48YSBocmVmPSIuY2FjaGUvIj4uY2FjaGUvPC9hPjwvbGk+CjxsaT48YSBocmVmPSIucHJvZmlsZSI+LnByb2ZpbGU8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5weXRob25faGlzdG9yeSI+LnB5dGhvbl9oaXN0b3J5PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuc3NoLyI+LnNzaC88L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWwiPi5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWw8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii52aW1pbmZvIj4udmltaW5mbzwvYT48L2xpPgo8bGk+PGEgaHJlZj0iYmFkLnBjYXAiPmJhZC5wY2FwPC9hPjwvbGk+CjxsaT48YSBocmVmPSJiYWQudHh0Ij5iYWQudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGkvIj5lc3hjbGkvPC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyIj5lc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyPC9hPjwvbGk+CjxsaT48YSBocmVmPSJnb29kLnBjYXAiPmdvb2QucGNhcDwvYT48L2xpPgo8bGk+PGEgaHJlZj0iZ29vZC50eHQiPmdvb2QudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJuc3gtaW5ncmVzcy1wYWNrZXQtY2FwdHVyZS8iPm5zeC1pbmdyZXNzLXBhY2tldC1jYXB0dXJlLzwvYT48L2xpPgo8L3VsPgo8aHI+CjwvYm9keT4KPC9odG1sPgo=</base64>

14:21:35.198438 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5689: Flags [.], ack 3560708797, win 501, length 0
<base64>AlBWVkRSAlBWVkRVCABFAAAoAABAAD8GSz8KcOQKCgv4Cx+QFjnn1uPo1DwavVAQAfXMywAA</base64>

14:21:35.207312 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.5691: Flags [S.], seq 3906696770, ack 2993992614, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
<base64>AlBWVkRSAlBWVkRVCABFAAA0AABAAD8GSzMKcOQKCgv4Cx+QFjvo23ZCsnSzpoAS+vCIegAAAgQFtAEBBAIBAwMH</base64>

14:21:35.226928 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5690: Flags [.], ack 2, win 501, length 0
<base64>AlBWVkRSAlBWVkRVCABFAAAoAABAAD8GSz8KcOQKCgv4Cx+QFjqjvXxI26KKiFAQAfUBUwAA</base64>

14:21:35.231042 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5691: Flags [.], ack 164, win 501, length 0
<base64>AlBWVkRSAlBWVkRVCABFAAAoIWRAAD8GKdsKcOQKCgv4Cx+QFjvo23ZDsnS0SVAQAfXBpQAA</base64>

14:21:35.231839 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.5691: Flags [P.], seq 1:156, ack 164, win 501, length 155: HTTP: HTTP/1.0 200 OK
<base64>AlBWVkRSAlBWVkRVCABFAADDIWVAAD8GKT8KcOQKCgv4Cx+QFjvo23ZDsnS0SVAYAfU0KwAASFRUUC8xLjAgMjAwIE9LDQpTZXJ2ZXI6IFNpbXBsZUhUVFAvMC42IFB5dGhvbi8zLjkuNQ0KRGF0ZTogVGh1LCAyMCBKYW4gMjAyMiAxNDoyMzoxOSBHTVQNCkNvbnRlbnQtdHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04DQpDb250ZW50LUxlbmd0aDogMTA5MQ0KDQo=</base64>

14:21:35.231858 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.5691: Flags [FP.], seq 156:1247, ack 164, win 501, length 1091: HTTP
<base64>AlBWVkRSAlBWVkRVCABFAARrIWZAAD8GJZYKcOQKCgv4Cx+QFjvo23besnS0SVAZAfVFtQAAPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvVFIvaHRtbDQvc3RyaWN0LmR0ZCI+CjxodG1sPgo8aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgo8dGl0bGU+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L3RpdGxlPgo8L2hlYWQ+Cjxib2R5Pgo8aDE+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L2gxPgo8aHI+Cjx1bD4KPGxpPjxhIGhyZWY9Ii5iYXNoX2hpc3RvcnkiPi5iYXNoX2hpc3Rvcnk8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5iYXNoX2xvZ291dCI+LmJhc2hfbG9nb3V0PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuYmFzaHJjIj4uYmFzaHJjPC9hPjwvbGk+CjxsaT48YSBocmVmPSIuY2FjaGUvIj4uY2FjaGUvPC9hPjwvbGk+CjxsaT48YSBocmVmPSIucHJvZmlsZSI+LnByb2ZpbGU8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5weXRob25faGlzdG9yeSI+LnB5dGhvbl9oaXN0b3J5PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuc3NoLyI+LnNzaC88L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWwiPi5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWw8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii52aW1pbmZvIj4udmltaW5mbzwvYT48L2xpPgo8bGk+PGEgaHJlZj0iYmFkLnBjYXAiPmJhZC5wY2FwPC9hPjwvbGk+CjxsaT48YSBocmVmPSJiYWQudHh0Ij5iYWQudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGkvIj5lc3hjbGkvPC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyIj5lc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyPC9hPjwvbGk+CjxsaT48YSBocmVmPSJnb29kLnBjYXAiPmdvb2QucGNhcDwvYT48L2xpPgo8bGk+PGEgaHJlZj0iZ29vZC50eHQiPmdvb2QudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJuc3gtaW5ncmVzcy1wYWNrZXQtY2FwdHVyZS8iPm5zeC1pbmdyZXNzLXBhY2tldC1jYXB0dXJlLzwvYT48L2xpPgo8L3VsPgo8aHI+CjwvYm9keT4KPC9odG1sPgo=</base64>

14:21:35.235813 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 66: 10.112.228.10.8080 > 10.11.248.11.5692: Flags [S.], seq 2380187894, ack 4002408216, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
<base64>AlBWVkRSAlBWVkRVCABFAAA0AABAAD8GSzMKcOQKCgv4Cx+QFjyN3sj27o/nGIAS+vAhNQAAAgQFtAEBBAIBAwMH</base64>

10 packets captured
117 packets received by filter
0 packets dropped by kernel

Next the other side of the T0 DR Downlink interface is checked. This is the T1 SR Uplink interface.

Checking Logical Router: SR-TNT24-T1
Router Type: SERVICE_ROUTER_TIER1
Checking Uplinks
Checking Interface uplink: TNT24-T0-TNT24-T1-t1_lrp ifuuid: ae3c1d38-8333-4a3e-b139-94972aa01386
14:21:41.931694 02:50:56:56:44:52 > 02:50:56:56:44:55, ethertype IPv4 (0x0800), length 54: 10.11.248.11.5859 > 10.112.228.10.8080: Flags [.], ack 3977768762, win 2049, length 0
<base64>AlBWVkRVAlBWVkRSCABFAAAomvNAAHsGdEsKC/gLCnDkChbjH5BY1YJ27RfvOlAQCAHJMAAA</base64>

14:21:41.931699 02:50:56:56:44:52 > 02:50:56:56:44:55, ethertype IPv4 (0x0800), length 217: 10.11.248.11.5859 > 10.112.228.10.8080: Flags [P.], seq 0:163, ack 1, win 2049, length 163: HTTP: GET / HTTP/1.1
<base64>AlBWVkRVAlBWVkRSCABFAADLmvRAAHsGc6cKC/gLCnDkChbjH5BY1YJ27RfvOlAYCAGsZwAAR0VUIC8gSFRUUC8xLjENClVzZXItQWdlbnQ6IE1vemlsbGEvNS4wIChXaW5kb3dzIE5UOyBXaW5kb3dzIE5UIDEwLjA7IGVuLVVTKSBXaW5kb3dzUG93ZXJTaGVsbC81LjEuMTc3NjMuMjI2OA0KSG9zdDogMTAuMTEyLjIyOC4xMDo4MDgwDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQoNCg==</base64>

14:21:41.931700 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5859: Flags [.], ack 163, win 501, length 0
<base64>AlBWVkRSAlBWVkRVCABFAAAo5yBAAD8GZB4KcOQKCgv4Cx+QFuPtF+86WNWDGVAQAfXOmQAA</base64>

14:21:41.932288 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 166: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 3678608424:3678608536, ack 393273502, win 4014, length 112
<base64>AlBWVkRSAlBWVkRVCABFEACYrAVAAD8GnrkKcOQKCgv4CwAWBpvbQxwoF3DgnlAYD64WFwAAvZEp6EN5OR409jRkt9IR7oWfKRtfpslKfFjohWvU8ec4FvshJ8AiycQjp15HcpFgw6ztV5EFcXH9gAmxhM1cUx86bQ9dt76qQjkZ9ZuzEoIVP1vlAud0qpZtuMBVW73Rox0bhKfZ/PY7GvXTok/esA==</base64>

14:21:41.932313 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 118: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 112:176, ack 1, win 4014, length 64
<base64>AlBWVkRSAlBWVkRVCABFEABorAZAAD8GnugKcOQKCgv4CwAWBpvbQxyYF3DgnlAYD66R4QAAXJhCpzE1EeThbX/R8w3HqnJ5Zvg6l38SWwjUMaz4LRSjCsmb/F4Uzk3PW+ttJuRj12l5na28QrgU6wRCrb4KnA==</base64>

14:21:41.932342 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.5859: Flags [P.], seq 1:156, ack 163, win 501, length 155: HTTP: HTTP/1.0 200 OK
<base64>AlBWVkRSAlBWVkRVCABFAADD5yFAAD8GY4IKcOQKCgv4Cx+QFuPtF+86WNWDGVAYAfVEHgAASFRUUC8xLjAgMjAwIE9LDQpTZXJ2ZXI6IFNpbXBsZUhUVFAvMC42IFB5dGhvbi8zLjkuNQ0KRGF0ZTogVGh1LCAyMCBKYW4gMjAyMiAxNDoyMzoyNiBHTVQNCkNvbnRlbnQtdHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04DQpDb250ZW50LUxlbmd0aDogMTA5MQ0KDQo=</base64>

14:21:41.932364 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.5859: Flags [FP.], seq 156:1247, ack 163, win 501, length 1091: HTTP
<base64>AlBWVkRSAlBWVkRVCABFAARr5yJAAD8GX9kKcOQKCgv4Cx+QFuPtF+/VWNWDGVAZAfVSqQAAPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvVFIvaHRtbDQvc3RyaWN0LmR0ZCI+CjxodG1sPgo8aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgo8dGl0bGU+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L3RpdGxlPgo8L2hlYWQ+Cjxib2R5Pgo8aDE+RGlyZWN0b3J5IGxpc3RpbmcgZm9yIC88L2gxPgo8aHI+Cjx1bD4KPGxpPjxhIGhyZWY9Ii5iYXNoX2hpc3RvcnkiPi5iYXNoX2hpc3Rvcnk8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5iYXNoX2xvZ291dCI+LmJhc2hfbG9nb3V0PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuYmFzaHJjIj4uYmFzaHJjPC9hPjwvbGk+CjxsaT48YSBocmVmPSIuY2FjaGUvIj4uY2FjaGUvPC9hPjwvbGk+CjxsaT48YSBocmVmPSIucHJvZmlsZSI+LnByb2ZpbGU8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5weXRob25faGlzdG9yeSI+LnB5dGhvbl9oaXN0b3J5PC9hPjwvbGk+CjxsaT48YSBocmVmPSIuc3NoLyI+LnNzaC88L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWwiPi5zdWRvX2FzX2FkbWluX3N1Y2Nlc3NmdWw8L2E+PC9saT4KPGxpPjxhIGhyZWY9Ii52aW1pbmZvIj4udmltaW5mbzwvYT48L2xpPgo8bGk+PGEgaHJlZj0iYmFkLnBjYXAiPmJhZC5wY2FwPC9hPjwvbGk+CjxsaT48YSBocmVmPSJiYWQudHh0Ij5iYWQudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGkvIj5lc3hjbGkvPC9hPjwvbGk+CjxsaT48YSBocmVmPSJlc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyIj5lc3hjbGktNi43LjAtMTMwMDQ3ODctbGluNjQudGFyPC9hPjwvbGk+CjxsaT48YSBocmVmPSJnb29kLnBjYXAiPmdvb2QucGNhcDwvYT48L2xpPgo8bGk+PGEgaHJlZj0iZ29vZC50eHQiPmdvb2QudHh0PC9hPjwvbGk+CjxsaT48YSBocmVmPSJuc3gtaW5ncmVzcy1wYWNrZXQtY2FwdHVyZS8iPm5zeC1pbmdyZXNzLXBhY2tldC1jYXB0dXJlLzwvYT48L2xpPgo8L3VsPgo8aHI+CjwvYm9keT4KPC9odG1sPgo=</base64>

14:21:41.948615 02:50:56:56:44:52 > 02:50:56:56:44:55, ethertype IPv4 (0x0800), length 54: 10.11.248.11.5860 > 10.112.228.10.8080: Flags [.], ack 3694913316, win 2049, length 0
<base64>AlBWVkRVAlBWVkRSCABFAAAomvVAAHsGdEkKC/gLCnDkChbkH5DbYyRU3DvnJFAQCAG9tQAA</base64>

14:21:41.948620 02:50:56:56:44:52 > 02:50:56:56:44:55, ethertype IPv4 (0x0800), length 217: 10.11.248.11.5860 > 10.112.228.10.8080: Flags [P.], seq 0:163, ack 1, win 2049, length 163: HTTP: GET / HTTP/1.1
<base64>AlBWVkRVAlBWVkRSCABFAADLmvZAAHsGc6UKC/gLCnDkChbkH5DbYyRU3DvnJFAYCAGg7AAAR0VUIC8gSFRUUC8xLjENClVzZXItQWdlbnQ6IE1vemlsbGEvNS4wIChXaW5kb3dzIE5UOyBXaW5kb3dzIE5UIDEwLjA7IGVuLVVTKSBXaW5kb3dzUG93ZXJTaGVsbC81LjEuMTc3NjMuMjI2OA0KSG9zdDogMTAuMTEyLjIyOC4xMDo4MDgwDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQoNCg==</base64>

14:21:41.948622 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 54: 10.112.228.10.8080 > 10.11.248.11.5860: Flags [.], ack 163, win 501, length 0
<base64>AlBWVkRSAlBWVkRVCABFAAAowjxAAD8GiQIKcOQKCgv4Cx+QFuTcO+ck22Mk91AQAfXDHgAA</base64>

10 packets captured
319 packets received by filter
0 packets dropped by kernel

Next the T1 DR Downlink to the segments attached to the T1 are checked. I check all of them because I didn’t want to do the API calls to NSX to find the correct interface ID on the correct segment. Or because i’m lazy.

Checking Downlinks
-------
Router Type: DISTRIBUTED_ROUTER_TIER1
Checking Uplinks
Checking Downlinks
Checking Interface downlink: infra-TNT24-HCX-UPLINK-dlrp ifuuid: 81449bdc-8934-4229-a6c3-ef116854c969
Timeout
-------
Checking Interface downlink: infra-segment-1-dlrp ifuuid: efd2baf7-1c77-4c03-bf4c-5f92237c53d8
14:22:07.088033 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 66: 10.11.248.11.6501 > 10.112.228.10.8080: Flags [S], seq 863093262, win 64240, options [mss 1418,nop,wscale 8,nop,nop,sackOK], length 0
<base64>AFBWlvjTAlBWVkRSCABFAQA0qP1AAHoGZzQKC/gLCnDkChllH5AzccIOAAAAAIAC+vBVQwAAAgQFigEDAwgBAQQC</base64>

14:22:07.089244 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 54: 10.11.248.11.6500 > 10.112.228.10.8080: Flags [.], ack 539260963, win 2049, length 0
<base64>AFBWlvjTAlBWVkRSCABFAAAoqP5AAHoGZ0AKC/gLCnDkChlkH5ADCz/MICR4I1AQCAGjLwAA</base64>

14:22:07.089255 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 217: 10.11.248.11.6500 > 10.112.228.10.8080: Flags [P.], seq 0:163, ack 1, win 2049, length 163: HTTP: GET / HTTP/1.1
<base64>AFBWlvjTAlBWVkRSCABFAADLqP9AAHoGZpwKC/gLCnDkChlkH5ADCz/MICR4I1AYCAGGZgAAR0VUIC8gSFRUUC8xLjENClVzZXItQWdlbnQ6IE1vemlsbGEvNS4wIChXaW5kb3dzIE5UOyBXaW5kb3dzIE5UIDEwLjA7IGVuLVVTKSBXaW5kb3dzUG93ZXJTaGVsbC81LjEuMTc3NjMuMjI2OA0KSG9zdDogMTAuMTEyLjIyOC4xMDo4MDgwDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQoNCg==</base64>

14:22:07.112239 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 54: 10.11.248.11.6501 > 10.112.228.10.8080: Flags [.], ack 1610637187, win 2049, length 0
<base64>AFBWlvjTAlBWVkRSCABFAAAoqQBAAHoGZz4KC/gLCnDkChllH5AzccIPYABfg1AQCAHJSAAA</base64>

14:22:07.112250 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 217: 10.11.248.11.6501 > 10.112.228.10.8080: Flags [P.], seq 0:163, ack 1, win 2049, length 163: HTTP: GET / HTTP/1.1
<base64>AFBWlvjTAlBWVkRSCABFAADLqQFAAHoGZpoKC/gLCnDkChllH5AzccIPYABfg1AYCAGsfwAAR0VUIC8gSFRUUC8xLjENClVzZXItQWdlbnQ6IE1vemlsbGEvNS4wIChXaW5kb3dzIE5UOyBXaW5kb3dzIE5UIDEwLjA7IGVuLVVTKSBXaW5kb3dzUG93ZXJTaGVsbC81LjEuMTc3NjMuMjI2OA0KSG9zdDogMTAuMTEyLjIyOC4xMDo4MDgwDQpDb25uZWN0aW9uOiBLZWVwLUFsaXZlDQoNCg==</base64>

14:22:07.113500 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 54: 10.11.248.11.6500 > 10.112.228.10.8080: Flags [.], ack 1248, win 2044, length 0
<base64>AFBWlvjTAlBWVkRSCABFAAAoqQNAAHoGZzsKC/gLCnDkChlkH5ADC0BvICR9AlAQB/ydsgAA</base64>

14:22:07.135474 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 54: 10.11.248.11.6500 > 10.112.228.10.8080: Flags [F.], seq 163, ack 1248, win 2044, length 0
<base64>AFBWlvjTAlBWVkRSCABFAAAoqQRAAHoGZzoKC/gLCnDkChlkH5ADC0BvICR9AlARB/ydsQAA</base64>

14:22:07.136400 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 54: 10.11.248.11.6501 > 10.112.228.10.8080: Flags [.], ack 1248, win 2044, length 0
<base64>AFBWlvjTAlBWVkRSCABFAAAoqQZAAHoGZzgKC/gLCnDkChllH5AzccKyYABkYlAQB/zDywAA</base64>

14:22:07.146053 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 66: 10.11.248.11.6502 > 10.112.228.10.8080: Flags [S], seq 500278107, win 64240, options [mss 1418,nop,wscale 8,nop,nop,sackOK], length 0
<base64>AFBWlvjTAlBWVkRSCABFAQA0qQdAAHoGZyoKC/gLCnDkChlmH5Ad0aNbAAAAAIAC+vCJlQAAAgQFigEDAwgBAQQC</base64>

14:22:07.163272 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 54: 10.11.248.11.6501 > 10.112.228.10.8080: Flags [F.], seq 163, ack 1248, win 2044, length 0
<base64>AFBWlvjTAlBWVkRSCABFAAAoqQhAAHoGZzYKC/gLCnDkChllH5AzccKyYABkYlARB/zDygAA</base64>

10 packets captured
125 packets received by filter
0 packets dropped by kernel


-------
Checking Interface downlink: TNT24-T1-dhcp-dlrp ifuuid: 67e061b9-6ecd-49c6-ad78-fe037e47a3e2
Timeout

Notice the traffic is only seen on the infra-segment-1-dlrp interface. This is the downlink for the segment the destination VMs is attached to. After this step the next edge VM is checked for the same traffic in the same places. While doing all of this VMware was helping troubleshoot another issues that showed the packets leaving the T0 downlink towards the T1 uplink were different. There were drops in the traffic. I don’t have a root cause yet but either the firewall (set to default allow any any) or flow cache misses are the culprit. This shows up in the stats on the edge interfaces fie the rx_firewall_drops counter and the flow_cache_misses counter. In either even the rx_firewall_drops counter increases and is easy to read so I put a check in for that as well. This check is on the T1 SR uplink interface. To capture the data you can run one of two commands:

get firewall {ifuuid} interface stats
get logical-router {vrid} interfaces | json

I chose the second for ease. Then loop through the json object and check the T1 SR uplink interface. I record the inital value and check again in 20 seconds to see if it is increasing.

Check T1 Firewall Drops
Edge VM: TNT24-EVM01 Host: 10.112.225.4
Connect to 10.112.224.7
Checking Logical Router: DR-TNT24-T1
Checking Interface uplink: TNT24-T0-TNT24-T1-t1_lrp ifuuid: ae3c1d38-8333-4a3e-b139-94972aa01386
RX Firewall Drops: 9
RX Firewall Drops: 9
rx_fw_drops not increasing

If this is increasing and there are no firewall rules something is broken. You can disable the T1 firewall as a workaround but you should call VMware to find out why you are seeing drops. Finally the script checks the destination VM host switch port for the same traffic

DST VM: ubuntu-template Host: 10.112.225.2
Connect to 10.112.225.3
World ID: 2107955
Switchport ID: 33554443
Timeout
Switchport ID: 33554444
Timeout
Switchport ID: 50331658
14:26:30.124277 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 60: 10.11.248.11.1691 > 10.112.228.10.22: Flags [.], ack 3679224712, win 4098, length 0
14:26:30.124278 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 60: 10.11.248.11.10506 > 10.112.228.10.8080: Flags [.], ack 2117387531, win 2044, length 0
14:26:30.129502 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 60: 10.11.248.11.10507 > 10.112.228.10.8080: Flags [.], ack 2602021991, win 2049, length 0
14:26:30.129503 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 217: 10.11.248.11.10507 > 10.112.228.10.8080: Flags [P.], seq 0:163, ack 1, win 2049, length 163: HTTP: GET / HTTP/1.1
14:26:30.129553 00:50:56:96:f8:d3 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 60: 10.112.228.10.8080 > 10.11.248.11.10507: Flags [.], ack 163, win 501, length 0
14:26:30.130330 00:50:56:96:f8:d3 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 166: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 1:113, ack 0, win 4014, length 112
14:26:30.130366 00:50:56:96:f8:d3 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 118: 10.112.228.10.22 > 10.11.248.11.1691: Flags [P.], seq 113:177, ack 0, win 4014, length 64
14:26:30.130434 00:50:56:96:f8:d3 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 209: 10.112.228.10.8080 > 10.11.248.11.10507: Flags [P.], seq 1:156, ack 163, win 501, length 155: HTTP: HTTP/1.0 200 OK
14:26:30.130478 00:50:56:96:f8:d3 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 1145: 10.112.228.10.8080 > 10.11.248.11.10507: Flags [FP.], seq 156:1247, ack 163, win 501, length 1091: HTTP
14:26:30.144443 02:50:56:56:44:52 > 00:50:56:96:f8:d3, ethertype IPv4 (0x0800), length 60: 10.11.248.11.10506 > 10.112.228.10.8080: Flags [F.], seq 0, ack 1, win 2044, length 0

The repo is here https://github.com/khensler/nsx-ingress-packetcapture. It has the community sames tools from pyvmomi in it for a couple helper functions.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: