Extremely Weird Ping Behaviour

[TotalGriffLock]

This is probably an OS issue rather than an OPNsense issue but here goes...

I have 8 OPNsense firewalls (4 pairs, 2 in each location). The firewalls are Deciso hardware. They are connected together via a layer 3 cloud (think MPLS/VPLS type service). Presentation to the firewalls is GBE copper as a tagged VLAN. This service uses a /24 for IP addressing, each firewall has its own IP and each pair shares a CARP IP. Layout is :

10.x.x.1  Site 1 VIP
10.x.x.2  Site 1 FW 1
10.x.x.3  Site 1 FW 2
10.x.x.4  Site 2 VIP
10.x.x.5  Site 2 FW 1
10.x.x.6  Site 2 FW 2
10.x.x.10 Site 3 VIP
10.x.x.11 Site 3 FW 1
10.x.x.12 Site 3 FW 1
10.x.x.13 Site 4 VIP
10.x.x.14 Site 4 FW 1
10.x.x.15 Site 5 FW 2

Each firewall has a firewall rule on the interface for this zone that permits all ICMP. Each firewall can ping (either from the CLI or dpinger via gateway monitoring) each other individual firewall. Each firewall can ping each other individual VIP except the one ending in .1, where pinging this address results in 128 duplicate packets being sent. The number 128 seems significant to me.

I did a packet capture on Site 3 FW 1 and what I see is the below:

Code Select Expand

14:14:35.094594 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097248 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097250 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097520 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097566 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097813 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097815 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097816 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097817 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097817 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097846 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.097847 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098106 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098107 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098108 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098109 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098110 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098111 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098112 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098112 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098135 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098387 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098389 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098389 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098390 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098391 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098392 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098393 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098394 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098394 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098708 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098710 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098711 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098712 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098713 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098713 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098714 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098715 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098743 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098744 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.098791 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099046 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099047 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099048 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099049 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099050 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099051 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099051 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099052 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099053 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099083 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099084 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099341 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099342 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099343 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099344 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099345 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099346 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099347 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099348 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099348 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099349 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099376 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099618 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099619 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099620 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099621 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099622 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099623 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099624 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099624 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099625 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099626 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099627 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099686 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099913 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099914 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099915 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099916 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099917 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099918 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099918 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099919 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099920 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.099957 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100211 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100212 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100213 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100214 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100215 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100216 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100217 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100218 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100218 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100219 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100220 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100221 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100222 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100253 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100532 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100533 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100534 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100535 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100536 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100537 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100537 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100538 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100539 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100568 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100826 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100827 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100828 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100829 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100830 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100830 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100831 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100832 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100833 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100834 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100869 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.100870 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.101140 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.101142 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.101143 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.101144 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.101145 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9
14:14:35.101145 IP 10.x.x.11 > 10.x.x.1: ICMP echo request, id 42659, seq 26, length 9


Important observations, same sequence number on all 128 pings, and these are being sent from the firewall initiating the ping. These are duplicate requests being generated by the device I run tcpdump on, they are not duplicate responses. This capture was performed on the device ending .11. Both CLI/OS ping command and dpinger exhibit the same response (I expect dpinger is just executing ping behind the scenes). All other firewalls connected to this service exhibit the same behaviour when pinging .1, and only .1.

WTF?!

[pfry]

[...]
The number 128 seems significant to me. [...]

Right, the immediate thought is off-by-one or integer rounding or truncation. I can't think of an obvious way for this to occur (no good place to start).
Another thought would be some sort of packet repetition killed by TTL, but I would expect OPNsense's TTL to be 64.

[TotalGriffLock]

My money is on either the NIC driver or something being introduced through FBSD's VLAN handling. Or a combination of the two. It has to be happening on the source device when the packet is originally generated so it has to be OS, I think.