Hi,
I have an opnsense cluster (21.1.4) with one node being physical and the other one being virtualized on VMWare ESXi 7.0.1.
The vSwitches have only one uplink, promiscuous mode, forged transmits and MAC changes are allowed.
Now, when I ping between two virtualized hosts on the same subnet, and CARP is enabled, the virtualized firewall duplicates all packets. If I disable CARP, all is well.
10.0.0.253 -> ping -> 10.0.0.253, the firewall is 10.0.0.3
I am pretty sure that this is not the known problem with VMWare and multiple uplinks, since I only have a single uplink on those vSwitches.
The ping:
64 bytes from 10.0.0.252: icmp_seq=38 ttl=128 time=0.338 ms
64 bytes from 10.0.0.252: icmp_seq=38 ttl=128 time=0.578 ms (DUP!)
64 bytes from 10.0.0.252: icmp_seq=38 ttl=127 time=0.667 ms (DUP!)
64 bytes from 10.0.0.252: icmp_seq=38 ttl=127 time=0.716 ms (DUP!)
TCPDUMP on the pinging host:
14:07:31.856468 00:0c:29:5a:57:ba > 00:0c:29:21:61:69, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 15016, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.0.253 > 10.0.0.252: ICMP echo request, id 24, seq 41, length 64
14:07:31.857056 00:0c:29:21:61:69 > 00:0c:29:5a:57:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 128, id 10011, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.252 > 10.0.0.253: ICMP echo reply, id 24, seq 41, length 64
14:07:31.857079 00:0c:29:e9:22:72 > 00:0c:29:5a:57:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 127, id 10011, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.252 > 10.0.0.253: ICMP echo reply, id 24, seq 41, length 64
14:07:31.857083 00:0c:29:21:61:69 > 00:0c:29:5a:57:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 128, id 10012, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.252 > 10.0.0.253: ICMP echo reply, id 24, seq 41, length 64
14:07:31.857202 00:0c:29:e9:22:72 > 00:0c:29:5a:57:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 127, id 10012, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.252 > 10.0.0.253: ICMP echo reply, id 24, seq 41, length 64
TCPDUMP on the firewall:
14:07:31.857652 00:0c:29:e9:22:72 > 00:0c:29:21:61:69, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 15016, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.0.253 > 10.0.0.252: ICMP echo request, id 24, seq 41, length 64
14:07:31.857716 00:0c:29:21:61:69 > 00:0c:29:5a:57:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 128, id 10011, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.252 > 10.0.0.253: ICMP echo reply, id 24, seq 41, length 64
14:07:31.857751 00:0c:29:e9:22:72 > 00:0c:29:5a:57:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 127, id 10011, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.252 > 10.0.0.253: ICMP echo reply, id 24, seq 41, length 64
14:07:31.857907 00:0c:29:21:61:69 > 00:0c:29:5a:57:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 128, id 10012, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.252 > 10.0.0.253: ICMP echo reply, id 24, seq 41, length 64
14:07:31.857936 00:0c:29:e9:22:72 > 00:0c:29:5a:57:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 127, id 10012, offset 0, flags [none], proto ICMP (1), length 84)
10.0.0.252 > 10.0.0.253: ICMP echo reply, id 24, seq 41, length 64
Best Wishes
Just use the VM as a cold standby and restore the config from time to time. Having different interface drivers is not recommended
That may well be, but is not at all the point of my post.