Messages

Hi,

I am having an issue where OPNSense is creating duplicate packets every time there is unknown unicast on our network. The cause seems to be the firewalls attempting to forward any packets that arrive on their interface, even if those packets aren't sent to the firewall's MAC address.
I found this topic, but there was no resolution. This differs slightly from our setup, as our firewalls are running on physical hardware.

This caused an outage for us in the following scenario:

In our DMZ we have two border routers running linux, an HA pair of OPNSense office firewalls running CARP, and a few DMZ hosts.

The border routers were responding to ARP requests for the gateway address twice, once with the physical interface MAC address and once with a linux bridge MAC address, but would only send packets with the physical interface MAC.

If a host learnt the bridge MAC, it would send packets to that MAC address, but because there would be no replies from that MAC address, the switches would eventually time out their MAC table entry, and would start flooding the unknown unicast.

The flooded packets would arrive at (both of) the OPNSense firewalls, and they would forward them on to the gateway, creating duplicate packets.

However, if the OPNSense firewalls also learnt the bridge MAC address, their forwarded packets would also be broadcast. This would create a routing loop, where we would generate 128 copies of every packet, as well as a bunch of TTL exceeded ICMP.

We have resolved this issue by stopping the gateways from responding twice to ARP requests, but potentially we could see a similar problem any time there is unknown unicast on the network, for instance if a host went down.

Here's a (anonymised) tcpdump (this was taken after the outage, during testing, when the firewall had learnt the correct MAC address).

Code Select Expand

root@office-fwa:~ # tcpdump -nevi lagg0 host 192.0.2.1 and icmp                 
tcpdump: listening on lagg0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:44:29.101744 22:22:22:22:22:22 > 44:44:44:44:44:44, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 25992, offset 0, flags [DF], proto ICMP (1), length 84)                   
    192.0.2.2 > 192.0.2.1: ICMP echo request, id 40050, seq 403, length 64     
12:44:29.101747 66:66:66:66:66:66 > 88:88:88:88:88:88, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 25992, offset 0, flags [DF], proto ICMP (1), length 84)                   
    192.0.2.2 > 192.0.2.1: ICMP echo request, id 40050, seq 403, length 64In this example 192.0.2.1 is the gateway, 192.0.2.2 is the host. 22:22:22:22:22:22 is the host MAC, 44:44:44:44:44:44 is the router bridge MAC, 88:88:88:88:88:88 is the router physical interface MAC, and 66:66:66:66:66:66 is the firewall MAC.

ifconfig suggests we have the IFF_PROMISC flag set on this interface (and, as best as I can tell, not IFF_PPROMISC):

Code Select Expand

root@office-fwa:~ # ifconfig lagg0
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500It seems that this is because we are running CARP.
Looking through the source code, this comment suggests that promiscuous packets shouldn't be forwarded. And the logic for that appears to be here.

However, for whatever reason, it seems that these packets are being forwarded.

I am running:
Type opnsense-business
Version 26.4_6
Architecture amd64

And we have these plugins:

os-caddy (installed)
os-cpu-microcode-intel
os-frr (installed)
os-lldpd (misconfigured)
os-net-snmp (installed)
os-OPNBEcore (installed)
os-OPNWAF (installed)
os-relayd (installed)
os-zabbix7-agent (installed)


Do you have any suggestions what would be causing these packets to be forwarded?