Messages

Not sure if I can really go into the specifics here (in part because I am not really across them, my brief is getting the office firewalls working).

But in the general case you might have a mismatch between the performance, scaling, service levels, operational requirements, etc. required for the office and what's required for the services on those hosts, and putting a host behind a server running PF isn't necessarily a significantly more secure set-up than running PF on the host itself.

Thanks, but I believe you are mistaken. There is no switching loop and Flooding unknown unicast traffic occurs naturally on every layer 2 network. The excessive unknown unicast is being generated when the firewalls route packets with mismatched ethernet destinations that they should be discarding. The TCPDump above demonstrates this happening.

Moreover, if there was a switching loop we would see a lot broader impact, we wouldnt see TTL expired messages, we would see more than just a couple of duplicate packets most of the time, and the issue wouldn't come and go regularly in the way that it was.

Storm control on the firewall interfaces would interfere with CARP. UUFB also would interfere with connectivity to the firewalls, especially as they use two MAC addresses (the interface MAC and the CARP MAC).

Forwarding incorrect packets is a documented bug in FreeBSD that will affect any OPNSense deployment that enables netflow and CARP on the same interface.

It appears the issue is this bug:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=295415

OPNSense is using NG_Netflow, and Netgraph bypasses the logic that marks promiscuous packets so they aren't forwarded by upper layers.

The only practical resolution I can see is to craft firewall rules to prevent the traffic from being forwarded.

Logically its a flat layer 2 network, with the two firewalls, the hosts and the gateways all connected to some Cisco switches running spanning tree.

So effectively:

Code Select Expand

border-routera   border-routerb   dmzhosta    dmzhost<x>
      |                 |            |           |
      --------------------------------------------
            |               |              |
         office-fwa    office-fwa       dmzhostb
The default gateway for everything points at a VRRP address shared between border-router1 and border-router2.
The office firewalls run CARP, and the border routers and the hosts have routes to the office-fw1 address ranges pointing at the CARP address.

Hi,

I am having an issue where OPNSense is creating duplicate packets every time there is unknown unicast on our network. The cause seems to be the firewalls attempting to forward any packets that arrive on their interface, even if those packets aren't sent to the firewall's MAC address.
I found this topic, but there was no resolution. This differs slightly from our setup, as our firewalls are running on physical hardware.

This caused an outage for us in the following scenario:

In our DMZ we have two border routers running linux, an HA pair of OPNSense office firewalls running CARP, and a few DMZ hosts.

The border routers were responding to ARP requests for the gateway address twice, once with the physical interface MAC address and once with a linux bridge MAC address, but would only send packets with the physical interface MAC.

If a host learnt the bridge MAC, it would send packets to that MAC address, but because there would be no replies from that MAC address, the switches would eventually time out their MAC table entry, and would start flooding the unknown unicast.

The flooded packets would arrive at (both of) the OPNSense firewalls, and they would forward them on to the gateway, creating duplicate packets.

However, if the OPNSense firewalls also learnt the bridge MAC address, their forwarded packets would also be broadcast. This would create a routing loop, where we would generate 128 copies of every packet, as well as a bunch of TTL exceeded ICMP.

We have resolved this issue by stopping the gateways from responding twice to ARP requests, but potentially we could see a similar problem any time there is unknown unicast on the network, for instance if a host went down.

Here's a (anonymised) tcpdump (this was taken after the outage, during testing, when the firewall had learnt the correct MAC address).

Code Select Expand

root@office-fwa:~ # tcpdump -nevi lagg0 host 192.0.2.1 and icmp                 
tcpdump: listening on lagg0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:44:29.101744 22:22:22:22:22:22 > 44:44:44:44:44:44, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 25992, offset 0, flags [DF], proto ICMP (1), length 84)                   
    192.0.2.2 > 192.0.2.1: ICMP echo request, id 40050, seq 403, length 64     
12:44:29.101747 66:66:66:66:66:66 > 88:88:88:88:88:88, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 25992, offset 0, flags [DF], proto ICMP (1), length 84)                   
    192.0.2.2 > 192.0.2.1: ICMP echo request, id 40050, seq 403, length 64In this example 192.0.2.1 is the gateway, 192.0.2.2 is the host. 22:22:22:22:22:22 is the host MAC, 44:44:44:44:44:44 is the router bridge MAC, 88:88:88:88:88:88 is the router physical interface MAC, and 66:66:66:66:66:66 is the firewall MAC.

ifconfig suggests we have the IFF_PROMISC flag set on this interface (and, as best as I can tell, not IFF_PPROMISC):

Code Select Expand

root@office-fwa:~ # ifconfig lagg0
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500It seems that this is because we are running CARP.
Looking through the source code, this comment suggests that promiscuous packets shouldn't be forwarded. And the logic for that appears to be here.

However, for whatever reason, it seems that these packets are being forwarded.

I am running:
Type opnsense-business
Version 26.4_6
Architecture amd64

And we have these plugins:

os-caddy (installed)
os-cpu-microcode-intel
os-frr (installed)
os-lldpd (misconfigured)
os-net-snmp (installed)
os-OPNBEcore (installed)
os-OPNWAF (installed)
os-relayd (installed)
os-zabbix7-agent (installed)


Do you have any suggestions what would be causing these packets to be forwarded?