Messages

I am doing outbound filtering on some interfaces. And I seem to have struck a similar issue to this person.

Eg. I have this rule on an interface:                                               
                                                                               

Code Select Expand

root@opnsense-test:~ # pfctl -vvs rules | grep -A 3 "drop out log quick on ixl0"
@195 block drop out log quick on ixl0 reply-to (ixl0 192.0.2.1) inet all label "83448bc8-ca97-401d-86f2-57d67fb16fb2"
  [ Evaluations: 211       Packets: 211       Bytes: 22618       States: 0     ]
  [ Inserted: uid 0 pid 90386 State Creations: 0     ]                         
  [ Last Active Time: Wed Aug 26 05:53:29 2026 ]                               
                                                                                                                                                               
I can allow traffic out of that interface with the following rule:             
                                                                               

Code Select Expand

root@opnsense-test:~ # pfctl -vvs rules | grep -A 3 5443b2cb-c3ca-4e0c-81cf-88c04c85f00f
@120 pass out log quick inet from (self:10) to any flags S/SA keep state label "5443b2cb-c3ca-4e0c-81cf-88c04c85f00f"
  [ Evaluations: 11        Packets: 18        Bytes: 1946        States: 4     ]
  [ Inserted: uid 0 pid 24 State Creations: 11    ]                             
  [ Last Active Time: Wed Aug 26 05:45:02 2026 ]                                
                                                                                                                                                               
And then I can establish a TCP connection:                                     
                                                                               

Code Select Expand

root@opnsense-test:~ # telnet 192.0.2.49 8443                                   
Trying 192.0.2.49...                                                           
Connected to 192.0.2.49.                                                       
Escape character is '^]'.                                                       
^]                                                                             
telnet> Connection closed.                                                      
                                                                               
And I can see the states in PF:                                                 
                                                                               

Code Select Expand

root@opnsense-test:~ # pfctl -vvs states | grep -A3 192.0.2.49                 
all tcp 192.0.2.47:45666 -> 192.0.2.49:8443       ESTABLISHED:ESTABLISHED       
   [3811272169 + 16711936] wscale 7  [181588979 + 65792] wscale 7               
   age 00:00:02, expires in 23:59:58, 2:1 pkts, 112:60 bytes, rule 120         
   id: bddd8c6a00000000 creatorid: 3610e121                                     
                                                                                                                                                             
However, if I replace the above floating rule with one that matches the interface:       
                                                                               

Code Select Expand

@148 pass out quick on ixl0 reply-to (ixl0 192.0.2.1) inet all flags S/SA keep state label "ed13d8d2-e403-4cf7-ba41-6b698f76643e"
  [ Evaluations: 36        Packets: 33        Bytes: 3751        States: 22    ]
  [ Inserted: uid 0 pid 70434 State Creations: 22    ]                         
  [ Last Active Time: Wed Aug 26 06:14:36 2026                                  
                                                                                                                                                               
I get responses, but the local firewall never completes the TCP handshake:         
                                                                               

Code Select Expand

root@opnsense-test:~ # telnet 192.0.2.49 8443                                   
Trying 192.0.2.49...                                                           
^C                                                                             
root@opnsense-test:~ # pfctl -vvs states | grep -A 3 192.0.2.49                 
all tcp 192.0.2.47:59825 -> 192.0.2.49:8443       ESTABLISHED:SYN_SENT         
   [3105104415 + 4294902016] wscale 7  [4173125645 + 4294902016] wscale 7       
   age 00:00:10, expires in 00:00:28, 2:5 pkts, 120:300 bytes, rule 148         
   id: 07de8c6a00000000 creatorid: 3610e121 reply-to: 192.0.2.1@ixl0

Code Select Expand

root@opnsense-test:~ # tcpdump -nevi ixl0 host 192.0.2.49   
tcpdump: listening on ixl0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
06:17:38.604709 38:68:dd:66:ac:e8 > 38:68:dd:66:7d:b0, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl
64, id 0, offset 0, flags [DF], proto TCP (6), length 60)                       
    192.0.2.47.50028 > 192.0.2.49.8443: Flags [S], cksum 0xff66 (correct), seq 4278843326, win 652
28, options [mss 1460,nop,wscale 7,sackOK,TS val 2155498607 ecr 0], length 0   
06:17:38.604815 38:68:dd:66:7d:b0 > 38:68:dd:66:ac:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 6
4, id 0, offset 0, flags [DF], proto TCP (6), length 60)                       
    192.0.2.49.8443 > 192.0.2.47.50028: Flags [S.], cksum 0x5bb6 (correct), seq 1043354308, ack 42
78843327, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2956288629 ecr 2155498607], length 0
06:17:39.628574 38:68:dd:66:ac:e8 > 38:68:dd:66:7d:b0, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl
64, id 0, offset 0, flags [DF], proto TCP (6), length 60)                       
    192.0.2.47.50028 > 192.0.2.49.8443: Flags [S], cksum 0xfb66 (correct), seq 4278843326, win 652
28, options [mss 1460,nop,wscale 7,sackOK,TS val 2155499631 ecr 0], length 0   
06:17:39.628651 38:68:dd:66:7d:b0 > 38:68:dd:66:ac:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 6
4, id 0, offset 0, flags [DF], proto TCP (6), length 60)                       
    192.0.2.49.8443 > 192.0.2.47.50028: Flags [S.], cksum 0x57b6 (correct), seq 1043354308, ack 4$
78843327, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2956289653 ecr 2155498607], length 0
06:17:39.628762 38:68:dd:66:7d:b0 > 38:68:dd:66:ac:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 6
4, id 0, offset 0, flags [DF], proto TCP (6), length 60)                       
    192.0.2.49.8443 > 192.0.2.47.50028: Flags [S.], cksum 0x53b6 (correct), seq 1043354308, ack 42
78843327, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2956289653 ecr 2155499631], length 0
06:17:40.690948 38:68:dd:66:7d:b0 > 38:68:dd:66:ac:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 6
4, id 0, offset 0, flags [DF], proto TCP (6), length 60)                       
    192.0.2.49.8443 > 192.0.2.47.50028: Flags [S.], cksum 0x4f90 (correct), seq 1043354308, ack 42
78843327, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2956290715 ecr 2155499631], length 0
This is even worse when attempting to connect to something not on the same subnet as my device, where I never even see the Syn packet:
[Edit] I successfully recreated this now.

Code Select Expand

root@opnsense-test:~ # telnet 198.51.100.1 8443
Trying 198.51.100.1...
^C

root@opnsense-test:~ # pfctl -vvs states | grep -A 3 198.51             
No ALTQ support in kernel                                                       
ALTQ related functions disabled                                                 
all tcp 192.0.2.47:18469 -> 198.51.100.1:8443       ESTABLISHED:SYN_SENT       
   [1406893479 + 4294705408] wscale 7  [2561759781 + 4294902016] wscale 7       
   age 00:01:11, expires in 00:00:23, 7:12 pkts, 420:720 bytes, rule 146       
   id: 8de98c6a00000000 creatorid: 3610e121 reply-to: 192.0.2.1@ixl0


Code Select Expand

root@opnsense-test:~ # tcpdump -nevi ixl0 tcp port 8443                                       
tcpdump: listening on ixl0, link-type EN10MB (Ethernet), snapshot length 262144 bytes               
21:55:52.902296 38:68:dd:66:ac:e8 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.0.2.47.61030 > 198.51.100.1.8443: Flags [S], cksum 0xef3a (correct), seq 2484399761, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 416987146 ecr 0], length 0
21:55:52.912149 e2:39:b3:4e:bc:47 > 38:68:dd:66:ac:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    198.51.100.1.8443 > 192.0.2.47.61030: Flags [S.], cksum 0xd0d9 (correct), seq 3967036082, ack 2484399762, win 64492, options [mss 1414,sackOK,TS val 3376431350 ecr 416987146,nop,wscale 7], length 0
21:55:53.902416 38:68:dd:66:ac:e8 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.0.2.47.61030 > 198.51.100.1.8443: Flags [S], cksum 0xeb52 (correct), seq 2484399761, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 416988146 ecr 0], length 0
21:55:53.912051 e2:39:b3:4e:bc:47 > 38:68:dd:66:ac:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    198.51.100.1.8443 > 192.0.2.47.61030: Flags [S.], cksum 0xccf1 (correct), seq 3967036082, ack 2484399762, win 64492, options [mss 1414,sackOK,TS val 3376432350 ecr 416987146,nop,wscale 7], length 0
21:55:54.940966 e2:39:b3:4e:bc:47 > 38:68:dd:66:ac:e8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    198.51.100.1.8443 > 192.0.2.47.61030: Flags [S.], cksum 0xc8ec (correct), seq 3967036082, ack 2484399762, win 64492, options [mss 1414,sackOK,TS val 3376433379 ecr 416987146,nop,wscale 7], length 0
^C Though, I have sometimes seen the same behaviour as with local addresses (Syns and Syn-Acks but nothing more), I just haven't been able to recreate it for the sake of this post.
                                                                               
The issue appears to be that any rule on an interface with a gateway configured (inbound or outbound) will have a reply-to value created automatically by OPNsense. You can disable this with the Disable reply-to check-box in the Edit-rule pop-up, which fixes the problem.                                                                           
                                                                               
But from my reading of man PF:

Code Select Expand

reply-to
   The reply-to  option is similar to route-to, but routes packets that
   pass in the opposite direction (replies) to the specified  interface.
   Opposite  direction is only defined in the context of a state entry,
   and reply-to is useful only in rules that create state.   It  can  be
   used  on systems with multiple external connections to route all out-
   going packets of a connection through the interface the incoming con-
   nection arrived through (symmetric routing enforcement).It seems to me that the reply-to field on an outbound rule doesn't make sense. Shouldn't replies always be coming back to the firewall itself for an outbound rule? Or am I misunderstanding the purpose of the reply-to field here?

For reference:

Code Select Expand

Type opnsense-business
Version 26.4.1p2
Architecture amd64

Not sure if I can really go into the specifics here (in part because I am not really across them, my brief is getting the office firewalls working).

But in the general case you might have a mismatch between the performance, scaling, service levels, operational requirements, etc. required for the office and what's required for the services on those hosts, and putting a host behind a server running PF isn't necessarily a significantly more secure set-up than running PF on the host itself.

Thanks, but I believe you are mistaken. There is no switching loop and Flooding unknown unicast traffic occurs naturally on every layer 2 network. The excessive unknown unicast is being generated when the firewalls route packets with mismatched ethernet destinations that they should be discarding. The TCPDump above demonstrates this happening.

Moreover, if there was a switching loop we would see a lot broader impact, we wouldnt see TTL expired messages, we would see more than just a couple of duplicate packets most of the time, and the issue wouldn't come and go regularly in the way that it was.

Storm control on the firewall interfaces would interfere with CARP. UUFB also would interfere with connectivity to the firewalls, especially as they use two MAC addresses (the interface MAC and the CARP MAC).

Forwarding incorrect packets is a documented bug in FreeBSD that will affect any OPNSense deployment that enables netflow and CARP on the same interface.

It appears the issue is this bug:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=295415

OPNSense is using NG_Netflow, and Netgraph bypasses the logic that marks promiscuous packets so they aren't forwarded by upper layers.

The only practical resolution I can see is to craft firewall rules to prevent the traffic from being forwarded.

Logically its a flat layer 2 network, with the two firewalls, the hosts and the gateways all connected to some Cisco switches running spanning tree.

So effectively:

Code Select Expand

border-routera   border-routerb   dmzhosta    dmzhost<x>
      |                 |            |           |
      --------------------------------------------
            |               |              |
         office-fwa    office-fwa       dmzhostb
The default gateway for everything points at a VRRP address shared between border-router1 and border-router2.
The office firewalls run CARP, and the border routers and the hosts have routes to the office-fw1 address ranges pointing at the CARP address.

Hi,

I am having an issue where OPNSense is creating duplicate packets every time there is unknown unicast on our network. The cause seems to be the firewalls attempting to forward any packets that arrive on their interface, even if those packets aren't sent to the firewall's MAC address.
I found this topic, but there was no resolution. This differs slightly from our setup, as our firewalls are running on physical hardware.

This caused an outage for us in the following scenario:

In our DMZ we have two border routers running linux, an HA pair of OPNSense office firewalls running CARP, and a few DMZ hosts.

The border routers were responding to ARP requests for the gateway address twice, once with the physical interface MAC address and once with a linux bridge MAC address, but would only send packets with the physical interface MAC.

If a host learnt the bridge MAC, it would send packets to that MAC address, but because there would be no replies from that MAC address, the switches would eventually time out their MAC table entry, and would start flooding the unknown unicast.

The flooded packets would arrive at (both of) the OPNSense firewalls, and they would forward them on to the gateway, creating duplicate packets.

However, if the OPNSense firewalls also learnt the bridge MAC address, their forwarded packets would also be broadcast. This would create a routing loop, where we would generate 128 copies of every packet, as well as a bunch of TTL exceeded ICMP.

We have resolved this issue by stopping the gateways from responding twice to ARP requests, but potentially we could see a similar problem any time there is unknown unicast on the network, for instance if a host went down.

Here's a (anonymised) tcpdump (this was taken after the outage, during testing, when the firewall had learnt the correct MAC address).

Code Select Expand

root@office-fwa:~ # tcpdump -nevi lagg0 host 192.0.2.1 and icmp                 
tcpdump: listening on lagg0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:44:29.101744 22:22:22:22:22:22 > 44:44:44:44:44:44, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 25992, offset 0, flags [DF], proto ICMP (1), length 84)                   
    192.0.2.2 > 192.0.2.1: ICMP echo request, id 40050, seq 403, length 64     
12:44:29.101747 66:66:66:66:66:66 > 88:88:88:88:88:88, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 25992, offset 0, flags [DF], proto ICMP (1), length 84)                   
    192.0.2.2 > 192.0.2.1: ICMP echo request, id 40050, seq 403, length 64In this example 192.0.2.1 is the gateway, 192.0.2.2 is the host. 22:22:22:22:22:22 is the host MAC, 44:44:44:44:44:44 is the router bridge MAC, 88:88:88:88:88:88 is the router physical interface MAC, and 66:66:66:66:66:66 is the firewall MAC.

ifconfig suggests we have the IFF_PROMISC flag set on this interface (and, as best as I can tell, not IFF_PPROMISC):

Code Select Expand

root@office-fwa:~ # ifconfig lagg0
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500It seems that this is because we are running CARP.
Looking through the source code, this comment suggests that promiscuous packets shouldn't be forwarded. And the logic for that appears to be here.

However, for whatever reason, it seems that these packets are being forwarded.

I am running:
Type opnsense-business
Version 26.4_6
Architecture amd64

And we have these plugins:

os-caddy (installed)
os-cpu-microcode-intel
os-frr (installed)
os-lldpd (misconfigured)
os-net-snmp (installed)
os-OPNBEcore (installed)
os-OPNWAF (installed)
os-relayd (installed)
os-zabbix7-agent (installed)


Do you have any suggestions what would be causing these packets to be forwarded?