Messages

[Symptoms of bug:]

Good Morning,

Several months ago I ran into an issue on my opnsense firewall. I noticed that "sometimes" I would be unable to create new VLANs without rebooting the host device.

The host device has an Intel I226-V 4 port NIC on it. I chalked it up to likely be a known incompatibility with this NIC (example with more info: https://forum.opnsense.org/index.php?topic=38055.0)

Recently - I redeployed this firewall as a high availability pair. This new pair has Intel X540-AT2 2 port NICs on them. The X540 appears to be *far* better supported without any known issues (that I can find) https://bsd-hardware.info/?id=pci:8086-1528-15d9-1528

Symptoms of bug:

Using tcpdump or similar:

When creating a new VLAN interface and enabling it with IP address configured, the interface is unable to communicate to any other hosts on the same VLAN.

When viewing traffic on the VLAN interface from the opnsense host the ARP packets are observed as being sent.

When viewing traffic on the switch (or any other host on the same VLAN), the ARP packets are observed as being sent by opnsense. From the switch (or ping target host), it can also be observed that ARP packets are being sent back to opnsense.

When viewing traffic on the VLAN interface on opnsense the response packets are never received. When viewing traffic on the master interface for the VLAN the responses are received (and tagged correctly), but are never received by the VLAN interface.

When viewing dmesg, or the serial console, the following are printed as long as traffic is being directed towards opnsense on the newly created VLAN interface:

Code Select Expand

**REPEATING INFINITELY**
056.634313 [ 323] freebsd_generic_rx_handler Warning: RX packet intercepted, but no emulated adapter
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
057.651332 [ 323] freebsd_generic_rx_handler Warning: RX packet intercepted, but no emulated adapter
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
058.331450 [ 323] freebsd_generic_rx_handler Warning: RX packet intercepted, but no emulated adapter
059.671444 [ 323] freebsd_generic_rx_handler Warning: RX packet intercepted, but no emulated adapter
060.686962 [ 323] freebsd_generic_rx_handler Warning: RX packet intercepted, but no emulated adapter
061.131317 [ 323] freebsd_generic_rx_handler Warning: RX packet intercepted, but no emulated adapter
**REPEATING INFINITELY**

Disabling and re-enabling the VLAN interface does not restore connectivity.

Rebooting the opnsense device does restore connectivity.

Alternatively: Disabling the suricata service entirely, disabling the newly created VLAN interface, and reenabling the newly creating VLAN interface also restores connectivity; however, this is not 100% reliable, sometimes this does not work, still requiring a reboot.

Full reproduction steps:

WITH Suricata enabled in IDS AND Promiscuous Mode on the master interface for the VLANs (as recommended for use with VLANs) - https://docs.opnsense.org/manual/ips.html

when using VLAN's, enable IPS on the parent

Create a new VLAN interface and attach to parent

Assign interface and configure IP address settings

Attempt to communicate with other hosts on the VLAN

Communication fails

EITHER: Reboot opnsense node, or disable suricata and disable / enable new VLAN interface (not 100% reliable).

Communication now works

EXCEPTION: When reenabling suricata WITHOUT a reboot, communication will again cease to function AND the error messages on console / dmesg return. A reboot is required to fully restore all expected functionality.

WORKAROUND: To keep suricata enabled and for network protection when adding new VLANs, ensure the CARP is setup properly for all gateway addresses for the VLANs.

Temporarily disable CARP to gracefully failover to the backup system

Reboot master

All VLANs and Suricata will be working as expected after reboot.

FINAL NOTES

I hope that this provide enough clarity to fix the issue. This has persisted since at least 24.1.9_4 and is still existing on 24.7.10_2

Thanks

Hi Franco,

I'm not suggesting a change to the default.

I would like to add help-text to the web GUI in a few places, and maybe even update the online documentation to better clarify the default behavior, and why it may need to be turned off.

I totally understand that this is how FreeBSD, pfSense, and opnsense have behaved for years.

This is a continuation of this thread https://forum.opnsense.org/index.php?topic=15900 started in February 2020


This reply brings up RFC 1122, specifically this section: https://forum.opnsense.org/index.php?topic=15900.msg73251#msg73251

Code Select Expand


   3.3  SPECIFIC ISSUES

      3.3.1  Routing Outbound Datagrams

         The IP layer chooses the correct next hop for each datagram it
         sends.  If the destination is on a connected network, the
         datagram is sent directly to the destination host; otherwise,
         it has to be routed to a gateway on a connected network.

         3.3.1.1  Local/Remote Decision

            To decide if the destination is on a connected network, the
            following algorithm MUST be used [see IP:3]:

            (a)  The address mask (particular to a local IP address for
                 a multihomed host) is a 32-bit mask that selects the
                 network number and subnet number fields of the
                 corresponding IP address.

            (b)  If the IP destination address bits extracted by the
                 address mask match the IP source address bits extracted
                 by the same mask, then the destination is on the
                 corresponding connected network, and the datagram is to
                 be transmitted directly to the destination host.

            (c)  If not, then the destination is accessible only through
                 a gateway.  Selection of a gateway is described below
                 (3.3.1.2).


It appears that opnsense / pfsense / FreeBSD all want to default to sending packets only replying to the upstream gateway on all packets received by interfaces with a defined upstream gateway.

Good, bad, or indifferent on your feelings about how opnsense handles this... it appears that opnsense does not follow the RFC correctly.

Instead of trying to argue over the fact that "this is how x does it," that's completely irrelevant to me.

I would like to propose a compromise solution that does not involve changing default settings and possibly upsetting anyone that relies on this "reply-to" default in opnsense.

Would it be possible to add something to the WebGUI help text, maybe in the WAN firewall rules? Maybe on the WAN interface? My only opinion on it is that it should be somewhere obvious on a page commonly used, and not buried somewhere in... Firewall -> Settings -> Advanced (which, yes, of course, was the "last place I looked," and only happened to find it, because of the original thread from 2020!)

This is extremely unintuitive from my (and seemingly others) understanding of how networks work. I totally understand that this is a default in opnsense and has been for many years. I get it. I'm not trying to change that.

I just want to make "future me's" life easier, and a little "Hey, it looks like you're trying to allow connections in on the WAN? Might they happen to be from the WAN subnet? You may need to change your reply-to settings!" - Clippy might be helpful in saving time, for those, who like me, spend hours of troubleshooting... just to find out it's this setting.

I'll even put the whole PR together myself if I can get *someone* from opnsense to agree that adding the help text to the UI is acceptable!

Edit: This thread also calls out the issue: https://forum.opnsense.org/index.php?topic=8833.0