Upgrade from 16.7.14: Firewall rules doesn't works as before

[franco]

Hi Framura,

Two people mentioned IPsec in here, it was for them. I'm looking at your rules now and get back soon. In any case thanks for the mail, it makes it a lot easier.

In your case we could maybe also try to replace the kernel to an older 17.1 state (beta maybe) just to make sure i's not something we've added there. In any case, will report back.


Cheers,
Franco

[Trebor1]

I have the same problem with rules and default gateway. My problem is not linked to IPSec either.

thanks.

[franco]

Alright, policy routing problems are like this:

Make sure your gateway policies are no-overlapping floating rules with "non-quick" and/or direction "in" (for non-floating all of these already apply). Changes in our kernel allow the two FreeBSD firewalls to share forwarding decisions, but that also means that previous routing decisions can be overruled.

If that doesn't help, we are going to need more details about the rules/gateway setups in order to be of help.


Thanks,
Franco

[mmoser]

Hi Franco!

Thanks for the feedback.

I have "Failover" configured based on that HOW-TO here:
https://docs.opnsense.org/manual/how-tos/multiwan.html

On top of that I have
- No floating rules
- LAN rules where one rule should hit a dedicated WAN Gateway
- No special WAN rules

The LAN rules are attached, 1 WAN rule is attached and is same for both WAN Gateways
(not added all of them due to size restrictions)

Hope this helps, let me know if you need more input.

Thanks,
Martinez

[franco]

For IPsec TCP session interruption, you guys should try the following as per indication of a FreeBSD developer:

# sysctl net.inet.ipsec.filtertunnel=1

[djGrrr]

I've done quite a bit of testing on this tonight.

From what i can tell, the route-to part of "in rules" is simply being ignored (i've tested enabling logging of packets on this rule and they do get logged, meaning it is matching on the correct rule), and if you use route-to with "out rules", the matched packets go into a void. I have manually edited /tmp/rules.debug many times and reloaded pf with the route-to rules in various locations in the file, even right at the top or bottom of pass/block rules, with no real change to routing.

I've also tested this on stock freebsd 11 (11.0-RELEASE-p2) and the route-to rules work exactly as expected, in both the in and out direction. Therefore I suspect changes specific to OPNsense kernel are the cause of this bug.

[djGrrr]

I have now confirmed with 100% certainty that this is a kernel bug in the OPNsense kernel. By copying over the stock 11.0-RELEASE-p2 kernel and booting opnsense with it (oddly enough i had to remove the os-upnp plugin and miniupnpd pakages as they caused a lockup during the boot process), the route-to rules now work perfectly without any firewall rule changes.

edit: i suspect something in this commit is to blame:
https://github.com/opnsense/src/commit/e92bed1aa6b78a9d1286445dde9570fbff68209c

[mmoser]

Thank djGrrr for investing the time and confirming that the issue is not being the FW rules itself!

Martinez

[franco]

It's indeed the commit, thanks for analysis and testing to djGrrr and Martinez!

The issue is a bit tricky. I think we're seeing something new in the network stack. On FreeBSD the packages for specific gateways were hi-jacked and never saw the rest of the stack, which made them completely unusable with the Captive Portal or Traffic Shaping. Since the routing is now only tagged, there is a priority issue with whether the policy route is being enforced or not. In this case not so much anymore.

In any case, this kernel will retain the old behaviour:

# opnsense-update -kr 17.1-noroute
# /usr/local/etc/rc.reboot

This is a priority item for 17.1.1 and something that did not come up in testing and all through RC1. djGrrr, do you know why this could be? It's part of a configuration difference that's not clear yet.


Cheers,
Franco

[virusmoere]

# opnsense-update -kr 17.1-noroute
# /usr/local/etc/rc.reboot

This is working for me. Thanks!

[mickbee]

It's indeed the commit, thanks for analysis and testing to djGrrr and Martinez!

The issue is a bit tricky. I think we're seeing something new in the network stack. On FreeBSD the packages for specific gateways were hi-jacked and never saw the rest of the stack, which made them completely unusable with the Captive Portal or Traffic Shaping. Since the routing is now only tagged, there is a priority issue with whether the policy route is being enforced or not. In this case not so much anymore.

In any case, this kernel will retain the old behaviour:

# opnsense-update -kr 17.1-noroute
# /usr/local/etc/rc.reboot

This is a priority item for 17.1.1 and something that did not come up in testing and all through RC1. djGrrr, do you know why this could be? It's part of a configuration difference that's not clear yet.


Cheers,
Franco

Hi Franco - i reported this while 17.1 was in beta and rc:
https://forum.opnsense.org/index.php?topic=4313.0

in any case, 17.1 stable still has the issue, IPSEC rules don't trigger UNLESS i put a rule on the top with any any any but that's kind of not the point.

I will try the other kernel tonight once there's no traffic on that one box. Holding off with upgrading the few other boxes until this is fixed. Let me know if i can help and provide logs to support the troubleshooting!

Thanks!

[franco]

I missed this thread, sorry :(

Try the IPsec sysctl fix too:

# sysctl net.inet.ipsec.filtertunnel=1

There are some fixes we're testing right now, takes some time to gather conclusive data. But we'll report back soon. The noroute kernel works in the meantime.


Cheers,
Franco

[mickbee]

I missed this thread, sorry :(

Try the IPsec sysctl fix too:

# sysctl net.inet.ipsec.filtertunnel=1

There are some fixes we're testing right now, takes some time to gather conclusive data. But we'll report back soon. The noroute kernel works in the meantime.


Cheers,
Franco

No worries at all Franco, feel free to close the duplicate thread.

I tried the sysctl command and makes no difference. I will run the any any any on IPSEC for the time being (as it seems to work) and wait for the kernel level fix in the next release.

[cbb09]

Franco,

I can report the same issue:

I have one regular WAN gateway and another gateway (WAN-VPN) that's connected to an OpenVPN Client. VLAN 20 traffic is specifically routed through the WAN-VPN gateway, at least it used to be. In 16.7.14 it works fine, in 17.1 VLAN 20 traffic goes out the default gateway although the WAN-VPN gateway is specified.

No other changes were made. I took a snapshot of my OPNsense VM right before upgrading to 17.1. The issue disappears when I revert back to the snapshot.

Hope this helps

[dragon2611]

The noroute kernel or and the sysctl variable didn't work for me

Short of a quick rule that's pretty much ipv4 any/any allow on the IPSEC interface I can't get IPSEC to work.

Worth noting my LAN traffic is coming from the WAN interface as far as opnsense is concerned