OPNsense Forum

Archive => 17.1 Legacy Series => Topic started by: framura on January 31, 2017, 08:18:29 pm

Title: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: framura on January 31, 2017, 08:18:29 pm
HI,

I just upgraded from 16.7.14 version and all seems works very well but not firewall rules.

I have some firewall rules (LAN Tab) to force VPN use on my LAN net: with 16.7.14 all works well but with 17.1 (I haven't modify any configuration) same rules doesn't works anymore.

Can you give me some advice?

Thanks in advance

P.S.: Now I reverted to 16.7.14 (Vmware machine) but I will try also with a fresh installation
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on January 31, 2017, 09:06:59 pm
Hi,

We've had a similar situation here in 17.1-BETA:

https://github.com/opnsense/core/issues/1331

Can you send us a copy of the file /tmp/rules.debug on the working 16.7.14 *and* the broken 17.1?

If yes we can fix this in no time. Send via mail: project AT opnsense DOT org


Thanks,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: Andreas on January 31, 2017, 10:43:16 pm
Hi.
updated also and this behavior also. My IPSEC Connection is now blocked - the rules are set and it worked before 17.1

thx
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mircsicz on February 01, 2017, 08:25:03 am
Hi hi, just upgraded three of my installation's two of them went smooth a third has firewall rule issue's as mentioned!
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 01, 2017, 09:09:26 am
At least in one of these cases it was a pfSense config.xml. We're investigating the rules generation now.
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: virusmoere on February 01, 2017, 09:22:41 am
Hi,

I have the same problem. Some rules are not working after upgrate to 17.1.


Regards
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mmoser on February 01, 2017, 09:27:39 am
Hi,

I have the same problem. I had a rule that routes a VPN connection to a dedicated WAN interface and it's not working anymore.

I've attached the rule itself that worked before the upgrade to 17.1.

Thanks,
Martin
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: dragon2611 on February 01, 2017, 06:35:04 pm
Same here ipsec doesn't appear to work unless I disable PF.

I did import from a pfsense config though so totally posisble that's screwed something
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 01, 2017, 07:13:50 pm
Can you guys confirm this is happening specifically to IPsec? If yes, which software/appliance is on the other end? Just to make sure we pin this down correctly and I think we can exclude the possibility of compatibility issues with pfSense imports.

I'm having trouble with a FortiGate, it seems that TCP state is not properly tracked and thus times out prematurely. Anything larger than a few seconds is killed by pf / the default rule. These blocks show up in my firewall log with the default rule as the culprit, which is odd, because the IPsec rule says allow all and ICMP/UDP fully works and TCP kind of works. Something is wrong with the state tracking it seems, settings in the GUI didn't help so far.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: Trebor1 on February 01, 2017, 07:22:35 pm
Hi,

I have 3 WAN and after upgrading to the 17.1 I do not respect me the GW's in rules. Always it use default gateway.

Thanks
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: framura on February 01, 2017, 07:59:02 pm
Hi,

I just sent via email my two rules.debug files, one from 16.7 (working) and other from 17.1 (not working).

Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mmoser on February 01, 2017, 08:54:02 pm
Hi,

I just tested to add a rule that should route IPv4* traffic from any source to a specific IP address using a dedicated Gateway. The rule is not applying and the default Gateway is being used instead. So it's not limited to IPSec.

Thanks,
Martin
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: framura on February 02, 2017, 05:29:33 pm
Hi,

these is any news about this?

Thanks
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 02, 2017, 05:45:53 pm
The IPsec instability in TCP is caused by a faulty state tracking. I have no idea where to look at the moment but I found that the state tracking can be disabled using the following workaround for the IPsec connections:

You can check whether you're affected by verifying this via command line:

# pfctl -s info

"state-mismatch" will increase frequently, one time for every reset TCP session.

Create a floating rule with "pass" and "quick" (default), interface "IPsec", direction "out", source is your local IPsec reachable network (or an alias if you have multiple networks), destination is your remote IPsec network (or an alias if you have multiple networks), go to the bottom and click "advanced options", set "state type" to "sloppy state". Save + Apply.

Please let us know if that helps your issue(s). We are looking for a way to fix this within FreeBSD itself if that is possible.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: framura on February 02, 2017, 05:54:39 pm
Thanks franco,

but my problem is not linked to IPSec: I use some rules on LAN side with a specific gateway group (I sent yesterday my two /tmp/rules.debug files, one for 16.7 and one for 17.1).

Thanks
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 02, 2017, 06:19:24 pm
Hi Framura,

Two people mentioned IPsec in here, it was for them. I'm looking at your rules now and get back soon. In any case thanks for the mail, it makes it a lot easier.

In your case we could maybe also try to replace the kernel to an older 17.1 state (beta maybe) just to make sure i's not something we've added there. In any case, will report back.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: Trebor1 on February 02, 2017, 09:35:15 pm
I have the same problem with rules and default gateway. My problem is not linked to IPSec either.

thanks.
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 02, 2017, 10:40:27 pm
Alright, policy routing problems are like this:

Make sure your gateway policies are no-overlapping floating rules with "non-quick" and/or direction "in" (for non-floating all of these already apply). Changes in our kernel allow the two FreeBSD firewalls to share forwarding decisions, but that also means that previous routing decisions can be overruled.

If that doesn't help, we are going to need more details about the rules/gateway setups in order to be of help.


Thanks,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mmoser on February 02, 2017, 10:58:44 pm
Hi Franco!

Thanks for the feedback.

I have "Failover" configured based on that HOW-TO here:
https://docs.opnsense.org/manual/how-tos/multiwan.html

On top of that I have
- No floating rules
- LAN rules where one rule should hit a dedicated WAN Gateway
- No special WAN rules

The LAN rules are attached, 1 WAN rule is attached and is same for both WAN Gateways
(not added all of them due to size restrictions)

Hope this helps, let me know if you need more input.

Thanks,
Martinez
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 02, 2017, 11:04:06 pm
For IPsec TCP session interruption, you guys should try the following as per indication of a FreeBSD developer:

# sysctl net.inet.ipsec.filtertunnel=1
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: djGrrr on February 03, 2017, 12:57:31 am
I've done quite a bit of testing on this tonight.

From what i can tell, the route-to part of "in rules" is simply being ignored (i've tested enabling logging of packets on this rule and they do get logged, meaning it is matching on the correct rule), and if you use route-to with "out rules", the matched packets go into a void. I have manually edited /tmp/rules.debug many times and reloaded pf with the route-to rules in various locations in the file, even right at the top or bottom of pass/block rules, with no real change to routing.

I've also tested this on stock freebsd 11 (11.0-RELEASE-p2) and the route-to rules work exactly as expected, in both the in and out direction. Therefore I suspect changes specific to OPNsense kernel are the cause of this bug.

Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: djGrrr on February 03, 2017, 03:04:50 am
I have now confirmed with 100% certainty that this is a kernel bug in the OPNsense kernel. By copying over the stock 11.0-RELEASE-p2 kernel and booting opnsense with it (oddly enough i had to remove the os-upnp plugin and miniupnpd pakages as they caused a lockup during the boot process), the route-to rules now work perfectly without any firewall rule changes.

edit: i suspect something in this commit is to blame:
https://github.com/opnsense/src/commit/e92bed1aa6b78a9d1286445dde9570fbff68209c
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mmoser on February 03, 2017, 08:10:49 am
Thank djGrrr for investing the time and confirming that the issue is not being the FW rules itself!

Martinez
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 03, 2017, 12:07:04 pm
It's indeed the commit, thanks for analysis and testing to djGrrr and Martinez!

The issue is a bit tricky. I think we're seeing something new in the network stack. On FreeBSD the packages for specific gateways were hi-jacked and never saw the rest of the stack, which made them completely unusable with the Captive Portal or Traffic Shaping. Since the routing is now only tagged, there is a priority issue with whether the policy route is being enforced or not. In this case not so much anymore.

In any case, this kernel will retain the old behaviour:

# opnsense-update -kr 17.1-noroute
# /usr/local/etc/rc.reboot

This is a priority item for 17.1.1 and something that did not come up in testing and all through RC1. djGrrr, do you know why this could be? It's part of a configuration difference that's not clear yet.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: virusmoere on February 03, 2017, 12:20:26 pm

# opnsense-update -kr 17.1-noroute
# /usr/local/etc/rc.reboot


This is working for me. Thanks!
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mickbee on February 03, 2017, 01:03:03 pm
It's indeed the commit, thanks for analysis and testing to djGrrr and Martinez!

The issue is a bit tricky. I think we're seeing something new in the network stack. On FreeBSD the packages for specific gateways were hi-jacked and never saw the rest of the stack, which made them completely unusable with the Captive Portal or Traffic Shaping. Since the routing is now only tagged, there is a priority issue with whether the policy route is being enforced or not. In this case not so much anymore.

In any case, this kernel will retain the old behaviour:

# opnsense-update -kr 17.1-noroute
# /usr/local/etc/rc.reboot

This is a priority item for 17.1.1 and something that did not come up in testing and all through RC1. djGrrr, do you know why this could be? It's part of a configuration difference that's not clear yet.


Cheers,
Franco

Hi Franco - i reported this while 17.1 was in beta and rc:
https://forum.opnsense.org/index.php?topic=4313.0

in any case, 17.1 stable still has the issue, IPSEC rules don't trigger UNLESS i put a rule on the top with any any any but that's kind of not the point.

I will try the other kernel tonight once there's no traffic on that one box. Holding off with upgrading the few other boxes until this is fixed. Let me know if i can help and provide logs to support the troubleshooting!

Thanks!
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 03, 2017, 01:20:51 pm
I missed this thread, sorry :(

Try the IPsec sysctl fix too:

# sysctl net.inet.ipsec.filtertunnel=1

There are some fixes we're testing right now, takes some time to gather conclusive data. But we'll report back soon. The noroute kernel works in the meantime.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mickbee on February 03, 2017, 05:29:49 pm
I missed this thread, sorry :(

Try the IPsec sysctl fix too:

# sysctl net.inet.ipsec.filtertunnel=1

There are some fixes we're testing right now, takes some time to gather conclusive data. But we'll report back soon. The noroute kernel works in the meantime.


Cheers,
Franco

No worries at all Franco, feel free to close the duplicate thread.

I tried the sysctl command and makes no difference. I will run the any any any on IPSEC for the time being (as it seems to work) and wait for the kernel level fix in the next release.
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: cbb09 on February 04, 2017, 10:58:34 pm
Franco,

I can report the same issue:

I have one regular WAN gateway and another gateway (WAN-VPN) that's connected to an OpenVPN Client. VLAN 20 traffic is specifically routed through the WAN-VPN gateway, at least it used to be. In 16.7.14 it works fine, in 17.1 VLAN 20 traffic goes out the default gateway although the WAN-VPN gateway is specified.

No other changes were made. I took a snapshot of my OPNsense VM right before upgrading to 17.1. The issue disappears when I revert back to the snapshot.

Hope this helps
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: dragon2611 on February 05, 2017, 12:30:38 am
The noroute kernel or and the sysctl variable didn't work for me

Short of a quick rule that's pretty much ipv4 any/any allow on the IPSEC interface I can't get IPSEC to work.

Worth noting my LAN traffic is coming from the WAN interface as far as opnsense is concerned
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mw01 on February 05, 2017, 12:57:10 pm
OpenVPN Client with specific firewall client rules routed through the WAN-VPN gateway no longer works the way it used to - the default gateway WAN-DHCP is always utilized. 

OpenVPN advanced configuration with "route-nopull; route-noexec;" following OpenVPN stop/start results in broken pipe log message - probably why it does not work.  Removing these results in OpenVPN added routes and all traffic from all VLANs is routed through the WAN-VPN gateway.

Feb 5 06:49:21    openvpn[19626]: MANAGEMENT: Client disconnected
Feb 5 06:49:21    openvpn[19626]: MANAGEMENT: TCP send error: Broken pipe
Feb 5 06:49:21    openvpn[19626]: MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Feb 5 06:49:21    openvpn[19626]: Initialization Sequence Completed
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 05, 2017, 01:36:51 pm
Dragon, I have the same issue here locally for incoming IPsec traffic. It could be FreeBSD 11.0 in this case indeed. I fixed it up temporarily by allowing IPsec subnets from WAN to LAN, disabling blocking of private networks. We're investigating this, too.
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: dragon2611 on February 06, 2017, 06:46:54 pm
I'll be redoing my network at somepoint soon so the design may change.

At the moment the Opensense is doing the Site2site VPNS and IDS/IPS for the IOT vlan.

The other Vlans go directly to a routerOS vm which also has the PPP termination from the ISP (I couldn't seem to get the performance on my N3150 in KVM with IPS enabled, I think because FreeBSD's network drivers are a bit shit in KVM, have to use E1000's as VirtIO + IDS/IPS = Crash) Anyway that's why the LAN/Wan interface are the same as far as opnsense is concerned.

Going from ~68/17 VDSL2 to 200/20 cable due to a move sadly, probably with most of my traffic limited to 100/20 by an L2TP tunnel (Old ISP lets me L2TP-in capped at 100M and I want to keep my static IP's)  Now normally people would consider going from 68 > 200 an upgrade but I'm essentially going from a Decent ISP to a not so Decent one.

Might have to get an ADSL2+ line (About 15/1 if I'm lucky) from the current ISP if the cable proves to be congested in that area (No VDSL2 service available :( )
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mow4cash on February 09, 2017, 04:53:18 am
I think I'm having the same issue? I route an alias through Opnvpn gateway and have a rule to block the alias on the default wan gateway. Something failed and the traffic is getting routed through the wan. This is a huge security issues, I'm not sure if it's something I did or not.
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mw01 on February 09, 2017, 01:06:19 pm
Thank you all.  Upgraded to OPNsense 17.1 to 17.1.1, no issues. 

Started up OpenVPN and broken pipe fixed.  Initially could not resolve address (tracert) through VPN (DNS servers specified on Windows box).  Started/stopped OpenVPN and everything lined up.  Works great!
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: Andreas on February 21, 2017, 12:27:56 pm
Any News? Any new fix? actually i dont have a functional IPSec
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: miclan on February 21, 2017, 01:31:43 pm
Me too, still waiting for a fix...
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 21, 2017, 05:17:22 pm
Workarounds for different issues are available. Unfortunately, there are a number of interleaving issues in this thread. I am not sure which one you are waiting for. :)

Some have been addressed for 17.1.1. We're adding more for 17.1.2 including going back to the default incompatible routing behaviour of pf and ipfw. The feature can still be used, but needs to be enabled in the GUI.

This should bring is back to a state that is unmodified, not counting problems with could still be present in the base system due to the FreeBSD 10.3 to 11.0 switch. At this point, it's unclear how much changed, but we know of an IPsec state tracking issue that is new there and likely also IPv6 policy routing problems. We are actively investigating all of those.

The bottom line is: there is a reason we are seeing these problems now going forward, because they are incredibly hard to catch. We are not alone in this, as such issues pop up in other FreeBSD-related projects as well, even in the FreeBSD bug tracker. If we decided not to go forward with an OS update, the downsides become more and more with time, increasing the amount of time it takes to adapt to another OS update in the future. It's a slippery slope.

If anything, we need to avoid standing still.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: pbolduc on February 21, 2017, 06:15:18 pm
I wanted to mention I have recently setup a clean install of 17.1 & experienced most concerns mentioned above. (Traffic not flowing through IPsec tunnels). I had performed the 17.1.1 update out of desperation hoping it would address the problem and it did for about 2 minutes then the traffic stopped flowing through the tunnels again. It wasn't until I performed a factory reset back to factory defaults using the latest firmware 17.1.1 and setup the IPsec tunnels again before the problems were completely resolved.

Going forward, my concern would be after upgrading to 17.1.2 down the road will it require us to factory default the router before things work as expected or can we still use our existing configs without needing to reconfigure everything over again?
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 22, 2017, 07:16:59 am
I can't say for sure, mostly be cause there are no details about your setup. But I can say that 17.1.2 should behave the same as 17.1.1 with one exception in the policy routing domain: we will be using the stock FreeBSD behaviour by default again. If your setup is affected by this because you run Policy/Gateway Routing in firewall rules *and* use the traffic shaper or captive portal, this option needs to be reenabled under firewall: settings: advanced: "Use shared forwarding between packet filter, traffic shaper and captive portal".


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: miclan on February 22, 2017, 05:47:27 pm
I upgraded to 17.1.2 and my situation with IPsec tunnel site to site is:

VPN site A (main) 17.1.2
VPN site B (remote office 1) 17.1.2
VPN site C (remote office 2) 16.7.14

From A to B connection is OK, but no traffic on LAN
From A to C connection and lan traffic OK

I understood that this problem is very difficult to isolate, but what can I try to have lan traffic between A and B?
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on February 22, 2017, 05:59:44 pm
On B, add a floating rule for interface "IPsec", set it to direction "any", go to advanced, choose "sloppy state". Save and apply the rules.

Then, from the console run:

# sysctl net.inet.ipsec.filtertunnel=0

You should now be able to make connections from A to B?
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: Andreas on February 23, 2017, 07:52:17 am
Hi,
updated on 17.1.2 still having problems
firewall blocks all communication and i see that in the logs
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: Andreas on February 28, 2017, 10:37:42 am
HI Franco
i tested your solution in the post from February the 22 - at my setup it doenst work :(
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: lordwarlock on February 28, 2017, 04:47:06 pm
Same here,

Workarround 
sysctl net.inet.ipsec.filtertunnel=1
doesnt work anymore after upgrade from 17.1.1 to 17.1.2

tried

sysctl net.inet.ipsec.filtertunnel=0 + Floting Rule does not work

Connection from LAN to IPSEC possible
Connecitons from IPSEC to LAN (Ping -> Possible, everything else -> Impossible)

Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: lordwarlock on February 28, 2017, 04:49:16 pm
just tried Setting

""Use shared forwarding between packet filter, traffic shaper and captive portal"."

reactivates Workarround 

sysctl net.inet.ipsec.filtertunnel=1
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: sln on March 19, 2017, 08:41:38 pm
Hi,

is there any news on this? I'm using 17.1.3 and still have problems with IPSEC traffic being blocked by the firewall. Is there any workaround (except for possibly creating a hole in the fw by allowing bogus IPs on WAN)?
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: djGrrr on March 20, 2017, 03:04:17 am
Hi,

is there any news on this? I'm using 17.1.3 and still have problems with IPSEC traffic being blocked by the firewall. Is there any workaround (except for possibly creating a hole in the fw by allowing bogus IPs on WAN)?

I would suggest checking out this thread with a test kernel to try:
https://forum.opnsense.org/index.php?topic=4804.0
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: sln on March 22, 2017, 12:18:16 pm
I would suggest checking out this thread with a test kernel to try:
https://forum.opnsense.org/index.php?topic=4804.0
Thanks for the advise! Sadly this kernel doesn't fix the issue (at least for me) with IPsec traffic getting filtered by the firewall despite rules saying otherwise.
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: guest15510 on March 22, 2017, 02:52:11 pm
I would suggest checking out this thread with a test kernel to try:
https://forum.opnsense.org/index.php?topic=4804.0
Thanks for the advise! Sadly this kernel doesn't fix the issue (at least for me) with IPsec traffic getting filtered by the firewall despite rules saying otherwise.

Hey, tried this?
https://forum.opnsense.org/index.php?topic=4313.msg19025#msg19025

I'm currently updating my 12 FW's for my company and only this solution works for me.
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on May 17, 2017, 07:04:09 pm
Hi all,

I have the same problem with 17.1.6.

00:00:00.000000 rule 88/0(match): pass in on igb2: 192.168.11.23.64782 > 172.18.210.10.443: Flags , seq 3434102374, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
00:00:00.000080 rule 77/0(match): pass out on enc0: 192.168.11.23.64782 > 172.18.210.10.443: Flags , seq 3434102374, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
00:00:00.023806 rule 12/0(match): block in on enc0: 172.18.210.10.443 > 192.168.11.23.64782: Flags [S.], seq 4228346538, ack 3434102375, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
00:00:03.003031 rule 12/0(match): block in on enc0: 172.18.210.10.443 > 192.168.11.23.64782: Flags [S.], seq 4228346538, ack 3434102375, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
00:00:06.006306 rule 12/0(match): block in on enc0: 172.18.210.10.443 > 192.168.11.23.64782: Flags [S.], seq 4228346538, ack 3434102375, win 65535, options [mss 1460,nop,nop,sackOK], length 0
00:00:11.996356 rule 12/0(match): block in on enc0: 172.18.210.10.443 > 192.168.11.23.64782: Flags [R], seq 4228346539, win 0, length 0

I tried the floating roule but the issue persist. The ICMP working fine but TCP/UDP will be blocked by PF.


Regards,
Liberomic
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on May 18, 2017, 03:00:26 pm
Hi All,

we have replicated the configurations on different site and the issue persist.
The difference between my office site to datacenter where the IPSEC workingfine and two branch office, the wan interfaces is NATed.

Regards,
Liberomic

   
 
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on May 18, 2017, 05:29:57 pm
Hi Franco,

I have found this workaround, but would not be permanent (if I will change a firewall rules... restart the appliance... this rules will be deleted).

I have deleted this line from /tmp/rules.debug
block in  log inet from {any} to {any} label "Default deny rule"
block in  log inet6 from {any} to {any} label "Default deny rule"

I have added this line at the end of file  (all interface without IPSEC "enc0")

block in  log on $WAN inet from {any} to {any} label "Default deny rule"
block in  log on $WAN inet6 from {any} to {any} label "Default deny rule"
block in  log on $LAN inet from {any} to {any} label "Default deny rule"
block in  log on $LAN inet6 from {any} to {any} label "Default deny rule"

# pfctl -f /tmp/rules.debug

Do you have a solution for this PF issue?

Many thanks for your support.  :'( :'( :'(

Liberomic
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on May 19, 2017, 05:22:21 pm
Hi all,

I have checked in 17.1.7 and the issue persist.

Regards,
Liberomic
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on May 22, 2017, 08:14:26 am
What's your rule on the IPsec tab? Isn't it easier to use any -> any there?
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on May 22, 2017, 10:45:16 am
Hi Franco,

on IPSEC interface we have checked all combinations.

ANY--ANY--Accept
SurceVPN subnet--Local subnet--Accept

But the issue persist......

I have replicated the issue on different site and this issue will be replicable.

To clarify the issue I am writing network scheme, I have four site connected by IPSEC to central Office (HO).

- Office1 (opnsense) to Head Office: in this site working fine the wan interface of opnsense is Public IP
- Office2 (opnsense) to Head Office: I have WAN interface NATed and the inbound traffic will be blocked on enc0 interface
- Office3 (opnsense) to Head Office: I have WAN interface NATed and the inbound traffic will be blocked on enc0 interface

for Office2 and Office3 I have applyed my workaround for inbound traffic coming from Head Office, because without my workaround working only ICMP traffic and TCP/UDP will be blocked.

Note: on Office2 and Office3 I have enabled Nat Traversal and the router forward all ports to opnsense WAN interface. I have upgraded all opnsense to 17.1.7.

Thanks for your support
Liberomic
 




Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on May 29, 2017, 03:46:10 pm
Hi All,

do you have news for this PF issue?

Regards,
Liberomic
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: opnsensebeb on June 05, 2017, 12:34:12 pm
Hi,

i have the same issue, since 17.1.1 no Roules for IPSEC trigger (now actual 17.1.8). I try all the hints, but nothing works for me.

Is there any news about this topic?

Regars
Sven
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on June 05, 2017, 05:17:36 pm
Hi All,

this issue is very bad, with my workaround the incoming traffic working fine....
But this change in the file /tmp/rules.debug will be lost, when you modify firewall rules or restart the appliance....

Regards,
Liberomic
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on June 26, 2017, 12:53:53 pm
UP!

 ;) ;) ;)
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mickbee on July 08, 2017, 02:40:13 am
What's your rule on the IPsec tab? Isn't it easier to use any -> any there?

Hi guys and Franco, any news on this? this is an issue since last december already and no sign of a fix for this :(

@Franco, i never thought that i'll see the day when someone suggest using any>any rules on an opensource/firewall product forum!?
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: kyferez on July 11, 2017, 07:09:50 pm
@Franco, i never thought that i'll see the day when someone suggest using any>any rules on an opensource/firewall product forum!?
I agree, but perhaps he was meaning as a troubleshooting step to verify it's not a issue with a rule not working as intended?

Which brings me to: Why do the rule logs not show Which Rule was hit? :(
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on July 12, 2017, 10:59:52 am
So just quickly I spent a whole week in total trying to chase this down in FreeBSD 11 but have been unable to find the bug in the kernel so far. If someone is up for the task that would be tremendously helpful.

The filtering in the IPsec tab is for "incoming" IPsec traffic only. If you want to prohibit traffic from entering the tunnel it's the wrong place.

Establishing an IPsec tunnel to another end places trust into it. I do not think it counts as leaving the door open to anything and all possible threats. The filtering here is convenient, but having had 6 months with no conclusion may also point to experts who could fix this judging that it's not worth fixing.

I'm no such expert. I don't know. We can surely find a fix for this eventually, but between spending a week and spending the amount necessary there is a big gap that a community will either have to help close or feel comfortable not closing.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on July 13, 2017, 10:57:08 am
Hi Franco,

thanks for your reply, do you have a tecnique to set my workaround permanent?

I have deleted this line from /tmp/rules.debug
block in  log inet from {any} to {any} label "Default deny rule"
block in  log inet6 from {any} to {any} label "Default deny rule"

I have added this line at the end of file  (all interface without IPSEC "enc0")

block in  log on $WAN inet from {any} to {any} label "Default deny rule"
block in  log on $WAN inet6 from {any} to {any} label "Default deny rule"
block in  log on $LAN inet from {any} to {any} label "Default deny rule"
block in  log on $LAN inet6 from {any} to {any} label "Default deny rule"

# pfctl -f /tmp/rules.debug

Regards,
Liberomic
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on July 17, 2017, 04:53:58 pm
There is a new test kernel that restores inbound logging / filtering for me...

# opnsense-update -kr 17.1.9-ipsec
# /usr/local/etc/rc.reboot

To go back to the normal kernel, use...

# opnsense-update -k
# /usr/local/etc/rc.reboot

Feedback welcome, maybe able to merge in time for 17.7 :)
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on July 19, 2017, 01:36:37 pm
I think I have an unproblematic solution now with the kernel posted above. It's really crucial to get feedback to get this into 17.7 in time... ;)
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on July 20, 2017, 12:49:32 pm
Hi Franco,

IT WORKS!!!!  ;D ;D ;D

I have tested in my lab and work fine!!!!

In my production enviroment I have the version 17.1.6, Do you suggest doing any updates first to 17.1.10 and then changing the kernel?

Many thanks
Liberomic

Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mickbee on July 20, 2017, 01:37:37 pm
I have one device i can test and rollback (vm) without disrupting some realtime stuff - i can do that tomorrow morning CET and feed back immediately. This is exciting!

@Franco, really appreciate your efforts and help in this space!
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on July 20, 2017, 02:40:24 pm
@liberomic it should work by only replacing the kernel. but don't forget to reboot.

@mickbee thanks, looking forward to your feedback.

To be honest, the issue is still obscure: the packet filter receives "garbage" as the IP header, and that only for inbound packets from the tunnel.... it may still not work for filtering TCP ports, I haven't checked. What I could find was that this affects a subset of IPsec connections, namely the ones that do have their IPsec not connected to WAN so NAT-T handling in IPsec plays a heavy role in solving this in a way that compares to the state of FreeBSD 10.3.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on July 21, 2017, 10:28:58 am
BTW, the patch is this, it will be available in 17.7.

https://github.com/opnsense/src/commit/0de7c3a57


Cheers,
Franco

Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: mickbee on July 21, 2017, 11:38:38 am
Hi Franco, again thanks for following up!

I tried the upgraded kernel and it doesn't seem to change much but then again, the VM machine sits behind a physical opensense box which I can't really touch (several thousand km away and no one to help fix it in case it becomes irresponsive) and which has the issue in question.

I have another one next to me which suffers from the same bug but this one needs to be up for the next 7 hours. I'll try as soon as people go home and report back.

Still, since the patch will already be included, i suppose thorough testing can wait for the weekend?

regards,
m.

Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on July 21, 2017, 11:45:58 am
Hi m,

We have tested a number of deployments with and without NAT-T, the behaviour is not different without it and considerably better when NAT-T is used.

If there are reasons to pull it from 17.7 please speak up. All testing is appreciated to see where we are at to make a balanced decision.


Cheers,
Franco
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: liberomic on July 28, 2017, 06:35:27 pm
Hi Franco,

we want test this on new device in production but the file is missed.

opnsense-update -kr 17.1.9-ipsec
Fetching kernel-17.1.9-ipsec-amd64.txz: ...opnsense-verify: Unable to open /var/cache/opnsense-update/69564/kernel-17.1.9-ipsec-amd64.txz: No such file or directory
 failed

We have updated to 17.1.11, this fix is included?

Regards,
Liberomic
Title: Re: Upgrade from 16.7.14: Firewall rules doesn't works as before
Post by: franco on July 28, 2017, 07:44:43 pm
Liberomic,

Sorry, I removed the test kernel. You can use the upcoming 17.7 release kernel safely:

# opnsense-update -kr 17.7


Cheers,
Franco