Messages

Historically, I've had to do one of two things to resolve this, and have never had luck with sloppy state.

1) SEEMS OK: Set a floating rule, include the GRE interface, enable quick, target outbound only, set state to none and TCP flags to any, and use action pass. Traffic is controlled by knowing that only the router and any traffic it lets in to forward to send outbound on that interface.

2) BAD CLEARLY: Set a pair of stateless floating rules for each stateful rule I would normally configure, for example if my floating rule was to allow devices over the GRE tunnel in 192.168.0.0/24 to reach 192.168.99.10 over RDP (3389) locally, I would create the following two rules:
a) int lan,gre0 direction any src 192.168.0.0/24 srcport any dst 192.168.99.10 dstport 3389 flags=any state=none
b) int lan,gre0 direction any src 192.168.99.10 srcport 3389 dst 192.168.0.0/24 dstport any flags=any state=none

I'm open to better ideas. 2 is terrible as it allows any traffic from srcport 3389 on .99.10 to anything, obviously. 1 doesn't seem bad given if someone owns the firewall we're in trouble anyway.

As in you tried it in FreeBSD via a port?

After struggling with zerotier performance, I'm getting back into running routing protocols over GRE, with IPSec in transport mode. I have a pair on 17.7 (no point upgrades) seemingly working fine, but with my boxes on the current 17.7.5 point release, with the same configs, I have a few pairs that get good SAs, SPs, but cannot pass any traffic and show the following in the log whenever a packet tries to go out:

Code Select Expand

charon: 07[KNL] received invalid PF_ROUTE message
Searching for this ^ returns only a reference to the strongswan source code :)

When I ping one WAN IP from another (leaving GRE out entirely), I get:

Code Select Expand

ping: sendto: Permission denied

Does anyone have IPSec transport mode working on 17.7.5?

Hi all - I was hoping that OPNSense would resolve this issue I've had long-standing on the pfSense side. Essentially, the firewall is unable to properly set states on traffic coming over a GRE that's using an IPSec transport. Here's the pfSense issue:

https://redmine.pfsense.org/issues/4479

Does anyone have stateful firewalling over GRE/IPSec(transport) working? It's possible I'm just missing something simple :)

@franco - Perhaps worth moving into a request for feedback GitHub issue on "making multiWAN awesome"?

Ah, turns out they have an online demo. Use admin/admin at:

https://balancedemo.peplink.com/cgi-bin/MANGA/index.cgi

It's not the prettiest UI, but the multi-WAN stuff works really well. Take a look at outbound policies in particular.

Generally, I think if you could tag a gateway group as "default gateway" for a box, that covered both internal and client traffic, that would be a great user experience - given the power gateway groups already have with parameters for "health", load balancing vs. failover, etc.

Default gateway switching seems to be working OK right now. The local services (DNS, Zerotier, etc.) angle makes this the only real workable option for me ATM.

Franco's earlier notes are making much more sense to me - I know one of the big pains for people moving from Sonicwall / Fortinet / Watchguard boxes to OPNSense / pfSense / etc. is "multi-WAN is hard". Providing a guided UI to simplify multi-WAN would help quite a bit in these scenarios.

Peplink is one vendor that, while their boxes are pretty simple and don't do much, do the multi-WAN UI simplification (and heck, Multi-WAN itself) fairly well. Franco, I'd be happy to provide someone from OPNSense access if you'd like to peek and don't have any around.

I'll be configuring something in our test lab this week to debug further, and will report back!

Sorry to pack so much in, but if_ipsec looks really interesting: https://www.reddit.com/r/freebsd/comments/6pmdrm/freebsd_111release_announcement_i_saw_this_and_i/

Another note, perhaps it's incredibly hard and/or impossible given how a gateway group in defined, but allowing usage of a gateway group as the system's default gateway would be awesome.

I expect that "switch default route" uses multiple route tables in bsd (setfib), but again I'm working from near-ignorance and trying to catch up here :)

@franco To be honest, I'm not sure what you mean by "fixed interface routes for local services". If there is no global default gw switching, how does local outbound DNS traffic know which gateway to go out on?

@whitwye I agree. I'm curious if we could have a rules tab for "local" or be able to select lo as an interface in floating rules.

Leaving IPSec out of it for now - the workaround we usually use is IPSec transport between pairs of connections, with GRE tunnels inside and OSPF for failover. I'm not comfortable enough with the internals of IPSec to really get deep into that use-case.

I was thinking about OpenVPN in this case (and in the future, ZeroTier :)), both of which seem to be more forgiving than the rats nest (useful, performant rats nest) that is IPSec.

One that's hitting us right now is DNS outbound from the router. When the primary link goes down, if OPNSense is the DNS server / forwarder, it can't get out to the servers it relies on. Am I missing a way to policy route this? Same for NTP, etc.

Multi-WAN in OpnSense works perfectly, and makes perfect logical sense, from a LAN-originated traffic perspective:

• Create the group
• Create a policy router
• Traffic goes out per the group's configuration depending on gateway health

However, for router-originated traffic (outbound VPN tunnel connections, outbound DNS queries, updates, etc.) failover doesn't seem to work. I recall being able to target router-originated traffic in floating rules in another FreeBSD-based firewall system, but I could be mistaken.

I'm curious about the deprected "Gateway Switching" feature in Firewall > Settings > Advanced. That seems, from my reading, to suggest it would emulate the behavior of lower-end edge devices with Multi-WAN, which just change their system default gateway based on gateway health. In fact, if that's what it does, it removes policy routing as a requirement, which would be nice.

If this feature works like I expect it to, is there a reason it is deprecated? I suppose one question would be how we determine the priority of the gateways, as we'd want it to swap back to the "primary" when it's healthy again.

In my dream world, the default gateway for the system as a whole could be set as a gateway group, but I could be missing something massive.