OpnSense after three months use

[Monviech (Cedrik)]

I have used Wireguard very extensively on the DEC hardware.

I had multiple instances listening (wg1, wg2, wg3 .... wgN)

Each of them had multiple site2site tunnels. And also A few different road warrior setups.

So all in all I had like 100 or more tunnels running.

Most of the tunnels were dual stack configurations. I had multiple Gigabytes of traffic every day.

Maybe that was too much for wireguard, I don't know.

[Monviech (Cedrik)]

What I tried out was VXLAN over a Policy Based IPsec VPN to connect two firewalls with each other. I got that to work. When I have time I'll try out if I can match NAT rules on the VXLAN interface, that could potentially mitigate the problem of not having a route based NAT IPsec tunnel.

T.W.I.M.C.

I just tried out the vxlan over IPsec between two OPNsense.
Spanned it over a policy based IPsec that had loopback interfaces as endpoints on both OPNsense.
Put a small /31 transfer net on the vxlan adapters on both sides.
I could route, NAT, everything (on the vxlan interfaces instead of using ipsecXX interfaces). So there's this way to have policy based and route based VPNs on the same opnsense box without using wireguard.

Edit: No it doesn't yet. I just judged the packet captures. The nated icmp echo request travels to the host on side b, and comes back into the vxlan interface as echo reply all the way back to the vxlan interface on site a. But there it isn't NATed back (NAT Table doesnt match somehow), it just gets blackholed there. So it's totally in line with what I've also seen with the VTI IPsec behavior. Crud. I give up here. Let's use wireguard ^^

Further Information: https://forum.opnsense.org/index.php?topic=36319.0

[tverweij]

Is see suddenly a lot of posts with problems using Wireguard.
I also see that @Monviech respons in one of these topics that Wireguard is UDP based - this is a big problem for me as my experience is that intercintinental VPN (more that 8000 km in a straight line) using UDP gives big instability problems.

And as I have NO problems whatsoever with IPSEc (apart from the SNAT problem), Wireguard won't be the solution for me.
I am working at the automation of my own work around using a Windows machine (generated interface proxies from a PF rule); this will be the solution for me for the time-being.

[franco]

> Is see suddenly a lot of posts with problems using Wireguard.

Confirmation bias does that. It's not more or less for any other VPN. Even the "simple" WireGuard protocol can be challenging.

And of course there is always a bug hidden somewhere. We can't find it without users reporting problems. ;)


Cheers,
Franco

[Seimus]

> Is see suddenly a lot of posts with problems using Wireguard.

Confirmation bias does that. It's not more or less for any other VPN. Even the "simple" WireGuard protocol can be challenging.

And of course there is always a bug hidden somewhere. We can't find it without users reporting problems. ;)


Cheers,
Franco

I can only agree on this as per my experience with Enterprise vendors.

As well this is not vendor exclusive. Basically as for any vendor and any feature they are providing. Can say I've seen many weird stuff happening with many vendors.

Certain scenarios and use cases trigger certain bugs/problems.

Regards,
S.

[CJ]

The main problems I see with WG are due to people not leaving Unbound set to all interfaces or changing the default allow setting.

The Keepalive 25 change has been known for quote a while and keeps connections from getting stale and works well.

[franco]

Should we change the keepalive default then perhaps?

[Monviech (Cedrik)]

Keepalive has to be set on the wireguard peer that is behind NAT. That means, it's almost always the client peer, almost never the OPNsense peer. So the current default in the OPNsense is the default that Wireguard calls best practice.

https://www.wireguard.com/quickstart/

Because NAT and stateful firewalls keep track of "connections", if a peer behind NAT or a firewall wishes to receive incoming packets, he must keep the NAT/firewall mapping valid, by periodically sending keepalive packets. This is called persistent keepalives. When this option is enabled, a keepalive packet is sent to the server endpoint once every interval seconds. A sensible interval that works with a wide variety of firewalls is 25 seconds. Setting it to 0 turns the feature off, which is the default, since most users will not need this, and it makes WireGuard slightly more chatty.

The value of 25 seconds is used because the UDP state/session timeout in most firewalls is 30 seconds. So if the mobile phone of a roadwarrior wireguard user sends a packet all 25 seconds to the opnsense, their own "socket" will stay open for the return packets.

EDIT:

I think most problems come from users who choose wireguard because they think it's easier to use. That means less users will choose ipsec, and thus there are less people having problems with it (ipsec), since they don't use it.

But here is an example that even a very simple wireguard setup can pose a big challenge because some vendors like AVM implement it badly: https://forum.opnsense.org/index.php?topic=36273.0

Sorry even more EDIT:

There's also the thing about Wireguard not using TCP-MSS Clamping by default. That creates a lot of hard to troubleshoot scenarios too. https://github.com/opnsense/docs/pull/498

[tverweij]

For the list of this topic (I will edit the fort post), I have to add:

Firewall broke for unknown reasons, leaving all LAN segments isolated from the Internet.
The only thing that helped was restoring the Virtual Harddisk from a backup of 4 days ago. I did not restore the virtual hardware, what means that the problem was on the disk.
I never had to restore a firewall before in 25 years, so this sounds like instability in OpnSense.

[franco]

I believe some people are truly unlucky running into all the (unknown) issues with the software. Most of the time, though, people simply don't like OPNsense for one reason or another and whatever fits that narrative is good enough as a justification that it must be bad.

Use the tools that work for you. There is a lot of good software (free or paid) out there.


Cheers,
Franco

[tverweij]

I believe some people are truly unlucky running into all the (unknown) issues with the software. Most of the time, though, people simply don't like OPNsense for one reason or another and whatever fits that narrative is good enough as a justification that it must be bad.

Use the tools that work for you. There is a lot of good software (free or paid) out there.

Just had to restore the second firewall (Running in Amsterdam, the first one was on Curacao), with exactly the same problem, 8 hours later.

With this as an extra problem, I have to second guess the OpnSense solution; I will look at the stability in the coming time, but if this becomes normal, I have to look further.
Paid is no problem, I am coming from a paid solution and I wanted to license the Business Verion of OpnSense for all firewaals (I already have one, the rest is wating for the next version with the OpenVPN instances).

But now I wait with the licensing until I know this was only a fluke ...

[franco]

I believe if you have these serious issues and no way to describe or pinpoint them this will not get any better.


Cheers,
Franco

[tverweij]

I am afraid of that too...

[tverweij]

If some of the developers of Deciso B.V. are reading this, if you contact me I can deliver 2 disks with an installation (HyperV) that have the problem for investigation.

[Patrick M. Hausen]

Hyper-V is not a particularly recommended platform for FreeBSD. ESXi is a different story altogether. Good support.