OpnSense after three months use

[Monviech (Cedrik)]

Yeah it can't do both. For very complex IPsec VPN and NAT scenarios I still use Juniper.

But it's not the fault of OPNsense. It's an upstream bug of FreeBSD:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474

To get NAT working, the tunables set are this mitigation of the upstream bug:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474#c39

But sadly that breaks enc0

[tverweij]

List of what I use for VPN:

- Client connections (laptops): OpenVPN instances
- Client network connections: IPSEC IPv4 IKE ESP IPv4 Tunnel
- Inter-firewall connections: IPSEC IPv4 IKEv2 ESP Route-based

This means I am in a catch-22 position:
1. I can not do SNAT on the WAN as inbound SNAT is not supported
2. I can not do SNAT on the inter-firewall connections as SNAT is not supported on route based IPSec
3. I can not upgrade to the solution of Monviech as I need IPSec ESP Ipv4 Tunnels

[elvinmammadov]

This is a good topic.

1. Previously I have used pfSense and I was very satisfied with that. In my new company, we are using Opnsense. I would say, working with Aliases is really headache. Creating hosts with names and combining them in one aliase takes too much time and it is also annoying.

2. In pfSense, in Firewall Rules, there is a separator which you can seperate the rules easily with different colors. It would be nice to have the same functionality in Opnsense.

3. Live view in Logs is not also user friendly and missing a lot of functions. In pfSense it is very well prepared.

4. IPsec section in Opnsense is also not very well prepared. We have almost hundred VPN connections. Working with them is not easy. First I click on Phase 1, and searching for Phase 2 which is located on the buttom of the page. Beside this, it is not possible to see the running packets, every time I need to refresh the page to see if the packets are running. In pfSense it is very nice and easy to use.

[Monviech (Cedrik)]

List of what I use for VPN:

- Client connections (laptops): OpenVPN instances
- Client network connections: IPSEC IPv4 IKE ESP IPv4 Tunnel
- Inter-firewall connections: IPSEC IPv4 IKEv2 ESP Route-based

This means I am in a catch-22 position:
1. I can not do SNAT on the WAN as inbound SNAT is not supported
2. I can not do SNAT on the inter-firewall connections as SNAT is not supported on route based IPSec
3. I can not upgrade to the solution of Monviech as I need IPSec ESP Ipv4 Tunnels

What I tried out was VXLAN over a Policy Based IPsec VPN to connect two firewalls with each other. I got that to work. When I have time I'll try out if I can match NAT rules on the VXLAN interface, that could potentially mitigate the problem of not having a route based NAT IPsec tunnel.

With this setup, both sites have a vxlan interface with an IP address, and a local loopback interface which the vxlan tunnel uses as local and remote IP. The Policy Based IPsec has a traffic selector with the local loopback interfaces as local and remote nets. Then you could put routes on the vxlan interfaces, and potentially also NAT into them.

Because VXLAN is Layer 2 over Layer 3 the Policy Based IPsec only needs to allow the IP addresses the VXLAN interfaces use to send and receive the traffic from.

[tverweij]

Thanks, but it keeps coming to work arounds, just because a simple function (incoming SNAT) is not implemented.
For a second generation firewall - other than a simple router - both DNAT and SNAT should be able to work in both directions.

[Monviech (Cedrik)]

I'm also not a fan of this. But its a FreeBSD limitation. PFsense has the same issue:

https://redmine.pfsense.org/issues/11395

In my opinion this feature only working one or the other way is a real issue to say IPsec on FreeBSD Platforms is "enterprise ready".

I don't know what happens in this regard though, maybe there could be sponsors who pay FreeBSD developers to improve this in the future?

[Patrick M. Hausen]

I have switched to WireGuard which implements a proper tunnel interface and routing for all company internal office to office VPNs. Of course that does not help with the customers and their "enterprise" gear 

Fortunately I do not need any NAT shenanigans with customer connections. And I can assure you that you can play all sorts of tricks including NPT6 with WireGuard interfaces just fine.

Kind regards,
Patrick

[tverweij]

Ok, lets try wireguard - it will be a few weeks before I have the time, but I will share the results.

[Patrick M. Hausen]

WireGuard lacks documentation and the terms they use to denote certain technical components are ... weird. But it's in the end all rather straightforward and simple. Very orthogonal (I like that!) to all other OPNsense features.

Feel free to reach out, but preferably on the public forum.

[Monviech (Cedrik)]

I'm also not a fan of this. But its a FreeBSD limitation. PFsense has the same issue:

I have to improve this comment I made before.

IPsec NAT with PF = limitations
IPsec NAT with IPFW = no limitations

It's not a limitation of FreeBSD. It's a limitation of "pf" Firewall in Freebsd. The "ipfw" Firewall ( a different firewall implementation in FreeBSD ) doesn't have those problems, as stated by multiple users in the bug report.

There you can mix IPsec VTI and Policy Based and NAT however you like.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474#c7
if_ipsec works with ipfw's NAT, we have many of such installations for years.
I think you use PF and it won't work in some configurations, because of its design and how IPsec handled in FreeBSD.

https://docs.freebsd.org/de/books/handbook/firewalls/#firewalls-ipfw
https://docs.freebsd.org/de/books/handbook/firewalls/#in-kernel-nat

Though since OPNsense and PFsense both use PF (it's even in the name of one) I don't know if it's likely for it ever to change.

In my opinion it's a major limitation that should be revealed much more proactively. Since I've seen multiple users (myself included) stumble over this.

Wireguard:

Also additionally, Wireguard in FreeBSD has a very troubled history, and I myself had stability problems with it before (kernel panics).

https://arstechnica.com/gadgets/2021/03/buffer-overruns-license-violations-and-bad-code-freebsd-13s-close-call/

(I really like Wireguard in Linux. I use it extensively for peer to peer connections between VMs)

[Patrick M. Hausen]

Wireguard:

Also additionally, Wireguard in FreeBSD has a very troubled history, and I myself had stability problems with it before (kernel panics).

I never had any issues, neither with the Golang implementation nor with the rewritten kernel module that is now in the standard installation.

Do you remember the version that gave you stability issues?

Kind regards,
Patrick

[Monviech (Cedrik)]

@Patrick

Yes I made a thread about it when it happened. Happened multiple times. Since then I have removed all wireguard from all DEC Hardware I use and only use IPsec and SSLVPN.

https://forum.opnsense.org/index.php?topic=35513.0

[Patrick M. Hausen]

Thanks, I run the community edition everywhere. That at least explains how a completely different experience is possible.

[Seimus]

Interesting stuff with the Wireguard,

I am using WG as well on Community editions OPN, basically cant live without out it, yet never experienced such issues. I am as well not using official HW but a miniPC knockoff. Also my WG implementation is RA. I believe you are using as well persistent WG tunnel? My is more or less on-demand used only in case I am out of the Home network.

Would be interesting to test it in a persistent way.

Thanks for your post, I'll personally keep an eye on this.

Regards,
S.

[Patrick M. Hausen]

All persistent tunnels connecting our two office locations to the data centre and my home network to the Karlsruhe office. I push TrueNAS ZFS replication through that tunnel every day.