I use OpnSense now for three months, and generally speaking I am very content with this firewall.
I come from Kerio, which I used for about 25 years.
If I look at the things I encountered, two points need improvement in OpnSense.
1. The Aliases. It works very well , but documentation of a Hosts alias is almost not possible (yes, I know, make an alias for each IP and then one hosts alias - but sorry, too much work and absolutely not workable as the amount of aliasses explodes. I use now URL Table (IPS) as a work around for this problem ....
2. The outbound. As the name says: it won't work for inbound. This really costs me a lot of headaches as a lot of rules I had in Kerio were impossible to implement on OpnSense, because it can not change the source IP for an inbound rule. I finally worked around this by setting up a Windows machine. I do the port forwarding to this machine and in this machine I do the address translation and reroute to the real destination using "netsh interface portproxy". It works like a charm, but it makes things less transparent and shouldn't be neccerary,
Other than those 2 problems, I think you make a really great product.
EDIT, number 3:
Firewall broke for unknown reasons, leaving all LAN segments isolated from the Internet.
The only thing that helped was restoring the Virtual Harddisk from a backup of 4 days ago. I did not restore the virtual hardware, what means that the problem was on the disk.
I never had to restore a firewall before in 25 years, so this sounds like instability in OpnSense.
Edit: restoring the second firewall with the same problem 8 hours later ...
For inbound connections that should use a real TCP relay, you can use HAproxy.
But possibly I did not quite get your 2. point. Could you provide an example of what you are routing/relaying where? I am pretty confident that can be solved in OPNsense alone.
Number 2 has got me scratching my head too, maybe two problems not one.
Explanation of Number 2:
I have 3 HyperV machines, two in the same rack and one on another continent.
Each physical machine runs it's own OpnSense firewall and all firewalls are connected to each other using IPSEC tunnels.
To make it simple (in reality, lots of VM's are running), in this picture I have 2 physical machines and 2 VM's, one on each physical machine:
PhM1 has VM1, IP 10.10.10.5 - this machine has an IPSEC tunnel to phM2, gateway 10.20.10.2
PhM2 has VM2, IP 172.25.0.7 - this machine has an IPSEC tunnel to phM1, gateway 10.20.10.3
I get a connection from the internet on PhM1 to port 95 from IP x.x.x.x
On the WAN of PhM1, I defined the following NAT rule: ThisFirewall on port 95 Forward to 172.25.0.7 on port 80.
In the routes, I defined that the network 172.25.0.0/24 has to go through gateway 10.20.10.2.
What happens:
1. The forward sees that it should route it to the gateway 10.20.10.2 (so far so good)
2. PhM2 is entered and it sends it to the correct machine (172.25.0.7)(again, works like a charm)
3. The destination machine answers and as the source address is an address on the internet, the data is routed to it's own internet gateway instead of routing it back to the IPSEC tunnel, breaking the connection flow.
Request from x.x.x.x:
x.x.x.x -> WAN-PhM1 -> 10.20.10.2 -> 10.20.10.3 -> 172.25.0.7
Answer from 172.25.0.7:
172.25.0.7 -> WAN-PhM2 -> x.x.x.x
Result: communication is impossible.
When the source address is an address within PhM1, everything works 100% correct, but as soon as the source is an internet address, the communication fails because it is impossible to ensure the the correct route back is followed.
When an Inbound SNAT is possible, I can translate the internet address to a local address of PhM1 and that would solve the problem.
With SNat
Request from x.x.x.x:
x.x.x.x -> WAN-PhM1 -> SNAT change source to y.y.y.y _>10.20.10.2 -> 10.20.10.3 -> 172.25.0.7
Answer from 172.25.0.7:
172.25.0.7 -> 10.20.10.3 -> 10.20.10.2 -> SNAT restore source to x.x.x.x -> WAN-PhM1 -> x.x.x.x
And then the flow is correct and communication works.
I hope this explanation makes things clear.
Quote from: tverweij on October 04, 2023, 10:52:06 PM
When an Inbound SNAT is possible, I can translate the internet address to a local address of PhM1 and that would solve the problem.
You can trivially achieve that with a TCP relay by e.g. HAproxy.
I would need to experiment to find out if port forwarding combined with outbound NAT on the tunnel would work, too. But I suspect it would.
That looks like about the same as I did now with a Windows instance that takes care of the SNAT ...
I suspect not quite as there is a haproxy plugin for OPN so if introducing it + rules work, then they can coexist in an OPN installation, cutting out a windows machine. If that was of value to you of course.
No, the value of my course is to make it easy and transparent. An extra Windows machine is not a problem.
Needing to use a proxy in any way is, as SNAT is invented for this.
And as I never, ever used SNAT for outgoing rules, other than the default SNAT on the WAN adapter, I really don't understand that this is missing in OpnSense. I found out that this was missing after I crossed the line of no return and luckely I found a work around, as this was really a point that, if I found out earlier, would have lead me to disregard OpnSense and search for another alternative.
But as said, beside those two points that make my life less easy (as the VM networks regurarly move from one host to another, based on resources), I like the product.
fair enough
It's not missing. You can define SNAT for inbound connections by creating an outbound NAT rule on the tunnel interface. Which makes sense if you visualise the flow of packets.
The terms outbound and inbound do not apply to to/from the Internet or the private networks. They are applied on the interface level.
A packet originating on your LAN directed to the Internet is inbound on the LAN interface. A packet port forwarded via NAT from the Internet down some tunnel is outbound on the tunnel interface.
Quote from: Patrick M. Hausen on October 05, 2023, 12:25:19 AM
It's not missing. You can define SNAT for inbound connections by creating an outbound NAT rule on the tunnel interface. Which makes sense if you visualise the flow of packets.
Yes I thought of that. And I tried that. And it failed.
As soon as I do that, I see in the live Log that it works, but no data is going through the tunnel anymore.
I searched the forum for this, and found that SNAT is not working on a IPSEC interface / tunnel that uses routing.
Quote from: tverweij on October 05, 2023, 01:22:10 PM
Quote from: Patrick M. Hausen on October 05, 2023, 12:25:19 AM
It's not missing. You can define SNAT for inbound connections by creating an outbound NAT rule on the tunnel interface. Which makes sense if you visualise the flow of packets.
Yes I thought of that. And I tried that. And it failed.
As soon as I do that, I see in the live Log that it works, but no data is going through the tunnel anymore.
I searched the forum for this, and found that SNAT is not working on a IPSEC interface / tunnel that uses routing.
It was true once that SNAT didn't match on VTI, but I just tested it today and its not true anymore.
Test Setup: https://forum.opnsense.org/index.php?topic=36254.msg176819#msg176819
SNAT Test: https://forum.opnsense.org/index.php?topic=36254.msg176976#msg176976
This means I have to recreate the tunnels using "Connections"?
Because I use the "Tunnel Settings" to create an IPv4 IKEv2 ESP Route based tunnel.
Yes I think that's what you'll have to do. I never got it to work with the legacy tunnels either.
But be careful, this setup I described will make the OPNsense "ONLY" work with VTI tunnels. So no other policy based tunnels will work. (At least not with filter rules and NAT rules. That functionality is then tethered to the ipsec interfaces, and the enc0 interface will be stripped of that functionality)
Only use this setup if you only have route based VPN tunnels and don't plan on any policy based ones in the future.
- I on the other hand only use policy based tunnels and don't use VTI at all. So it's a design choice.
EDIT: I also added packet captures to my proof of concept in the post above to have evidence of SNAT indeed working.
Quote from: Monviech on October 05, 2023, 02:43:14 PM
Only use this setup if you only have route based VPN tunnels and don't plan on any policy based ones in the future.
That means I can not use it, as I have other tunnel modes for customer connections.
Yeah it can't do both. For very complex IPsec VPN and NAT scenarios I still use Juniper.
But it's not the fault of OPNsense. It's an upstream bug of FreeBSD:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474
To get NAT working, the tunables set are this mitigation of the upstream bug:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474#c39
But sadly that breaks enc0 :)
List of what I use for VPN:
- Client connections (laptops): OpenVPN instances
- Client network connections: IPSEC IPv4 IKE ESP IPv4 Tunnel
- Inter-firewall connections: IPSEC IPv4 IKEv2 ESP Route-based
This means I am in a catch-22 position:
1. I can not do SNAT on the WAN as inbound SNAT is not supported
2. I can not do SNAT on the inter-firewall connections as SNAT is not supported on route based IPSec
3. I can not upgrade to the solution of Monviech as I need IPSec ESP Ipv4 Tunnels
This is a good topic.
1. Previously I have used pfSense and I was very satisfied with that. In my new company, we are using Opnsense. I would say, working with Aliases is really headache. Creating hosts with names and combining them in one aliase takes too much time and it is also annoying.
2. In pfSense, in Firewall Rules, there is a separator which you can seperate the rules easily with different colors. It would be nice to have the same functionality in Opnsense.
3. Live view in Logs is not also user friendly and missing a lot of functions. In pfSense it is very well prepared.
4. IPsec section in Opnsense is also not very well prepared. We have almost hundred VPN connections. Working with them is not easy. First I click on Phase 1, and searching for Phase 2 which is located on the buttom of the page. Beside this, it is not possible to see the running packets, every time I need to refresh the page to see if the packets are running. In pfSense it is very nice and easy to use.
Quote from: tverweij on October 05, 2023, 03:58:48 PM
List of what I use for VPN:
- Client connections (laptops): OpenVPN instances
- Client network connections: IPSEC IPv4 IKE ESP IPv4 Tunnel
- Inter-firewall connections: IPSEC IPv4 IKEv2 ESP Route-based
This means I am in a catch-22 position:
1. I can not do SNAT on the WAN as inbound SNAT is not supported
2. I can not do SNAT on the inter-firewall connections as SNAT is not supported on route based IPSec
3. I can not upgrade to the solution of Monviech as I need IPSec ESP Ipv4 Tunnels
What I tried out was VXLAN over a Policy Based IPsec VPN to connect two firewalls with each other. I got that to work. When I have time I'll try out if I can match NAT rules on the VXLAN interface, that could potentially mitigate the problem of not having a route based NAT IPsec tunnel.
With this setup, both sites have a vxlan interface with an IP address, and a local loopback interface which the vxlan tunnel uses as local and remote IP. The Policy Based IPsec has a traffic selector with the local loopback interfaces as local and remote nets. Then you could put routes on the vxlan interfaces, and potentially also NAT into them.
Because VXLAN is Layer 2 over Layer 3 the Policy Based IPsec only needs to allow the IP addresses the VXLAN interfaces use to send and receive the traffic from.
Thanks, but it keeps coming to work arounds, just because a simple function (incoming SNAT) is not implemented.
For a second generation firewall - other than a simple router - both DNAT and SNAT should be able to work in both directions.
I'm also not a fan of this. But its a FreeBSD limitation. PFsense has the same issue:
https://redmine.pfsense.org/issues/11395
In my opinion this feature only working one or the other way is a real issue to say IPsec on FreeBSD Platforms is "enterprise ready".
I don't know what happens in this regard though, maybe there could be sponsors who pay FreeBSD developers to improve this in the future?
I have switched to WireGuard which implements a proper tunnel interface and routing for all company internal office to office VPNs. Of course that does not help with the customers and their "enterprise" gear ;)
Fortunately I do not need any NAT shenanigans with customer connections. And I can assure you that you can play all sorts of tricks including NPT6 with WireGuard interfaces just fine.
Kind regards,
Patrick
Ok, lets try wireguard - it will be a few weeks before I have the time, but I will share the results.
WireGuard lacks documentation and the terms they use to denote certain technical components are ... weird. But it's in the end all rather straightforward and simple. Very orthogonal (I like that!) to all other OPNsense features.
Feel free to reach out, but preferably on the public forum.
Quote from: Monviech on October 05, 2023, 07:28:57 PM
I'm also not a fan of this. But its a FreeBSD limitation. PFsense has the same issue:
I have to improve this comment I made before.
IPsec NAT with PF = limitations
IPsec NAT with IPFW = no limitationsIt's not a limitation of FreeBSD. It's a limitation of "pf" Firewall in Freebsd. The "ipfw" Firewall ( a different firewall implementation in FreeBSD ) doesn't have those problems, as stated by multiple users in the bug report.
There you can mix IPsec VTI and Policy Based and NAT however you like.
Quote
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474#c7
if_ipsec works with ipfw's NAT, we have many of such installations for years.
I think you use PF and it won't work in some configurations, because of its design and how IPsec handled in FreeBSD.
https://docs.freebsd.org/de/books/handbook/firewalls/#firewalls-ipfw
https://docs.freebsd.org/de/books/handbook/firewalls/#in-kernel-nat
Though since OPNsense and PFsense both use PF (it's even in the name of one) I don't know if it's likely for it ever to change.
In my opinion it's a major limitation that should be revealed much more proactively. Since I've seen multiple users (myself included) stumble over this.
Wireguard:Also additionally, Wireguard in FreeBSD has a very troubled history, and I myself had stability problems with it before (kernel panics).
https://arstechnica.com/gadgets/2021/03/buffer-overruns-license-violations-and-bad-code-freebsd-13s-close-call/
(I really like Wireguard in Linux. I use it extensively for peer to peer connections between VMs)
Quote from: Monviech on October 06, 2023, 08:02:41 AM
Wireguard:
Also additionally, Wireguard in FreeBSD has a very troubled history, and I myself had stability problems with it before (kernel panics).
I never had any issues, neither with the Golang implementation nor with the rewritten kernel module that is now in the standard installation.
Do you remember the version that gave you stability issues?
Kind regards,
Patrick
@Patrick
Yes I made a thread about it when it happened. Happened multiple times. Since then I have removed all wireguard from all DEC Hardware I use and only use IPsec and SSLVPN.
https://forum.opnsense.org/index.php?topic=35513.0
Thanks, I run the community edition everywhere. That at least explains how a completely different experience is possible.
Interesting stuff with the Wireguard,
I am using WG as well on Community editions OPN, basically cant live without out it, yet never experienced such issues. I am as well not using official HW but a miniPC knockoff. Also my WG implementation is RA. I believe you are using as well persistent WG tunnel? My is more or less on-demand used only in case I am out of the Home network.
Would be interesting to test it in a persistent way.
Thanks for your post, I'll personally keep an eye on this.
Regards,
S.
All persistent tunnels connecting our two office locations to the data centre and my home network to the Karlsruhe office. I push TrueNAS ZFS replication through that tunnel every day.
I have used Wireguard very extensively on the DEC hardware.
I had multiple instances listening (wg1, wg2, wg3 .... wgN)
Each of them had multiple site2site tunnels. And also A few different road warrior setups.
So all in all I had like 100 or more tunnels running.
Most of the tunnels were dual stack configurations. I had multiple Gigabytes of traffic every day.
Maybe that was too much for wireguard, I don't know.
Quote from: Monviech on October 05, 2023, 05:49:26 PM
What I tried out was VXLAN over a Policy Based IPsec VPN to connect two firewalls with each other. I got that to work. When I have time I'll try out if I can match NAT rules on the VXLAN interface, that could potentially mitigate the problem of not having a route based NAT IPsec tunnel.
T.W.I.M.C.
I just tried out the vxlan over IPsec between two OPNsense.
Spanned it over a policy based IPsec that had loopback interfaces as endpoints on both OPNsense.
Put a small /31 transfer net on the vxlan adapters on both sides.
I could route, NAT, everything (on the vxlan interfaces instead of using ipsecXX interfaces). So there's this way to have policy based and route based VPNs on the same opnsense box without using wireguard.
Edit: No it doesn't yet. I just judged the packet captures. The nated icmp echo request travels to the host on side b, and comes back into the vxlan interface as echo reply all the way back to the vxlan interface on site a. But there it isn't NATed back (NAT Table doesnt match somehow), it just gets blackholed there. So it's totally in line with what I've also seen with the VTI IPsec behavior. Crud. I give up here. Let's use wireguard ^^
Further Information: https://forum.opnsense.org/index.php?topic=36319.0
Is see suddenly a lot of posts with problems using Wireguard.
I also see that @Monviech respons in one of these topics that Wireguard is UDP based - this is a big problem for me as my experience is that intercintinental VPN (more that 8000 km in a straight line) using UDP gives big instability problems.
And as I have NO problems whatsoever with IPSEc (apart from the SNAT problem), Wireguard won't be the solution for me.
I am working at the automation of my own work around using a Windows machine (generated interface proxies from a PF rule); this will be the solution for me for the time-being.
> Is see suddenly a lot of posts with problems using Wireguard.
Confirmation bias does that. It's not more or less for any other VPN. Even the "simple" WireGuard protocol can be challenging.
And of course there is always a bug hidden somewhere. We can't find it without users reporting problems. ;)
Cheers,
Franco
Quote from: franco on October 10, 2023, 09:09:49 PM
> Is see suddenly a lot of posts with problems using Wireguard.
Confirmation bias does that. It's not more or less for any other VPN. Even the "simple" WireGuard protocol can be challenging.
And of course there is always a bug hidden somewhere. We can't find it without users reporting problems. ;)
Cheers,
Franco
I can only agree on this as per my experience with Enterprise vendors.
As well this is not vendor exclusive. Basically as for any vendor and any feature they are providing. Can say I've seen many weird stuff happening with many vendors.
Certain scenarios and use cases trigger certain bugs/problems.
Regards,
S.
The main problems I see with WG are due to people not leaving Unbound set to all interfaces or changing the default allow setting.
The Keepalive 25 change has been known for quote a while and keeps connections from getting stale and works well.
Should we change the keepalive default then perhaps?
Keepalive has to be set on the wireguard peer that is behind NAT. That means, it's almost always the client peer, almost never the OPNsense peer. So the current default in the OPNsense is the default that Wireguard calls best practice.
https://www.wireguard.com/quickstart/
Quote
Because NAT and stateful firewalls keep track of "connections", if a peer behind NAT or a firewall wishes to receive incoming packets, he must keep the NAT/firewall mapping valid, by periodically sending keepalive packets. This is called persistent keepalives. When this option is enabled, a keepalive packet is sent to the server endpoint once every interval seconds. A sensible interval that works with a wide variety of firewalls is 25 seconds. Setting it to 0 turns the feature off, which is the default, since most users will not need this, and it makes WireGuard slightly more chatty.
The value of 25 seconds is used because the UDP state/session timeout in most firewalls is 30 seconds. So if the mobile phone of a roadwarrior wireguard user sends a packet all 25 seconds to the opnsense, their own "socket" will stay open for the return packets.
EDIT:
I think most problems come from users who choose wireguard because they think it's easier to use. That means less users will choose ipsec, and thus there are less people having problems with it (ipsec), since they don't use it.
But here is an example that even a very simple wireguard setup can pose a big challenge because some vendors like AVM implement it badly: https://forum.opnsense.org/index.php?topic=36273.0
Sorry even more EDIT:
There's also the thing about Wireguard not using TCP-MSS Clamping by default. That creates a lot of hard to troubleshoot scenarios too. https://github.com/opnsense/docs/pull/498
For the list of this topic (I will edit the fort post), I have to add:
Firewall broke for unknown reasons, leaving all LAN segments isolated from the Internet.
The only thing that helped was restoring the Virtual Harddisk from a backup of 4 days ago. I did not restore the virtual hardware, what means that the problem was on the disk.
I never had to restore a firewall before in 25 years, so this sounds like instability in OpnSense.
I believe some people are truly unlucky running into all the (unknown) issues with the software. Most of the time, though, people simply don't like OPNsense for one reason or another and whatever fits that narrative is good enough as a justification that it must be bad.
Use the tools that work for you. There is a lot of good software (free or paid) out there.
Cheers,
Franco
Quote from: franco on October 11, 2023, 07:13:16 PM
I believe some people are truly unlucky running into all the (unknown) issues with the software. Most of the time, though, people simply don't like OPNsense for one reason or another and whatever fits that narrative is good enough as a justification that it must be bad.
Use the tools that work for you. There is a lot of good software (free or paid) out there.
Just had to restore the second firewall (Running in Amsterdam, the first one was on Curacao), with exactly the same problem, 8 hours later.
With this as an extra problem, I have to second guess the OpnSense solution; I will look at the stability in the coming time, but if this becomes normal, I have to look further.
Paid is no problem, I am coming from a paid solution and I wanted to license the Business Verion of OpnSense for all firewaals (I already have one, the rest is wating for the next version with the OpenVPN instances).
But now I wait with the licensing until I know this was only a fluke ...
I believe if you have these serious issues and no way to describe or pinpoint them this will not get any better.
Cheers,
Franco
I am afraid of that too...
If some of the developers of Deciso B.V. are reading this, if you contact me I can deliver 2 disks with an installation (HyperV) that have the problem for investigation.
Hyper-V is not a particularly recommended platform for FreeBSD. ESXi is a different story altogether. Good support.
The non internet problem continues in https://forum.opnsense.org/index.php?topic=36382.0
Quote from: Monviech on October 11, 2023, 05:09:27 PM
Keepalive has to be set on the wireguard peer that is behind NAT. That means, it's almost always the client peer, almost never the OPNsense peer. So the current default in the OPNsense is the default that Wireguard calls best practice.
https://www.wireguard.com/quickstart/
Quote
Because NAT and stateful firewalls keep track of "connections", if a peer behind NAT or a firewall wishes to receive incoming packets, he must keep the NAT/firewall mapping valid, by periodically sending keepalive packets. This is called persistent keepalives. When this option is enabled, a keepalive packet is sent to the server endpoint once every interval seconds. A sensible interval that works with a wide variety of firewalls is 25 seconds. Setting it to 0 turns the feature off, which is the default, since most users will not need this, and it makes WireGuard slightly more chatty.
The value of 25 seconds is used because the UDP state/session timeout in most firewalls is 30 seconds. So if the mobile phone of a roadwarrior wireguard user sends a packet all 25 seconds to the opnsense, their own "socket" will stay open for the return packets.
EDIT:
I think most problems come from users who choose wireguard because they think it's easier to use. That means less users will choose ipsec, and thus there are less people having problems with it (ipsec), since they don't use it.
But here is an example that even a very simple wireguard setup can pose a big challenge because some vendors like AVM implement it badly: https://forum.opnsense.org/index.php?topic=36273.0
Sorry even more EDIT:
There's also the thing about Wireguard not using TCP-MSS Clamping by default. That creates a lot of hard to troubleshoot scenarios too. https://github.com/opnsense/docs/pull/498
I've been behind NAT using WG in a road warrior setup the entire time. Previously, the WG tunnel would reconnect whenever new traffic was sent down it. After an OPNSense update, that was no longer the case and Keepalive 25 became required to prevent the connection from dropping and requiring a reconnect. So I don't believe this is due to NAT.
I'll try to remember to dig back through my history and see if I can determine roughly which update caused the change. If I get time I'll try testing with 0 as well but I'll be surprised if that works considering the recent forum posts.