Hello,With almost an year experience with OpnSense trying to accomplish the most simple enterprise setup ever (Headquarter with 2 OpnSense boxes for HA and ~10 small remote branches with a single OpnSense box, all these are single-homed to one Service Provider providing L2 ethernet service to the HQ), I would like to provide my negative feedback, mainly for folks who may consider it for similar future projects.
HA setup - who the hell may think of scale out solution and implement active-active or even active-active-active...? (Keep in mind that OpnSense is based on FreeBSD implementation of PF and CARP which are slightly different than the OpenBSD implementation, OpenBSD has better HA features)
IPSec implementation (...if your branch office lost connectivity for a day - don't expect IPSec to re-establish again in case you haven't modified the config files and rely only on FrontEnd UI)
Lack of ECMP feature .... yes, you can't have 2 routes (static or dynamic) for the same destination towards 2 different gateways, so you can not do traffic load-sharing
Lack of ability to fine tune the PF firewall rules....unless you want to spend a week bugging with ...php scripts
Impossibility to add an additional VLAN without bouncing the main physical interface, even though it is a LACP bonding - "well the interruption is a really short...and we're not adding vlans every day, are we?
Jumbo frames? Yes, it's supported, even it works, but only if you enable it at the very beginning. Of course if the jumbo frame MTU requirement comes a little bit late and you have already deployed your vlan subinterfaces - you'll need to start over with a fresh install
We are not considering to have a DHCP relay for a remote/branch office pointing to the DHCP server located in the Headquarter over VTI IPSec interface, aren't we? Yes, the buggy daemon doesn't like ipsecXXX interface type and cannot even start
Dynamic Routing using FRR - Yes, but don't expect to use the FrontEnd UI, unless you want your BGP daemon to restart each time when you have to make a small modification like adding a new BGP neighbor, or even modifying a simple route-map. OSPFv2/v3 is not different. More advanced routing setup like BGP Route-Reflector or Route-Server - forget it, these may be implemented in the UI in 2122 if we're lucky (and yes, they won't work as expected)
Multicast routing? PIM? IGMPv1/2/3? .... Forget it all,moreover the highly limited igmp-proxy doesn't like Lagg interface types (and even vlans in older versions), so can't bind and start at all and even if you can make it to work somehow - don't expect any functionality to replicate the multicast state between 2 HA OpnSense devices..and again, who is looking for HA?
VRF functionality, at least for the management traffic - no, not supported by BSD (although JunOS is running on top of FreeBSD for centuries). So, be careful what you're doing with routing, PF, badly documented check boxes on the UI just to avoid locking yourself
Lack of documentation? Really? Try to understand something as simple as how the static routing "gateway" concept is working and you'll understand what I mean. During that time, expect your statically configured default gateway suddenly to start pointing to a different interface or being overwritten by dynamic protocol
. I would firmly say it's not even being develop with "more than 2 OpnSense boxes in a network" mindset.
Hi, Just add some comments reading up on this. I am not sure you are quite familiar to know what OPNSense really is... It's a firewall first and not a router.Now on some things for others to know in regards to DHCP. It is mandatory for any organization to have a local DHCP. No one under any circumstances ought to use DHCP relay across a WAN interface EVEN if it's L2 ethernet (YOU DON'T OWN THE WIRE!!!) To add to this, I can create an AToM (Any Transport over MPLS) across Frame/ATM/ISDN and give you a L2 ethernet. So, NO! you got no clue what you got... LOLAs for other Routing features, if anything not working, please file your complaints to https://frrouting.org/.you do have one or two nice complaints, but all others... Meh!edit: just outta curiosity, what did you end up replacing OPNSense with?
Quote from: lilsense on October 09, 2021, 10:07:41 pmHi, Just add some comments reading up on this. I am not sure you are quite familiar to know what OPNSense really is... It's a firewall first and not a router.Now on some things for others to know in regards to DHCP. It is mandatory for any organization to have a local DHCP. No one under any circumstances ought to use DHCP relay across a WAN interface EVEN if it's L2 ethernet (YOU DON'T OWN THE WIRE!!!) To add to this, I can create an AToM (Any Transport over MPLS) across Frame/ATM/ISDN and give you a L2 ethernet. So, NO! you got no clue what you got... LOLAs for other Routing features, if anything not working, please file your complaints to https://frrouting.org/.you do have one or two nice complaints, but all others... Meh!edit: just outta curiosity, what did you end up replacing OPNSense with?Hi lilsense, First of all thanks for the comment. Few points that you've mentioned - DHCP relay - do you really think that feature should not work? I'm coming from the Enterprise and Service provider segments and for the last almost 20 years I haven't seen a vendor where DHCP relay doesn't work. Technically it is so simple - just listen on broadcast DHCP messages and forward it to an unicast destination (based on the configured unicast IP and routing table) and vice versa (and of course do a slight modification of few DHCP fields in the packet format)Regarding the design decision to rely on central DHCP server in the HQ and not on local one - it really depends. Yes, I agree it is not wise all hosts in a branch office to lose IP addresses and not to be able to communicate locally, because of WAN failure. But why you believe this is the case here? As I said - the setup I'm working on is quite simple, so simple that all hosts (literally few) in each remote location are configured with static addressing. The DHCP relay was intended to be used only the deployment phase (PXE).Regarding the firewall vs routing platform - I'm trying to accomplish really simple things which are widely available in the last 20 years in open source implementations like Zebra/Quagga/FRR! I'm not talking about MPLS and its applications like L2VPNs, L3VPNs, TE, DiffServ Aware TE, AToM, CsC, Inter-AS options (like A,B,C,AB). I'm not talking about segment routing. I'm not talking about DMVPN or vendor specific solutions like GetVPN, FlexVPN, etc) I'm not talking about various SD-WAN solutions that each vendor (including Firewall Vendors) provides nowadays. I'm not talking about really specific BGP and OSPF features like MP-BGP transporting IPv6 over IPv4 AFI or vice versa, selective BGP next-hop address tracking, BGP confederations, BGP as-path ignore or relax, complex multi-area OSPF area stub and NSSA areas, OSPF Forward Address tricks, OSPF Type7-to-Type5 translator selection, OSPFv3 Authentication Trailer and so on. All I'm talking about is a CORE FUNCTIONALITY which is not missing from FRR (yes, it's there, it was there in Quagga, it was there even in Zebra in 2004), but it is missing in OpnSense FrontEnd. Tell me a single firewall vendor which doesn't support IBGP Route Reflector? (is it a good design of using a firewall as a BGP RR is completely different story). An year ago it was even not possible to change the keepalive/holdtime timers for the BGP session via GUI and we had to rely on the peer device to do it! Going back to the firewall functionality - the only reason of having less complaints for it in the list above is just because the setup I'm using is really, really simple. And actually it cannot evolves to more complex one, definitely not with OpnSense. I can't imagine dealing with more than 100 rules and with more than 5 interfaces! Imagine I need to move a rule to the middle or rearrange the rules a little bit.... I'm pretty sure that editing the pf.conf with VI will be much faster than trying to accomplish this with the Front-End. And of course - you can still generate a completely invalid rule with the Front-End and try to insert it (the same thing I can do with the text editor,right?). I'm also grateful that I don't need to play with NATs in that deployment. What did I end up replacing OPNSense with - I haven't replaced that yet, and the only reason for that is the minority of changes expected after the go-live. My thought process when we selected OpnSense was to check the support for the required features - does it support DHCP relay - Yes (is it working for me - NO, but that's not written anywhere), does it support dynamic routing - Yes (is it working for me as it is - NO and again that's not documented), does it support HA - Yes (does it for active-active - No, again not documented), does it support VLANs - yes (do I need to interrupt the traffic when I need to add additional VLAN - yes, and again lacks in the documentation) For future low cost projects I would definitely prefer a fresh Linux based distribution with iproute2, frr, iptables/nftables. At least I'll be able to achieve the routing part. For firewall part - probably I'll go with OpenBSD (In case I can't make the HA as I want in linux). Will it be with a shiny web based UI - No, it won't. Will it work - Yes, it will. Will I have to spend time debugging Front-End and Back-End interaction just to generate a simple config file for 3rd party open source product usually with a good documentation and tons of examples in Internet - no.
First of all, let me clarify that I have no relationship with OPNSense and that it is not my interest to have a discussion, but I think it’s a partial view to state that the product cannot be used in a large company. May be no the right product for you, sure.We personally migrated a couple of old OpenBSD firewalls to OPNSense and it is working perfectly.It is not so small. It has 3 internet access (separating different traffic for each one), several internal interfaces, plus several VPNs with OpenVPN and IPSec. There are about 400 servers on the network and about 3000 users who use it. In adition 400 remote OpenVPN users. Indeed we had some inconvenience (I don't remember which one now) with options that were not available in the VRRP administration interface. We analyzed modifying the plugin and decided that it was simpler for us to use vrrp directly.I have many years of Linux / Unix experience and I appreciate that FreeBSD is underneath.We were also able to add external scripts and cron jobs for certain very specific things that were used in the old firewalls, such as upadate a dynamic DNS when an internet link stops working or parse some site to obtain a list of IP address, etc etc We plan in the future add IDS/IPS or Sensei to the firewall.It has limitations like any product, but its Open Source base allows it to be adapted with more or less effort. I thinks a closed product it’s not so versatile.It is true that it is faster to edit pf.conf, but if the user (as is our case) is not a Linux / Unix specialist, he is deeply grateful to have a friendly interface to add a firewall rule or simply validate it. Or check the traffic or even add a VPN user, without needing to edit a single file.Sure, it would be nice to be able to drag a ruler to place it in a given position or put a separator to facilitate reading, but for us it’s not the end of the world.One last comment regarding the OpenBSD CARP active/active. It works perfect, but did you try to use it in a firewall? For us was a real headache to get it to work, and is related in the way OpenBSD select which cluster member takes the traffic. In fact already in OpenBSD we had switched to active/passive.While OPNSense may have been originally intended for a small business environment, it is perfectly adaptable to a much more demanding one. If we enter the cost into the equation (even paying for support from Deciso), the product goes from good to excellent.Sorry for the long message and if something is not totally clear, explaining all this was a lot for my english.RegardsNorberto
