OPNsense Forum

English Forums => General Discussion => Topic started by: pmladenov on October 09, 2021, 07:52:17 pm

Title: OpnSense in small Enterprise segment - negative feedback
Post by: pmladenov on October 09, 2021, 07:52:17 pm
Hello,

With almost an year experience with OpnSense trying to accomplish the most simple enterprise setup ever (Headquarter with 2 OpnSense boxes for HA and ~10 small remote branches with a single OpnSense box, all these are single-homed to one Service Provider providing L2 ethernet service to the HQ), I would like to provide my negative feedback, mainly for folks who may consider it for similar future projects.

Overall I'm disappointed.
I'm disappointed with, but not limited to:
 
I understand that some of the limitations listed above are not related to OpnSense itself, but with FreeBSD or some 3rd party plugins.
I also understand that the product is mainly for home users bragging about having "c00l f1r3w@ll @home" but it's far, far, far away for enterprise ready solution.
I would firmly say it's not even being develop with "more than 2 OpnSense boxes in a network" mindset.

On a positive side - I would say - if you are not expecting to change anything after the initial deployment and have all the requirements in advance (and the listed caveats above) - it works and is stable.

What else I'm missing here? What is your experience?
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: lilsense on October 09, 2021, 10:07:41 pm
Hi,
    Just add some comments reading up on this. I am not sure you are quite familiar to know what OPNSense really is... It's a firewall first and not a router.

Now on some things for others to know in regards to DHCP. It is mandatory for any organization to have a local DHCP. No one under any circumstances ought to use DHCP relay across a WAN interface EVEN if it's L2 ethernet (YOU DON'T OWN THE WIRE!!!) To add to this, I can create an AToM (Any Transport over MPLS) across Frame/ATM/ISDN and give you a L2 ethernet. So, NO! you got no clue what you got... LOL

As for other Routing features, if anything not working, please file your complaints to https://frrouting.org/.

you do have one or two nice complaints, but all others... Meh!


edit: just outta curiosity, what did you end up replacing OPNSense with?
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: mimugmail on October 10, 2021, 08:03:47 am
Hello,

With almost an year experience with OpnSense trying to accomplish the most simple enterprise setup ever (Headquarter with 2 OpnSense boxes for HA and ~10 small remote branches with a single OpnSense box, all these are single-homed to one Service Provider providing L2 ethernet service to the HQ), I would like to provide my negative feedback, mainly for folks who may consider it for similar future projects.

This is really easy.

Quote
  • HA setup - who the hell may think of scale out solution and implement active-active or even active-active-active...? (Keep in mind that OpnSense is based on FreeBSD implementation of PF and CARP which are slightly different than the OpenBSD implementation, OpenBSD has better HA features)

This is how CARP works in FreeBSD, you should complain upstream

Quote
  • IPSec implementation (...if your branch office lost connectivity for a day - don't expect IPSec to re-establish again in case you haven't modified the config files and rely only on FrontEnd UI)

Set keyingtries to -1

Quote
  • Lack of ECMP feature .... yes, you can't have 2 routes (static or dynamic) for the same destination towards 2 different gateways, so you can not do traffic load-sharing

ECMP will work with FBSD13, so it will come with 22.1

Quote
  • Lack of ability to fine tune the PF firewall rules....unless you want to spend a week bugging with ...php scripts

Example please

Quote
  • Impossibility to add an additional VLAN without bouncing the main physical interface, even though it is a LACP bonding - "well the interruption is a really short...and we're not adding vlans every day, are we?

Thats trues, 1-2 seconds, if you dont disable STP it's 20 seconds.

Quote

  • Jumbo frames? Yes, it's supported, even it works, but only if you enable it at the very beginning. Of course if the jumbo frame MTU requirement comes a little bit late and you have already deployed your vlan subinterfaces - you'll need to start over with a fresh install

WHO is using jumbo frames on a firewall?? In a small setup like you describe??

Quote
  • We are not considering to have a DHCP relay for a remote/branch office pointing to the DHCP server located in the Headquarter over VTI IPSec interface, aren't we? Yes, the buggy daemon doesn't like ipsecXXX interface type and cannot even start

This is open source, it lives from it's community, did you try to open an issue via github?

Quote
  • Dynamic Routing using FRR - Yes, but don't expect to use the FrontEnd UI, unless you want your BGP daemon to restart each time when you have to make a small modification like adding a new BGP neighbor, or even modifying a simple route-map. OSPFv2/v3 is not different. More advanced routing setup like BGP Route-Reflector or Route-Server - forget it, these may be implemented in the UI in 2122 if we're lucky (and yes, they won't work as expected) 

That's how templating in a framework works. You set somthing in UI and it's generating a config.
FRR on itself has a sloppy pyton script for reload ... really sloppy. I dont think OPN will change it's way to work cause of this. I can't imagine you need route reflectors in such a small setup.
I use OpenVPN to connect to multiple DC's, branch can have unlimited WANs, eBGP on the outside and OSPF inside datacenter .. all via UI. When a new branch opens it needs a restart, but this happens less ofter than an update will be done.

Quote
  • Multicast routing? PIM? IGMPv1/2/3? .... Forget it all,moreover the highly limited igmp-proxy doesn't like Lagg interface types (and even vlans in older versions), so can't bind and start at all and even if you can make it to work somehow - don't expect any functionality to replicate the multicast state between 2 HA OpnSense devices..and again, who is looking for HA?

Tried udpbroadcastrelay plugin?
I can add pimd plugin if you know this will fit your needs.

Quote
  • VRF functionality, at least for the management traffic - no, not supported by BSD (although JunOS is running on top of FreeBSD for centuries). So, be careful what you're doing with routing, PF, badly documented check boxes on the UI just to avoid locking yourself

You don't need to be an expert to block such traffic. Don't forget this firewall doesn't cost anything, are you aware of any free firewall supporting VRFs?

Quote
  • Lack of documentation? Really? Try to understand something as simple as how the static routing "gateway" concept is working and you'll understand what I mean. During that time, expect your statically configured default gateway suddenly to start pointing to a different interface or being overwritten by dynamic protocol

To set a route you need t add a gateway, for gateway failover you have priorities since 20.1. This is ran by scripts which of course will interfere with FRR. If you build a box for such cases it has to be limited in features like an ASA or Sophos, but community loves to be free :)

Quote
.
I would firmly say it's not even being develop with "more than 2 OpnSense boxes in a network" mindset.

I would firmly say it's only your frustration :)

Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: pmladenov on October 10, 2021, 08:30:40 am
Hi,
    Just add some comments reading up on this. I am not sure you are quite familiar to know what OPNSense really is... It's a firewall first and not a router.

Now on some things for others to know in regards to DHCP. It is mandatory for any organization to have a local DHCP. No one under any circumstances ought to use DHCP relay across a WAN interface EVEN if it's L2 ethernet (YOU DON'T OWN THE WIRE!!!) To add to this, I can create an AToM (Any Transport over MPLS) across Frame/ATM/ISDN and give you a L2 ethernet. So, NO! you got no clue what you got... LOL

As for other Routing features, if anything not working, please file your complaints to https://frrouting.org/.

you do have one or two nice complaints, but all others... Meh!


edit: just outta curiosity, what did you end up replacing OPNSense with?

Hi lilsense,

First of all thanks for the comment.

Few points that you've mentioned - DHCP relay - do you really think that feature should not work? I'm coming from the Enterprise and Service provider segments and for the last almost 20 years I haven't seen a vendor where DHCP relay doesn't work. Technically it is so simple - just listen on broadcast DHCP messages and forward it to an unicast destination (based on the configured unicast IP and routing table) and vice versa (and of course do a slight modification of few DHCP fields in the packet format)
Regarding the design decision to rely on central DHCP server in the HQ and not on local one - it really depends. Yes, I agree it is not wise all hosts in a branch office to lose IP addresses and not to be able to communicate locally, because of WAN failure. But why you believe this is the case here? As I said - the setup I'm working on is quite simple, so simple that all hosts (literally few) in each remote location are configured with static addressing. The DHCP relay was intended to be used only the deployment phase (PXE).

Regarding the firewall vs routing platform - I'm trying to accomplish really simple things which are widely available in the last 20 years in open source implementations like Zebra/Quagga/FRR! I'm not talking about MPLS and its applications like L2VPNs, L3VPNs, TE, DiffServ Aware TE, AToM, CsC, Inter-AS options (like A,B,C,AB). I'm not talking about segment routing. I'm not talking about DMVPN or vendor specific solutions like GetVPN, FlexVPN, etc)
 I'm not talking about various SD-WAN solutions that each vendor (including Firewall Vendors) provides nowadays. I'm not talking about really specific BGP and OSPF features like MP-BGP transporting IPv6 over IPv4 AFI or vice versa, selective BGP next-hop address tracking, BGP confederations, BGP as-path ignore or relax, complex multi-area OSPF area stub and NSSA areas, OSPF Forward Address tricks, OSPF Type7-to-Type5 translator selection, OSPFv3 Authentication Trailer and so on. All I'm talking about is a CORE FUNCTIONALITY which is not missing from FRR (yes, it's there, it was there in Quagga, it was there even in Zebra in 2004), but it is missing in OpnSense FrontEnd. Tell me a single firewall vendor which doesn't support IBGP Route Reflector? (is it a good design of using a firewall as a BGP RR is completely different story). An year ago it was even not possible to change the keepalive/holdtime timers for the BGP session via GUI and we had to rely on the peer device to do it! 

Going back to the firewall functionality - the only reason of having less complaints for it in the list above is just because the setup I'm using is really, really simple. And actually it cannot evolves to more complex one, definitely not with OpnSense. I can't imagine dealing with more than 100 rules and with more than 5 interfaces! Imagine I need to move a rule to the middle or rearrange the rules a little bit.... I'm pretty sure that editing the pf.conf with VI will be much faster than trying to accomplish this with the Front-End. And of course - you can still generate a completely invalid rule with the Front-End and try to insert it (the same thing I can do with the text editor,right?). I'm also grateful that I don't need to play with NATs in that deployment.
 
What did I end up replacing OPNSense with - I haven't replaced that yet, and the only reason for that is the minority of changes expected after the go-live.
My thought process when we selected OpnSense was to check the support for the required features - does it support DHCP relay - Yes (is it working for me - NO, but that's not written anywhere), does it support dynamic routing - Yes (is it working for me as it is - NO and again that's not documented), does it support HA - Yes (does it for active-active - No, again not documented), does it support VLANs - yes (do I need to interrupt the traffic when I need to add additional VLAN - yes, and again lacks in the documentation)   

For future low cost projects I would definitely prefer a fresh Linux based distribution with iproute2, frr, iptables/nftables. At least I'll be able to achieve the routing part. For firewall part - probably I'll go with OpenBSD (In case I can't make the HA as I want in linux). Will it be with a shiny web based UI - No, it won't. Will it work - Yes, it will. Will I have to spend time debugging Front-End and Back-End interaction just to generate a simple config file for 3rd party open source product usually with a good documentation and tons of examples in Internet - no.




Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: mimugmail on October 10, 2021, 09:36:34 am
You can also use frr pkg without GUI and all the features. You compare CLI-only with OPNsense?  ::)
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: franco on October 10, 2021, 11:55:28 am
I'm really curious what your corporate firewall budget is for a year?


Cheers,
Franco
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: bimbar on October 10, 2021, 01:59:11 pm
I think compared to most other small firewalls, be it open or closed source, opnsense compares quite favorably.
If you have so many problems with it, maybe it's not the right tool for the job?
It can often be a good idea to operate a router for the router stuff and a firewall for the packet filtering.
For advanced BGP setups, maybe take a look at mikrotik?
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: allebone on October 10, 2021, 08:28:20 pm
For the HA option are you just saying you want active-backup option?
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: lilsense on October 11, 2021, 01:26:23 am
Hi,
    Just add some comments reading up on this. I am not sure you are quite familiar to know what OPNSense really is... It's a firewall first and not a router.

Now on some things for others to know in regards to DHCP. It is mandatory for any organization to have a local DHCP. No one under any circumstances ought to use DHCP relay across a WAN interface EVEN if it's L2 ethernet (YOU DON'T OWN THE WIRE!!!) To add to this, I can create an AToM (Any Transport over MPLS) across Frame/ATM/ISDN and give you a L2 ethernet. So, NO! you got no clue what you got... LOL

As for other Routing features, if anything not working, please file your complaints to https://frrouting.org/.

you do have one or two nice complaints, but all others... Meh!


edit: just outta curiosity, what did you end up replacing OPNSense with?

Hi lilsense,

First of all thanks for the comment.

Few points that you've mentioned - DHCP relay - do you really think that feature should not work? I'm coming from the Enterprise and Service provider segments and for the last almost 20 years I haven't seen a vendor where DHCP relay doesn't work. Technically it is so simple - just listen on broadcast DHCP messages and forward it to an unicast destination (based on the configured unicast IP and routing table) and vice versa (and of course do a slight modification of few DHCP fields in the packet format)
Regarding the design decision to rely on central DHCP server in the HQ and not on local one - it really depends. Yes, I agree it is not wise all hosts in a branch office to lose IP addresses and not to be able to communicate locally, because of WAN failure. But why you believe this is the case here? As I said - the setup I'm working on is quite simple, so simple that all hosts (literally few) in each remote location are configured with static addressing. The DHCP relay was intended to be used only the deployment phase (PXE).

Regarding the firewall vs routing platform - I'm trying to accomplish really simple things which are widely available in the last 20 years in open source implementations like Zebra/Quagga/FRR! I'm not talking about MPLS and its applications like L2VPNs, L3VPNs, TE, DiffServ Aware TE, AToM, CsC, Inter-AS options (like A,B,C,AB). I'm not talking about segment routing. I'm not talking about DMVPN or vendor specific solutions like GetVPN, FlexVPN, etc)
 I'm not talking about various SD-WAN solutions that each vendor (including Firewall Vendors) provides nowadays. I'm not talking about really specific BGP and OSPF features like MP-BGP transporting IPv6 over IPv4 AFI or vice versa, selective BGP next-hop address tracking, BGP confederations, BGP as-path ignore or relax, complex multi-area OSPF area stub and NSSA areas, OSPF Forward Address tricks, OSPF Type7-to-Type5 translator selection, OSPFv3 Authentication Trailer and so on. All I'm talking about is a CORE FUNCTIONALITY which is not missing from FRR (yes, it's there, it was there in Quagga, it was there even in Zebra in 2004), but it is missing in OpnSense FrontEnd. Tell me a single firewall vendor which doesn't support IBGP Route Reflector? (is it a good design of using a firewall as a BGP RR is completely different story). An year ago it was even not possible to change the keepalive/holdtime timers for the BGP session via GUI and we had to rely on the peer device to do it! 

Going back to the firewall functionality - the only reason of having less complaints for it in the list above is just because the setup I'm using is really, really simple. And actually it cannot evolves to more complex one, definitely not with OpnSense. I can't imagine dealing with more than 100 rules and with more than 5 interfaces! Imagine I need to move a rule to the middle or rearrange the rules a little bit.... I'm pretty sure that editing the pf.conf with VI will be much faster than trying to accomplish this with the Front-End. And of course - you can still generate a completely invalid rule with the Front-End and try to insert it (the same thing I can do with the text editor,right?). I'm also grateful that I don't need to play with NATs in that deployment.
 
What did I end up replacing OPNSense with - I haven't replaced that yet, and the only reason for that is the minority of changes expected after the go-live.
My thought process when we selected OpnSense was to check the support for the required features - does it support DHCP relay - Yes (is it working for me - NO, but that's not written anywhere), does it support dynamic routing - Yes (is it working for me as it is - NO and again that's not documented), does it support HA - Yes (does it for active-active - No, again not documented), does it support VLANs - yes (do I need to interrupt the traffic when I need to add additional VLAN - yes, and again lacks in the documentation)   

For future low cost projects I would definitely prefer a fresh Linux based distribution with iproute2, frr, iptables/nftables. At least I'll be able to achieve the routing part. For firewall part - probably I'll go with OpenBSD (In case I can't make the HA as I want in linux). Will it be with a shiny web based UI - No, it won't. Will it work - Yes, it will. Will I have to spend time debugging Front-End and Back-End interaction just to generate a simple config file for 3rd party open source product usually with a good documentation and tons of examples in Internet - no.


Oh boy...DMVPN, flexvpn??? WOW!!!! Did you want OPNSense to have Cisco IOS command line style or Cisco Cat OS, or Cisco IOS XR, or Cisco ASA, etc... Cisco seems to have lost what they want to do...

Now trying to set up dhcp relay on any VPN tunnel is not easy and needs a lot of delicacies to get it working just like OPNSense, but why torture one's self to do this while you can use the local dhcp server which dhcp relay works marvelously.

I think that anytime someone is setting up a firewall as an BGP RR they have no idea what they are doing... As you have mentioned IBGP ( is internal not external) firewall would be an edge device hence, external which means , no workie router reflector on a BGP on a Firewall IBGP... LOL

There a lot of terms you throw around that would outside the scope of OPNSense. Even some of the best firewalls fail miserably at these to name a few, Fortinet, Palo, and Juniper.

It's always best to NEVER keep all your eggs in one basket, i.e. don't use an iPhone to take pictures, there are cameras that do that. Now you are more than welcome to throw all your eggs in one, but because you know the result stop complaining about the outcome. You should have proper switches to do switching, proper routers to do routing and proper firewalls (OPNSense) to the everything else...

as for asking Cisco which ASA is the fastest firewall, they tell you it's an ASR router... ROFL...
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: naltalef on October 14, 2021, 12:26:34 am
First of all, let me clarify that I have no relationship with OPNSense and that it is not my interest to have a discussion, but I think it’s a partial view to state that the product cannot be used in a large company. May be no the right product for you, sure.

We personally migrated a couple of old OpenBSD firewalls to OPNSense and it is working perfectly.
It is not so small. It has 3 internet access (separating different traffic for each one), several internal interfaces, plus several VPNs with OpenVPN and IPSec.
There are about 400 servers on the network and about 3000 users who use it. In adition 400 remote OpenVPN users.

Indeed we had some inconvenience (I don't remember which one now) with options that were not available in the VRRP administration interface.
We analyzed modifying the plugin and decided that it was simpler for us to use vrrp directly.
I have many years of Linux / Unix experience and I appreciate that FreeBSD is underneath.

We were also able to add external scripts and cron jobs for certain very specific things that were used in the old firewalls, such as upadate a dynamic DNS when an internet link stops working or parse some site to obtain a list of IP address, etc etc 
We plan in the future add IDS/IPS or Sensei to the firewall.

It has limitations like any product, but its Open Source base allows it to be adapted with more or less effort. I thinks a closed product it’s not so versatile.

It is true that it is faster to edit pf.conf, but if the user (as is our case) is not a Linux / Unix specialist, he is deeply grateful to have a friendly interface to add a firewall rule or simply validate it. Or check the traffic or even add a VPN user, without needing to edit a single file.
Sure, it would be nice to be able to drag a ruler to place it in a given position or put a separator to facilitate reading, but for us it’s not the end of the world.

One last comment regarding the OpenBSD CARP active/active. It works perfect, but did you try to use it in a firewall? For us was a real headache to get it to work, and is related in the way OpenBSD select which cluster member takes the traffic. In fact already in OpenBSD we had switched to active/passive.

While OPNSense may have been originally intended for a small business environment, it is perfectly adaptable to a much more demanding one.
If we enter the cost into the equation (even paying for support from Deciso),  the product goes from good to excellent.

Sorry for the long message and if something is not totally clear, explaining all this was a lot for my english.

Regards
Norberto
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: naltalef on October 14, 2021, 12:52:37 am
Sorry.
Indeed we had some inconvenience (I don't remember which one now) with options that were not available in the VRRP administration interface.
We analyzed modifying the plugin and decided that it was simpler for us to use vrrp directly.

Must be read as:

Indeed we had some inconvenience (I don't remember which one now) with options that were not available in the FRR administration interface.
We analyzed modifying the plugin and decided that it was simpler for us to use FRR directly.

First of all, let me clarify that I have no relationship with OPNSense and that it is not my interest to have a discussion, but I think it’s a partial view to state that the product cannot be used in a large company. May be no the right product for you, sure.

We personally migrated a couple of old OpenBSD firewalls to OPNSense and it is working perfectly.
It is not so small. It has 3 internet access (separating different traffic for each one), several internal interfaces, plus several VPNs with OpenVPN and IPSec.
There are about 400 servers on the network and about 3000 users who use it. In adition 400 remote OpenVPN users.

Indeed we had some inconvenience (I don't remember which one now) with options that were not available in the VRRP administration interface.
We analyzed modifying the plugin and decided that it was simpler for us to use vrrp directly.
I have many years of Linux / Unix experience and I appreciate that FreeBSD is underneath.

We were also able to add external scripts and cron jobs for certain very specific things that were used in the old firewalls, such as upadate a dynamic DNS when an internet link stops working or parse some site to obtain a list of IP address, etc etc 
We plan in the future add IDS/IPS or Sensei to the firewall.

It has limitations like any product, but its Open Source base allows it to be adapted with more or less effort. I thinks a closed product it’s not so versatile.

It is true that it is faster to edit pf.conf, but if the user (as is our case) is not a Linux / Unix specialist, he is deeply grateful to have a friendly interface to add a firewall rule or simply validate it. Or check the traffic or even add a VPN user, without needing to edit a single file.
Sure, it would be nice to be able to drag a ruler to place it in a given position or put a separator to facilitate reading, but for us it’s not the end of the world.

One last comment regarding the OpenBSD CARP active/active. It works perfect, but did you try to use it in a firewall? For us was a real headache to get it to work, and is related in the way OpenBSD select which cluster member takes the traffic. In fact already in OpenBSD we had switched to active/passive.

While OPNSense may have been originally intended for a small business environment, it is perfectly adaptable to a much more demanding one.
If we enter the cost into the equation (even paying for support from Deciso),  the product goes from good to excellent.

Sorry for the long message and if something is not totally clear, explaining all this was a lot for my english.

Regards
Norberto
Title: Re: OpnSense in small Enterprise segment - negative feedback
Post by: mimugmail on October 14, 2021, 06:40:21 am
Thx for correcting the initial post.
I'm always here to help if someone wants to add features. The Framework is so easy that you can also add content with simple if/then/else experience.
Thats the only thing I can offer since due to Corona my time for Open source dramatically decreased