Production use

[peksi]

I would appreciate your opinions on using Opnsense in production business environment.

I have 2 dual Xeon CentOS 7 virtual hosts and I am considering using KVM virtualized Opnsense with VLAN tagging to switches via 10G ethernet.

In your opinion what are the main differences / benefits / warnings between hardware and virtualized Opnsense? And if anyone has experience on commercial firewalls then how would you compare reliability and overall usage between Opnsense and them? There is a major difference in price tags so I am not going to commercial side with light grounds nor I am willing to risk my environment. But I have done well the past 15 years with Linux routing, iptables and openvpn so I am leaning towards open source route.

Thanks for your help in advance.

[FrenchFries]

IMHO:

OPNsense offers FreeBSD+pf firewall.

pf v.s. iptables
pf is superior to iptables, especially for logging and managing long rules. Iptables is one large blog and when logging you have to create an INPUT/OUTPUT chain for each rule, so technically speaking it will triple lenght of rules. Each time you add or remove a rule, you need to reload it and will loose stated. iptables is a technology of the 1990s, released in 1998. If you jump an iptables firewall, nobody will notice, as most people don't really log.

Linux v.s. FreeBSD.
FreeBSD networking and security design is way superior to Linux.
I used both in production environments and I can confirm I now have a preference for FreeBSD.

Now about virtualization.

Obviously, you will loose power, but it will work.
If you have some room, buy a small appliance or use your own dedicated hardware.
If this is an essential device for the sake of you company, don't virtualize because Linux still does not isolate root users very well and could have a lof of zero day attacks.

The fact is that the combination of OPNSense and FreeBSD (hardened) is now rock-solid.

[franco]

Hi there,

KVM works, but network performance is not so great for multiple reasons (FreeBSD "hardware" offloading can be buggy on top of hypervisors; sometimes the drivers are causing compatibility issues), especially if you plan to push 10G. Linux gives you more "bang for the buck", hands down.

Use e1000 NIC emulation if you have trouble with OPNsense and evaluate from there if the result is what you expect.


Cheers,
Franco

[whitwye]

iptables
... Each time you add or remove a rule, you need to reload it and will loose stated. iptables is a technology of the 1990s, released in 1998. If you jump an iptables firewall, nobody will notice, as most people don't really log.

Not true. Netfilter/iptables can add or remove one rule at a time. No need to reload at all. And this has been the case since the beginning. Also, I suspect most everyone logs, and standard log analysis tools have recipes to extract from those logs.

Linux v.s. FreeBSD.
FreeBSD networking and security design is way superior to Linux.
I used both in production environments and I can confirm I now have a preference for FreeBSD.

The main advantage from my POV is that OPNsense has a good GUI, with integrated failover functions and so on. You can build a very capable firewall on Linux -- especially using a front-end like FireHOL -- but you have to comfortable at the console, and with doing your own integration and logic for failover and the like. And you won't have a pretty set of screens to share with CLI-adverse fellow admins.

On the other and FreeBSD is less documented than Linux, so when you run into a question about the system behind OPNsense, you're not so likely to quickly find the answer through Google.

[FrenchFries]

Not true. Netfilter/iptables can add or remove one rule at a time. No need to reload at all. And this has been the case since the beginning. Also, I suspect most everyone logs, and standard log analysis tools have recipes to extract from those logs.

Sorry, I meant iptablse source code:

iptables rules are treated internally as a single blob that needs to be reloaded automatically. This is why adding a rule or removing a rule is so long with complex firewalling rules. But I agree this is done automatically.

Read this about iptables (my post):
https://lede-project.org/docs/howto/log-rejected-packets#logging_limitation_in_openwrt_and_lede

For example, to log iperf traffic, you will need to write:

 #IPv4
$IPTABLES -A In_RULE_6  -j LOG  --log-level info --log-prefix "VAIO 6 -- ACCEPT "
$IPTABLES -A Out_RULE_6  -j LOG  --log-level info --log-prefix "VAIO 6 -- ACCEPT "

The same is written for IPv6:

$IPTABLES6 -A In_RULE_6  -j LOG  --log-level info --log-prefix "VAIO 6 -- ACCEPT "
$IPTABLES6 -A Out_RULE_6  -j LOG  --log-level info --log-prefix "VAIO 6 -- ACCEPT "

You need to create a single INPUT and OUTPUT target for each rule.
Multiply by 2 if you want ipv6.

iptables was not designed for mass-logging.

FreeBSD pf is a mature firewalling system, which can reject and log on the same line.

[franco]

On the other and FreeBSD is less documented than Linux, so when you run into a question about the system behind OPNsense, you're not so likely to quickly find the answer through Google.

Maybe if you use "documentation" as a synonym for stack overflow through a google search.

Not in terms of actual documentation, namely manual pages and handbooks.

One of the reasons why I prefer BSD over Linux any day.

I agree it's a little hard to learn to experience this, and thinking back 15 years, typing "man ifconfig" and actually starting to read is really hard at first.


Cheers,
Franco

[FrenchFries]

FreeBSD is very well documented and well-written (style).
Visit FreeBSD.org

GNU/Linux has serious design and security flaws.

Let's not start a flamewar.

[whitwye]

Hey, I'm not trying to start a flame war. I'm friendly to OPNsense.

But I've read the docs at FreeBSD.org. They're shallow and incomplete. But then documentation quality has gone down in all areas of IT since the 90s, when even MS-DOS came with hundreds of pages of manuals.

Admittedly there's no single source for the perfect Linux doc either. But there are a lot more docs about. The Arch Linux project on the whole is has the best quality. I've been a Linux sysadmin for 22 years. It's not like the terminology is opaque to me.

[fabian]

You may have a look on nftables which does not have some of the disadvantages of iptables anymore (man page: http://www.netfilter.org/projects/nftables/manpage.html) and replaces iptables. You can write log into the rule if you like. The syntax is more like the one of pf.

In my opinion, Linux is not less secure but it is not as conservative as FreeBSD is. Which means, it is more likely that an API changes in Linux that in FreeBSD (note that Linux usually keeps support for old APIs). If you feel well on Linux there is no reason not to use it. The advantage of an appliance is, that you don't have to touch the configs directly and it manages all the stuff for you like updating config files, input validaton, etc. and an appliance usually includes the glue code between the components.
For example, in OPNsense is a PAM module included, which lets you authenticate against any auth server. The same is valid for the GUI and VPN. If you would use a plain Linux VM, you may have to write a script that does this for you but it would fit your needs better than any open source or commercial appliance - which means you will have to find a trade off between work you have to put into it and functionality (pull requests for extension are welcome).

[Alphabet Soup]

Let me throw my 2 cents at your question about comparing OPNsense to commercial offerings.  Where I work I have replaced some fairly expensive big-name gear with OPNsense.  Overall, the result has been great.  Not perfect, but neither was the big-name stuff either.  My opinions, not global truths:

Pro:

• Cost.  If you've got a pool of money for wading in, this won't matter to you.  It mattered to us and we're able to do other stuff with the money we're not spending on big-name products.
• Simplicity of licensing.  I like to be license-compliant, but I really really hate having to navigate big-name license schemes full of self-invented buzzwords.
  
• HW / SW independence.  We bought hardware from applianceshop.eu, but really anything that runs FreeBSD is a candidate for OPNsense.

Con:

• Popularity.  Put out a job ad for big-name knowledgable tech person, lots of responses.  Job ad for OPNsense or FreeBSD knowledgable tech person, fewer responses.
• Manpower.  The OPNsense team and community certainly test and fix and code, but their resources are smaller than big-names.  I should mention that OPNsense does offer paid support options!
• GUI vs CLI.  I am grateful for how easy the GUI made some tasks, and occasionally irritated at having to wander around the GUI trying to find my way back to some screen I remember seeing last month.  The Search helps, but only if you're bang on with your keywords.

[peksi]

Thanks a lot for your comments. I've been a Linux admin since 1995 when kernel was version 1.1 and I am very comfortable with shell stuff and I do all my stuff in command line if a GUI is not required. The Linux firewall I've used so far has performed well, hardly uses any cpu and has been without problems.

But the biggest issues are problem situations and the amount time I use with it. If it fails and I am on another country there is nobody to fix it. Or if I want to add another openvpn cert user I need to make a csr, download it, issue it, upload it, modify openvpn configurations, create a client conf file and cert package and maybe client works. Or I forgot something as this is something i do seldom enough to forget things

With OpnSense I am hoping to delegate even with problem scenarios (HA, easy recovery, even some automatic recovery?) and do the above things a bit easier and faster. How is openvpn administration in OpnSense in practice, using the above as an example?

I've never done *BSD but it sounds interesting to learn.

[franco]

We don't have flame wars here. To be honest this is nice, I will check out the Arch docs now that you mentioned them curious to see. Thanks.

I used Debian and Gentoo in the past. I just dislike the fragmented user land, with BSD the kernel and the user land tools just fit together. Ambiguity in tooling is mostly manual pages not being updated, but that is easily verified / fixed.

And what I've seen is that companies build products on top of Ubuntu because they happen to use the Desktop at work. It's really not optimal.

@peksi:

Every task in OPNsense can be handled from the GUI in a matter of minutes. Issuing certificates (even Let's Encrypt integration), OpenVPN setup for client or server is very simple, CARP / HA sync is built in. An audit / recovery trail for the (global) configuration.

I also like SSH remote access, you can just port-forward the GUI to access it...

It's true that the CLI component is not working as well, coming from a point in time where m0n0wall and later pfSense thrived on the idea that CLI is not everything and GUI should be the primary focus. We're half-amending this with an encompassing API, but that is a long road as well and still doesn't bring back CLI, but is also just as viable for automation / scripting.

Hope this helps.


Cheers,
Franco

[peksi]

I think I will give OpnSense a spin. Still not sure about hardware solution since it requires more hardware and is a single point of failure. Do you think that virtualized OpnSense will be a bottleneck with 20mbps symmetrical connection? Most important internal stuff (iscsi) will reside in same 10g ethernet subnet so it should not be affected, only the outside connections and workstation-server connections would pass through OpnSense but there is no critical stuff there.

edit: and VLAN is something that would play a major role in virtualized solution. I suppose that is not a problem in OpnSense.

[franco]

Do you think that virtualized OpnSense will be a bottleneck with 20mbps symmetrical connection?

No. VLANs, LAGGs are ok too.