Choosing a border router/firewall for an (W)ISP

[ggallo]

Hi All!

I'm new to this forum, so greeting to you all! 

TL;DR: Is OPNsense is good enough for a 2500 customer ISP as central gateway (router/firewall) to/from internet working at 700 Mbit/s rate at peak hours (and growing)? If used in an active/passive HA setup in two VMs?

Detailed version:

As a foreword, I already using OPNsense on my own home server from about 2015 (in a VM on Hyper-V), which started as a pilot/test project using this instead of pfSense, and left on my machine because it fullfills all my needs, and a very good testing point for my networking expertises. Above this, I'm using bare FreeBSD from 5.4 for routing and pf based firewalling at out other (small enterprise) customers.

I'm hired by a local WISP to modernize their backbone (not in a physical way, "only" the networking layer) and services (mail and web servers, monitoring, ticketing, etc.). I have good moderate experience in enterprise networing (20 years), but never worked for an ISP, which requires totally different networking setup and have totally different load on routers.

The company have about 2500 customers, and currently have one ISP grade, fiber backed uplink, rated at 1000 Mbit/s, utilizing about 70% in peak hours. In the next 1 or 2 years, I think we will go up to a 10GbE physical connection from the actual 1 GbE, and buy 1500-2000 Mbit/s internet bandwidth.

I'm planned the backbone and services already, and I reached the task to select the proper border router/firewall. On the term "border router" I mean the router between the ISP and the upstream internet provider.

The current solution (which is horrible, but working at all) is an old, Xeon based server with 8GB RAM and Debian 6 OS with manually edited iptables configuration. This machine doing the routing, the firewalling for all customers, and NAT for about 2/3 of customers. No IDS/IPS, proxy or any other higher level service running.

So, after this "little" introduction, I'd like to ask here your opinions on using OPNsense for the main gateway to the internet.

I'd like to have redundant setup (active/passive is good, no need for active/active-loadbalace now), and I'd like to avoid using true hardware routers in this place now for additional flexibility (we can buy e.g. two more CCR-1036's if needed, not the budget is the limiting factor). In the backbone, we mainly use Mikrotik equipment.
In the new network setup, area-central Mikrotik CCRs (currently we have 4 of them) will do the NAT, bandwidth limiting, PPPoE server, etc. functionality. The border router only do central routing to/from internet and CCRs/services, and whole-ISP firewalling (maybe IDS/IPS for the services) workload.
We have two Dell servers with VMware ESXi as a base (all "new" services running on them in active/passive setup).

I can't decide to use Mikrotik's CHR (no stateful firewall connection sync between two), pfSense (maybe licensing change in future?), OPNsense (you know the caveats, I'm here to ask :-) or some sort of bare Linux (Ubuntu preferred, but no GUI means nobody can manage it in the company, except me), with hand-installed SW stack (maybe Shorewall). The main problem is that I can't change border router too often, because it causes internet disruption for customers because of the only one uplink. So I can't install and try every option, let alone the production network. But at lab environment, I can't simulate 2500 customer's traffic and behavior.
Maybe two CCRs (but same applies as CHR), if someone have experiece that this load not suitable for software firewalls, but we don't want to buy expensive Cisco, Juniper, or other big brand routers.

If it matters, no BGP peering needed, so no big dynamic routing tables, etc.

So, if anyone have any experience on this topic, or similar load/connected clients, please share his/her thoughts with me!

Thank you for reading and commenting! Feel free to ask more details before answering!

[mimugmail]

How many Firewall rules so you run? It's no fun to work with 1000 rules. For everything else OPN is quite fine

[ggallo]

About 100 rules running on current firewall, but I will use address-lists for some filter rules, so much less rules will be used on new firewall.

[mimugmail]

I'd really consider running it on hardware.