Messages

Hi All!

I have a problem with our home2home IPsec VPN tunnel (site2site mode). I know IPsec with dynamic IPs on both ends do not play well (even not supposed to), but we cannot get static public IPs from out ISP. Above this, we must use IPsec, because the other end have a Mikrotik router, wich supports IPsec AES hardware accelerationm but do not support OpenVPN hardware acceleration. Because this "limitation", with IPsec, we have about 280-300 Mbps throughput (on 1000/300 Mbit internet connections both sides), with OpenVPN, only 3-5 Mbps available, which is not enough for us.

I'm running OPNsense 19.1.7 at home (on a Xeon E3, AES-NI capable CPU), with a PPPoE internet connection, which has dynamic public IPv4 address, with DynDNS refresh service enabled, which changes about 3 days (ISP session timneout).
The other end have a Mikrotik HEX (RB750Gr3), with AES hardware acceleration support on IPsec. That connection is the same, PPPoE dynamic IPv4 address with DynDNS.

The problem is, when the other end (the Mikrotik's) IPv4 address changes, my side did not recognise this (DPD activated both sides), and does not update/reinit/restart the IPsec stack to refresh the IP address associated with the FQDN I used in the configuration. This happen even I switch on the "Dynamic gateway" options in phase 1 settings. The config page suggests that the other end's address refreshed on startup/update, but this only happens when I press Save on the page (even without any changes) and the press Apply at the top. FQDN to address never ever updated any other ways (automatically).

At the other end, I wrote a little script for the MT router, which checks the tunnel availability, and if no answer received, it refreshes the IP addresses in IPsec configuration on the MT. This way, when my IP address changes, the other end reconnects in a minute automatically.

Are there any (semi)official solution exists or planned like this on OPNsense side, or I must start to write a cron-managed script for my side, that works like the MT side one?
I can't restart IPsec on a timely basis (for ex. at night), because the IP address changes do not happen at the same time.

Do any body have any solution on this?

About 100 rules running on current firewall, but I will use address-lists for some filter rules, so much less rules will be used on new firewall.

[TL;DR]

Hi All!

I'm new to this forum, so greeting to you all!  :)

TL;DR: Is OPNsense is good enough for a 2500 customer ISP as central gateway (router/firewall) to/from internet working at 700 Mbit/s rate at peak hours (and growing)? If used in an active/passive HA setup in two VMs?

Detailed version:

As a foreword, I already using OPNsense on my own home server from about 2015 (in a VM on Hyper-V), which started as a pilot/test project using this instead of pfSense, and left on my machine because it fullfills all my needs, and a very good testing point for my networking expertises. Above this, I'm using bare FreeBSD from 5.4 for routing and pf based firewalling at out other (small enterprise) customers.

I'm hired by a local WISP to modernize their backbone (not in a physical way, "only" the networking layer) and services (mail and web servers, monitoring, ticketing, etc.). I have good moderate experience in enterprise networing (20 years), but never worked for an ISP, which requires totally different networking setup and have totally different load on routers.

The company have about 2500 customers, and currently have one ISP grade, fiber backed uplink, rated at 1000 Mbit/s, utilizing about 70% in peak hours. In the next 1 or 2 years, I think we will go up to a 10GbE physical connection from the actual 1 GbE, and buy 1500-2000 Mbit/s internet bandwidth.

I'm planned the backbone and services already, and I reached the task to select the proper border router/firewall. On the term "border router" I mean the router between the ISP and the upstream internet provider.

The current solution (which is horrible, but working at all) is an old, Xeon based server with 8GB RAM and Debian 6 OS with manually edited iptables configuration. This machine doing the routing, the firewalling for all customers, and NAT for about 2/3 of customers. No IDS/IPS, proxy or any other higher level service running.

So, after this "little" introduction, I'd like to ask here your opinions on using OPNsense for the main gateway to the internet.

I'd like to have redundant setup (active/passive is good, no need for active/active-loadbalace now), and I'd like to avoid using true hardware routers in this place now for additional flexibility (we can buy e.g. two more CCR-1036's if needed, not the budget is the limiting factor). In the backbone, we mainly use Mikrotik equipment.
In the new network setup, area-central Mikrotik CCRs (currently we have 4 of them) will do the NAT, bandwidth limiting, PPPoE server, etc. functionality. The border router only do central routing to/from internet and CCRs/services, and whole-ISP firewalling (maybe IDS/IPS for the services) workload.
We have two Dell servers with VMware ESXi as a base (all "new" services running on them in active/passive setup).

I can't decide to use Mikrotik's CHR (no stateful firewall connection sync between two), pfSense (maybe licensing change in future?), OPNsense (you know the caveats, I'm here to ask :-) or some sort of bare Linux (Ubuntu preferred, but no GUI means nobody can manage it in the company, except me), with hand-installed SW stack (maybe Shorewall). The main problem is that I can't change border router too often, because it causes internet disruption for customers because of the only one uplink. So I can't install and try every option, let alone the production network. But at lab environment, I can't simulate 2500 customer's traffic and behavior.
Maybe two CCRs (but same applies as CHR), if someone have experiece that this load not suitable for software firewalls, but we don't want to buy expensive Cisco, Juniper, or other big brand routers.

If it matters, no BGP peering needed, so no big dynamic routing tables, etc.

So, if anyone have any experience on this topic, or similar load/connected clients, please share his/her thoughts with me!

Thank you for reading and commenting! Feel free to ask more details before answering!