I have OPNSense running on a Pico PC (Intel® E3845 4xLAN) and it's been rock-solid for months with one very big notable exception. This is relating to my dual WAN setup. I have 2 x VDSL circuits (both providing c.20Mbps downstream) both of these services are using the providers original routers and these two units are connected upstream to 2 x Ethernet interfaces on my OPNsense.
When I have a single gateway in OPNsense enabled (either WAN1 or WAN2 doesn't matter), everything is rock solid. However if I configure any load-balancing then things seem to get unpredictable and unreliable, i.e. loss of Internet connectivity. I've even had some issues when I have OPNSense configured for just failover.
I even resorted to using each of the single broadband connections each from their local subnet as a test (so no OPNsense) and both circuits are stable, no errors and long up times.
In OPNsense, I've double checked my configuration but I'm thinking I must have done something stupid...
My gut feel is it's something to do with the "Monitor IP" configuration on the gateways. In that regard I have tried using different IPs for the monitoring (I originally used 8.8.8.8 & 8.8.4.4). I even tried disabling this yesterday and that didn't seem to help.
I have even changed my downstream WAN2 router to be in bridge mode (thus eliminating the WAN2 router) so now I have OPNsense configured for PPPoE. In this configuration the gatway is showing Down in OPNSense but it is actually up and working.
Any thoughts/guidance appreciated.
I had a similar issue on my old Asus router, lots of services don't like having connections from different IP's, website acted weird/slowly, services were slow or failed to connect - everything was fine on failover though.
There is a workaround for this, I remember something like a persistent connection setting - it may be worth searching for this in settings/google. I didn't look into it further as load balancing isn't ideal for my setup.m
Under Firewall, Advanced, Multi-wan you'll need to enable "Sticky Connections". You'll also probably want to set a reasonable value in the timeout - say 10 or 15 minutes.