Messages

I'm not exactly sure what's going on here. But I'm having some struggles getting multi wan to fail back to primary interface once network is restored. 
...

Did you get this resolved?  I am having the same issue, and Sticky Connections is not set.

Not 100% disabling sticky connections helped.  With full on gateway outage, cable unplugged or isp fully down.  However when the isp flaps or doesn't go totally down, think heavy packet loss, it doesn't fail over either direction very well.  Not sure how to resolve it.

Sent from my IN2025 using Tapatalk

Mine works OK. You can change the thresholds for latency and packet loss on the System->Gateways->Single page. and choose which value triggers a switch System->Gateways->Group->Trigger Level.

My understanding of how it works (or can be configured) is as follows:

Member Down means triggers when either Packet Loss or High Latency exceeds higher threshold (so called To or Down status)
Packet Loss means triggers when Packet Loss exceeds lower threshold (so called From or Alert status)
High Latency means triggers when High Latency exceeds lower threshold (so called From or Alert status)
Packet Loss or High Latency means triggers when either Packet Loss or High Latency exceeds lower threshold (so called From or Alert status).

I use Member Down as the trigger with all the Thresholds set to their default values.

Also, I have System->Gateways->Single->Priority set to 255 for both the Tier1 and Tier2 gateway. You only need different priority if you have multiple gateways on the same Tier (e.g. when load balancing)

Can't find the english language version anymore, but I found this guide invaluable:

https://www.thomas-krenn.com/de/wiki/OPNsense_Multi_WAN

I have a problem with traffic graphs not showing any data where IDS/IPS is used on the interface. Turned off IDS/IPS on WAN (still enabled on LAN) so graphs at least show WAN traffic. Other than that, I have no issues. I haven't noticed any performance issues or increased CPU usage. Intel i350 network hardware and 900Mbps up and down.

Regarding IPS:
https://github.com/opnsense/docs/issues/278

I tried that, didn't make any difference (and yes I did reboot after), graphs still broken.

Did you ever get this resolved and are you running Suricata? If I disable IDS/Suricata, the graphs work...

I am running Suricata and like you If I disable Suricata the graphs work.

FreeBSD bug with netmap integration... sometimes it counts packets twice or not at all depending on the driver used.

Strange as it was working fine right up to and including 20.1.9. Also, you'd think they would have tested any changes on something as popular as the Intel i350 series.

As per the title the traffic graph widget on the Dashboard is broken after upgrade from 20.1 to 20.7. LAN and WAN data is selectable but data is missing from the graph. Similarly LAN and WAN data missing from Reporting Traffi.c

I have a symmetric 900 mbps service from Gigaclear and using either their own speedtest or speedtest.org my ping times are typically 5-7ms using default destinations (for the latter IOMART Maidenhead). I can get different ping times on speedtest.org by choosing different destinations (i.e. XILO Maidenhead is 7ms, University of Oxford, IT services 8ms, Vodafone Bracknell 8ms, KCOM Kingston Upon Hull 15ms, BT Lancashire 25ms). Generally the further away the longer the ping times. If I use ThinkBroadband the ping time is around 21-23ms. In my case the ping times haven't changed noticeably between the different versions of OPNsense (Currently on 20.1.8_1) as its never been better than 5-7ms.

I use an i5 4670K with 8GB RAM. My Internet service is rated at 900Mbps (up and down) and I use IDS/IPS with about 40,000 rules but no VPN. I can get 900+ with CPU load peaking at around 2.8.  As it's a 4 core cpu you can divide that by 4 so that leaves plenty in reserve.

I had a problem with getting a DHCP lease on a VLAN and it was related to the Intrusion Detection service. With the Intrusion Detection service enabled devices connecting on a network associated with a VLAN couldn't get a DHCP lease. With the Intrusion Detection service disabled they could. I didn't want to disable the Intrusion Detection service so I eventually found that I could leave it enabled if I disabled VLAN Hardware Filtering (Interfaces->Settings->VLAN Hardware Filtering=Disable VLAN Hardware Filtering).

Not sure from your diagram and description how you managed to follow the instructions for configuring a Multi-Wan setup as you appear to only have one WAN into OPNsense. That said, if you do manage to configure a Multi-Wan setup in a load balance configuration, it will monitor the gateways and mark as down any that fail so that all traffic can then go via any remaining gateway(s).

Have you looked at System->Gateways->Log File?

You can change the Latency and Packet Loss thresholds in System->Gateways->Single

I have a multi-wan setup with both WANs behind other routers (i.e. double NAT) and my setup works flawlessly and I haven't had to resort to allowing private networks on WAN (i.e. the Block private networks option is checked for both WAN interfaces). Everything still worked when I upgraded from 20.1.2 to 20.1.3 as it has done through every upgrade from 19.7.3 right up to 20.1.5 (I haven't upgraded to 20.1.6 yet). So you might like to look to see if anything else has changed in your setup when you went from 20.1.2 to 20.1.3. Have you, for example, changed the Firewall->NAT->Outbound rule generation from Automatic?

Have you got a Cron job (System->Settings->Cron) that performs an "Automatic firmware update"?

I have a similar setup with a Fibre connection for my primary WAN and a mobile (4G) for backup WAN. I haven't checked the data consumed from my mobile provider, but the Interface Statistics for my backup WAN connection work out at around 70MB per month. I have the probe interval for my backup WAN connection set to 2 instead of the default 1 so it would follow that if you have the default value set you would use twice as much data as me.

You could try increasing the probe interval (System->Gateways->Single->Edit Wan->Probe Interval) on your backup 4G WAN from 1 (Default) to something higher. I use 2 but as it is purely a backup, it doesn't really need to be checked that often.

Have you tried using Sticky connections (Firewall->Settings->Advanced).