24.1.3 and .4 new issue with daily drops of WAN

[p0lar]

ever since I upgraded to 24.1.3 and then 24.1.4, my failover gateways have been going offline about 1 to 2 times daily.  I don't see much in the logs besides 100% packet loss.  If I go to the interfaces/overview and RELOAD the down interfaces, the net will come back up and function for another 12 +- hours.  i disabled the failover gateway config and disabled using them in the firewall rules but it still happened. 

Any ideas of what to look for?

send_interval 1000ms loss_interval 4000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 0ms loss_alarm 0% alarm_hold 10000ms dest_addr 8.8.4.4 bind_addr 7x.1xx.9x.192 identifier "XFINITYWAN_DHCP "

[mimugmail]

IPS running?

[p0lar]

Yes.  Crowdsec, Suricata, and Zenarmor.   Too many???  maybe I should only run Zenarmor?

[mimugmail]

You disable one after another to find the problem

[p0lar]

OK.  ill report back

[Greg_E]

All three hardware accelerations turned off for the network cards?

I have all three of these running too, though I'm not sure if Crowdsec is really doing anything (still learning).

Suricata (open ET rules plus some others) on WAN, Zenarmor (free version for now) on 2 of my LAN, 2 other LAN get nothing right now (lab and direct connect management).

[p0lar]

same exact setup as you.  All three are turned off.  i disabled Crowdsec and it hung. i then disabled IDS/IPS suricata and am waiting for the next hang.  its about every 12 hours ish.   Zenarmor is still running. 

[jonm]

12 hours is suspicious, when I had wan drops every 12 hours it was to do with dhcp on the wan interface renewing.

[kixx09]

I have the same issue and also looks it's begun after update to 24.1.3
Seeing the same "12 hour"-ish pattern.

However, I do not run IPS/IDS at all.

Changing DHCP timings between "Basic, FreeBSD default and OPNSense default" on WAN interface doesn't look to make any difference.

[p0lar]

I disabled Crowdsec and still stopped +-12 hours in.  I disabled IDS/IPS Suriata and its been running for more than 12 hours so far.  Zenarmor is STILL running as well.   I'm looking into the Suricata logs to see if I can find something.   

[mimugmail]

Ok, one step further, hope you find something, maybe on the console?

[p0lar]

Okay, I didn't make any changes yesterday (that I remember )—it was a sick day—but it failed again this morning at 1 a.m. 

I'll check the console when I get home.  any good logs to see what the interfaces are doing? 

[p0lar]

so last night I removed the failover gateway and all that config.  using the default route out one ISP for now. 

[axsdenied]

Skimming through this is looks like 2 problems.  1. the primary gateway going down and then 2nd. it's not failing back when it's available again.

I'll assume you have the primary gateway configured to down itself at a particular packet loss percentage? Then it fails over but then doesn't fail back until you reload the interfaces right?

1. When it goes down, is it really down? i.e. there is a real problem with that provider or do you believe it's being artificially downed for some unknown reason?
2. Do you have the primary gateway set as the upstream gateway?

I used to have the same issue with problem statement 2 but did some reconfiguring, and setting the upstream was one of them and now it works perfectly.  I am also running 24.1.4.

[p0lar]

Thanks for the questions.  here is what I know so far.

Q = 1. When it goes down, is it really down? i.e. there is a real problem with that provider or do you believe it's being artificially downed for some unknown reason?

A = I don't think there is a problem with the Xfinity modem or internet connection. I don't have much visibility into the modem, but the network and Opnsense come back online immediately when I refresh the interface from inside the GUI.  Currently, I think the gateway config/software is killing the connection and saying that it is 100% packet loss.  it is killing BOTH my Xfinity and ATT FirstNet hotspots by marking them down.  the ATT hotspot works fine if it's not connected to the Opnsense FW, as I use it for work remotely all the time with no loss.

Q = 2. Do you have the primary gateway set as the upstream gateway?

A = yes I did before I removed the gateway failover config and set all the firewall rules back to "default" vs the failover groups.