Gateway Monitoring

[supercm]

When I have gateway monitoring on, i get random alerts that an interface is down. However, it is not actually down (from what I can tell). What can I investigate to get to the bottom of this?

[mimugmail]

Can you post the logs of gateways?

[supercm]

Sure, here you go.

[supercm]

Did those logs help? Any thoughts here?

[RedVortex]

I wonder if you may be affected by the same issue I had where some static routes were added by dhcp client when I was getting my WAN IPs and overriding the routes and gateway monitoring I had.

You may want to try this, it fixed it for me (and will make its way to the next 22.1.x release anyways)

https://forum.opnsense.org/index.php?topic=26765.msg129675#msg129675

You will have to reboot after the patch so that your routing tables clears up properly or re-save your WANPUBLIC and WIRELESS interfaces so they re-update themselves but reboot makes sure everything starts clean after the patch.

Also, send us your routing table (while the problem is there) just for the fun of it (Just the IPv4 (top) part, not everything).

[supercm]

Thank you, I will dig through that post shortly.

RE: Sending the routing table is that just the Status under Routes in the GUI or something else?

[RedVortex]

Yes, only the first few 5-6 lines should be good.

My problem showed up at lines 2 and 3 where my DNS servers, that I use for gateway monitoring (like you seem to be doing), appeared routed to the wrong place instead of being assigned to the right gateways as it used to before 22.1.

Franco created a quick fix that will make it way to the next 22.1 release that fixes this and brings back the pre-22.1 behaviour (which was the right way).

You can run this, in command line, in your pfsense to add this patch if you want to have it before the next release.

# opnsense-patch 02dc1ebd93

And then reboot. My problem was gone and I could then see the right routes to the DNS servers and gateway monitoring IPs.

[supercm]

Thank you.

I have applied the patch to one of my nodes and I was going to compare the data once I see the alert come through. Right now they look the same except for one item (which I expect because I have one interface configured as a pass through but can only do that on a single unit at a time).

[RedVortex]

Make sure to reboot or re-save any interface that is configured in DHCP (usually WAN interfaces) after the patch or else nothing will change. The affected script is only called when an interface comes up/gets a DHCP ip.

[supercm]

I rebooted after patching.

[supercm]

I just got an alert but the tables look the same. Wan12 is the one I was alerted on.

No Patch

ipv4   default   192.168.12.1   UGS   NaN   1500   hn4   Wireless2       
ipv4   1.0.0.1   192.168.252.254   UGHS   NaN   1492   hn2   Wan12       
ipv4   1.1.1.1   192.168.250.1   UGHS   NaN   1500   hn3   Wireless1       
ipv4   8.8.4.4   192.168.12.1   UGHS   NaN   1500   hn4   Wireless2       
ipv4   8.8.8.8   192.168.254.254   UGHS   NaN   1492   hn1   Wan18       

Patch
ipv4   default   192.168.12.1   UGS   NaN   1500   hn4   Wireless2       
ipv4   1.0.0.1   192.168.252.254   UGHS   NaN   1492   hn2   Wan12       
ipv4   1.1.1.1   192.168.250.1   UGHS   NaN   1500   hn3   Wireless1       
ipv4   8.8.4.4   192.168.12.1   UGHS   NaN   1500   hn4   Wireless2       
ipv4   8.8.8.8   192.168.254.254   UGHS   NaN   1492   hn1   Wan18       

[supercm]

The patched node is continuing to experience this issue. Any other suggestions?

[RedVortex]

If you try to ping the monitoring IP of the gateway yourself from the router itself (while opnsense says it is down) does it ping ?

ie: let's say you have gateway Wan12 that uses 1.0.0.1 as the monitoring IP and opnsense says that it is currently down, can you ping 1.0.0.1 from opnsense itself if you run ping 1.0.0.1 in command line in opnsense ? Also make sure to ping the gateway itself to make sure it is also down and/or up.

Also, when it show as "down", is it because it sees high packet loss or 100% faillure or the interface down (in opnsense UI when you check the status) ?

Last thing, was the routes you sent above during a gateway down or while it was considered up ? If not during down, can you try to capture the routes when it says it is down ?

We're investigating here, I have no idea why this would be happening. I also have many public WANs and LANs and many have external monitoring IP as well and I'm not affected by this so I'm very curious as to what would trigger this behaviour.

[mimugmail]

I had a similar thing last year, the provider was rate limiting ping so I opened advanced section in gateway and set to only ping every 5 seconds

[supercm]

I just turned it back on and I got an email immediately that it failed. I pinged the gateway and the ip that I have in monitoring from the interface that is down, and the ping succeeds.

The alert that I get is that the gateway is down. The UI shows that everything is fine at this time.

The routes are the same regardless if it down or not.