Interfaces randomly go down/unroutable

[n1nja]

I have this issue where every so often (a bit on the spontaneous side, unfortunately) I lose internet.  I can't ping my IPv4 LAN facing gateway.  I can't ping my management port (which I created for troubleshooting this problem).  Looking at /var/log/system.log, I see this:

Code Select Expand

Nov 26 13:08:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:08:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:08:27 OPNsense dhclient[52573]: Creating resolv.conf
Nov 26 13:08:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:13:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:13:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:13:27 OPNsense dhclient[88047]: Creating resolv.conf
Nov 26 13:13:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:16:00 OPNsense root[28797]: reload filter for configured schedules
Nov 26 13:18:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:18:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:18:27 OPNsense dhclient[87230]: Creating resolv.conf
Nov 26 13:18:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:23:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:23:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:23:27 OPNsense dhclient[74715]: Creating resolv.conf
Nov 26 13:23:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:28:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:28:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:28:27 OPNsense dhclient[65189]: Creating resolv.conf
Nov 26 13:28:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:31:00 OPNsense root[30800]: reload filter for configured schedules
Nov 26 13:33:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:33:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:33:27 OPNsense dhclient[84129]: Creating resolv.conf
Nov 26 13:33:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:33:32 OPNsense kernel: em1: link state changed to DOWN
Nov 26 13:33:32 OPNsense kernel: em1_vlan35: link state changed to DOWN
Nov 26 13:33:32 OPNsense kernel: em1_vlan10: link state changed to DOWN
Nov 26 13:33:32 OPNsense kernel: em1_vlan30: link state changed to DOWN
Nov 26 13:33:32 OPNsense kernel: em2: link state changed to DOWN
Nov 26 13:33:32 OPNsense opnsense[93736]: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for lan
Nov 26 13:33:33 OPNsense opnsense[63600]: /usr/local/etc/rc.linkup: Hotplug event detected for Ooma(opt2) but ignoring since interface is configured with static IP (10.35.0.254 ::)
Nov 26 13:33:33 OPNsense opnsense[75016]: /usr/local/etc/rc.linkup: Hotplug event detected for WirelessGuest(opt3) but ignoring since interface is configured with static IP (10.10.0.254 ::)
Nov 26 13:33:34 OPNsense opnsense[80883]: /usr/local/etc/rc.linkup: Hotplug event detected for WirelessTrust(opt1) but ignoring since interface is configured with static IP (10.30.0.254 ::)
Nov 26 13:33:34 OPNsense opnsense[91909]: /usr/local/etc/rc.linkup: Hotplug event detected for MGMT(opt6) but ignoring since interface is configured with static IP (10.255.255.254 ::)
Nov 26 13:36:03 OPNsense kernel: em1: link state changed to UP
Nov 26 13:36:03 OPNsense kernel: em1_vlan35: link state changed to UP
Nov 26 13:36:03 OPNsense kernel: em1_vlan10: link state changed to UP
Nov 26 13:36:03 OPNsense kernel: em1_vlan30: link state changed to UP
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for lan
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface lan
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: ROUTING: entering configure using 'lan'
Nov 26 13:36:03 OPNsense kernel: em2: link state changed to UP
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure ipsec (,lan)
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure ipsec (execute task : ipsec_configure_do(,lan))
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dhcp ()
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dns ()
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dns (execute task : dnsmasq_configure_do())
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dns (execute task : unbound_configure_do())
Nov 26 13:36:06 OPNsense opnsense[191]: /usr/local/etc/rc.linkup: Hotplug event detected for Ooma(opt2) but ignoring since interface is configured with static IP (10.35.0.254 ::)
Nov 26 13:36:06 OPNsense opnsense[6297]: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em1_vlan35'


I found another forum post that says marking their interface as the gateway solved their problem, but mines already set this way.

For now I've disabled gateway monitoring to see if it makes any difference, but I'm not sure why I'd lose my LAN facing stuff in this case.

[alexroz]

Same here after upgrade to 21.7.6.
Thinking of rolling back to previous version.

[alexroz]

Don't now what exactly the problem was....
But rolling back to 21.7.5 from a console resolved the issue.

Code Select Expand

opnsense-revert -r 21.7.5 opnsense
https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert
So far so good....

[4Saken]

Hey guys, just reading up on this. I just reverted to. Been having issues with interfaces this way to, after upgrading to 21.7.6.

Did you guys perhaps have suricata running on those interfaces? My issues seem to be resolved when i disable suricata or taking the interface out of the config.

I was experiencing this on the lan side btw, since i dont have suricata on wan side. My lagg interface seems to be doing just fine with suricata enabled.
Sidenote: rss enabled. intel i210.

I suspect this issue to have something to do with.
Suricata 6.0.4 with an additional change for the Netmap API version 14. not sure  :-X

[franco]

Hello confirmation bias, my old friend...

I don't see any hints in the release notes and also can't recall anything that drastic. Reverting 21.7.5 core package obviously would leave newer Suricata in place so it's not that specifically or the newer ET rulesets give you trouble (easy enough to test if enough info was given).

What seems like a joke is:

Nov 26 13:08:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.

300 seconds is 5 minutes... then...

Nov 26 13:13:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67

That is EXACTLY 5 minutes later so I am unsure how this is thought to be not what it should be?

I get that something doesn't work, but the report doesn't say and jumps to conclusions. Never a good combo.


Cheers,
Franco

[alexroz]

Hey guys, just reading up on this. I just reverted to. Been having issues with interfaces this way to, after upgrading to 21.7.6.

Did you guys perhaps have suricata running on those interfaces? My issues seem to be resolved when i disable suricata or taking the interface out of the config.

I was experiencing this on the lan side btw, since i dont have suricata on wan side. My lagg interface seems to be doing just fine with suricata enabled.
Sidenote: rss enabled. intel i210.

I suspect this issue to have something to do with.
Suricata 6.0.4 with an additional change for the Netmap API version 14. not sure  :-X

Yep. I have Suricata on LAN side interfaces.

[4Saken]

Yep. I have Suricata on LAN side interfaces.

After upgrade to 21.7.6 i was facing issues where some interfaces became unreachable, also via setting a static ip. Gateway did not respond at all, dhcp did seem to reach the server. verified by the log. But thats was all. There seemed to be, something stuck. :o

I  noticed this on my management interface.

After removing the management interface from suricata it worked again.
After putting the interface back in the config, it worked like it did before, but did not survive a reboot. 

Yesterday i removed all rules from suricata and disabled suricata for ids/ips.
After downloading all rules and enabling ids/ips my issue has been solved!

[n1nja]

I've been mucking with this issue for weeks. I had it on both mentioned versions of the software. I initially blamed Sensei but even disabling it I had that problem. To be clean, yes my wan DHCP stuff seems weird in the log but I'm not concerned about that right now because my problem is lan facing. I can't ping my lan side. I can't ping my management port. I can't ssh to either. The thing seems dead. As soon as I initiate shutdown with hardware button press, there's a brief moment where about 12 pings make it through before it shuts down. In my case I also lose DHCP because I'm using OPNsense as a DHCP server

[Julien]

Yep. I have Suricata on LAN side interfaces.

After upgrade to 21.7.6 i was facing issues where some interfaces became unreachable, also via setting a static ip. Gateway did not respond at all, dhcp did seem to reach the server. verified by the log. But thats was all. There seemed to be, something stuck. :o

I  noticed this on my management interface.

After removing the management interface from suricata it worked again.
After putting the interface back in the config, it worked like it did before, but did not survive a reboot. 

Yesterday i removed all rules from suricata and disabled suricata for ids/ips.
After downloading all rules and enabling ids/ips my issue has been solved!

i have this behaivor too, IDS crashes from time to time didnt know the cause.
i've followed your steps seems to works for 30 min and after it crashes.

i am back to the old version.

[franco]

dmesg please

[chemlud]

I updated to 21.7.6 yesterday in the evening, this morning hell broke loose on the most important interface (suricata in IPS mode running on a total of 3 interfaces). All of the sudden no internet connection, dhcp delivers addresses, but devices are not reachable on the same LAN.

I rebooted, changed the physical interface (network card), I moved the whole network to another 21.7.6 install, I changed ALL dumb switches on the network, it helps for an hour and then the terror starts again. I have no idea where to start, nothing remarkable in the system log, dhcp log, unbound log, suricata log.

dmesg appended....

[chemlud]

Disabled suricata on the interface and traffic started flowing. Does this make sense at all for traffic an the SAME interface?

PS: changing the interface switched from em to igb, without any success....

[Qu0th]

I've had the exact same experience.

I updated to 21.7.6 yesterday in the evening, this morning hell broke loose on the most important interface (suricata in IPS mode running on a total of 3 interfaces). All of the sudden no internet connection, dhcp delivers addresses, but devices are not reachable on the same LAN.

I rebooted, changed the physical interface (network card), I moved the whole network to another 21.7.6 install, I changed ALL dumb switches on the network, it helps for an hour and then the terror starts again. I have no idea where to start, nothing remarkable in the system log, dhcp log, unbound log, suricata log.

dmesg appended....

[jt-socal]

I'm also having problems with WAN link going down and up.  This has not happened in the past, though I cannot be certain it is not my ISP.  dmesg attached

[chemlud]

For testing I enabled suricata IPS on the interface with problems yesterday, it took about 2 min to blow away the WAN interface

Code Select Expand

2021-11-30T15:03:59 opnsense[97742] /usr/local/etc/rc.linkup: Clearing states for stale wan route on em2
2021-11-30T15:03:59 dhclient[11453] exiting.
2021-11-30T15:03:59 dhclient[11453] connection closed
2021-11-30T15:03:59 opnsense[97742] /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
2021-11-30T15:03:55 opnsense[48849] plugins_configure hosts (execute task : unbound_hosts_generate())
2021-11-30T15:03:55 opnsense[48849] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2021-11-30T15:03:55 opnsense[48849] plugins_configure hosts ()
2021-11-30T15:03:54 opnsense[48849] /usr/local/etc/rc.newwanip: On (IP address: 10.110.122.1) (interface: Drack[opt2]) (real interface: igb0).
2021-11-30T15:03:54 opnsense[48849] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'igb0'
2021-11-30T15:03:53 opnsense[14602] /usr/local/etc/rc.linkup: Hotplug event detected for Drack(opt2) but ignoring since interface is configured with static IP (10.110.122.1 ::)
2021-11-30T15:03:52 opnsense[22399] /usr/local/etc/rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (10.11.12.1 ::)
2021-11-30T15:03:51 opnsense[54788] /usr/local/etc/rc.linkup: Hotplug event detected for iNET(opt1) but ignoring since interface is configured with static IP (10.157.11.1 ::)
2021-11-30T15:03:49 opnsense[76900] /usr/local/etc/rc.linkup: Hotplug event detected for Drack(opt2) but ignoring since interface is configured with static IP (10.20.22.1 ::)

Why the heck

Code Select Expand

/usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'igb0'

igb0 is the problematic LAN interface, not WAN (em2)....