After update, problems with LAN/WAN Interfaces

[oh_lawd]

Today it happened again. LAN went out completely and I needed to reboot the router. Now I was able to capture some logs:

Code Select Expand


<11>1 2022-02-14T23:16:37+01:00 router.local opnsense 33229 - [meta sequenceId="2"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic lan(igb1)
<27>1 2022-02-14T23:16:38+01:00 router.local dhcp6c 33998 - [meta sequenceId="3"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:16:39+01:00 router.local dhcp6c 33998 - [meta sequenceId="4"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:16:41+01:00 router.local dhcp6c 33998 - [meta sequenceId="5"] transmit failed: Network is down
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="6"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic lan(igb1)
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="7"] /usr/local/etc/rc.linkup: Accept router advertisements on interface igb1
<13>1 2022-02-14T23:16:42+01:00 router.local dhcp6c 6366 - [meta sequenceId="8"] RTSOLD script - Sending SIGHUP to dhcp6c
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'lan'
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: ROUTING: IPv6 default gateway set to lan
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
<13>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="14"] plugins_configure ipsec (,lan)
<13>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="15"] plugins_configure ipsec (execute task : ipsec_configure_do(,lan))
<13>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="16"] plugins_configure dhcp ()
<13>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="17"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2022-02-14T23:16:43+01:00 router.local opnsense 80163 - [meta sequenceId="18"] plugins_configure dns ()
<13>1 2022-02-14T23:16:43+01:00 router.local opnsense 80163 - [meta sequenceId="19"] plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2022-02-14T23:16:43+01:00 router.local opnsense 80163 - [meta sequenceId="20"] plugins_configure dns (execute task : unbound_configure_do())
<45>1 2022-02-14T23:17:05+01:00 router.local syslog-ng 4839 - [meta sequenceId="21"] syslog-ng shutting down; version='3.35.1'
<45>1 2022-02-14T23:17:57+01:00 router.local syslog-ng 17974 - [meta sequenceId="1"] syslog-ng starting up; version='3.35.1'
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="2"] plugins_configure loopback_prepare (1)
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="3"] plugins_configure loopback_prepare (execute task : loopback_configure_interface(1))
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="4"] plugins_configure openvpn_prepare (1)
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="5"] plugins_configure openvpn_prepare (execute task : openvpn_prepare(1))
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="6"] plugins_configure vxlan_prepare (1)
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="7"] plugins_configure vxlan_prepare (execute task : vxlan_configure_interface(1))
<11>1 2022-02-14T23:17:58+01:00 router.local opnsense 495 - [meta sequenceId="8"] /usr/local/etc/rc.bootup: Accept router advertisements on interface igb1
<13>1 2022-02-14T23:17:58+01:00 router.local dhcp6c 37343 - [meta sequenceId="9"] RTSOLD script - Starting dhcp6 client
<27>1 2022-02-14T23:17:59+01:00 router.local dhcp6c 38371 - [meta sequenceId="10"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:17:59+01:00 router.local dhcp6c 38371 - [meta sequenceId="11"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:18:00+01:00 router.local dhcp6c 38371 - [meta sequenceId="12"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:18:00+01:00 router.local dhcp6c 38371 - [meta sequenceId="13"] transmit failed: Network is down
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 53721 - [meta sequenceId="14"] New IP Address (igb0): <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 55210 - [meta sequenceId="15"] New Subnet Mask (igb0): 255.255.255.0
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 56598 - [meta sequenceId="16"] New Broadcast Address (igb0): <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 58168 - [meta sequenceId="17"] New Routers (igb0): <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 61390 - [meta sequenceId="18"] route add default <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 62734 - [meta sequenceId="19"] New Static Routes (igb0): 192.168.5.1 <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 65468 - [meta sequenceId="20"] Creating resolv.conf
<11>1 2022-02-14T23:18:02+01:00 router.local opnsense 66978 - [meta sequenceId="21"] /usr/local/etc/rc.newwanip: IP renewal deferred during boot on 'igb0'
<27>1 2022-02-14T23:18:02+01:00 router.local dhcp6c 38371 - [meta sequenceId="22"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:18:02+01:00 router.local dhcp6c 38371 - [meta sequenceId="23"] transmit failed: Network is down
<11>1 2022-02-14T23:18:02+01:00 router.local opnsense 495 - [meta sequenceId="24"] /usr/local/etc/rc.bootup: Accept router advertisements on interface igb0
<13>1 2022-02-14T23:18:02+01:00 router.local dhcp6c 79303 - [meta sequenceId="25"] RTSOLD script - Sending SIGHUP to dhcp6c
<13>1 2022-02-14T23:18:02+01:00 router.local opnsense 495 - [meta sequenceId="26"] plugins_configure ipsec_prepare (1)
<13>1 2022-02-14T23:18:02+01:00 router.local opnsense 495 - [meta sequenceId="27"] plugins_configure ipsec_prepare (execute task : ipsec_configure_vti(1))
<27>1 2022-02-14T23:18:02+01:00 router.local dhcp6c 38371 - [meta sequenceId="28"] transmit failed: Can't assign requested address
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="29"] plugins_configure early (1)
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="30"] plugins_configure early (execute task : openssh_configure_do(1))
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="31"] plugins_configure early (execute task : unbound_cache_flush(1))
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="32"] plugins_configure early (execute task : webgui_configure_do(1))
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="33"] /usr/local/etc/rc.bootup: ROUTING: entering configure using defaults
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="34"] /usr/local/etc/rc.bootup: ROUTING: IPv4 default gateway set to wan
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="35"] /usr/local/etc/rc.bootup: ROUTING: setting IPv4 default route to <REDACTED>
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="36"] /usr/local/etc/rc.bootup: ROUTING: keeping current default gateway '<REDACTED>'
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="37"] /usr/local/etc/rc.bootup: ROUTING: IPv6 default gateway set to lan
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="38"] /usr/local/etc/rc.bootup: ROUTING: skipping IPv6 default route
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="39"] plugins_configure hosts ()
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="40"] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="41"] plugins_configure hosts (execute task : unbound_hosts_generate())
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="42"] plugins_configure dhcp (1)
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="43"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="44"] plugins_configure dhcrelay (1)
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="45"] plugins_configure dhcrelay (execute task : dhcpd_dhcrelay_configure(1))
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="46"] plugins_configure dns (1)
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="47"] plugins_configure dns (execute task : dnsmasq_configure_do(1))
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="48"] plugins_configure dns (execute task : unbound_configure_do(1))
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="49"] plugins_configure monitor (1)
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="50"] plugins_configure monitor (execute task : dpinger_configure_do(1))
<11>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="51"] /usr/local/etc/rc.bootup: The LAN_DHCP6 monitor address is empty, skipping.
<11>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="52"] /usr/local/etc/rc.bootup: The WAN_DHCP6 monitor address is empty, skipping.
<11>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="53"] /usr/local/etc/rc.bootup: The WAN_DHCP monitor address is empty, skipping.


The redacted parts are the public IP and gateway given by the modem.

[RedVortex]

Out of curiosity, anything on the console when it happens ?

Anything on the console or output of dmesg ?

Also, can you post the output of

ifconfig igb0
ifconfig igb1

And the output of

dmesg | grep igb0
dmesg | grep igb1

[oh_lawd]

Nothing super suspicious. I've been trying to go through all the logs, but I see that one bad dn_init result that doesn't look too concerning, then the interfaces going down and up:

Code Select Expand


root@router:~ # ifconfig igb0
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
ether <REDACTED>
inet <REDACTED> netmask 0xffffff00 broadcast <REDACTED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@router:~ # ifconfig igb1
igb1: flags=28963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
description: LAN
options=4900028<VLAN_MTU,JUMBO_MTU,NETMAP,NOMAP>
ether <REDACTED>
inet 10.0.0.1 netmask 0xffff0000 broadcast 10.0.255.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@router:~ # dmesg |grep igb0
igb0: <Intel(R) I210 (Copper)> port 0x8000-0x801f mem 0xa1600000-0xa167ffff,0xa1680000-0xa1683fff irq 16 at device 0.0 on pci1
igb0: EEPROM V3.25-0 eTrack 0x800005cf
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
igb0: Ethernet address: <REDACTED>
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb0: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80001dd8800), ignoring.
igb0: link state changed to DOWN
igb0: link state changed to UP
root@router:~ # dmesg | grep igb1
igb1: <Intel(R) I210 (Copper)> port 0x7000-0x701f mem 0xa1500000-0xa157ffff,0xa1580000-0xa1583fff irq 17 at device 0.0 on pci2
igb1: EEPROM V3.25-0 eTrack 0x800005cf
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
igb1: Ethernet address: <REDACTED>
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb1 (ifp 0xfffff80003079800), ignoring.
igb1: link state changed to DOWN
igb1: link state changed to UP
igb1: permanently promiscuous mode enabled
igb1: link state changed to DOWN
igb1: link state changed to UP

This started to happen with the freebsd 13 upgrade, so might be a config value this network card doesn't like. I tried to reduce the MSI-X queues to 1 if it's a concurrency issue, but that didn't have any effect.

[RedVortex]

Nobody does usually but I suppose you have nothing in the maximum states, etc ? Firewall/Settings/Advanced (screenshot) and that the numbers for your system are high enough (not sure what are the specs (CPU, memory) of your router...

And no, I don't see anything suspicious either. I wonder if they updated the drivers in BSD13 for igb, maybe there is something in there or in the release notes.

[oh_lawd]

State limits are not set. See attached screenshots: https://imgur.com/a/l0xrLHX. I do run and subscribe to Zenarmor, and I do run a Wireguard client for all outgoing connections. Both of these use a bit of CPU, which is a quad core i5 with eight gigabytes of RAM (that has been more than enough for all of this). And, to be honest, I'd be extremely surprised if Sensei or Wireguard would kill the LAN randomly...

I'm trying to reduce the things I don't need, starting with disabling IPv6 in the internal network, if it has any help.

[oh_lawd]

One suspicious setting I have is the MSS setting in the LAN interface, that's set to 1380. This is due to Wireguard and how many sites didn't work with it if not set or set to the normal 1460. Might be something I'll check later if it has any effect, or if it's needed now when the distribution changed to FreeBSD in 22.1.

[oh_lawd]

Now it breaks almost exactly every 24h. When I trigger the shutdown the LAN comes back momentarily which I find weird.

[RedVortex]

Do you still have your router in your arp table when it stops working and is it the mac address of your router ?

arp -an

Also, if you ping something in your LAN from the opnsense console/command line, does it work ?

Also, instead of shutting down/rebooting opnsense, can you simply reload the firewall rules to see what happens ?

From memory it is something like:

configctl filter reload

I'm wondering about the last one since I think the rules are unloaded when rebooting so this may explain why it temporarily works while it reboots.

[oh_lawd]

I should check that next. I just need a monitor I can plug knto the box. It has no IPMI and naturally the LAN is down.

[oh_lawd]

If it is ythe firewall, two services I might want to stop now to see if one of them causes this: upnp and suricata...

[RedVortex]

I should check that next. I just need a monitor I can plug knto the box. It has no IPMI and naturally the LAN is down.

To be clear, the arp -an command is to be run on the client side. I simply wonder if your clients don't "see" the firewall at all anymore or that it doesn't even answer to arp requests.

[oh_lawd]

It looks like the 21.1.1 patch release fixed this issue for me. Haven't had this problem since I updated last week.

[oh_lawd]

I don't know if this is the right place to write about this, but from 22.1.1 forward the LAN problem seems to be fixed, but there's almost every day something that just breaks down the WAN. Today I was in the office, and around noon the WAN went down. I'm having a wireguard connection from the router to a remote host, and the issue is fixed by reconnecting and restarting unbound. Also what is weird is the logs are spammed with the following the whole time the connection is down:

https://gist.github.com/pimeys/e47b5ab4ca07ca9599906f28cd66f430

I was not home, so my partner restarted the router and everything went back to normal. I route all traffic over wireguard, using the kmod and having keepalive set to 25.

[Bytechanger]

Hm, I could have same problems, also using wireguard and WAN went down...

https://forum.opnsense.org/index.php?topic=26929.msg131462#msg131462

Using monit to restart wireguard if IP of destination change...


Greets

Byte

[oh_lawd]

This is happening more rarely nowadays, but is still an issue. I think the topic here is a bit mixed due to similar errors with WireGuard, which I got solved by removing the kmod and installing the go version...

Now here I have a problem happening every 1-2 weeks where the LAN (as far as I know, WAN might be down too) goes down and all machines in the house lose their IP address, wired and wireless. It happened last night, and from the machines connected with an ethernet I cannot even enable the interface, just getting "no carrier" errors. Reboot fixes everything.

I'm really trying hard to look into the logs, and there is nothing super interesting in there. System log has interfaces going up and down a few times, so this is the possible reason. I've disabled almost everything in the router, still running zenarmor but suricata has been off for over a month already. No MAC-spoofing or no VLANs.

From the logs we just start seeing errors how the router cannot connect to services in my LAN such as Grafana, meaning for the router all internal addresses are not there.