Messages

Happens to me too with OPNsense 23.1.1_2... The problem is I run nextdns in the router, and when this link state thing happens, nextdns is unable to resolve for a few minutes and I either need to wait with no DNS or restart nextdns when that happens.

This is happening more rarely nowadays, but is still an issue. I think the topic here is a bit mixed due to similar errors with WireGuard, which I got solved by removing the kmod and installing the go version...

Now here I have a problem happening every 1-2 weeks where the LAN (as far as I know, WAN might be down too) goes down and all machines in the house lose their IP address, wired and wireless. It happened last night, and from the machines connected with an ethernet I cannot even enable the interface, just getting "no carrier" errors. Reboot fixes everything.

I'm really trying hard to look into the logs, and there is nothing super interesting in there. System log has interfaces going up and down a few times, so this is the possible reason. I've disabled almost everything in the router, still running zenarmor but suricata has been off for over a month already. No MAC-spoofing or no VLANs.

From the logs we just start seeing errors how the router cannot connect to services in my LAN such as Grafana, meaning for the router all internal addresses are not there.

Could it be the same issue that me and a few others are having over here?

https://forum.opnsense.org/index.php?topic=26929.0

The problem still persists in 22.1.2. WAN comes down quite often now when I have a cloud sync task running, using the upstream of my connection for hours to transfer hundreds of gigabytes to the backup service.

I've had the connection dropping now every hour or so. What I see is there's lots of activity in pf.searches value during the upload. Restarting the Wireguard connection makes the connection to work again.

Looks like my problem. Do you have a easy setup for your wireguard restart script until this is fixed?

I don't know if this is the right place to write about this, but from 22.1.1 forward the LAN problem seems to be fixed, but there's almost every day something that just breaks down the WAN. Today I was in the office, and around noon the WAN went down. I'm having a wireguard connection from the router to a remote host, and the issue is fixed by reconnecting and restarting unbound. Also what is weird is the logs are spammed with the following the whole time the connection is down:

https://gist.github.com/pimeys/e47b5ab4ca07ca9599906f28cd66f430

I was not home, so my partner restarted the router and everything went back to normal. I route all traffic over wireguard, using the kmod and having keepalive set to 25.

It looks like the 21.1.1 patch release fixed this issue for me. Haven't had this problem since I updated last week.

If it is ythe firewall, two services I might want to stop now to see if one of them causes this: upnp and suricata...

I should check that next. I just need a monitor I can plug knto the box. It has no IPMI and naturally the LAN is down.

Now it breaks almost exactly every 24h. When I trigger the shutdown the LAN comes back momentarily which I find weird.

One suspicious setting I have is the MSS setting in the LAN interface, that's set to 1380. This is due to Wireguard and how many sites didn't work with it if not set or set to the normal 1460. Might be something I'll check later if it has any effect, or if it's needed now when the distribution changed to FreeBSD in 22.1.

State limits are not set. See attached screenshots: https://imgur.com/a/l0xrLHX. I do run and subscribe to Zenarmor, and I do run a Wireguard client for all outgoing connections. Both of these use a bit of CPU, which is a quad core i5 with eight gigabytes of RAM (that has been more than enough for all of this). And, to be honest, I'd be extremely surprised if Sensei or Wireguard would kill the LAN randomly...

I'm trying to reduce the things I don't need, starting with disabling IPv6 in the internal network, if it has any help.

Nothing super suspicious. I've been trying to go through all the logs, but I see that one bad dn_init result that doesn't look too concerning, then the interfaces going down and up:

Code Select Expand


root@router:~ # ifconfig igb0
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
ether <REDACTED>
inet <REDACTED> netmask 0xffffff00 broadcast <REDACTED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@router:~ # ifconfig igb1
igb1: flags=28963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
description: LAN
options=4900028<VLAN_MTU,JUMBO_MTU,NETMAP,NOMAP>
ether <REDACTED>
inet 10.0.0.1 netmask 0xffff0000 broadcast 10.0.255.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@router:~ # dmesg |grep igb0
igb0: <Intel(R) I210 (Copper)> port 0x8000-0x801f mem 0xa1600000-0xa167ffff,0xa1680000-0xa1683fff irq 16 at device 0.0 on pci1
igb0: EEPROM V3.25-0 eTrack 0x800005cf
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
igb0: Ethernet address: <REDACTED>
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb0: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80001dd8800), ignoring.
igb0: link state changed to DOWN
igb0: link state changed to UP
root@router:~ # dmesg | grep igb1
igb1: <Intel(R) I210 (Copper)> port 0x7000-0x701f mem 0xa1500000-0xa157ffff,0xa1580000-0xa1583fff irq 17 at device 0.0 on pci2
igb1: EEPROM V3.25-0 eTrack 0x800005cf
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
igb1: Ethernet address: <REDACTED>
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb1 (ifp 0xfffff80003079800), ignoring.
igb1: link state changed to DOWN
igb1: link state changed to UP
igb1: permanently promiscuous mode enabled
igb1: link state changed to DOWN
igb1: link state changed to UP

This started to happen with the freebsd 13 upgrade, so might be a config value this network card doesn't like. I tried to reduce the MSI-X queues to 1 if it's a concurrency issue, but that didn't have any effect.

Today it happened again. LAN went out completely and I needed to reboot the router. Now I was able to capture some logs:

Code Select Expand


<11>1 2022-02-14T23:16:37+01:00 router.local opnsense 33229 - [meta sequenceId="2"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic lan(igb1)
<27>1 2022-02-14T23:16:38+01:00 router.local dhcp6c 33998 - [meta sequenceId="3"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:16:39+01:00 router.local dhcp6c 33998 - [meta sequenceId="4"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:16:41+01:00 router.local dhcp6c 33998 - [meta sequenceId="5"] transmit failed: Network is down
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="6"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic lan(igb1)
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="7"] /usr/local/etc/rc.linkup: Accept router advertisements on interface igb1
<13>1 2022-02-14T23:16:42+01:00 router.local dhcp6c 6366 - [meta sequenceId="8"] RTSOLD script - Sending SIGHUP to dhcp6c
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'lan'
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: ROUTING: IPv6 default gateway set to lan
<11>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
<13>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="14"] plugins_configure ipsec (,lan)
<13>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="15"] plugins_configure ipsec (execute task : ipsec_configure_do(,lan))
<13>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="16"] plugins_configure dhcp ()
<13>1 2022-02-14T23:16:42+01:00 router.local opnsense 80163 - [meta sequenceId="17"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2022-02-14T23:16:43+01:00 router.local opnsense 80163 - [meta sequenceId="18"] plugins_configure dns ()
<13>1 2022-02-14T23:16:43+01:00 router.local opnsense 80163 - [meta sequenceId="19"] plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2022-02-14T23:16:43+01:00 router.local opnsense 80163 - [meta sequenceId="20"] plugins_configure dns (execute task : unbound_configure_do())
<45>1 2022-02-14T23:17:05+01:00 router.local syslog-ng 4839 - [meta sequenceId="21"] syslog-ng shutting down; version='3.35.1'
<45>1 2022-02-14T23:17:57+01:00 router.local syslog-ng 17974 - [meta sequenceId="1"] syslog-ng starting up; version='3.35.1'
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="2"] plugins_configure loopback_prepare (1)
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="3"] plugins_configure loopback_prepare (execute task : loopback_configure_interface(1))
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="4"] plugins_configure openvpn_prepare (1)
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="5"] plugins_configure openvpn_prepare (execute task : openvpn_prepare(1))
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="6"] plugins_configure vxlan_prepare (1)
<13>1 2022-02-14T23:17:57+01:00 router.local opnsense 495 - [meta sequenceId="7"] plugins_configure vxlan_prepare (execute task : vxlan_configure_interface(1))
<11>1 2022-02-14T23:17:58+01:00 router.local opnsense 495 - [meta sequenceId="8"] /usr/local/etc/rc.bootup: Accept router advertisements on interface igb1
<13>1 2022-02-14T23:17:58+01:00 router.local dhcp6c 37343 - [meta sequenceId="9"] RTSOLD script - Starting dhcp6 client
<27>1 2022-02-14T23:17:59+01:00 router.local dhcp6c 38371 - [meta sequenceId="10"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:17:59+01:00 router.local dhcp6c 38371 - [meta sequenceId="11"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:18:00+01:00 router.local dhcp6c 38371 - [meta sequenceId="12"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:18:00+01:00 router.local dhcp6c 38371 - [meta sequenceId="13"] transmit failed: Network is down
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 53721 - [meta sequenceId="14"] New IP Address (igb0): <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 55210 - [meta sequenceId="15"] New Subnet Mask (igb0): 255.255.255.0
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 56598 - [meta sequenceId="16"] New Broadcast Address (igb0): <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 58168 - [meta sequenceId="17"] New Routers (igb0): <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 61390 - [meta sequenceId="18"] route add default <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 62734 - [meta sequenceId="19"] New Static Routes (igb0): 192.168.5.1 <REDACTED>
<13>1 2022-02-14T23:18:02+01:00 router.local dhclient 65468 - [meta sequenceId="20"] Creating resolv.conf
<11>1 2022-02-14T23:18:02+01:00 router.local opnsense 66978 - [meta sequenceId="21"] /usr/local/etc/rc.newwanip: IP renewal deferred during boot on 'igb0'
<27>1 2022-02-14T23:18:02+01:00 router.local dhcp6c 38371 - [meta sequenceId="22"] transmit failed: Can't assign requested address
<27>1 2022-02-14T23:18:02+01:00 router.local dhcp6c 38371 - [meta sequenceId="23"] transmit failed: Network is down
<11>1 2022-02-14T23:18:02+01:00 router.local opnsense 495 - [meta sequenceId="24"] /usr/local/etc/rc.bootup: Accept router advertisements on interface igb0
<13>1 2022-02-14T23:18:02+01:00 router.local dhcp6c 79303 - [meta sequenceId="25"] RTSOLD script - Sending SIGHUP to dhcp6c
<13>1 2022-02-14T23:18:02+01:00 router.local opnsense 495 - [meta sequenceId="26"] plugins_configure ipsec_prepare (1)
<13>1 2022-02-14T23:18:02+01:00 router.local opnsense 495 - [meta sequenceId="27"] plugins_configure ipsec_prepare (execute task : ipsec_configure_vti(1))
<27>1 2022-02-14T23:18:02+01:00 router.local dhcp6c 38371 - [meta sequenceId="28"] transmit failed: Can't assign requested address
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="29"] plugins_configure early (1)
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="30"] plugins_configure early (execute task : openssh_configure_do(1))
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="31"] plugins_configure early (execute task : unbound_cache_flush(1))
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="32"] plugins_configure early (execute task : webgui_configure_do(1))
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="33"] /usr/local/etc/rc.bootup: ROUTING: entering configure using defaults
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="34"] /usr/local/etc/rc.bootup: ROUTING: IPv4 default gateway set to wan
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="35"] /usr/local/etc/rc.bootup: ROUTING: setting IPv4 default route to <REDACTED>
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="36"] /usr/local/etc/rc.bootup: ROUTING: keeping current default gateway '<REDACTED>'
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="37"] /usr/local/etc/rc.bootup: ROUTING: IPv6 default gateway set to lan
<11>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="38"] /usr/local/etc/rc.bootup: ROUTING: skipping IPv6 default route
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="39"] plugins_configure hosts ()
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="40"] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
<13>1 2022-02-14T23:18:03+01:00 router.local opnsense 495 - [meta sequenceId="41"] plugins_configure hosts (execute task : unbound_hosts_generate())
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="42"] plugins_configure dhcp (1)
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="43"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="44"] plugins_configure dhcrelay (1)
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="45"] plugins_configure dhcrelay (execute task : dhcpd_dhcrelay_configure(1))
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="46"] plugins_configure dns (1)
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="47"] plugins_configure dns (execute task : dnsmasq_configure_do(1))
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="48"] plugins_configure dns (execute task : unbound_configure_do(1))
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="49"] plugins_configure monitor (1)
<13>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="50"] plugins_configure monitor (execute task : dpinger_configure_do(1))
<11>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="51"] /usr/local/etc/rc.bootup: The LAN_DHCP6 monitor address is empty, skipping.
<11>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="52"] /usr/local/etc/rc.bootup: The WAN_DHCP6 monitor address is empty, skipping.
<11>1 2022-02-14T23:18:04+01:00 router.local opnsense 495 - [meta sequenceId="53"] /usr/local/etc/rc.bootup: The WAN_DHCP monitor address is empty, skipping.


The redacted parts are the public IP and gateway given by the modem.

This morning again the whole LAN went down. It seems to happen quite often now. I tried a few settings from the earlier posts:

Code Select Expand


sysctl dev.igb.0.eee_control=0
sysctl dev.igb.1.eee_control=0


Disabling energy saving just turned off the network interfaces and I needed to reboot.

Code Select Expand


dev.igb.1.iflib.disable_msix=1
dev.igb.0.iflib.disable_msix=1


If disabling msi-x helped with the realtek drivers, maybe it'll help here too. These settings didn't break the connection, so let's see if they have any effect to the problem.

Here's the full output of pciconf:

Code Select Expand


igb0@pci0:1:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0xffff subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 5 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 00f421ffff6833bb
    ecap 0017[1a0] = TPH Requester 1