OPNsense behind Fritz!Box reboots

[zaphod]

***EDIT*** Title

The problem occurs with the following setup using OPNsense 22.7.5:

Internet
     |
     |
Fritz!Box [AVM Internet Router] (IP 192.168.0.1/24, Exposed Hosts set to 192.168.0.2/24)
     |
     |
OPNsense (WAN: 192.168.0.2/24, Upstream Gateway 192.168.0.1, LAN; 192.168.1.1/24)
     |
     |
Managed Switch (192.168.1.2/24)
     |
     |
Client (192.168.1.3/24)

Also I have disabled "Block private networks" for the WAN interface since I use the Fritz!Box to connect to the internet.

It working as intended so far, but I get regular occurrences of the following which forces a connection loss and the OPNsense to perform some kind of "reset" since I hear the startup beep (da da di da da).

Code Select Expand

2022-10-06T11:07:16 Notice syslog-ng Configuration reload finished;
2022-10-06T11:07:16 Notice syslog-ng Configuration reload request received, reloading configuration;
2022-10-06T11:07:14 Error opnsense /usr/local/etc/rc.routing_configure: The WAN_GW monitor address is empty, skipping.
2022-10-06T11:07:14 Notice opnsense plugins_configure monitor (execute task : dpinger_configure_do(1,))
2022-10-06T11:07:14 Notice opnsense plugins_configure monitor (1,)
2022-10-06T11:07:14 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: keeping current default gateway '192.168.0.1'
2022-10-06T11:07:14 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: setting IPv4 default route to 192.168.0.1
2022-10-06T11:07:14 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: IPv4 default gateway set to wan
2022-10-06T11:07:14 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
2022-10-06T11:07:12 Notice php plugins_configure bootup (execute task : unbound_configure_do(1))
2022-10-06T11:07:12 Notice php plugins_configure bootup (execute task : opendns_configure_do(1))
2022-10-06T11:07:12 Notice php plugins_configure bootup (execute task : ntpd_configure_do(1))
2022-10-06T11:07:12 Notice php plugins_configure bootup (1)
2022-10-06T11:07:12 Error php /usr/local/etc/rc.bootup: Resyncing OpenVPN instances.
2022-10-06T11:07:12 Notice php plugins_configure vpn (execute task : openvpn_configure_do(1))
2022-10-06T11:07:10 Notice php plugins_configure vpn (execute task : ipsec_configure_do(1))
2022-10-06T11:07:10 Notice php plugins_configure vpn (1)
2022-10-06T11:07:10 Error php /usr/local/etc/rc.bootup: The WAN_GW monitor address is empty, skipping.
2022-10-06T11:07:10 Notice php plugins_configure monitor (execute task : dpinger_configure_do(1,,1))
2022-10-06T11:07:10 Notice php plugins_configure monitor (1,,1)
2022-10-06T11:07:09 Notice php plugins_configure dns (execute task : unbound_configure_do(1))
2022-10-06T11:07:09 Notice php plugins_configure dns (execute task : dnsmasq_configure_do(1))
2022-10-06T11:07:09 Notice php plugins_configure dns (1)
2022-10-06T11:07:09 Notice php plugins_configure dhcrelay (execute task : dhcpd_dhcrelay_configure(1))
2022-10-06T11:07:09 Notice php plugins_configure dhcrelay (1)
2022-10-06T11:07:09 Notice php plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))
2022-10-06T11:07:09 Notice php plugins_configure dhcp (1)
2022-10-06T11:07:09 Error php /usr/local/etc/rc.bootup: ROUTING: creating /tmp/re0_defaultgw using '192.168.0.1'
2022-10-06T11:07:09 Error php /usr/local/etc/rc.bootup: ROUTING: setting IPv4 default route to 192.168.0.1
2022-10-06T11:07:09 Error php /usr/local/etc/rc.bootup: ROUTING: IPv4 default gateway set to wan
2022-10-06T11:07:09 Error php /usr/local/etc/rc.bootup: ROUTING: entering configure using defaults
2022-10-06T11:07:09 Notice php plugins_configure early (execute task : webgui_configure_do(1))
2022-10-06T11:07:09 Notice php plugins_configure early (execute task : unbound_cache_flush(1))
2022-10-06T11:07:08 Notice php plugins_configure early (execute task : openssh_configure_do(1))
2022-10-06T11:07:08 Notice php plugins_configure early (1)
2022-10-06T11:07:06 Notice syslog-ng syslog-ng starting up; version='3.38.1'

The Fritz!Box router does not support bridge mode and I cannot replace it with a DSL modem right now.
Researching indicates that the above setup should basically be working, which is does aside from the mentioned instability.

Since I can't make any heads and tails of it, please give me a hint if you can.
Any help would be appreciated!

Thanks.

[cookiemonster]

Try if you can using a different network card. Realtek interfaces can still get lockups and drops from freebsd systems. I _think_ there are alternative drivers that can be used in OPN. Don't quote me on that, I'm going by what I might incorrectly recall from seeing over time.

[zaphod]

I switched to the manufacturer's drivers for the Realtek card yesterday using the os-realtek-re plugin.

I have an Intel PRO/1000PT Quad Port NIC installed as well (em0...3), so I switched the WAN interface from re0 to em1.

Sadly the error is exactly the same on the em1 interface.

The config was WAN re0, LAN em0 which I now changed to WAN em1 and LAN em1.

[phoenix]

The Fritz!Box router does not support bridge mode and I cannot replace it with a DSL modem right now.

That's correct but it does support a 'passthru' mode. Is your connection a PPPoE connection? If it is then it's faIrly trivial to pass that through to your OPNsense for it to connect via PPPoE, it works fine for me.

[cookiemonster]

It's most likely the routines you see there are what normally happens when the connection drops. So you're diagnosing the connection drops. Harware and drivers is where you're at. Those might not be the problem.
The joys of diagnosing.

[zaphod]

After plugging a monitor into the OPNsense I discovered that it simply reboots for no reason.
That's why I heard the boot beep in the first place.
I still see the log entry, but it seems now that this is just a reaction to the reboot.

Maybe it's a hardware issue, but I have no replacement parts right now.

Thanks for the help so far, I have edited the title.

[lilsense]

I'd recommend to reset everything to default and restart fresh.

[zaphod]

Just give an update:
The issue is caused by hardware incompatibility between the ThinClient (Fujitsu Futro S920) I was using as base for the OPNSense and the Intel PRO/1000PT Quad Port.
The Futro S920 comes in multiple versions, mine has the AMD GX-222GC SoC which is incompatible with the Intel NIC.
The incompatibility is based on the AMD's APU and the Active State Power Mgmt (ASPM) on the NIC.
See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240944
Disabling ASPM (hw.pci.enable_aspm = 0) did not solve the problem.

I decided to switch the ThinClient for an earlier version with an AMD GX-415GA SoC.
The issue was never seen with this ThinClient/NIC combo and OPNsense is running stable ever since  :)