LAN "disconnected" after high load

[andreaslink]

Hej guys,
I really need some help as I recognized some strange behavior of my OPNsense setup (OPNsense 19.7.3-amd64, FreeBSD 11.2-RELEASE-p14-HBSD, OpenSSL 1.0.2s 28 May 2019).

I hope I provide enough information so you understand the issue. I'm running my OPNsense as a KVM machine on quite powerful server (Ubuntu 18.04) with dedicated passthrough network cards (4CPUs, 16 GB RAM). Currently I have assigned two NICs, one for WAN and one for LAN and "other LANs" with different VLANs. All works as expected with different DHCP servers, firewall rules etc. and I have very low load (only three clients as being in test mode).

See attached pic: 2019-09-05_OPNsense-IFs.png

Yesterday I started a "huge" test load, so I opened a port on WAN and forwarded it via NAT Port Forward to the SSH port of the client in the usual LAN network. Connections work like a charm and I was able to connect to the SSH server via port forward from WAN network - all as it should be.

Then I started quite some load (rsync via SSH) on only two connections (two rsync processes running equally). And around 500 GB were planned to be transferred from WAN to LAN (1GBit connection bandwidth).

See attached pic: 2019-09-05_OPNsense_LoadDuringTest.png

It all ran quite well, even though I was surprised about the high CPU load with my setup. I do not have intrusion detection, ntopng, NetFlow, Insight etc. set up right now, so packets are expected to only be guided through the firewall rules.

Then I let it run, went to bed and today in the morning (when I was at work already), the connection was surprisingly dead, so I was not able to reach the LAN from the WAN anymore. So I connected from my phone via wireguard to OPNsense and connection went smooth, load was around 0,30 so all seemed to be fine on OPNsense and I tried to check, if I could see anything from here. I used the diagnostic tools to ping the SSH host directly from OPNsense as "being in the local LAN" then and was not able to ping it. So I thought my target system died, but it never came to my mind that the LAN IF died .

When I cam home, I recognized my other host (also in the same LAN) was offline as well and I was not able to reach OPNsense from LAN anymore - but I was still able to connect via wireguard from my phone. So I rebooted OPNsense from my phone and did a clean reboot and immediately after the reboot my two hosts in the LAN were back online again as you can see in the graph (I started one rsync again in the early evening to test the connection again).

See attached pic: 2019-09-05_Traffic_LAN-Info.png
See attached pic: 2019-09-05_Traffic_WAN-Info.png

Conclusion so far:
During normal low lazy load from several LAN clients, it all runs stable. But having "heavy load" from WAN into LAN over a longer period made by OPNsense "somehow invisibly die" on the LAN side and I have no clue what happened here. This also happend once before but there I thought it was my fault due to other changes.

Does anyone have an idea or need more details?
There are no error packages logged, there are no related log entries, I could find. Could that be KVM setup related in relation to high load over a longer period? I can also always reproduce this.

Thanks in advance!
/Andreas