Messages

[Dienst "netdata"]

Ich bin hier ein wenig weiter gekommen. Ich habe mit den folgenden Tunnables für meine Broadcom Karten experimentiert:

kern.ipc.nmbclusters="2543660"
hw.bce.tso_enable="0"
hw.pci.enable_msix="0"

Mein kern.ipc.nmbclusters war vorher schon Größer als 1 Mio., somit hatte ich hier nur einfach nochmal mehr drauf gelegt.
Ich habe allerdings damit leider nichts verändert. Meine MBUF size ist damit nun erheblich Größer, aber meine MBUF usage liegt weiterhin bei 0% mit 6340/2543660 - auch wenn der Fehler auftritt. Die Zahlen kommen also an, scheinen aber keinen Zusammenhang mit der besagten Fehlermeldung zu haben:

2020-09-07T22:22:10   130.255112 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-07T22:22:04   124.883013 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-07T22:21:59   119.759379 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload

Sobald die Fehlermeldung auftritt, ist meine IPv6 connectivität jedenfalls schonmal tot.

Ich denke aber, ich habe nun die Ursache des Problemes gefunden, diese liegt bei dem Dienst "netdata". Dieser war nach einem Reboot nicht mit hochgekommen und starb mittels:

kernel   pid 53819 (netdata), jid 0, uid 302: exited on signal 11 (core dumped)

Im Log finde ich an selber Stelle unmittelbar davor auch:

configctl[62556]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out

configctl[96731]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out

Fazit:
Lasse ich netdata ausgeschaltet, dann habe ich keine Probleme, starte ich den Service, stirbt er immer wieder und kommt dann nach dem 3. oder 5. mal anklicken trotzdem hoch, dann kommt aber relativ Zeit der besagte Fehler "netmap_transmit bce0 drop mbuf that needs checksum offload" in enormer Geschwindigkeit in die Logs geflutet.

Kannst du oder jemand anders, diesen Fall auch nachstellen?

PS: Warum ich mein IPv6 GW nicht pingen kann, habe ich noch nicht geklärt, scheint aber ein anderer Fall zu sein und habe ich hier einmal gesondert angesprochen.

[OPNsense 20.7.1-amd64]

Thanks banym for your fast reaction, I hope my following words answer your questions.

I'm connected via FTTH behind a FritzBox (FB) and running IPv4 fine. I also got IPv6 some weeks ago within a dual stack setup and a common /56 to delegate and so to use on my own, which is assigned towards my FB. Within the FB IPv6 setup I activated "DNS-Server und IPv6-Präfix (IA_PD) zuweisen" aka allowing it to share (parts of) the /56 via DHCPv6 further on with other routers in the LAN. This works so far. Behind the FB I've my OPNsense (OPNsense 20.7.1-amd64) running, where all my clients are connected to.

It is set up with:

• Activated IPv6 witin OPNsense
• Set "IPv6 Configuration Type" on WAN (bce0) IF to DHCPv6
• Set within the basic "DHCPv6 client configuration":
• Request only an IPv6 prefix --> true
• Prefix delegation size --> 60 (As I got a /56 and I just wanted to have "some" (4 Bytes aka 16) subnets available on OPNsense (some more I can experiment on another router later)
• Send IPv6 prefix hint --> true
• Use IPv4 connectivity --> false
• Use VLAN priority --> Disabled
• On the LAN interface (bce1) I defined "IPv6 Configuration Type" as "None" (before I tested with "Track Interface" and further setup, turned it off for now, until IPv6 works towards WAN, before I start announcing the prefix into LAN)
• Deactivated "Block private networks" as well as "Block bogon networks" on LAN IF (as the LAN behind the FB obviously falls under these rules)
• Setup a Firewall rule to allow all ICMPv6 travel IN from WAN as well as for LAN (to cover all IPv6 ping and MTU-size requirements etc.)

With this setup, WAN got a decent IPv6 assigned from the FB as well as the /60. So this works fine and looks OK so far and as far as I can evaluate.

Please see the network plan, where I added the main parts:


            +------------------------------------------------------------------------------------+
            |                                                                                    |
            |                                    Internet                                        |
            |                                                                                    |
            +---------------------------------------+--------------------------------------------+
                                                    |
                                                    |
                                                    |
                                                    |
                      +-----------------------------+--------------------------+
                      |  FritzBox                                              |
                      |  fe80::c225:6ff:feff:820d                              |
                      |  Provider IPv6: 2a02:2f4:yyyy:yyyy:c225:6ff:feff:820d  |
                      |  IPv6-Prefix: 2a02:2f4:xxxx:xxxx::/56                  |
                      |  192.168.0.254                                         |
                      +---------------------+----------------------------------+
                                            |
                                            |
                                            |
                                            |
                      +---------------------+--------------------+
                      |  OPNsense                                |
                      |  WAN:                                    |
                      |   fe80::221:5eff:fec8:be88               |
                      |   2a02:2f4:xxxx:xxxx:221:5eff:fec8:be88  |
                      |   2a02:2f4:xxxx:zzzz::/60                |
                      |   192.168.0.100                          |
                      |  LAN:                                    |
                      |   DHCPv4: 192.168.123.10..100/24         |                     
                      |                                          |
                      +--+----+-------------------+---------+----+
                         |    |                   |         |
         +---------------+    |                   |         +-----+
         |                    |                   |               |
+--------+-------+    +-------+-------+   +-------+-------+   +---+---------+
| Client 1       |    | Client 2      |   | Client 3      |   |[...]        |
| 192.168.123.10 |    | 192.168.123.11|   | 192.168.123.12|   |             |
+----------------+    +---------------+   +---------------+   +-------------+


I also have further interfaces for IoT and Guests, but they are currently all setup comparable with LAN and no forther IPv6 subnets are delegated, so I skipped them in the drawing for now. And as said before, I first wanted to ensure that my WAN setup works with IPv6 and my OPNsense has full IPv6 connection before routing other IPv6 networks. Until then I stay with IPv4 for the LAN clients.

BTW: I have a wireguard VPN up and running on OPNsense as a working side2side connection, where the other side connects in via IPv6 directly onto the OPNsense wireguard service. This works without any problems as well.

Looking forward to further comments and hints :).

[BUT]

I have a strange IPv6 behavior running my OPNsense 20.7.2-amd64 with IPv6 behind a FritzBox.

I got an IPv6 address as well as /60 sub net assigned to my WAN, but when I try to ping the gateway directly from the firewall, all ICMPv6s get lost. I had opened firewall for all ICMPv6 on WAN on all directions.

This is what happens:

fe80::c225:6ff:feff:820d = FritzBox Link local address, correctly set as default IPv6 gateway
bce0 = WAN infterface

I cannot directly ping my router aka FritzBox :o:

root@OPNsense:~ # ping6 -c 3 fe80::c225:6ff:feff:820d%bce0
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> fe80::c225:6ff:feff:820d%bce0

--- fe80::c225:6ff:feff:820d%bce0 ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss


I cannot ping google's ipv6 dedicated host:

root@OPNsense:~ # ping6 -c3 ipv6.google.com
PING6(56=40+8+8 bytes) 2a02:2f4:xxxx:xxxx:221:5eff:fec8:be88 --> 2a00:1450:4001:81b::200e
--- ipv6.l.google.com ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss


BUT I can make a fine traceroute6 to that address, that works as expected (done via UDP):

root@OPNsense:~ # traceroute6 ipv6.google.com
traceroute6 to ipv6.l.google.com (2a00:1450:4001:81b::200e) from 2a02:2f4:xxxx:xxxx:221:5eff:fec8:be88, 64 hops max, 20 byte packets
1  2a02:2f4:xxxx:xxxx:c225:6ff:feff:820d  0.510 ms  0.468 ms  0.388 ms
2  2a02:2f0:0:72::  4.589 ms  14.459 ms  21.283 ms
3  2a02:2f0:0:34::  4.682 ms  4.618 ms  4.451 ms
4  2a02:2f0:4002::5d32:a0  7.877 ms  4.728 ms  4.649 ms
5  2001:4860:0:12e6::4  5.377 ms
    2001:4860:0:12e3::3  4.899 ms
    2001:4860:0:12e4::2  5.358 ms
6  2001:4860::c:4001:ec6  5.069 ms
    2001:4860::c:4001:ebe  15.328 ms
    2001:4860::c:4001:ec6  4.939 ms
7  2001:4860::c:4001:9920  15.494 ms
    2001:4860::c:4001:5c4  10.797 ms
    2001:4860::c:4001:9920  15.498 ms
8  2001:4860::8:0:cb95  14.999 ms
    2001:4860::c:4000:f873  14.720 ms *
9  2001:4860::1:0:d0d8  15.346 ms
    2001:4860::9:4001:31f1  14.559 ms  14.683 ms
10  2001:4860:0:1::673  14.393 ms  14.732 ms
    2001:4860:0:1::671  14.432 ms
11  fra15s16-in-x0e.1e100.net  14.496 ms
    2001:4860:0:1::671  14.465 ms  14.405 ms


Then I just ask for all the router in my local network via multicast request and I suddenly get an answer from the Fritzbox :o, this really puzzles me:

#All Routers Address:
root@OPNsense:~ # ping6 -c 2 ff02::2
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> ff02::2
16 bytes from fe80::c225:6ff:feff:820d%bce0, icmp_seq=0 hlim=64 time=0.562 ms
16 bytes from fe80::c225:6ff:feff:820d%bce0, icmp_seq=1 hlim=64 time=0.629 ms

--- ff02::2 ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.562/0.595/0.629/0.034 ms


So it obviously can receive and answer on ICMPv6, so I ping it directly again, but it does not answer:


root@OPNsense:~ # ping6 -c 3  fe80::c225:6ff:feff:820d%bce0
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> fe80::c225:6ff:feff:820d%bce0

--- fe80::c225:6ff:feff:820d%bce0 ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss


I really need a good advise, what is wrong here?

Summary:

• Multicast ping aka ICMPv6 works
• Direct ping does not work
• traceroute6 into internet works fine
• ping into internet does not work

Hej, hast du das Problem hier gelöst. Ich habe hier einen langen Post gemacht und nun nach vielen weiteren Tests festgestellt, dass ich vermutlich genau dein Problem habe und dazu auch ein sehr ähnliches Setup. Hast du auch keine Connectivität mehr, wenn das Problem auftritt? Bei mir bricht neben dem IPv6 ping bzw. der Nutzung vor allem auch mein Wireguard Side-2-side VPN weg, was aber auch über IPv6 getunnelt wird und daher immer ein signifikantes Indiz für einen Verlust der funktionierenden Verbindung ist.

Bei mir bringt eine Eingabe zur Aktivierung der Options mittels ifconfig bce0 txcsum rxcsum txcsum6 rxcsum6 leider nichts, ich habe auch nichts gefunden, wie ich den Fehler jetzt wegbekommen können sollte. Testweise könnte man Surricata aka IPS/IDS abschalten, ich habe zudem noch Sensei laufen. Alles das lief mit IPv4 only einwandfrei, erst seit mein IPv6 stack mit dazu gekommen ist und ich einen delegierten Prefix hinter der FB mitnutzen will habe ich massive Probleme und bin heute erst auf diesen Fehler hier aufmerksam geworden.
Ich bin inzwischen auf OPNsense 20.7.2-amd64, aber das hat nichts geändert. In diesem Bereich wurde ja auch laut Changelog nicht wirklich was geändert.

Was mich am meisten verwirrt, warum ich mein default IPv6 nicht von der OPNsense aus pingen kann:

Code Select Expand

root@OPNsense:~ # ping6 -c 2 fe80::c225:6ff:feff:820d%bce0
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> fe80::c225:6ff:feff:820d%bce0

--- fe80::c225:6ff:feff:820d%bce0 ping6 statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss


fe80::c225:6ff:feff:820d ist meine Fritzbox
bce0 ist mein WAN interface.

Und das obwohl das die default route ist:

Code Select Expand


root@OPNsense:~ # netstat -nr
Routing tables
[...]
Internet6:
Destination                       Gateway                       Flags     Netif Expire
default                           fe80::c225:6ff:feff:820d%bce0 UGS        bce0
::1                               link#8                        UH          lo0
[...]


Also gern bin ich an dieser Problemlösung mit interessiert :).

[20.7.2]

Looong story I entered here, and today I updated to the freshly released OPNsense 20.7.2-amd64 and since then I did all the setup again and it works right out of the box. So this was the same last time, meaning I'm not trusting it yet :). But I'm monitoring it...

[IPv6]

I have a problem getting IPv6 up and running in a stable way. I'm connected via FTTH behind a FritzBox (FB) and running IPv4 fine. Since some days my f***** provider forced me silently behind a Carrier Gate NAT but added - at least - a public IPv6 within the same change now. So I got a common /56 to use on my own, which is assigned towards my FB. Within the FB IPv6 setup I activated "DNS-Server und IPv6-Präfix (IA_PD) zuweisen" aka allowing it to share (parts of) the /56 via DHCPv6 further on with other routers in the LAN.

Behind the FB I've my OPNsense (OPNsense 20.7.1-amd64) running, where all my clients are, so I'm routing from my computer (Ubuntu 20.04 Linux) via OPNsense via FB into the Web. I know I'm double NATted within IPv4 (with CGNAT I guess even three times, but who cares...).  This forced me now to get deeper into IPv6 again so I also activated IPv6 within OPNsense according to common descriptions and examples in the web.

This means, what I did so far - beside the FB setup as explained before:

• Activated IPv6 witin OPNsense
• Set "IPv6 Configuration Type" on WAN (bce0) IF to DHCPv6
• Set within the basic "DHCPv6 client configuration":
• Request only an IPv6 prefix --> true
• Prefix delegation size --> 60 (As I got a /56 and I just wanted to have "some" (4 Bytes aka 16) subnets available on OPNsense (some more I can experiment on another router later)
• Send IPv6 prefix hint --> true
• Use IPv4 connectivity --> false
• Use VLAN priority --> Disabled
• On the LAN interface (bce1) I defined "IPv6 Configuration Type" as "Track Interface"
• Deactivated "Block private networks" as well as "Block bogon networks" on LAN IF (as the LAN behind the FB obviously falls under these rules)
• Under "Track IPv6 Interface" I set the value "WAN" for parameter "IPv6 Interface"
• Left the "IPv6 Prefix ID" unchanged at 0
• "Manual configuration" I left "false"
• Setup a Firewall rule to allow all ICMPv6 travel IN from WAN as well as for LAN (to cover all IPv6 ping and MTU-size requirements etc.)

With this setup, LAN got a decent IPv6 assigned from the FB as well as the /60. So this works fine and all clients within the OPNsense LAN got IPv6 addresses from the first subnet assigned. This looked OK so far and as far as I can evaluate.

Problem:
Now we come to the problem, sometimes I can ping the web and sometimes and I can only ping the OPNsense firewall from my computer. Then OPNsense cannot ping the FB via fe80 addr and then it works out of sudden. I would swear, I did not change the setup, I either rebooted or I only "refreshed" an interface, like e.g. WAN interface by pressing safe/apply without any changes and then it worked - sometimes. I'm annoyed as I cannot find a pattern for "sometimes" and my setup is fairly common with all I googled so far. So why does the behaviour flip so much? When it works it usually works really long. Last time I destroyed and reset me setup again when we had the Cloudfare aka transit provider issue in the internet as I first thought it was me here at home - bad timing :).

First there was no chance for the OPNsense to ping the FB, but then I read somewhere else that OPNsense is usually not following the entries in the routing table but enforces a GW per IF. So activated the option "Disable force gateway" on 'Firewall --> Settings --> Advanced' which leads to the route table to be used/evaluated. Right after this it worked out of the box, but I was never able to reproduce this step.

I then restored everything from a working backup, did the steps again and got another behaviour. Today I stopped experimenting and turned IPv6 of again on LAN so far as many website did not work with IPv6 not being correctly routed into the web.

I also had sometimes the impression that I made changes, which did not really have an effect or immediate impact. Then I thought I have it, restarted the firewall and after the boot it did not work out again or was behaving differently again, while it was working fine before I rebooted.

Solution approaches:
Beside many changes in the setup and some reboots I focussed closer on the firewall log. I added a logging IPv6 rule allowing all in via LAN, I named it "Default allow LAN IPv6 to any rule" and it allows any IPv6 protocol from everywhere to eveywhere in LAN, so more or less opening up the firewall. Just to be sure I also added the same rule for "out" on LAN in case I need to secure the way back and log it. Similar rules are defined for WAN, so I had a quite oben IPv6 net allowing all between OPNsense-LAN and Fritzbox-LAN aka OPNsense-WAN.

On my machine (ending on ::14cf) I have a client running which constanly tries to reach a secured (port tcp 443) web service on a server in the web (ending with ::59) and as you can see, sometimes packages are blocked randomly by the default deny rule, even though I explicitly opened all ports here. ICMPv6 is working fine as well. So even though the firewall was "mainly green", in this case the connection was not working - but traceroute6 (UDP) showed a perfect route from start to end - so routing in general seems to work correctly.




I have further screenshots, where I also show the details of the rules, while kicking in:

Block-Details:


Accept-Details:


And it does not matter, when I change the service or the target in the web, I can always monitor the same behaviour.
BTW, traceroute6 is always sucvcessful and correct. So the route as such seems to be fully correct. DNS works perfect and route is correct, but the real tcp content is not routed as it should.


What I might need to add is my local machines routing table for IPv6 (parts):

route -n6:

Kernel-IPv6-Routentabelle
Destination                    Next Hop                   Flag Met Ref Use If
::1/128                        ::                         U    256 2     0 lo
2a02:xxx:xxxx:fcf0::/64        ::                         Ue   100 9     0 enp5s0_vlan3
fe80::/64                      ::                         U    256 10     0 enp5s0_vlan3
::/0                           fe80::221:5eff:fec8:be8a   UGe  100 9     0 enp5s0_vlan3
::1/128                        ::                         Un   0   11     0 lo
2a02:xxx:xxxx:fcf0::14cf/128   ::                         Un   0   10     0 enp5s0_vlan3
2a02:xxx:xxxx:fcf0:82ee:73ff:fe28:2165/128 ::                         Un   0   4     0 enp5s0_vlan3
fe80::82ee:73ff:fe28:2165/128  ::                         Un   0   9     0 enp5s0_vlan3
ff00::/8                       ::                         U    256 11     0 enp5s0_vlan3
::/0                           ::                         !n   -1  1     0 lo


With:
fe80::221:5eff:fec8:be8a --> Being my OPNsense LAN link local address

And with "ip a s" showing (parts):
enp5s0_vlan3@enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 80:ee:73:28:21:65 brd ff:ff:ff:ff:ff:ff
    inet 192.168.42.10/24 brd 192.168.42.255 scope global enp5s0_vlan3
       valid_lft forever preferred_lft forever
    inet6 2a02:xxx:xxxx:fcf0::14cf/128 scope global dynamic noprefixroute
       valid_lft 4094sec preferred_lft 435sec
    inet6 2a02:xxx:xxxx:fcf0:82ee:73ff:fe28:2165/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 85754sec preferred_lft 13754sec
    inet6 fe80::82ee:73ff:fe28:2165/64 scope link
       valid_lft forever preferred_lft forever


I have no clue, why there is this random behaviour of OPNsense with IPv6 while IPv4 is running like a charm. I cannot find a pattern or where to work on, were is the problem or is it a software issue?
Looking very much for anyones smart input or thoughts or own experience, after I have invested days now and a lot of googleing!
Thanks in advance!

PS: And thanks for reading until here :). This description became more and more now :-\.

Hah, I finally solved it by myself after hours of thoughts ;). The solution is to remove the Source "LAN net" and replace it with "any" as my usual local LAN is mainly IPv4 based and therefore also only defined as an IPv4 net.

I was so blind, because I always add the source net to the interface, just to be sure not to oversee and allow someone something, who does not belong onto the corresponding interface. But for IPv6 I need to think different.

Posted the answer here as someone else might benefit from it or at least from these thoughts as well. ;D

[IPv4 working]

Hej firewall experts, I go nuts as I have two times the same thing, but once in IPv4 working and once in IPv6 not working. This is only all about allowing mDNS broadcasts to the common broadcast addresses (224.0.0.251 and [ff02::fb]) on port 5353 via UDP from LAN.

I have set up two aliases including the hosts as described above and as it can be seen on the screenshots. First I had both addresses in one alias, but now I split it up for dedicated IPv4 and IPv6 targets and created two rules by also copying them. Both on the LAN interface. IPv4 always worked right from the beginning but the copied IPv6 one is not considered and then finally blocked.

What is wrong here or what is different for IPv6? I do not get or see it.

I'm running newest release (OPNsense 20.1.2-amd64, FreeBSD 11.2-RELEASE-p17-HBSD, OpenSSL 1.1.1d 10 Sep 2019).
Looking for some good ideas/feedback or what I'm overlooking. Please ask, if you need more details.
Thanks in advance.

Non-the-less I expected it to behave slightly different, which is explaining the case now. IPS seems to only be triggered by the full separated eicar file - which kind of makes sense as the fingerprint is generated of the whole file, isn't it? But when I open the website, where the eicar-string as such is only included, nothing happens (see screenshot) - and always opening insecure with http of course.

Thank you very much for sharing this @AdSchellevis, this gives way more input and I can immediately prove it's working. I've tested opening your URL in the browser on two different clients (iPhone Safari and in Linux Chromium) as you can see on the screenshot.

PS: There is no chance of embedding attached screenshots within my reply, is it?

@sjjh I have tested it now for some days with different constellations and I can second, this does not work. But I still don't think it's necessarily related to the IDS/IPS setup, I guess it's still about the test. Can anyone prove the OPNsense-eicar-test to be working? I can still see impacts from outside which appear in my log if I'm attacked from outside on WAN (see attached screenshot). But I cannot trigger eicar download prevention either.

I'll read more into surricata and rules generation and will create my own rule, as I think it's somehow related to the fingerprint within the test. But this is just an assumption.

Ah, sorry, you are right, there is bunch of new rules in the new version, which I haven't seen yet. I also downloaded and installed the test rules now, I did not prevent the access, but I moved it to "Alarm".
The content of the testrule is very limited:

opnsense.test.rules
drop http any any -> any any (msg:"OPNsense test eicar virus"; content:"|58 35 4f 21 50 25 40 41 50 5b 34 5c 50 5a 58 35 34 28 50 5e 29 37 43 43 29 37 7d 24 45 49 43 41 52 2d 53 54 41 4e 44 41 52 44 2d 41 4e 54 49 56 49 52 55 53 2d 54 45 53 54 2d 46 49 4c 45 21 24 48 2b 48 2a|"; fast_pattern; reference:url,www.eicar.org/anti_virus_test_file.htm; classtype:bad-unknown; sid:7999999; rev:1;)


So I'll follow your test and report back. Have you considered deactivating the "Promiscues Mode" for testing?

I'm not sure, but I somehow think your are testing wrong ;). You expect the eicar test virus to kick in, but that's not what IPS does, this requires AV to be setup in terms of using the proxy and so on.

For IPS you might consider the SHA-1 cert fingerprint test. I just added facebook's current cert fingerprint with an alarm. An then - with a fresh client, who has not yet cached the cert - you can surf on facebook and the log shows the alarm appearing (just let it run for some days with all your clients in the net surfing to facebook). That worked for me and proved it working.
Else my setup is pretty close to yours, just that I'm not using "Promiscues Mode", so that flag is deactivated as I had trouble with my network card ignoring some data when this was hooked.

Hej guys,
I really need some help as I recognized some strange behavior of my OPNsense setup (OPNsense 19.7.3-amd64, FreeBSD 11.2-RELEASE-p14-HBSD, OpenSSL 1.0.2s 28 May 2019).

I hope I provide enough information so you understand the issue. I'm running my OPNsense as a KVM machine on quite powerful server (Ubuntu 18.04) with dedicated passthrough network cards (4CPUs, 16 GB RAM). Currently I have assigned two NICs, one for WAN and one for LAN and "other LANs" with different VLANs. All works as expected with different DHCP servers, firewall rules etc. and I have very low load (only three clients as being in test mode).

See attached pic: 2019-09-05_OPNsense-IFs.png

Yesterday I started a "huge" test load, so I opened a port on WAN and forwarded it via NAT Port Forward to the SSH port of the client in the usual LAN network. Connections work like a charm and I was able to connect to the SSH server via port forward from WAN network - all as it should be.

Then I started quite some load (rsync via SSH) on only two connections (two rsync processes running equally). And around 500 GB were planned to be transferred from WAN to LAN (1GBit connection bandwidth).

See attached pic: 2019-09-05_OPNsense_LoadDuringTest.png

It all ran quite well, even though I was surprised about the high CPU load with my setup. I do not have intrusion detection, ntopng, NetFlow, Insight etc. set up right now, so packets are expected to only be guided through the firewall rules.

Then I let it run, went to bed and today in the morning (when I was at work already), the connection was surprisingly dead, so I was not able to reach the LAN from the WAN anymore. So I connected from my phone via wireguard to OPNsense and connection went smooth, load was around 0,30 so all seemed to be fine on OPNsense and I tried to check, if I could see anything from here. I used the diagnostic tools to ping the SSH host directly from OPNsense as "being in the local LAN" then and was not able to ping it. So I thought my target system died, but it never came to my mind that the LAN IF died :o.

When I cam home, I recognized my other host (also in the same LAN) was offline as well and I was not able to reach OPNsense from LAN anymore - but I was still able to connect via wireguard from my phone. So I rebooted OPNsense from my phone and did a clean reboot and immediately after the reboot my two hosts in the LAN were back online again as you can see in the graph (I started one rsync again in the early evening to test the connection again).

See attached pic: 2019-09-05_Traffic_LAN-Info.png
See attached pic: 2019-09-05_Traffic_WAN-Info.png

Conclusion so far:
During normal low lazy load from several LAN clients, it all runs stable. But having "heavy load" from WAN into LAN over a longer period made by OPNsense "somehow invisibly die" on the LAN side and I have no clue what happened here. This also happend once before but there I thought it was my fault due to other changes.

Does anyone have an idea or need more details?
There are no error packages logged, there are no related log entries, I could find. Could that be KVM setup related in relation to high load over a longer period? I can also always reproduce this.

Thanks in advance!
/Andreas