Messages

Hej, just throwing in your virtualization. Are you running your OPNsense VM on Proxmox? There is a known issue (I am trying to find a solution for as well) where Proxmox looses IPv6 routing/connectivity after ~20 minutes from and to all guests. I have tried several parameters, but nothing fixes the issue - testing is also cumbersome, as this requires a lot of waiting after each reboot to see it fail again. Every reboot fixes the issue, for the next ~20 minutes.
So, if you are not running on bare metal, you might search for your issue on Proxmox and not on OPNsense.

I can confirm this, first I was confused, what was triggering it, but your hint regarding wireguard answered it for me. But I need some WG interfaces to be part of it as well, as I had before the update; so I guess this requires an issue for a bugfix on github then?

Out of curiosity, is there any update here? I cannot yet find an official package/plugin in the repo, can I?
And can I (later) use CrowdSec next to surricata as an IPS system or am I mixing things up here?

[21.1.3_3]

Hi @andreas,

We think that this is not related to netmap and has something to do with wireguard's resetting interface flags; but did not find much time to have a look in detail.

I'll post an update once we have some more information.

I just want to bring this topic up again. I doubt you found the time @mb to look further into this, but since this is still existing with the exact same situation as before and in my case triggered via a foreign wireguard ping, I thought I bring it up again and ask again.
So the same problem still existing in 21.1.3_3 so far (did not go to 20.1.4 yet, due to major wireguard implementation changes).

Any news here or plan to progress?

Thank you Franco and you are right as I was finally able to solve. The kernel was not able to detect the usual boot medium as my RAID (HW-Raid controller) ran out of sync "somehow" during the boot, I guess and as long as it was out of sync the SAS RAID controller kept if offline.
I finally connected monitor and keyboard and restored the RAID (simple mirroring) by syncing the disks again (see screenshot) and finally after the sync the problem was solved. After restore, there was a disk again and I was able too boot as I expected it.

So this was very probably not related to the upgrade I made, but to my hardware aka environment and kicked by coincidence in during the reboot. Thanks for supporting and solving it. I mark this one as solved.

I've never ran into any issues with kernel updates (with the same BIOS)

That is good to know and (until today) I share the same experience - even though not over such a long period. That is why I was asking how to be able to restore the former kernel from here as I potentially also might have HW issues or a RAID-Controller issue by coincidence in parallel which was just triggered by the reboot only. It's not said that I'm connecting wrong events here to drill down to the error.

I will for sure investigate further on the HW part, but is there a way to downgrade to the former kernel, when I just boot from USB to somehow repair the system manually or is a booting always required? It looks like the kernel is trying to boot but can't proceed due to missing/unreadable hard disks, I guess?

With restoring I also loose all my historical logs, so that is my last way to go, I would prefer to find a real root cause here.
Is that the new kernel not supporting my hardware oder BIOS anymore?
I just would prefer to go back to the former kernel, where all was running fine and check this was still working.

I have updated my server as usual (and as I had done all times before) and as this update required a reboot I'm stuck now. The kernel does not boot on my hardware anymore, server is hanging with "Probing 13 blocks devices..." and that's it, see attached screenshot.

I'm running on bare metal and that is one hw-info from before: https://bsd-hardware.info/?probe=e161817fa3

I'm really looking for help, any idea how I could boot from USB and restore to former kernel somehow?

Is it possible that only XEN virtualized OPNsense instances could be affected?

No, I do not think so, I'm also running directly on bare metal and I was affected here as well. Franco pointed out the kernel to be responsible and this could be confirmed. So I can currently live with only IDS and no IPS activated until this is under better control, but downgrading kernel is in my mind currently the approbate solution in relation to the performance drop.

He guys,
any news here, to get this fixed?
I still have this error and I'm running on 21.1 now and I have turned off Sensei/ElasticSearch so far. So I'm not sure, if this is the root cause or triggering it? Just asking for fresh ideas.

[ET EXPLOIT TLS HeartBeat Request (Server Initiated) fb set]

I can second this as well.

I got some false positives (at least I hope they were false) on SID 2018375 "ET EXPLOIT TLS HeartBeat Request (Server Initiated) fb set", where I would see/test if it is making sense; so for testing I wanted to set this one from "Drop" to "Alert" only, the change is not saved. Even after applying and restart of service.

[with IDS/IPS]

I updated Thursday evening to OPNsense 21.1-amd64 and realized next morning that routed permanent video streams between LAN and WAN were significant slower until they broke very soon.

To the background, I have OPNsense running within my local network separating networks, so there is another router before reaching the Internet, which allows me 1GBit speedtests via OPNsense within my infrastructure.
So, in short I have LAN <-> OPNsense --> WAN <--> FritzBox --> Internet, this allows me stress tests from my LAN into the WAN without going through the slower internet connection of the provider. If it matters, I am running OPNsense on a 4 core Intel Xeon E5506, 20GB RAM, 2x Broadcom NetXtreme II BCM5709, 4x Intel 82580. Sensei currently deactivated.

After analysis I figured out that IDS/IPS is the root cause here. I came updated from 20.7.8_4 were everything was fine and as I read here https://opnsense.org/opnsense-21-1-marvelous-meerkat-released/ there are no changes made to Suricata within the release. I did not make any changes to the related setup or rules etc.

So, I made some interesting iperf3 measurements.

OPNsense v20.7.8_4:
Host LAN-net <-> Host WAN-net with IDS/IPS activated --> ~550 MBit/s

OPNsense v21.1:
Host LAN-net <-> Host WAN-net with IDS/IPS activated:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  1.68 GBytes   240 Mbits/sec  128             sender
[  5]   0.00-60.17  sec  1.68 GBytes   239 Mbits/sec                  receiver

Host LAN-net <-> Host WAN-net no IDS/IPS:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  2.54 GBytes   729 Mbits/sec  978             sender
[  5]   0.00-30.01  sec  2.54 GBytes   728 Mbits/sec                  receiver


OPNsense <-> Host WAN-net no IDS/IPS:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  3.29 GBytes   942 Mbits/sec  408             sender
[  5]   0.00-30.00  sec  3.29 GBytes   941 Mbits/sec                  receiver

As a result I can see within the update a performance dropped from ~550 Mbit/s down to ~240 Mbit/s, which is a performance drop of ~310 MBit/s aka 56%, which I cannot explain but measure. My overall routing power between LAN and WAN seems to be around 729 Mbit/s, which is acceptable for me as quite some video streams were passing through the firewall during measurement and where I do not have a comparable value from before the update.

Any suggestions what causes this IDS/IPS impact? Can someone second this behavior on his setup as well? I know for future only-internet-connections, this might be sufficient, but currently I feel unhappy with the result as it just came with the update to 21.1.
Looking forward for hints, ideas and comments.

[fe80::c225:6ff:feff:820d%bce0]

Thanks for your input, @marjohn56!

Regarding:

This causes an issue with dpinger gateway monitoring, a workaround is to use another target address such as one of the google DNS server addresses, note if you use multwan do not use the same address for both v6 monitors, also if you are using googles DNS for resolution then find some other WAN v6 targets.

I'm not sure, if this is the case as I do not use a monitoring IP yet in my via DHCPv6 assigned IPv6 WAN IF. I kept it default and then it is empty, so I consider this interface always as being up to not get into any trouble with dpinger. And I do not use multiwan (yet).

Regarding your hint, I tested that:

EDIT: @andreaslink - can you try disabling Static route filtering on the Firewall->Advanced page, I've seen this fix the issue for some.

I was so motivated, but this did not change anything.

Reading the posts from @gary201 I also checked some of the tables. Looking into ndp tables, I see my link local IPv6 gateway in there, so this seems to be a difference to your problem:


root@OPNsense:~ # ndp -a
Neighbor                             Linklayer Address  Netif Expire    S Flags
fe80::92e2:baff:fe68:cd75%igb1       90:e2:ba:68:cd:75   igb1 permanent R
fe80::92e2:baff:fe68:cd74%igb0       90:e2:ba:68:cd:74   igb0 permanent R
fe80::221:5eff:fec8:be8a%bce1        00:21:5e:c8:be:8a   bce1 permanent R
fd00:0:cafe:affe:221:5eff:fec8:be88  00:21:5e:c8:be:88   bce0 permanent R
OPNsense.lan                         00:21:5e:c8:be:88   bce0 permanent R
fe80::221:5eff:fec8:be88%bce0        00:21:5e:c8:be:88   bce0 permanent R
fe80::c225:6ff:feff:820d%bce0        c0:25:06:ff:82:0d   bce0 23h57m29s S R
2a02:xxx:xxxx:7700:307d:9cff:feca:7797 32:7d:9c:ca:77:97 bce0 8h55m29s  S
fe80::307d:9cff:feca:7797%bce0       32:7d:9c:ca:77:97   bce0 8h55m34s  S

root@OPNsense:~ # netstat -rn6
Routing tables

Internet6:
Destination                       Gateway                       Flags     Netif Expire
default                           fe80::c225:6ff:feff:820d%bce0 UG         bce0
::1                               link#8                        UH          lo0
2a02:xxx:xxxx:7700::/64           link#1                        U          bce0
2a02:xxx:xxxx:7700:221:5eff:fec8:be88 link#1                    UHS         lo0
fd00:0:cafe:affe::/64             link#1                        U          bce0
fd00:0:cafe:affe:221:5eff:fec8:be88 link#1                      UHS         lo0
fe80::%bce0/64                    link#1                        U          bce0
fe80::221:5eff:fec8:be88%bce0     link#1                        UHS         lo0
fe80::%bce1/64                    link#2                        U          bce1
fe80::221:5eff:fec8:be8a%bce1     link#2                        UHS         lo0
fe80::%igb0/64                    link#3                        U          igb0
fe80::92e2:baff:fe68:cd74%igb0    link#3                        UHS         lo0
fe80::%igb1/64                    link#4                        U          igb1
fe80::92e2:baff:fe68:cd75%igb1    link#4                        UHS         lo0
fe80::%lo0/64                     link#8                        U           lo0
fe80::1%lo0                       link#8                        UHS         lo0


All the rest still behaving the same way and I cannot connect to the internet via IPv6:
root@OPNsense:~ # ping6 -c 1 fe80::c225:6ff:feff:820d%bce0
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> fe80::c225:6ff:feff:820d%bce0

--- fe80::c225:6ff:feff:820d%bce0 ping6 statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss


root@OPNsense:~ # ping6 -c 1 ipv6.google.com
PING6(56=40+8+8 bytes) 2a02:xxx:xxxx:7700:221:5eff:fec8:be88 --> 2607:f8b0:4004:82a::200e

--- ipv6.l.google.com ping6 statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss


root@OPNsense:~ # ping6 -c 2 ff02::2%bce0
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> ff02::2%bce0
16 bytes from fe80::c225:6ff:feff:820d%bce0, icmp_seq=0 hlim=64 time=2.595 ms
16 bytes from fe80::c225:6ff:feff:820d%bce0, icmp_seq=1 hlim=64 time=1.674 ms

--- ff02::2%bce0 ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.674/2.135/2.595/0.460 ms


BTW, IPv4 works as it should, no probs at all.
I'm slightly puzzled, what could prevent the IPv6 connection here. I'm considering setting up a fresh install as this one is already updated since quite some years now.

Could it be, that some default settings have changed over time and the updater does not consider deleting/updating old ones or likewise? A fresh install would proof that; I read something similar here https://forum.opnsense.org/index.php?topic=11341.msg66960#msg66960 some time ago as I'm running out of ideas  :-\.

I'm planning on setting up traffic shaping with pipes and queues to limit bandwidth or prioritize traffic. I read through the documentation (https://docs.opnsense.org/manual/shaping.html) but I did not clearly unterstand if this always limits the bandwidth or only when the cable is under full load - if you understand what I mean?

For example, I have a 100 MBit internet connection and I want to prioritize bandwidth for a specific host (in a specific network) with always at least 40 MBit, which means, there are only 60 Mbit left for all other hosts, when all 100 Mbit are requested via WAN. Today I would assume there is an equal balance between all of them. But if no one is using the bandwidth I want everything - all 100 MBit - to be available for the specific host asking and do not want to limit it to 40 MBit only.

As I see it, I can setup a dedicated pipe according to (https://docs.opnsense.org/manual/how-tos/shaper.html) case 1 "Reserve dedicated bandwidth for a realtime traffic". Then it is reserved if all is requested, but will this host also get everything, if nothing else is used? This is not clear to me. There the setup seems to ensure, but also limit it, by creating upload and download pipes as it states "desired bandwidth" and not "minimum bandwidth" or likewise. I hope I can explain myself?

Or can/do I need to setup queues with pipes on top somehow? Such that I can kind of give a pipe a higher weight bevor it kicks in and if the weight is not requested, the pipe does not have any value?

What I would like to prevent is not to leave out available bandwidth just due to limiting down packets in relation to the corresponding pipe. If there is no traffic, I want to offer everything. How can I ensure this?

Thanks for the hint, I read that thread as well but this one is one step further already. I'm one stage before. I follow nearly the same approach as described in the beginning of the issue and get via DHCPv6 request send from my OPNsense an address from my IPv6 proving router, but I'm not at the point (yet) of advertising addresses further into my networks.

My example is still showing to be stuck at not being able to contact the providing GW again via link local address while I saw it perfectly answering on the multicast before.
I would guess, gary201 has the same problem, as he is also having the gateway as link local address in the routing table - exactly as I have. So there is nothing wrong that way, but why can't it be pinged directly from the WAN interface?