Messages

Today I DOWNgraded it to a 6.XX firmware that I still had - 'poof' - all issues seem to be gone. I will continue to

Fingers crossed ;-)

Der Client sagt immer aktiviert, aber selbst wenn ich beim Host www.google.de reinschreibe. Ich habe den Eindruck, sobald der DNS aufgelöst werden kann sagt er Verbindung aufgebaut. Im Log stehn dann natürlich Fehler.

Ne... er sagt auch nur das er einen Tunnelendpunkt für die von dir vorgegebene Config laufen hat. ;-) Aber ja, falls die DNS-Auflösung nicht klappt, würde er den Tunnelendpunkt auch nicht aufmachen.

Für das Verständnis vielleicht entscheidend: Wireguard baut nicht direkt einen ständigen Tunnel auf, überwacht nicht dessen "Durchgängig" oder sonstigen Zauber; sondern er startet erst mal nur einen Prozess, welcher bei "Bedarf" Pakete in den Tunnel ein- und ausleiten würde.
Der "Bedarf" würde erst dadurch entstehen das Pakete zum Routen am jeweiligen wg-Interface ankommen würden.

Was ist das denn für ein Quatsch ?

im Allgemeinen: "IT"


P.S.: Erst handshakes zeigen an das zwei Tunnelendpunkte erfolgreich ein Paket durch den Tunnel schieben konnten.

WG behauptet immer, dass die Verbindung steht, selbst wenn es unmöglich ist, und richtet auch das Routing entsprechend ein, auch wenn es ins Nirvana geht...

Wie würde es denn bei allowed ips: = 0/0 das routing einrichten? Mit einer zusätzlichen`host route` (/32) zum eigentlichen default gateway? Bei Android und auch bei Linux?

Ich würde eher darauf tippen wollen (und mich wahrscheinich gerade mal wieder richtig irren wollen), dass das routing der auschliesslich für in den Tunnel gedachten Packete nicht passt.

Versuchen würde ich es zunächst mit allowed ips: 192.168.0.0/16 und passender Route

Code Select Expand

ip route add 192.168.0.0/16 dev wg0 auf das wg-interface nur genau das richtige Zielnetz in den Tunnel zu packen.

Bg
Reza

Is there anything else that could be done? I am very open to suggestions.

Next of my shots into the dark light: We are reloading OPNsense, or just pf, a lot at the moment. But e.g. our laptop and other network devices, which has a lot of established TCP connections, stays "online" during this time.

So when OPNsense (pf) loses it's knowledge of all states (because of a reboot or config reload,...), the laptop still has the knowledge of it's already established connections.

When the laptop sends a TCP packet to another station with already established TCP state, it won't send a new SYN packet - it will just send acks (or maybe push acks) with sequence numbers.

OPNsense, seeing this traffic, does not know about this already established state and will log a debug message.
The more we test atm, the more we'll get this debug message.

And the packets will be blocked at "last rule", because of the state violation. "Works as designed" ;-)

Br
Reza

If you didn't back up the configuration, prepare a Linux boot disk and wait for help. You'll need to boot the rescue disk, find the config file, and copy it out to a drive. Then you can do the steps in the first paragraph. That file is in there somewhere, I'm not sure where though.

The configs are located on the partition under top-level directory /conf/, the file is named config.xml:

Code Select Expand


[user]@OPNsense:~ $ file /conf/config.xml
/conf/config.xml: XML 1.0 document, ASCII text, with very long lines (1101)


The "OPNsense Importer feature" is maybe helpfull to import the config file to a new installation: https://docs.opnsense.org/manual/install.html#opnsense-importer

Br
Reza

Hi,

sorry, I missed your post for days ...

The MTU I have set is (unfortunately) not the problem, as I am only testing between two local VLANs (MTU==1500) that are routed/filtered via opnsense. I could try jumbo frames, maybe it can get even worse ;-)

Yet you show results from a iperf3 test run against an internet IP?

Please dont beat me, but 198.18.0.0/15 are not public route-able IPs. (pssst: "bogus IPs" ;-] )

So there are alo VLANs and LAGGs in the mix? Maybe netmap and suricata as well? ???

I think of LAGGs and VLANs as very basic FW/Router interface types. Beside of pppoe I have not more in the mix. So... no netmap or suricata here.

Br
Reza

Hi!

Thanks, good to know.

I was a little bit disappointed at that moment.

Maybe it's a different (but somehow related) issue that did not surface in the same way until now.

I'm going to take a step back and look at the whole thing with some distance to the maybe blinding "FreeBSD 14 / IPv6 issue".

Do you also see the performance degredation/FW hits?

Unfortunately, yes.
But I did something just days before upgrading to OPNsense 27.4: I had previously done the bonding (lacp) of the interfaces and VLAN tagging on the host (promox) and put the resulting bond0.[vlan id] interfaces as separated virtio-networkcards to the OPNsense-VM. I just changed that, because I was unhappy with creating an interface for each new  (or changed) VLAN on the host and having to guess in which order it will be assigned to the OPNsense interfaces (this was the behavior under virtualbox).

So, I'm going back to 24.7 (no_SA kernel), but assemble again the bond- and vlan-interfaces on the host - assuming that linux (probably?) will have the better driver and working hardware offloading *fingers crossed* ;-)

Br
Reza

P.S.: Sorry, I'm a little bit sick at the moment and spending less time in front of my homelab at the moment ...

Hi,

short answer: Yes, it is the IP of the resolver. 127.0.0.1 is the 'local host' which is your OPNsense.

Br
Reza

(Interfaces -> Diagnostics -> DNS Lookup? ...)

Hallo,

kurz vorweg: Ich hab noch nie Multi-WAN mit OPNsens gemacht, aber eine Frage stellt sich mir sofort:

Mit wie vielen gleichzeitigen Destinations hast Du getestet? Ich meine es müssten zumindest zwei sein, da ansonsten weiterhin nur ein (physikalischer) WAN-Link genutzt werden kann.
Sprich: Du hast weiterhin 1x TCom Bitrate dazu 1x DG Bitrate, aber nicht 1x (TCom + DG) Bitrate.

Br
Reza

P.S.: Falls Du eine "provider-indepent" (eigene ASN, BGB-Routing...) WAN-IP hast, dann passt meine Frage natürlich nicht.

With the normal 24.7.3 kernel, I can confirm the "pf: ICMP error message too short (ip6)" messages - which go away with the no-sa kernel.

I can also confirm the "pf: loose state match" notices with both kernels.

I just went back to OPNsense 24.1 (imported config from 24.7)  and, with debug logging turned on,... taddahhh... I also see same 'pf: loose state match' notices.

...and wireguard is UDP only.

You could try with "-u" to see if UDP is faster. If it is, I would guess that your MTU is misconfigured. Can you try this to find your real maximum MTU?

Probably, setting "-M 1400" would show if this is the problem.

Many thanks for the answer.
The MTU I have set is (unfortunately) not the problem, as I am only testing between two local VLANs (MTU==1500) that are routed/filtered via opnsense. I could try jumbo frames, maybe it can get even worse ;-)

The reference to pppoe only referred to the fact that this is my slowest connection (100/40); however, the problem is not as obvious there as with my gigabit/VLAN/LAGG connections.

I would like to point out that this topic is about allowed traffic inexplicably hitting the "last rule", a problem with pf and TCP states is suspected, and that this is interfering with TCP connections.

Br
Reza

[TCP connections can be drastically slower than UDP connections]

I don't run the IGMP Proxy service (actually not even using IGMP at all in my network).
So, I would assume that this is not related.

Unfortunately, I could not confirm my suspicion of IGMP proxy. Would have been too nice (easy). ;)

As currently the other thread is more active I posted my assumption based on the current testing there just a few moments ago.
But I am very open to switch to this one, if we feel like the "ICMP error message too short (ip6)" logs can be targeted with the full-revert kernel (and are therefore manageble), while the state losses don't seem to be affected by the -no_sa kernel.

It seems to me that TCP connections can be drastically slower than UDP connections on the same route.

Reproducibly, some routes, e.g. over my slow pppoe interface, are only about 25% slower. I suspect with faster routes (1Gibt/s) it is 80% to almost 100% packet loss.
On the connections that don't show 100% loss anyway, however, there are noticeable dropouts lasting a few seconds every few minutes, during which 100% packet loss occurs.

That's why everything to do with TCP hardware offloading came into question for me. In the meantime, however, I have tried every "hw offlaodung turn off" combination without finding any significant differences.

In the firewall log you can find individual messages, such as the one in the topic, which indicate that traffic that is actually permitted is being dropped by last rule.

Together with the observed "debug" message: 'pf: loose state match...', however, it seems clear what is happening:
TCP packets are discarded because pf no longer recognises the TCP states. Each discarded packet slows down the TCP connection - each additional discarded packet slows it down even more.

I think that the underlying problem also explains why TCP connections have different speeds depending on the direction. And I don't just mean a small difference:

Code Select Expand


❯ iperf3 -c 198.18.50.136 -t 3 --bidir
Connecting to host 198.18.50.136, port 5201
[  5] local 198.18.178.160 port 42184 connected to 198.18.50.136 port 5201
[  7] local 198.18.178.160 port 42188 connected to 198.18.50.136 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec   201 KBytes  [color=red]1.64 Mbits/sec    2 [/color]  1.41 KBytes       
[  7][RX-C]   0.00-1.00   sec   111 MBytes   931 Mbits/sec                 
[  5][TX-C]   1.00-2.00   sec  0.00 Bytes  [color=red]0.00 bits/sec    1  [/color] 1.41 KBytes       
[  7][RX-C]   1.00-2.00   sec   111 MBytes   932 Mbits/sec                 
[  5][TX-C]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  7][RX-C]   2.00-3.00   sec   111 MBytes   932 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -


Br
Reza

Hi,

To briefly conclude this topic: Using the "no_SA" kernel prevents unwanted effects when processing ICMPv6 packets. The (debug) message 'pf: ICMP error message too short (ip6)' first observed by cloudz has disappeared; I think we all agree that other ICMPv6/ND6 issues are also fixed with the patched kernel.

Code Select Expand


opnsense-update -zkr 24.7.3-no_sa
fixes the problem.


If cloudz wants, he can mark this topic as [Solved] again ;-)

However, the observed symptom of TCP (IPv4) states being "lost" and thus blocking traffic still persists. I think that the (debug) message 'pf: loose state match...' actually points to the underlying problem.

TheDJ has already created a separate topic about this: Topic: Regular LAN Traffic hits Default deny / state violation rule since 24.7 (https://forum.opnsense.org/index.php?topic=42657.0).

Br Reza

[ICMP]

You can easily check if the SA is the culprit by trying the kernel with the SA completely removed via

Code Select Expand

opnsense-update -zkr 24.7.3-no_sa

and reboot, see this.

After switching kernel to version 14.1-RELEASE-p3 no_sa-n267804-164bfe67604) the debug message 'pf: ICMP error message too short (ip6)' disappeared.

It also looks to me that the no_SA kernel solves this (one of several?) [ironic]downstream[/ironic] problems.

Other, but perhaps unproblematic debug messages such as 'pf: loose state match...' and 'pf: dropping packet with ip options' are still often logged if debug logging is enabled.


Br
Reza