Messages

Gonzo, is that you?

Let's clarify this - those Netgate devs with FreeBSD commit bit seem to use every FreeBSD user as a guinea pig for their experiments lately. Such as the recent "stateful ICMP" SNAFU.

BTW, "blb" means a dumba*s in Czech. Nomen omen, it seems.

Anything productive to add to the discussion? Those "Netgate devs with commit privileges" were FreeBSD developers before working for Netgate. Judging by your post history and attitude, are you sure you are not talking about yourself? You seem to have a problem grasping a whole lot of things, and engage strangers online aggressively. Copium much?

I wasn't sure if I was reading gonzopancho or you Franco  ::)

No, really. Users can do free QA, are you also expecting them to be developers (kernel developers at that! -which you are not, also-)  for the foundations of your commercial product? That is really all that usually happens with OPNsense: the free users are guinea pigs for the commercial offering. In this aspect, pfSense is doing the same.

No need to be conceited or abrasive about it.

I'm happy to revisit this and test if you have updated minor revision images for the installer (USB/ISO).

Have you checked how much your fork differs from upstream's sys/kernel? I don't think expecting users to cherry pick commits (or go through your cherry picking history) is a realistic approach.

How is it not an OPNsense issue if other FreeBSD based systems (on the same major version) function properly?

@boom42 How are you using the Chelsio NICs? Did you configure it or the panic/doublefault happens regardless of whether they are in use? (test with no ports connected, link down, and no configuration using them -ex no interface assigned-).

If you have a serial port or SOL/IPMI console text log that would also be quite helpful to see if we have the same stack trace (the calls to functions up to the point where the first fault occurs before the panic). It's very likely the same issue.

Code Select Expand

opnsense-update -zkr kernel-dbg-24.7.1

This presumes a bootable environment, or are you suggesting running this from 24.1.10? I can setup a tunnel and use the BMC to get a working console, but like I said, there is a double fault at some point and a loop that makes kdb unusable. The debug symbols might help if present, but kdb won't be workable. It will be a few hours until I can do this, though.

Franco, OPNsense is great, but you do have a habit of both releasing unstable major versions (as far as more complex environments are involved, I don't expect a basic kvm or esxi setup falling apart in some odd "homelab") and failing to commit resources to providing LTS-like (long term stable, a la Debian) updates. This creates a burden on your users to produce actual QA as they become guinea pigs until all the issues are ironed out. More often than not, that time buffer creates problems of its own. This affected pfSense in the past too, although they have even less of an excuse than you would.

If you provided a buffer of time with updates for the previous major versions as some sort of LTS channel, this would be literally a non-issue. Making the stable prior major revision EOL before 24.7 has all the kinks ironed out is how you get a flood of posts from folks encountering problems.

This is not a personal attack, and merits a response that does not trail along ad hominems or attempts to shrug it off. It's also not grandstanding. You can do better with your devops approach as a business, let alone as a FOSS project.

I will see if I have time to get a serial console log from the person I'm helping out. Feel free to link or send a established diagnostics procedure she can follow, meanwhile. I'll do what I can.

You are dodging the "issue", if you take a honest look at your entire post history in this forum, you might find a pattern. I would not call it a case study in social ineptitude, but it comes close to it.

You don't need to explain yourself or bring up your personal circumstances in the thread. That's the cliff notes for you.

So, moving on and forward, if you have a Chelsio T320 and actually are curious to debug the problem, I can tell you how to configure it and replicate the BIOS settings.

I'm out of time for free QA today, but I did find some posts from other users that might hint at some kernel issues that need to be ironed out and they weren't. The Chelsio driver is one of the most stable NIC drivers in FreeBSD, written by a core developer. An out of bounds read (or a lock contention issue perhaps) in the driver indicates this is very likely an OPNsense mistake (without reviewing all the cherry picked patches Deciso has taken from upstream).

It needs proper debugging.

Code Select Expand

cxgbc0@pci0:2:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x0031 subvendor=0x1425 subdevice=0x0001
    vendor     = 'Chelsio Communications Inc'
    device     = 'T320 10GbE Dual Port Adapter'
    class      = network
    subclass   = ethernet

For the developers and anyone who actually has interest in diagnosing the issue:

A quick look at the panic log (there seems to be a double fault so kdb won't help) shows some stack frames that are related to the cxgbc0 task queuing (so, Chelsio driver).

I also tested on a production system with the same hardware (redundancy spare kept in storage), also with a T320, and the trap also kicks in. Again, double fault, then a loop, then a hard CPU reset.

Would you mind leaving your personal/subjective assumptions and trolling attempts out of the thread? A root cause is less likely to elude *you* if you are actually trying to diagnose it, instead of derailing a thread out of personal reasons (like picking arguments with strangers on the internet...).

Also, please enlighten us with that AS2Di you have "run for years". Sounds like BS. The AS2Di is not "years old" quite, although it is far from new (has not been completely superseded, just like the X10SDV line). What "10 major revisions"? There were breaking changes that make that impossible as a smooth upgrade path without reinstalls.

Just in case I can grab a kernel panic log later and this might be related:
https://forum.opnsense.org/index.php?topic=42194.msg208066#msg208066

I'm glad this is not on a production system yet. The host is a SYS-E300-9A-4C (A2SDi-4C-HLN4F motherboard).

Additional hardware: Chelsio T320, rest is standard for the model.

I cannot/could not get a capture of the kernel panic, but it happens immediately after importing the previous 24.1.10 good configuration. After "initializing.... done".

Tested from Live DVD, and also on a boot environment upgraded from 24.1.10 online. Using a LAGG of the SFP+ ports. Everything else is pretty much standard for any decent enterprise setup. WAN groups, IPSec, some client OVPN, and quite a few VLANs.

Considering the fact that Deciso's commercial offerings actually use the A2SDi platform, this is not great news. Chelsio T320s are also the most common/popular SFP+ NIC for FreeBSD hosts.

Sometimes I wish Deciso did not use us as guinea pigs for QA that should have been done in-house. No harm done on this one, but anyone else with a similar setup beware. Make sure you create and activate a boot env for the upgrade so you can revert if this hits you.

I hope the above comment is not taken personally (hi Franco). I'm just surprised this is the third time an upgrade causes issues. Prior to boot environments being properly supported it was a bigger deal.

TL;DR 24.1.10 to 24.7 = kernel panic on a A2SDi-4C-HLN4F system with QAT and a Chelsio NIC.

Hi,

I'm using a VPN provider that reuses subnets across different hosts, and I suspect they will likely do this for many if not most of the servers:

server1 172.21.26.3
server2 172.21.26.16
server3 172.21.28.50

I get the dreaded error on route() add, so only one of them can work at a time. I would like to know a workaround, if possible. In the past this seemed to work and pfsense apparently handles this internally (haven't looked into it, but a system using pfs with the same client connections actually works gracefully). I recently updated to the latest stable and this is when it broke (.x upgrade, same major revision).

Hi,

This has been mentioned in other posts for a while:
https://forum.opnsense.org/index.php?topic=22585.0

The symptoms are failed DNS resolution that persists for any given host using Unbound as resolver, with Unbound itself passing the requests to DNScrypt locally. I have tested this with Tor and Shadowsocks proxies and TCP only servers enabled. The jostle period sometimes might help, but it won't fix the problem. DNSSEC hardening disabled.

It manifests for clients as a persistent SERVFAIL response, whereas targeting DNScrypt directly will actually yield proper responses and the name is resolved successfully.

I have sometimes worked around the problem by forcing a cron job to restart Unbound periodically, but this is admittedly a crappy way to solve the symptoms and not the actual "disease".

Might want to look at this:
https://forum.opnsense.org/index.php?topic=25410.msg122060#msg122060

I'm not yet sure what the culprit is. Could you use some of my commands with UDP/-u mode and -Z -N whenever possible?
Also -P n where n is half your cores*2 count.  (just to avoid competing for resources elsewhere, leave some cores "free"). You can use all of them, though, but I suggest trying -P 2 first.

TL;DR run iperf3 with -u mode, it will show you packet loss. It's also relevant.

I made a new thread about this very same issue but with Proxmox guests in the mix:
https://forum.opnsense.org/index.php?topic=25410.msg122060#msg122060

I don't want to blame OPNsense 100% before I rule out OVS problems, but OVS has not had issues for me in the past :(

Hi,

This might be an issue related to what other folks have published in the past related to TP issues in opnsense 20.x due to FreeBSD kernel changes.

The setup:

• Brocade ICX with 10G SFP+ to Opnsense (using Chelsio NIC) and a L2 10G SFP+ switch that provides a trunk to the Proxmox host
• All flow control disabled in the SFP+ ports across all switches.
• OVS used in Proxmox for a dedicated vmbr using the trunk from the switch (interface has no IP assigned for the host, it just passes VLAN tagged traffic).
• lagg0 containing the cxgb 10G NIC and (unused) ix ports (I did it for testing and will have to revert those).

It's set to normal retention for states, just in case.

The firewall CPU:

Code Select Expand

CPU: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz (2200.07-MHz K8-class CPU)


This is what I am observing:

Inter-VLAN across Linux hosts, sender/client has 1g link:

Code Select Expand


$ iperf3 -c XXX -u -b 0 -N -Z -P4
Connecting to host XXX, port 5201
[  5] local ZZZ port 44857 connected to XXX port 5201
[  7] local ZZZ port 49631 connected to XXX port 5201
[  9] local ZZZ port 41755 connected to XXX port 5201
[ 11] local ZZZ port 36856 connected to XXX port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  25.2 MBytes   211 Mbits/sec  18090 
[  7]   0.00-1.00   sec  25.2 MBytes   211 Mbits/sec  18090 
[  9]   0.00-1.00   sec  25.2 MBytes   211 Mbits/sec  18090 
[ 11]   0.00-1.00   sec  25.2 MBytes   211 Mbits/sec  18090 
[SUM]   0.00-1.00   sec   101 MBytes   845 Mbits/sec  72360 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  25.2 MBytes   211 Mbits/sec  18080 
[  7]   1.00-2.00   sec  25.2 MBytes   211 Mbits/sec  18080 
[  9]   1.00-2.00   sec  25.2 MBytes   211 Mbits/sec  18080 
[ 11]   1.00-2.00   sec  25.2 MBytes   211 Mbits/sec  18080 
[SUM]   1.00-2.00   sec   101 MBytes   845 Mbits/sec  72320 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[  7]   2.00-3.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[  9]   2.00-3.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[ 11]   2.00-3.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[SUM]   2.00-3.00   sec   101 MBytes   844 Mbits/sec  72280 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[  7]   3.00-4.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[  9]   3.00-4.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[ 11]   3.00-4.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[SUM]   3.00-4.00   sec   100 MBytes   843 Mbits/sec  72150 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  7]   4.00-5.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  9]   4.00-5.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[ 11]   4.00-5.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[SUM]   4.00-5.00   sec   100 MBytes   843 Mbits/sec  72130 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  25.1 MBytes   210 Mbits/sec  18020 
[  7]   5.00-6.00   sec  25.1 MBytes   210 Mbits/sec  18020 
[  9]   5.00-6.00   sec  25.1 MBytes   210 Mbits/sec  18020 
[ 11]   5.00-6.00   sec  25.1 MBytes   210 Mbits/sec  18020 
[SUM]   5.00-6.00   sec   100 MBytes   842 Mbits/sec  72080 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  7]   6.00-7.00   sec  25.1 MBytes   210 Mbits/sec  18020 
[  9]   6.00-7.00   sec  25.1 MBytes   210 Mbits/sec  18020 
[ 11]   6.00-7.00   sec  25.1 MBytes   210 Mbits/sec  18020 
[SUM]   6.00-7.00   sec   100 MBytes   842 Mbits/sec  72090 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  7]   7.00-8.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[  9]   7.00-8.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[ 11]   7.00-8.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[SUM]   7.00-8.00   sec   100 MBytes   843 Mbits/sec  72140 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  7]   8.00-9.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  9]   8.00-9.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[ 11]   8.00-9.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[SUM]   8.00-9.00   sec   100 MBytes   842 Mbits/sec  72130 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec  18030 
[  7]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec  18030 
[  9]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec  18030 
[ 11]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec  18030 
[SUM]   9.00-10.00  sec   100 MBytes   842 Mbits/sec  72120 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   251 MBytes   211 Mbits/sec  0.000 ms  0/180450 (0%)  sender
[  5]   0.00-10.00  sec   251 MBytes   211 Mbits/sec  0.093 ms  0/180413 (0%)  receiver
[  7]   0.00-10.00  sec   251 MBytes   211 Mbits/sec  0.000 ms  0/180450 (0%)  sender
[  7]   0.00-10.00  sec   251 MBytes   211 Mbits/sec  0.091 ms  0/180413 (0%)  receiver
[  9]   0.00-10.00  sec   251 MBytes   211 Mbits/sec  0.000 ms  0/180450 (0%)  sender
[  9]   0.00-10.00  sec   251 MBytes   211 Mbits/sec  0.090 ms  0/180413 (0%)  receiver
[ 11]   0.00-10.00  sec   251 MBytes   211 Mbits/sec  0.000 ms  0/180450 (0%)  sender
[ 11]   0.00-10.00  sec   251 MBytes   211 Mbits/sec  0.096 ms  0/180412 (0%)  receiver
[SUM]   0.00-10.00  sec  1005 MBytes   843 Mbits/sec  0.000 ms  0/721800 (0%)  sender
[SUM]   0.00-10.00  sec  1005 MBytes   843 Mbits/sec  0.093 ms  0/721651 (0%)  receiver


Reversed:

Code Select Expand


[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  23.1 MBytes   194 Mbits/sec  0.075 ms  4027/20608 (20%) 
[  7]   0.00-1.00   sec  23.1 MBytes   194 Mbits/sec  0.068 ms  4042/20627 (20%) 
[  9]   0.00-1.00   sec  23.1 MBytes   194 Mbits/sec  0.082 ms  4038/20611 (20%) 
[ 11]   0.00-1.00   sec  23.1 MBytes   194 Mbits/sec  0.069 ms  4036/20618 (20%) 
[SUM]   0.00-1.00   sec  92.3 MBytes   774 Mbits/sec  0.074 ms  16143/82464 (20%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  28.4 MBytes   238 Mbits/sec  0.084 ms  58/20458 (0.28%) 
[  7]   1.00-2.00   sec  28.4 MBytes   238 Mbits/sec  0.064 ms  57/20439 (0.28%) 
[  9]   1.00-2.00   sec  28.5 MBytes   239 Mbits/sec  0.083 ms  58/20500 (0.28%) 
[ 11]   1.00-2.00   sec  28.5 MBytes   239 Mbits/sec  0.074 ms  59/20497 (0.29%) 
[SUM]   1.00-2.00   sec   114 MBytes   954 Mbits/sec  0.076 ms  232/81894 (0.28%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  28.4 MBytes   238 Mbits/sec  0.097 ms  52/20446 (0.25%) 
[  7]   2.00-3.00   sec  28.4 MBytes   239 Mbits/sec  0.078 ms  66/20490 (0.32%) 
[  9]   2.00-3.00   sec  28.5 MBytes   239 Mbits/sec  0.076 ms  45/20490 (0.22%) 
[ 11]   2.00-3.00   sec  28.4 MBytes   239 Mbits/sec  0.059 ms  72/20494 (0.35%) 
[SUM]   2.00-3.00   sec   114 MBytes   954 Mbits/sec  0.077 ms  235/81920 (0.29%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  28.4 MBytes   239 Mbits/sec  0.069 ms  48/20475 (0.23%) 
[  7]   3.00-4.00   sec  28.4 MBytes   238 Mbits/sec  0.071 ms  60/20472 (0.29%) 
[  9]   3.00-4.00   sec  28.4 MBytes   238 Mbits/sec  0.074 ms  63/20473 (0.31%) 
[ 11]   3.00-4.00   sec  28.4 MBytes   238 Mbits/sec  0.083 ms  61/20476 (0.3%) 
[SUM]   3.00-4.00   sec   114 MBytes   954 Mbits/sec  0.074 ms  232/81896 (0.28%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  28.3 MBytes   238 Mbits/sec  0.078 ms  70/20404 (0.34%) 
[  7]   4.00-5.00   sec  28.5 MBytes   239 Mbits/sec  0.059 ms  68/20528 (0.33%) 
[  9]   4.00-5.00   sec  28.5 MBytes   239 Mbits/sec  0.076 ms  51/20492 (0.25%) 
[ 11]   4.00-5.00   sec  28.5 MBytes   239 Mbits/sec  0.049 ms  53/20495 (0.26%) 
[SUM]   4.00-5.00   sec   114 MBytes   954 Mbits/sec  0.066 ms  242/81919 (0.3%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  28.3 MBytes   237 Mbits/sec  0.077 ms  183/20475 (0.89%) 
[  7]   5.00-6.00   sec  28.2 MBytes   237 Mbits/sec  0.072 ms  190/20471 (0.93%) 
[  9]   5.00-6.00   sec  28.2 MBytes   237 Mbits/sec  0.068 ms  190/20474 (0.93%) 
[ 11]   5.00-6.00   sec  28.2 MBytes   237 Mbits/sec  0.080 ms  199/20475 (0.97%) 
[SUM]   5.00-6.00   sec   113 MBytes   948 Mbits/sec  0.074 ms  762/81895 (0.93%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  26.7 MBytes   224 Mbits/sec  0.084 ms  1345/20505 (6.6%) 
[  7]   6.00-7.00   sec  26.7 MBytes   224 Mbits/sec  0.060 ms  1350/20501 (6.6%) 
[  9]   6.00-7.00   sec  26.5 MBytes   223 Mbits/sec  0.095 ms  1349/20406 (6.6%) 
[ 11]   6.00-7.00   sec  26.7 MBytes   224 Mbits/sec  0.052 ms  1350/20505 (6.6%) 
[SUM]   6.00-7.00   sec   107 MBytes   894 Mbits/sec  0.073 ms  5394/81917 (6.6%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  28.4 MBytes   238 Mbits/sec  0.073 ms  72/20468 (0.35%) 
[  7]   7.00-8.00   sec  28.5 MBytes   239 Mbits/sec  0.081 ms  59/20508 (0.29%) 
[  9]   7.00-8.00   sec  28.5 MBytes   239 Mbits/sec  0.078 ms  66/20506 (0.32%) 
[ 11]   7.00-8.00   sec  28.3 MBytes   238 Mbits/sec  0.088 ms  69/20423 (0.34%) 
[SUM]   7.00-8.00   sec   114 MBytes   954 Mbits/sec  0.080 ms  266/81905 (0.32%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  28.4 MBytes   239 Mbits/sec  0.066 ms  54/20479 (0.26%) 
[  7]   8.00-9.00   sec  28.4 MBytes   238 Mbits/sec  0.066 ms  84/20475 (0.41%) 
[  9]   8.00-9.00   sec  28.4 MBytes   238 Mbits/sec  0.090 ms  81/20470 (0.4%) 
[ 11]   8.00-9.00   sec  28.4 MBytes   238 Mbits/sec  0.068 ms  67/20479 (0.33%) 
[SUM]   8.00-9.00   sec   114 MBytes   953 Mbits/sec  0.073 ms  286/81903 (0.35%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  28.4 MBytes   238 Mbits/sec  0.080 ms  75/20477 (0.37%) 
[  7]   9.00-10.00  sec  28.4 MBytes   238 Mbits/sec  0.066 ms  60/20478 (0.29%) 
[  9]   9.00-10.00  sec  28.4 MBytes   238 Mbits/sec  0.064 ms  50/20468 (0.24%) 
[ 11]   9.00-10.00  sec  28.4 MBytes   238 Mbits/sec  0.063 ms  59/20478 (0.29%) 
[SUM]   9.00-10.00  sec   114 MBytes   954 Mbits/sec  0.069 ms  244/81901 (0.3%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec   285 MBytes   239 Mbits/sec  0.000 ms  0/204900 (0%)  sender
[  5]   0.00-10.00  sec   277 MBytes   232 Mbits/sec  0.080 ms  5984/204795 (2.9%)  receiver
[  7]   0.00-10.01  sec   286 MBytes   239 Mbits/sec  0.000 ms  0/205100 (0%)  sender
[  7]   0.00-10.00  sec   277 MBytes   232 Mbits/sec  0.066 ms  6036/204989 (2.9%)  receiver
[  9]   0.00-10.01  sec   285 MBytes   239 Mbits/sec  0.000 ms  0/205000 (0%)  sender
[  9]   0.00-10.00  sec   277 MBytes   232 Mbits/sec  0.064 ms  5991/204890 (2.9%)  receiver
[ 11]   0.00-10.01  sec   285 MBytes   239 Mbits/sec  0.000 ms  0/205040 (0%)  sender
[ 11]   0.00-10.00  sec   277 MBytes   232 Mbits/sec  0.063 ms  6025/204940 (2.9%)  receiver
[SUM]   0.00-10.01  sec  1.12 GBytes   957 Mbits/sec  0.000 ms  0/820040 (0%)  sender
[SUM]   0.00-10.00  sec  1.08 GBytes   929 Mbits/sec  0.069 ms  24036/819614 (2.9%)  receiver


Note the significant early packet loss and still a ~3% figure averaged.


From Linux host (over 1g link) to one of the LXC containers:

Code Select Expand


$ iperf3 -c YYY -u -b 0 -N -Z -P 4
Connecting to host YYY, port 5201
[  5] local ZZZ port 52927 connected to YYY port 5201
[  7] local ZZZ port 36209 connected to YYY port 5201
[  9] local ZZZ port 58231 connected to YYY port 5201
[ 11] local ZZZ port 45380 connected to YYY port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  24.9 MBytes   208 Mbits/sec  17850 
[  7]   0.00-1.00   sec  24.9 MBytes   208 Mbits/sec  17850 
[  9]   0.00-1.00   sec  24.8 MBytes   208 Mbits/sec  17840 
[ 11]   0.00-1.00   sec  24.8 MBytes   208 Mbits/sec  17840 
[SUM]   0.00-1.00   sec  99.4 MBytes   834 Mbits/sec  71380 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  7]   1.00-2.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  9]   1.00-2.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[ 11]   1.00-2.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[SUM]   1.00-2.00   sec   100 MBytes   843 Mbits/sec  72140 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  24.9 MBytes   208 Mbits/sec  17850 
[  7]   2.00-3.00   sec  24.9 MBytes   208 Mbits/sec  17850 
[  9]   2.00-3.00   sec  24.9 MBytes   208 Mbits/sec  17850 
[ 11]   2.00-3.00   sec  24.8 MBytes   208 Mbits/sec  17840 
[SUM]   2.00-3.00   sec  99.4 MBytes   834 Mbits/sec  71390 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[  7]   3.00-4.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[  9]   3.00-4.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[ 11]   3.00-4.00   sec  25.2 MBytes   211 Mbits/sec  18080 
[SUM]   3.00-4.00   sec   101 MBytes   844 Mbits/sec  72290 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[  7]   4.00-5.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[  9]   4.00-5.00   sec  25.1 MBytes   211 Mbits/sec  18060 
[ 11]   4.00-5.00   sec  25.1 MBytes   211 Mbits/sec  18060 
[SUM]   4.00-5.00   sec   101 MBytes   844 Mbits/sec  72260 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[  7]   5.00-6.00   sec  25.2 MBytes   211 Mbits/sec  18070 
[  9]   5.00-6.00   sec  25.2 MBytes   211 Mbits/sec  18080 
[ 11]   5.00-6.00   sec  25.2 MBytes   211 Mbits/sec  18080 
[SUM]   5.00-6.00   sec   101 MBytes   844 Mbits/sec  72300 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  7]   6.00-7.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[  9]   6.00-7.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[ 11]   6.00-7.00   sec  25.1 MBytes   211 Mbits/sec  18030 
[SUM]   6.00-7.00   sec   100 MBytes   842 Mbits/sec  72120 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  25.1 MBytes   211 Mbits/sec  18050 
[  7]   7.00-8.00   sec  25.1 MBytes   211 Mbits/sec  18050 
[  9]   7.00-8.00   sec  25.1 MBytes   211 Mbits/sec  18050 
[ 11]   7.00-8.00   sec  25.1 MBytes   211 Mbits/sec  18040 
[SUM]   7.00-8.00   sec   101 MBytes   843 Mbits/sec  72190 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  25.1 MBytes   211 Mbits/sec  18060 
[  7]   8.00-9.00   sec  25.1 MBytes   211 Mbits/sec  18060 
[  9]   8.00-9.00   sec  25.1 MBytes   211 Mbits/sec  18060 
[ 11]   8.00-9.00   sec  25.1 MBytes   211 Mbits/sec  18060 
[SUM]   8.00-9.00   sec   101 MBytes   844 Mbits/sec  72240 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec  18050 
[  7]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec  18050 
[  9]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec  18050 
[ 11]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec  18060 
[SUM]   9.00-10.00  sec   101 MBytes   844 Mbits/sec  72210 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   251 MBytes   210 Mbits/sec  0.000 ms  0/180130 (0%)  sender
[  5]   0.00-10.00  sec   250 MBytes   210 Mbits/sec  0.087 ms  310/180094 (0.17%)  receiver
[  7]   0.00-10.00  sec   251 MBytes   210 Mbits/sec  0.000 ms  0/180130 (0%)  sender
[  7]   0.00-10.00  sec   250 MBytes   210 Mbits/sec  0.087 ms  311/180094 (0.17%)  receiver
[  9]   0.00-10.00  sec   251 MBytes   210 Mbits/sec  0.000 ms  0/180130 (0%)  sender
[  9]   0.00-10.00  sec   250 MBytes   210 Mbits/sec  0.082 ms  311/180093 (0.17%)  receiver
[ 11]   0.00-10.00  sec   251 MBytes   210 Mbits/sec  0.000 ms  0/180130 (0%)  sender
[ 11]   0.00-10.00  sec   250 MBytes   210 Mbits/sec  0.082 ms  308/180093 (0.17%)  receiver
[SUM]   0.00-10.00  sec  1003 MBytes   842 Mbits/sec  0.000 ms  0/720520 (0%)  sender
[SUM]   0.00-10.00  sec  1001 MBytes   840 Mbits/sec  0.084 ms  1240/720374 (0.17%)  receiver


Reversed:

Code Select Expand

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  25.0 MBytes   209 Mbits/sec  0.095 ms  23791/41765 (57%) 
[  7]   0.00-1.00   sec  25.1 MBytes   210 Mbits/sec  0.152 ms  23663/41714 (57%) 
[  9]   0.00-1.00   sec  25.4 MBytes   212 Mbits/sec  0.099 ms  23487/41700 (56%) 
[ 11]   0.00-1.00   sec  25.2 MBytes   210 Mbits/sec  0.165 ms  23525/41631 (57%) 
[SUM]   0.00-1.00   sec   101 MBytes   841 Mbits/sec  0.128 ms  94466/166810 (57%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  28.3 MBytes   238 Mbits/sec  0.020 ms  22166/42526 (52%) 
[  7]   1.00-2.00   sec  28.4 MBytes   238 Mbits/sec  0.019 ms  22124/42486 (52%) 
[  9]   1.00-2.00   sec  28.4 MBytes   238 Mbits/sec  0.018 ms  22109/42489 (52%) 
[ 11]   1.00-2.00   sec  28.3 MBytes   238 Mbits/sec  0.022 ms  22112/42467 (52%) 
[SUM]   1.00-2.00   sec   113 MBytes   951 Mbits/sec  0.020 ms  88511/169968 (52%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  28.6 MBytes   240 Mbits/sec  0.028 ms  16409/36941 (44%) 
[  7]   2.00-3.00   sec  28.6 MBytes   240 Mbits/sec  0.022 ms  16325/36863 (44%) 
[  9]   2.00-3.00   sec  28.3 MBytes   238 Mbits/sec  0.027 ms  16468/36823 (45%) 
[ 11]   2.00-3.00   sec  28.1 MBytes   236 Mbits/sec  0.028 ms  16634/36803 (45%) 
[SUM]   2.00-3.00   sec   114 MBytes   953 Mbits/sec  0.026 ms  65836/147430 (45%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  28.0 MBytes   235 Mbits/sec  0.034 ms  20337/40437 (50%) 
[  7]   3.00-4.00   sec  28.3 MBytes   238 Mbits/sec  0.030 ms  20085/40426 (50%) 
[  9]   3.00-4.00   sec  27.9 MBytes   234 Mbits/sec  0.034 ms  20336/40397 (50%) 
[ 11]   3.00-4.00   sec  28.4 MBytes   238 Mbits/sec  0.033 ms  20003/40388 (50%) 
[SUM]   3.00-4.00   sec   113 MBytes   945 Mbits/sec  0.033 ms  80761/161648 (50%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  28.4 MBytes   239 Mbits/sec  0.017 ms  20901/41327 (51%) 
[  7]   4.00-5.00   sec  28.0 MBytes   235 Mbits/sec  0.015 ms  21113/41223 (51%) 
[  9]   4.00-5.00   sec  28.4 MBytes   238 Mbits/sec  0.015 ms  20822/41196 (51%) 
[ 11]   4.00-5.00   sec  28.1 MBytes   236 Mbits/sec  0.021 ms  20965/41175 (51%) 
[SUM]   4.00-5.00   sec   113 MBytes   947 Mbits/sec  0.017 ms  83801/164921 (51%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  28.3 MBytes   237 Mbits/sec  0.027 ms  19254/39580 (49%) 
[  7]   5.00-6.00   sec  28.5 MBytes   239 Mbits/sec  0.033 ms  19046/39513 (48%) 
[  9]   5.00-6.00   sec  28.6 MBytes   240 Mbits/sec  0.034 ms  18919/39460 (48%) 
[ 11]   5.00-6.00   sec  28.3 MBytes   237 Mbits/sec  0.028 ms  19106/39431 (48%) 
[SUM]   5.00-6.00   sec   114 MBytes   954 Mbits/sec  0.031 ms  76325/157984 (48%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  27.7 MBytes   233 Mbits/sec  0.029 ms  22041/41968 (53%) 
[  7]   6.00-7.00   sec  27.7 MBytes   232 Mbits/sec  0.034 ms  21953/41851 (52%) 
[  9]   6.00-7.00   sec  27.6 MBytes   232 Mbits/sec  0.030 ms  21967/41808 (53%) 
[ 11]   6.00-7.00   sec  27.5 MBytes   231 Mbits/sec  0.031 ms  22023/41799 (53%) 
[SUM]   6.00-7.00   sec   111 MBytes   928 Mbits/sec  0.031 ms  87984/167426 (53%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  25.0 MBytes   210 Mbits/sec  0.026 ms  22505/40485 (56%) 
[  7]   7.00-8.00   sec  25.0 MBytes   210 Mbits/sec  0.028 ms  22425/40372 (56%) 
[  9]   7.00-8.00   sec  25.1 MBytes   210 Mbits/sec  0.027 ms  22338/40335 (55%) 
[ 11]   7.00-8.00   sec  25.3 MBytes   212 Mbits/sec  0.029 ms  22136/40314 (55%) 
[SUM]   7.00-8.00   sec   100 MBytes   842 Mbits/sec  0.027 ms  89404/161506 (55%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  26.5 MBytes   222 Mbits/sec  0.028 ms  22171/41170 (54%) 
[  7]   8.00-9.00   sec  26.4 MBytes   221 Mbits/sec  0.025 ms  22177/41140 (54%) 
[  9]   8.00-9.00   sec  26.3 MBytes   221 Mbits/sec  0.028 ms  22179/41099 (54%) 
[ 11]   8.00-9.00   sec  26.5 MBytes   222 Mbits/sec  0.024 ms  22044/41060 (54%) 
[SUM]   8.00-9.00   sec   106 MBytes   886 Mbits/sec  0.026 ms  88571/164469 (54%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  28.3 MBytes   238 Mbits/sec  0.058 ms  18665/39021 (48%) 
[  7]   9.00-10.00  sec  28.4 MBytes   239 Mbits/sec  0.064 ms  18524/38951 (48%) 
[  9]   9.00-10.00  sec  28.3 MBytes   237 Mbits/sec  0.064 ms  18629/38942 (48%) 
[ 11]   9.00-10.00  sec  28.5 MBytes   239 Mbits/sec  0.063 ms  18433/38911 (47%) 
[SUM]   9.00-10.00  sec   114 MBytes   953 Mbits/sec  0.062 ms  74251/155825 (48%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.02  sec   566 MBytes   473 Mbits/sec  0.000 ms  0/406170 (0%)  sender
[  5]   0.00-10.00  sec   274 MBytes   230 Mbits/sec  0.058 ms  208240/405220 (51%)  receiver
[  7]   0.00-10.02  sec   565 MBytes   473 Mbits/sec  0.000 ms  0/405480 (0%)  sender
[  7]   0.00-10.00  sec   274 MBytes   230 Mbits/sec  0.064 ms  207435/404539 (51%)  receiver
[  9]   0.00-10.02  sec   564 MBytes   472 Mbits/sec  0.000 ms  0/405190 (0%)  sender
[  9]   0.00-10.00  sec   274 MBytes   230 Mbits/sec  0.064 ms  207254/404249 (51%)  receiver
[ 11]   0.00-10.02  sec   564 MBytes   472 Mbits/sec  0.000 ms  0/404920 (0%)  sender
[ 11]   0.00-10.00  sec   274 MBytes   230 Mbits/sec  0.063 ms  206981/403979 (51%)  receiver
[SUM]   0.00-10.02  sec  2.21 GBytes  1.89 Gbits/sec  0.000 ms  0/1621760 (0%)  sender
[SUM]   0.00-10.00  sec  1.07 GBytes   920 Mbits/sec  0.062 ms  829910/1617987 (51%)  receiver

Simple ping test to firewall from a qemu guest:

Code Select Expand


PING HHH (HHH) 56(84) bytes of data.
64 bytes from HHH: icmp_seq=1 ttl=64 time=0.350 ms
64 bytes from HHH: icmp_seq=2 ttl=64 time=0.393 ms
64 bytes from HHH: icmp_seq=3 ttl=64 time=0.414 ms

64 bytes from HHH: icmp_seq=15 ttl=64 time=0.421 ms
64 bytes from HHH: icmp_seq=16 ttl=64 time=0.320 ms
64 bytes from HHH: icmp_seq=17 ttl=64 time=0.307 ms
^C
--- HHH ping statistics ---
17 packets transmitted, 6 received, 64.7059% packet loss, time 16390ms
rtt min/avg/max/mdev = 0.307/0.367/0.421/0.044 ms

I'm still trying to figure out what is the culprit, but ruled out the switches already as local link tests work just fine. Now I'm doubting opnsense and OVS in Proxmox. The Proxmox host is a modern Xeon D system, and I tested with mitigations-off just to rule out any weird IO impact from the CPU mitigations.

Since other users mentioned problems here:
https://forum.opnsense.org/index.php?topic=18754.135
https://forum.opnsense.org/index.php?topic=19426.0

And it mimics very closely what I am observing, could anyone or even developers throw some advice at the problem, or any known issues we should be aware of?

Why some directions exhibit near-zero or zero packet loss?

Anyone else having such problems is also welcome to comment! Please remember to use -u mode for UDP with iperf... TCP will not show you any packet loss.

Thank you!