Messages

Thank you. That plus deleting all libfetch_crl. files in /tmp did the trick.
All working now.

I am also running into this issue:

Error logs when running the update:

Code Select Expand


***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10 at Fri Oct 18 10:20:45 PDT 2024
Fetching subscription information, please wait... Could not load CRL file /tmp/libfetch_crl.24101810
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/subscription: Authentication error
Fetching changelog information, please wait... Could not load CRL file /tmp/libfetch_crl.24101810
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/sets/changelog.txz: Authentication error
Updating OPNsense repository catalogue...
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/meta.txz: Authentication error
repository OPNsense has no meta file, using default settings
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.pkg: Authentication error
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
Could not load CRL file /tmp/libfetch_crl.24101810
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***


I can however access the ULR https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.txz with my browser (using my subscription ID instead of the variable).

Using CURL on the router itself to download the file also works and I can see when using update via the SSH menu that the correct URL is being used. Still gets an authentication error:

Code Select Expand


Fetching change log information, please wait... Could not load CRL file /tmp/libfetch_crl.24101810
fetch: https://opnsense-update.deciso.com/<removed>/FreeBSD:14:amd64/24.10/sets/changelog.txz: Authentication error
done


What you are seeing is pretty much expected when running iperf with a single stream/connection.

Can you run some tests using parallel streams? The -P option is what you need to add.
E.g. -P 10 would be 10 connections.

I still do see a performance issue with 24.7 that is not in 21.4 when it comes to 10gig performance but let's see what results you get with a more realistic test setup.

I just went to 24.4 to 24.7 again for a test of the performance in a real production environment as I was no longer seeing performance issues on 24.7 with RSS enabled in my lab.

Unfortunately took less than 24h for the performance to drop from 9.4gig (expected and normal using 24.4) down to less than 8G down and 6G upload.

I really cannot put my finger on what is causing the issue.
Take the exact same config file on 24.4 and no performance issues. A reboot of 24.7 restores the performance for a few hours before it degrades again.

I understand that there are likely very few people in this forum that run 24.7 in a 10gig WAN environment and on a DEC850 or any other A20 based (axgbe NIC) based system but would like to hear if somebody solved this or sees no performance issues.

Let's figure this out before a potential bug makes it into the business edition. This is likely a FreeBSD related issue but I hope that a few of you are willing to work with me on figuring this out.

EDIT: A few more observations and relevant details:
1. My 10Gig WAN has a fix IPv4 that is used with NAT and a fixed /56 IPv6 subnet not used with NAT. Essentially a direct BIDI connection that goes straight into my ISP's back-end router.
2. The issues happens with both IPv4 and IPv6 traffic ruling out issues with NAT.

Thanks,
Sven

I debugged this some more. There seems to be a change in how FreeBSD handles network load scheduling across cores. Can you try to set "net.inet.rss.enabled" to "1" in tunables and test again? I saw a noticeable improvement (back to line speed) after setting it.

I do run into the issue on Edge.

[Setup:]

I see a noticeable performance impact on 10G routing after upgrading my DEC850 to 24.7. Going back to 24.1 solves the issue.

Setup:
I am on a 10G internet connection and use a DEC850. AX0 is the WAN port and AX1 is used as LAN. AX1 has several v-lans. No other ports are in use.

The recommended tuning parameters to get 10G performance are set

Code Select Expand

dev.ax.0.iflib.override_nrxds = 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048
dev.ax.0.iflib.override_ntxds = 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048
dev.ax.0.rss_enabled = 1
Same settings for ax.1 as well.

Issue:
On prior release (including 24.1) is was able to get "line" performance while having relatively low CPU load on the 8 cores the DEC850 has. Since upgrading to 24.7 the network performance is inconsistent and no longer line performance. CPU core spike at 100% during transfer tests.

Test to repro
Using a device connected to the LAN, run a moderate network load to a WAN target that can provide consistent performance close to 10G line speed. For my example I used "iperf3 -c speedtest.sea11.us.leaseweb.net -p 5201-5210 -P10" which normally results in about 9.4 to 9.3 Gbits/sec and a well distributed load across CPU cores. Screenshot attached (24.1.jpg)

Running the same test on 24.7 results in only 8.8 Gbits/sec down and ~5 Gbits/sec up while maxing out most CPU cores. Screenshot attached (24.7.jpg).

I have two DEC850 units so that I was able to do a clean install on both for the test and apply the exact same configuration. 

Is anybody else seeing this with 10G or higher speed WAN connections?

Running into the same issue on a DEC850 which should have enough CPU resources.
There must be something else going on that is not caused by low performance systems.

I applied it earlier today to two of my servers and had no issues.
All working as expected.

Did the update via UFI using the built in console which is fast and easy.
The BMC update is a separate update that I applied via the web interface.

Take note of any BIOS settings as they will all be reset to default by the update.
The BMC update gives you the option to retain all BMC related settings and that worked out fine.

I don't have IPv6 issues (tested and all working) but do not see anything in the logs.
Running os-wireguard. Is that expected?

[Relevant Tunables:]

Here you go:

Relevant Tunables:

Code Select Expand

dev.ax.0.iflib.override_nrxds = 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048
dev.ax.0.iflib.override_ntxds = 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048
dev.ax.0.rss_enabled = 1
dev.ax.1.iflib.override_nrxds = 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048
dev.ax.1.iflib.override_ntxds = 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048
dev.ax.1.rss_enabled = 1
hw.ibrs_disable = 1
net.isr.bindthreads = 1
net.isr.maxthreads = -1
vm.pmap.pti = 0

I tested different SFP+ modules and it made no difference. Currently running
LAN:10Gtek SFP+ DAC Twinax Cable
WAN: Flexoptix BIDI module from my ISP

Are you getting 10G performance if you are just checking from test PC to test PC with no router in the middle?
Also, do you have flow control disabled on you switches?

No need to mess with PHP files.

Just run the following via an SSH connection (or serial):

opnsense-patch 23318015b 4ec97df4c

I am running on a DEC850 and see line performance on 10G.
https://www.speedtest.net/result/c/59052567-48a8-470c-8dd0-9ead1e3f4034

This is with RSS and all hardware offloading enabled.

Interfaces do show for me.

Same issue with most clients showing offline.
Debugging this a bit more, I see that all online client have static IPs and no client with a dynamic IP shows online.