Messages

1

General Discussion / Re: Monitoroing: Missing SNMP Agent support

« on: September 30, 2023, 08:36:33 am »

Nice @gcorre !
Can you confirm that bsnmpd runs fine on the current OPNsense , please ?

Me personally, searched forums and got the impression, that bsnmpd is not preferred in OPNsense (since some troubles), however it is default in FreeBSD.
I took this Zabbix template (https://www.zabbix.com/integrations/opnsense) and disabled all those BEGEMOT-MIB items, and basically lost PF monitoring. Instead of BEGEMOT-MIB items for PF , I've tried collectd and Telegraf features to PF monitoring . Both output Prometheus format and so pluggable into Zabbix.

Well, for now, I use Telegraf's PF metrics via Prometheus output to Zabbix. The metrics include PF state table only, that is less  than BEGEMOT-MIB.

Cheers

2

General Discussion / Re: Monitoroing: Missing SNMP Agent support

« on: September 11, 2023, 08:49:02 pm »

it looks like since 19.1 , bsnmp is superseded

https://forum.opnsense.org/index.php?topic=11398.msg51514#msg51514


has been discussed already

https://forum.opnsense.org/index.php?topic=19753.0


I'm dealing with Zabbix monitoring of OPNsense via snmp too these days.

T.

3

Hardware and Performance / Re: Any issues with i210 / i211 & i350 on OPNSense 20.x ?

« on: January 07, 2022, 04:03:13 pm »

Have you tried to install a switch between your OPNsense and ISP;s equipment ?
Namely some switch with controllable EEE , so you could additionally control EEE of the counterpart to your i211.

Seems to me that disabling EEE on your i211 NICs is necessary regardless how much the FreeBSD driver is being updated & bugfixed. So I would ruled out the driver from the list of possible causes.

4

Hardware and Performance / Re: Hardware Opinion? Supermicro Xeon D-1518 Mini 1U Rackmount

« on: January 07, 2022, 11:23:54 am »

Hi

I wonder how your Ethernet network card will deal with the SFP module. If you could share some results later ?

Because, IMHO, the card supports Ethernet standards only , IEEE 802.3* , like for example:

Code: [Select]

        Supported link modes:   1000baseT/Full
                                10000baseT/Full
                                1000baseX/Full
                                10000baseSR/Full
                                10000baseLR/Full
and your GPON ONT SFP thus might have matching physical&electrical SFP interface however not matching link/protocol towards to the Ethernet NIC.

Then there is a question what HW part will manage WDM/TDM for GPON ... interesting.

Happy hacking -- thumbs up !

5

Hardware and Performance / Re: 10GB LAN Performance

« on: January 01, 2022, 06:42:24 pm »

Bit of an update on this. After swapping the Chelsio cards for Intel X710-DA2s and getting more or less the same result I've figured out at least the iperf issue. iperf3 is single threaded, even if you run it with the -P option it still only hits one CPU core. If you want multithreaded operation you have to use iperf2.

I'd been checking CPU utilisation on the firewall dashboard while iperf3 was running and not seeing any significant numbers, but when I checked with top directly from the console the single CPU core being hit by iperf was running at close to 100%.

So I installed iperf2 and ran it multithreaded and boom! Near wire speed with 20+ concurrent threads!!

Running iperf continuously makes it easier to monitor top. For those who are interested, this runs the iperf2 client continuously with 50 threads:

Code: [Select]

iperf -c hostname -tinf -P 50
Then run top on the firewall like this:

Code: [Select]

top -PCH
But here's what I don't get. If I run iperf2 *through* the firewall to a server on the same 10Gbps network segment as the WAN, I get around 5Gbps with a single thread and 7-8Gbps multithreaded. But the same client running speedtest cli peaks at around 1Gbps. Looking at top on the firewall while speedtest is running doesn't show any significant CPU utilisation and anyway, if the firewall is only running a single thread for speedtest realistically it should be capable of way better than 1Gbps (half of that with WIN10!).

The obvious culprit is the ISP network but I'm still getting up to 8Gbps running speedtest directly from the firewall. I've also tested with mtr (no data loss and super low latency) and tracepath (no mtu issues all the way through to 1.1.1.1).

In summary, here is what I have found:

• There is not much difference I can tell in performance between the Intel X710-DA2 and the Chelsio T520-CRs
• The internal 10Gbps network and attached clients are healthy and can transfer data at close to wire speed
• The overhead from packet filtering on the firewall (passing iperf traffic) is 2-3Gbps which is bearable. Faster CPUs might reduce this, but with 10 cores engaged utilisation is only about 25-30%
• The ISP upstream network is healthy

So I'm not sure why there is such a big difference in firewall throughput between speedtest and iperf. I'm guessing speedtest uses tcp/443 and iperf defaults tcp/5001 (5201 for iperf3).

Unless the firewall is doing additional processing for tcp/443? I don't have any special rules set up for https and there is no IDS running at the moment. I'm going to have a close look at the proxy setup see if that leads anywhere.

Nice finding about iperf2 vs iperf3. Thanks.

I think "rungekutta" reported about similar forwarding performance as yours in iperf3 testing.
My ESX based testbed (only 10G capable I've got) runs also something over 5Gbps  with iperf3 but I haven't tweaked it much.

When you mentioned "10 cores engaged utilisation is only about 25-30%" , does that mean that each of the ten CPU cores is utilized at 25-30% ?

Few shots I would check:
if power management allows CPU to scale its frequency up ?

Code: [Select]

sysctl -a | grep cpu | grep freq

if this network tunables for multicore CPUs are on:

Code: [Select]

net.isr.maxthreads = "-1"
net.isr.bindthreads = "1"

if flow control per network interface is off ?

Code: [Select]

dev.ixl.#.fc = "0"

Further, I'd say that one may try to increase number of RX/TX queues and descriptors.
If not ixl(4) then iflib(4) based tunables might let you to do so. Check the sysctl values of 'nrxqs', 'ntxqs', 'nrxds' and 'ntxds' and see if you may override them to make them bigger/larger. Overrides require reboot , I guess.
Docs e.g. here:
https://www.freebsd.org/cgi/man.cgi?query=iflib&sektion=4&apropos=0&manpath=FreeBSD+12.2-RELEASE+and+Ports


This approach boosted forwarding performance on my ESX setup with vmx interfaces.


With regards the speedtest-cli to the Internet, I;d say to try to tcpdump/wireshark on both sides of firewall to see if the packets go nicely as expected or if there are resends, rubbish or something strange going on.

6

Hardware and Performance / Re: Poor routing speed on i5-7200U

« on: December 29, 2021, 12:09:34 pm »

So instead of single stream 770Mbps you are now getting 850Mbps . IMHO thats nice, not perfect but nice.

Networking tasks should now be spread over more CPU cores but still low. As the last time you have posted 33% sys CPU load at one core. IMHO healthy system. But this time it will spread over two plus cores.

I don't have benchmarks for single stream , I'm afraid. Always been using -P2 and resulting similarly to 950Mbps (either UP or DOWN load). Maybe I'll try later but can;t promise.

Maybe 850Mbps single-stream is just fine now but I'm not sure. There is always a performance decrease because of intentional non-offloading NICs and Netmap'ing within OPNsense, however the decrease is almost invisible with multiple streams/sessions in case of 1GbE. That is 950Mbps and more , in case of -P2, which is very nice considering MTUs , pps, and other limitations of source-destination NICs.

Reminds me, the OPNSense docs say , to keep TCP, UDP, LRO offloadings in the default = OFF.

And this might be useful too -- an example of "healthy" initialization of a powerful Intel 1GbE NIC (dmesg | grep igb0):

Code: [Select]

igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xc020-0xc03f mem 0xfe8a0000-0xfe8bffff,0xfe880000-0xfe89ffff,0xfe8c4000-0xfe8c7fff irq 40 at device 0.0 on pci3
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 2 RX queues 2 TX queues
igb0: Using MSI-X interrupts with 3 vectors
igb0: Ethernet address: 00:25:90:00:00:00
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
Important is: MSI on, more than one HW-queue, more than one netmap queues mapped in non-emulated mode.

Is the single stream TCP performance somehow crucial for you ?

TCP stack itself has got some tunables too -- both via Iperf3 tool and the kernel sysctl.

T.

7

Hardware and Performance / Re: Poor routing speed on i5-7200U

« on: December 26, 2021, 09:41:39 am »

I'd say, for the start , use these tunables

Code: [Select]

net.isr.maxthreads = "-1"
net.isr.bindthreads = "1"
to have multiple queues ;
And disable flow control on the Intel NICs, e.g.

Code: [Select]

dev.igb.0.fc = "0"
dev.igb.1.fc = "0"

The most 1GbE setups i've seen do the NAT  close to 1Gbps happily.

Is the CPU frequency scaling up/down ?

Code: [Select]

sysctl -a | grep cpu | grep freq


8

Hardware and Performance / Re: HP Proliant Throughput Issue

« on: December 21, 2021, 05:44:57 pm »

You mainly run speedtest-cli from LAN to a speedtest server in the Internet , don't you ?

What Is the type/technology of your connection  to ISP ?

T.

9

Hardware and Performance / Re: 10GB LAN Performance

« on: December 17, 2021, 08:59:58 am »

I'say : try connect another PC with 10GbE to your switch stack,
to clearly see if the bottleneck is proxmox/win OR the OPNsense server.

Throughput around 1Gbps on this 20G setup seems to me crazy low. Unless you have left some IPS or shaping settings on OPNsense -- what Is CPu load on OPNsense server when you test throughput ?

10

Hardware and Performance / Re: Poor Speedperformance with IntelXXV710 and Intel X550 10GBASE-T Cards

« on: November 23, 2021, 11:09:01 pm »

Performance of what precisely ?

IMHO your setup is non-standard and I don't see any diagnostics information in your post.

T.

11

Hardware and Performance / Re: Speed negotiation fail

« on: November 20, 2021, 05:33:40 pm »

Sounds like broken....

Have you booted some other operating system to rule out OPNsense ?

12

Hardware and Performance / Re: Poor 10Gb performance, need help troubleshooting

« on: November 20, 2021, 05:25:59 pm »

@rungekutta
Very nice info about your NIC setup. And throughput results.

Was the earlier  firmware in the NIC something like too old or it was perhaps customized ?
Because as people often reuse HW / NICs , it might not have a genuin firmware. For example customized by e.g. server vendor.

I'm only getting familiar with Suricata.... Is it like utilizing 100% CPU if enabled ?

13

Hardware and Performance / Re: vmxnet tuning advice

« on: November 14, 2021, 10:03:11 am »

I don't have such complex setup as you, but this post helped me:
https://forum.opnsense.org/index.php?topic=18754.msg90576#msg90576

Update:
Here for example the topic w.r.t. VMXNET tuning that confirms it works https://forum.opnsense.org/index.php?topic=25076.0 .

And my test VM shows 10Gbps for 8 vCPU (Xeon @2.4GHz) and 'iperf3 -P 5 -t60' and following tunables:

Code: [Select]

net.isr.maxthreads = -1
net.isr.bindthreads = 1
hw.pci.honor_msi_blacklist = 0
hw.ibrs_disable = 1
dev.vmx.0.iflib.override_ntxds="0,4096"
dev.vmx.0.iflib.override_nrxds="0,2048,0"
dev.vmx.1.iflib.override_ntxds="0,4096"
dev.vmx.1.iflib.override_nrxds="0,2048,0"

T.

14

Hardware and Performance / Re: Poor 10Gb performance, need help troubleshooting

« on: November 14, 2021, 09:52:58 am »

Thanks for the info. NB I’ve ordered a Chelsio T520-SO-CR on eBay. These are the (10Gb) cards that Netgate themselves sell for pfSense. And the FreeBSD crowd seems to love them. So can’t really get more “supported” than that.

Then there’s the question of CPU and chipset of course. The FreeBSD hardware list that you linked frankly leaves a lot to be desired. The latest “supported” AMD CPU on that list is from 2003. Let’s hope I shouldn’t read that literally…

Also, with regards to FreeBSD performance overall, I saw this: https://www.phoronix.com/scan.php?page=article&item=freebsd-13-beta1&num=1

… which also surprised me a bit. FreeBSD 13 is apparently now “closer to parity with Linux performance on the same hardware“ (note “closer”) and if you look at the results, 13 in turn is sometimes twice as fast, sometimes much more than that, compared to FreeBSD 12.

So in summary, netmap runs in emulated mode possibly due to lack of NIC support in drivers, already starting from possible question marks on FreeBSD 12 performance overall, possibly exacerbated further by HardenedBSD, and to round it off with question marks around how FreeBSD interacts with modern AMD CPUs…

I’ll report back when I have been able to try the T520.

I might have scared you with the Ryzen platform HW support too much.
It seems you have everything running stable even with FreeBSD12.x base, so it seems to me alright and the most important property afterall.

I'm generally in doubts about HW, namely because vendors implement vast amounts of "specifics" with which operating systems have to keep up.

AFAIK, FreeBSD 13 based OPNsense will be a major step forward w.r.t. both compatibility and performance of some HW setups.

15

Hardware and Performance / Re: WAN performance issue

« on: November 14, 2021, 09:36:15 am »

IMHO, the best is to iperf3 benchmark network throughput of your OPNsense box without your ISP;s connect -- only  with use of a LAN PC (iperf3 clients) and WAN PC (iperf3 server). And observe CPU usage within these tests. And optionally observe effects if tunables within these tests.

After you'll be happy with TCP performance, connect via ISP's box, because there might be need to adjust OPNsense w.r.t. to ISP.

I'd say the checklist of fully initialized network card is simple:
- HW driver queues for RX/TX
- netmap driver queues for RX/TX
- TSO, LRO all off based on OPNsense default settings

I don't see bxe driver or the Broadcom cards to be natively supported by netmap (4).
@rungekutta might soon report the difference in throughput between natively supported Chelsio NIC and NIC supported only in emulated netmapmode.