Hello folks and experts,
I have a question.. and need some advice.
I got an Intel Ethernet Controller E810-XXVDA2, with an SPF28 link and want to do LAGG with LACP. Later with CARP an HA cluster.
I use the standard driver from the kernel, with the standard configuration I get around 5-7 gbit/s between the two firewalls.
With some tuning like https://forum.opnsense.org/index.php?topic=34999.0 from this post I get up to 24.3 gbit/s, would expect for 2x 25gbit more like 45gbit/s (maybe prozessor to slow cause of 100%)
But as soon as I turn on the option net.inet.rss.enabled="1" I am not able to reach the other firewall with SSH or IPERF3.
On a production firewall I dont want to compile ports or driver, what would you advice me ?
- Get a Broadcom card
- Or install the port an compile
Later we want to use Zennarmor and I am not sure if for this driver netmap is availible ?!
Is RSS availible for an Intel E810-XXVDA2 or do I have to compile a custom kernel or port ??
How can I debug my problem?
Thanks in advance!!
Florian
Quote from: Netfloh on April 21, 2024, 07:09:50 PM
With some tuning like https://forum.opnsense.org/index.php?topic=34999.0 from this post I get up to 24.3 gbit/s, would expect for 2x 25gbit more like 45gbit/s (maybe prozessor to slow cause of 100%)
If you're looking for performance and your system is at 100%, you probably don't have to look further
Quote
Is RSS availible for an Intel E810-XXVDA2 or do I have to compile a custom kernel or port ??
How can I debug my problem?
In the article you linked is another link from lilsense to the complete RSS guide: sysctl -a | grep rss
The other most common pitfall when trying to do > 10Gb on a x64 platform is a capable PCIe slot. The E810 is a PCIe v4 X8 card, ideally it needs _all_ (v4) lanes connected and directly connected to the CPU. Not shared or behind a chipset link to get full performance.
Upgrade to the latest firmware, and don't forget your Jumbo's, that's your first ±15% CPU benefit...
Hello,
thank you for the quick reply.
So when RSS is enable, sysctl -a | grep rss looks like:
root@OPNsense:~ # sysctl -a | grep rss
net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3
net.inet.rss.enabled: 1
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 4
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 8
net.inet.rss.maxbits: 7
net.inet.rss.mask: 3
net.inet.rss.bits: 2
net.inet.rss.hashalgo: 2
hw.bxe.udp_rss: 0
hw.ix.enable_rss: 1
When I look for the RSS option on the device it says ...
root@OPNsense:~ # sysctl -a | grep dev.ice.0.rss
root@OPNsense:~ #
and when I try to start a connection between the two Firewalls it looks like:
root@OPNsense:~ # ssh 10.10.10.2
^C
root@OPNsense:~ # ssh 10.10.10.2
(root@10.10.10.2) Password:
root@OPNsense:~ # ssh 10.10.10.2
^C
root@OPNsense:~ # ssh 10.10.10.2
^C
root@OPNsense:~ # ssh 10.10.10.2
^C
root@OPNsense:~ # ssh 10.10.10.2
(root@10.10.10.2) Password:
when I disable RSS all works fine accept RSS is off lsusb
root@OPNsense:~ # netstat -Q
Configuration:
Setting Current Limit
Thread count 8 8
Default queue limit 8192 10240
Dispatch policy deferred n/a
Threads bound to CPUs enabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
igmp 2 8192 source default ---
rtsock 3 8192 source default ---
arp 4 8192 source default ---
ether 5 8192 cpu direct C--
ip6 6 1000 cpu hybrid C--
ip_direct 9 8192 cpu hybrid C--
ip6_direct 10 8192 cpu hybrid C--
Workstreams:
WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled
0 0 ip 0 2 0 531 0 6120 6651
0 0 igmp 0 0 0 0 0 0 0
0 0 rtsock 0 0 0 0 0 0 0
0 0 arp 0 0 0 0 0 0 0
0 0 ether 0 0 849 0 0 0 849
0 0 ip6 0 1 0 1 0 7 8
0 0 ip_direct 0 0 0 0 0 0 0
0 0 ip6_direct 0 0 0 0 0 0 0
1 1 ip 0 2 0 357 0 2266 2623
1 1 igmp 0 0 0 0 0 0 0
1 1 rtsock 0 0 0 0 0 0 0
1 1 arp 0 1 0 0 0 48 48
1 1 ether 0 0 511 0 0 0 511
1 1 ip6 0 1 0 1 0 13 14
1 1 ip_direct 0 0 0 0 0 0 0
1 1 ip6_direct 0 0 0 0 0 0 0
2 2 ip 0 2 0 1947 0 427 2374
2 2 igmp 0 0 0 0 0 0 0
2 2 rtsock 0 0 0 0 0 0 0
2 2 arp 0 1 0 0 0 39 39
2 2 ether 0 0 2000 0 0 0 2000
2 2 ip6 0 1 0 4 0 36 40
2 2 ip_direct 0 0 0 0 0 0 0
2 2 ip6_direct 0 0 0 0 0 0 0
3 3 ip 0 1 0 801 0 27 828
3 3 igmp 0 0 0 0 0 0 0
3 3 rtsock 0 0 0 0 0 0 0
3 3 arp 0 1 0 0 0 1 1
3 3 ether 0 0 945 0 0 0 945
3 3 ip6 0 1 0 0 0 6 6
3 3 ip_direct 0 0 0 0 0 0 0
3 3 ip6_direct 0 0 0 0 0 0 0
4 4 ip 0 0 0 0 0 0 0
4 4 igmp 0 0 0 0 0 0 0
4 4 rtsock 0 0 0 0 0 0 0
4 4 arp 0 0 0 0 0 0 0
4 4 ether 0 0 135 0 0 0 135
4 4 ip6 0 0 0 0 0 0 0
4 4 ip_direct 0 0 0 0 0 0 0
4 4 ip6_direct 0 0 0 0 0 0 0
5 5 ip 0 0 0 0 0 0 0
5 5 igmp 0 0 0 0 0 0 0
5 5 rtsock 0 0 0 0 0 0 0
5 5 arp 0 0 0 0 0 0 0
5 5 ether 0 0 0 0 0 0 0
5 5 ip6 0 0 0 0 0 0 0
5 5 ip_direct 0 0 0 0 0 0 0
5 5 ip6_direct 0 0 0 0 0 0 0
6 6 ip 0 0 0 0 0 0 0
6 6 igmp 0 0 0 0 0 0 0
6 6 rtsock 0 0 0 0 0 0 0
6 6 arp 0 0 0 0 0 0 0
6 6 ether 0 0 0 0 0 0 0
6 6 ip6 0 0 0 0 0 0 0
6 6 ip_direct 0 0 0 0 0 0 0
6 6 ip6_direct 0 0 0 0 0 0 0
7 7 ip 0 0 0 0 0 0 0
7 7 igmp 0 0 0 0 0 0 0
7 7 rtsock 0 2 0 0 0 31 31
7 7 arp 0 0 0 0 0 0 0
7 7 ether 0 0 0 0 0 0 0
7 7 ip6 0 0 0 0 0 0 0
7 7 ip_direct 0 0 0 0 0 0 0
7 7 ip6_direct 0 0 0 0 0 0 0
Firmware I updated to the newest version.
ice0: <Intel(R) Ethernet Network Adapter E810-XXV-2 - 1.37.11-k> mem 0x2807e000000-0x2807fffffff,0x28080010000-0x2808001ffff irq 118 at device 0.0 on pci17
ice0: Loading the iflib ice driver
ice0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.30.0, track id 0xc0000001.
..
..
ice1: <Intel(R) Ethernet Network Adapter E810-XXV-2 - 1.37.11-k> mem 0x2807c000000-0x2807dffffff,0x28080000000-0x2808000ffff irq 118 at device 0.1 on pci17
ice1: Loading the iflib ice driver
ice1: DDP package already present on device: ICE OS Default Package version 1.3.30.0, track id 0xc0000001.
ice1: fw 7.4.13 api 1.7 nvm 4.40 etid 8001c96c netlist 4.3.5000-1.14.0.99840ef4 oem 1.3534.0
I tried out the Port Intel-ice-kmod, but this seem not to get loaded ..
I am a bit stucked :-(
To finish this up ...
The Intel E810-XXVDA2 needs some developer love to work with OPNsense, I change to Broadcom P225P and this card works out of the box without compiling kernel or ports.
Thanks Netnut for your help !
Quote from: Netfloh on April 24, 2024, 10:36:00 PMTo finish this up ...
The Intel E810-XXVDA2 needs some developer love to work with OPNsense, I change to Broadcom P225P and this card works out of the box without compiling kernel or ports.
Thanks Netnut for your help !
Did you try 25.1 ?
ice_ddp driver has been upgraded to latest version