Messages

I confirm that the new BIOS version solved the issue for me. Many thanks!

Do you run Suricata or ZenArmor?

No, IDS is not enabled. I got the problem with the default configuration out of the box, with just two interfaces configured and basic firewall rules.

@AdSchellevis I would like to open a support case, but unfortunately our reseller didn't get us support when we bought the appliances... So we are now in the process of acquiring support, and in the meantime I opened this topic. It seems that I'm not the only one facing this problem, so if a solution is find, it would be great to share it.

our default test setup (which reaches reaches the advertised speeds), is setup like:

[client] --> [firewall] --> [server]

I confirm that we are not using the firewall's iperf server but two machines connecting to the firewall as described in your message. There was a switch initially but I just did the test again without any switch and the result is about the same (around 5 Gbit/s with parallel threads).

Code Select Expand

root@PROXMOX4:~# iperf3 -c 10.1.1.12 -p 5201 -P 8 -Z -t 5 -M 1500
Connecting to host 10.1.1.12, port 5201

[...]

[SUM]   0.00-5.00   sec  2.79 GBytes  4.79 Gbits/sec  873             sender
[SUM]   0.00-5.02   sec  2.77 GBytes  4.75 Gbits/sec                  receiver

iperf Done.


I checked "Disable reply-to on WAN rules" but it changed nothing. The tunable `ice_ddp_load` is set to `YES`.

How about some more "netstat"s: "-m", "-i", "-sp ip" - anything look odd?

I'm definitely not an expert, but I don't have packets drops or errors here. Same for the pciconf command:

Code Select Expand

root@OPNsense:~ # pciconf -lvc ice0
ice0@pci0:6:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1593 subvendor=0x8086 subdevice=0x0005
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller E810-C for SFP'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 512 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x8000]
    cap 10[a0] = PCI-Express 2 endpoint max data 512(512) FLR RO
                 max read 512
                 link x16(x16) speed 8.0(16.0)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 000e[148] = ARI 1
    ecap 0003[150] = Serial 1 000100ffff000000
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0100, VF RID Stride 0x0001
                     VF Device ID 0x1889
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
    ecap 0019[1d0] = PCIe Sec 1 lane errors 0
    ecap 0025[200] = Data Link Feature 1
    ecap 0026[210] = Physical Layer 16.0 GT/s 1
    ecap 0027[250] = Lane Margining at Receiver 1


The speed is so ridiculously close to exactly 1 Gbit/s that I would believe the negotiated link speed is at 1 GBit/s for one or both interfaces.

That was also my first thought, but they negotiate 25G, and if I use parallel streams it can get up to 5 Gb/s. But a single flow seems capped at around 1 Gb/s. I also changed settings about error corrections and flow control, but it has no effect.

Code Select Expand

ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
ice0: link state changed to UP
ice1: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
ice1: link state changed to UP


So long as you're not terminating iperf sessions on the firewall, about the only tunables/sysctls that'll help you are RSS: net.isr.bindthreads, net.isr.maxthreads, net.inet.rss.bits, net.inet.rss.enabled. Have a look at "netstat -Q" (IIRC) to check. I believe OPNsense sets most other necessary sysctls reasonably. But I believe RSS is mainly good for throughput.  How does CPU utilization look while running the tests?

I just reverted all the tunables to the default config that came with the device and I enabled this:

• net.isr.bindthreads = 1
• net.isr.maxthreads = -1
• net.inet.rss.enabled = 1
• net.inet.rss.bits = 4

After reboot, results are the same. The command netstat -Q shows a lot of workstreams now. If I use iperf with the option -P 16, I get up to 4.3 Gbit/s and a single stream is around 1.5Gbit/s. During transfer with 16 streams, some CPU cores are indeed used at 100%, global usage is: CPU:  0.0% user,  0.0% nice, 23.6% system, 30.0% interrupt, 46.4% idle.

I noticed an error regarding the ice interfaces while running sysctl command, do you think it's related?

Code Select Expand

root@OPNsense:~ # sysctl -a | grep rss
ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice1: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice2: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice3: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 8:8 9:9 10:10 11:11 12:12 13:13 14:14 15:15
net.inet.rss.enabled: 1
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 16
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 32
net.inet.rss.maxbits: 7
net.inet.rss.mask: 15
net.inet.rss.bits: 4
net.inet.rss.hashalgo: 2
hw.bxe.udp_rss: 0
hw.ix.enable_rss: 1
dev.ax.3.rss_enabled: 1
dev.ax.2.rss_enabled: 1
dev.ax.1.rss_enabled: 1
dev.ax.0.rss_enabled: 1

root@OPNsense:~ # dmesg | grep vectors
igc0: Using MSI-X interrupts with 5 vectors
igc1: Using MSI-X interrupts with 5 vectors
igc2: Using MSI-X interrupts with 5 vectors
igc3: Using MSI-X interrupts with 5 vectors
ice0: Using MSI-X interrupts with 33 vectors
ice1: Using MSI-X interrupts with 33 vectors
ice2: Using MSI-X interrupts with 33 vectors
ice3: Using MSI-X interrupts with 33 vectors
ax0: Using MSI-X interrupts with 16 vectors
ax1: Using MSI-X interrupts with 16 vectors
ax2: Using MSI-X interrupts with 6 vectors
ax3: Using MSI-X interrupts with 6 vectors

As far as I understand, RSS only helps with multiple flows. Is 1Gbit/s the best we can expect from this device for a single flow?

Hi,

I'm using DEC4280 appliances from Deciso with OPNsense preinstalled. The goal is to use them in HA with two link aggregations: two SFP28 interfaces for the internal VLANs and two SFP+ interfaces for the uplink.

Unfortunately during the tests, it was difficult to obtain multi-gigabit speeds. So I'm back using one appliance with a very basic configuration to perform tests:

• igc0 (assigned to LAN): just for management purpose
• ice0 (assigned opt1): plugged to a switch in access mode on a VLAN
• ice1 (assigned opt2): plugged to a switch in access mode on another VLAN

On interfaces opt1 and opt2, only one rule allow all traffic to pass. There is only one machine in each VLAN. With iperf3, I got this result:

Code Select Expand

# iperf3 -c 10.2.2.12 -p 5201
Connecting to host 10.2.2.12, port 5201
[  5] local 10.1.1.11 port 38024 connected to 10.2.2.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   129 MBytes  1.08 Gbits/sec   52   1.17 MBytes
[  5]   1.00-2.00   sec   125 MBytes  1.05 Gbits/sec    0   1.28 MBytes
[  5]   2.00-3.00   sec   126 MBytes  1.06 Gbits/sec    0   1.36 MBytes
[  5]   3.00-4.00   sec   126 MBytes  1.06 Gbits/sec    0   1.43 MBytes
[  5]   4.00-5.00   sec   126 MBytes  1.06 Gbits/sec    1   1.08 MBytes
[  5]   5.00-6.00   sec   125 MBytes  1.05 Gbits/sec    0   1.16 MBytes
[  5]   6.00-7.00   sec   126 MBytes  1.06 Gbits/sec    0   1.24 MBytes
[  5]   7.00-8.00   sec   126 MBytes  1.06 Gbits/sec    0   1.31 MBytes
[  5]   8.00-9.00   sec   126 MBytes  1.06 Gbits/sec    0   1.38 MBytes
[  5]   9.00-10.00  sec   125 MBytes  1.05 Gbits/sec    3   1.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.23 GBytes  1.06 Gbits/sec   56             sender
[  5]   0.00-10.00  sec  1.23 GBytes  1.06 Gbits/sec                  receiver

iperf Done.

But if I plug the two machines directly on the same VLAN, bypassing the firewall, I got this result:

Code Select Expand

# iperf3 -c 10.1.1.12 -p 5201
Connecting to host 10.1.1.12, port 5201
[  5] local 10.1.1.11 port 40454 connected to 10.1.1.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.63 GBytes  22.6 Gbits/sec   61   2.53 MBytes
[  5]   1.00-2.00   sec  2.66 GBytes  22.8 Gbits/sec    0   2.91 MBytes
[  5]   2.00-3.00   sec  2.66 GBytes  22.9 Gbits/sec    0   2.96 MBytes
[  5]   3.00-4.00   sec  2.68 GBytes  23.0 Gbits/sec    0   3.14 MBytes
[  5]   4.00-5.00   sec  2.52 GBytes  21.6 Gbits/sec   47   2.43 MBytes
[  5]   5.00-6.00   sec  2.66 GBytes  22.8 Gbits/sec    0   2.48 MBytes
[  5]   6.00-7.00   sec  2.66 GBytes  22.8 Gbits/sec    0   2.55 MBytes
[  5]   7.00-8.00   sec  2.69 GBytes  23.1 Gbits/sec    0   2.55 MBytes
[  5]   8.00-9.00   sec  2.67 GBytes  22.9 Gbits/sec    0   2.66 MBytes
[  5]   9.00-10.00  sec  2.66 GBytes  22.9 Gbits/sec    0   2.66 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  26.5 GBytes  22.7 Gbits/sec  108             sender
[  5]   0.00-10.00  sec  26.5 GBytes  22.7 Gbits/sec                  receiver

Increasing the number of parallel streams increased a bit the bandwidth but we are far from 10Gb/s (and we are using 25Gb/s interfaces). The only time I reached near 10Gb/s was with firewall disabled and a lot of parallel streams.

I tried multiple tunable settings found on this forum, tried to enable/disable HW offloading and I also tried to use the SFP+ interfaces, but same results.

The interfaces looks healthy and the counters for errors and collisions are at 0.

I'm running out of ideas to troubleshoot this. Anyone got similar issues?

Thanks.

EDIT: The new BIOS version (https://docs.opnsense.org/hardware/bios.html#dec4200-series) fixed the issue.