DEC4280 - low speed on multi-gig interfaces

[l1lz]

Hi,

I'm using DEC4280 appliances from Deciso with OPNsense preinstalled. The goal is to use them in HA with two link aggregations: two SFP28 interfaces for the internal VLANs and two SFP+ interfaces for the uplink.

Unfortunately during the tests, it was difficult to obtain multi-gigabit speeds. So I'm back using one appliance with a very basic configuration to perform tests:

• igc0 (assigned to LAN): just for management purpose
• ice0 (assigned opt1): plugged to a switch in access mode on a VLAN
• ice1 (assigned opt2): plugged to a switch in access mode on another VLAN

On interfaces opt1 and opt2, only one rule allow all traffic to pass. There is only one machine in each VLAN. With iperf3, I got this result:

Code Select Expand

# iperf3 -c 10.2.2.12 -p 5201
Connecting to host 10.2.2.12, port 5201
[  5] local 10.1.1.11 port 38024 connected to 10.2.2.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   129 MBytes  1.08 Gbits/sec   52   1.17 MBytes
[  5]   1.00-2.00   sec   125 MBytes  1.05 Gbits/sec    0   1.28 MBytes
[  5]   2.00-3.00   sec   126 MBytes  1.06 Gbits/sec    0   1.36 MBytes
[  5]   3.00-4.00   sec   126 MBytes  1.06 Gbits/sec    0   1.43 MBytes
[  5]   4.00-5.00   sec   126 MBytes  1.06 Gbits/sec    1   1.08 MBytes
[  5]   5.00-6.00   sec   125 MBytes  1.05 Gbits/sec    0   1.16 MBytes
[  5]   6.00-7.00   sec   126 MBytes  1.06 Gbits/sec    0   1.24 MBytes
[  5]   7.00-8.00   sec   126 MBytes  1.06 Gbits/sec    0   1.31 MBytes
[  5]   8.00-9.00   sec   126 MBytes  1.06 Gbits/sec    0   1.38 MBytes
[  5]   9.00-10.00  sec   125 MBytes  1.05 Gbits/sec    3   1.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.23 GBytes  1.06 Gbits/sec   56             sender
[  5]   0.00-10.00  sec  1.23 GBytes  1.06 Gbits/sec                  receiver

iperf Done.

But if I plug the two machines directly on the same VLAN, bypassing the firewall, I got this result:

Code Select Expand

# iperf3 -c 10.1.1.12 -p 5201
Connecting to host 10.1.1.12, port 5201
[  5] local 10.1.1.11 port 40454 connected to 10.1.1.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.63 GBytes  22.6 Gbits/sec   61   2.53 MBytes
[  5]   1.00-2.00   sec  2.66 GBytes  22.8 Gbits/sec    0   2.91 MBytes
[  5]   2.00-3.00   sec  2.66 GBytes  22.9 Gbits/sec    0   2.96 MBytes
[  5]   3.00-4.00   sec  2.68 GBytes  23.0 Gbits/sec    0   3.14 MBytes
[  5]   4.00-5.00   sec  2.52 GBytes  21.6 Gbits/sec   47   2.43 MBytes
[  5]   5.00-6.00   sec  2.66 GBytes  22.8 Gbits/sec    0   2.48 MBytes
[  5]   6.00-7.00   sec  2.66 GBytes  22.8 Gbits/sec    0   2.55 MBytes
[  5]   7.00-8.00   sec  2.69 GBytes  23.1 Gbits/sec    0   2.55 MBytes
[  5]   8.00-9.00   sec  2.67 GBytes  22.9 Gbits/sec    0   2.66 MBytes
[  5]   9.00-10.00  sec  2.66 GBytes  22.9 Gbits/sec    0   2.66 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  26.5 GBytes  22.7 Gbits/sec  108             sender
[  5]   0.00-10.00  sec  26.5 GBytes  22.7 Gbits/sec                  receiver

Increasing the number of parallel streams increased a bit the bandwidth but we are far from 10Gb/s (and we are using 25Gb/s interfaces). The only time I reached near 10Gb/s was with firewall disabled and a lot of parallel streams.

I tried multiple tunable settings found on this forum, tried to enable/disable HW offloading and I also tried to use the SFP+ interfaces, but same results.

The interfaces looks healthy and the counters for errors and collisions are at 0.

I'm running out of ideas to troubleshoot this. Anyone got similar issues?

Thanks.

[meyergru]

The speed is so ridiculously close to exactly 1 Gbit/s that I would believe the negotiated link speed is at 1 GBit/s for one or both interfaces. Did you inspect the settings? With these types of interfaces, often times it is neccessary to manually set link speeds.

[pfry]

[...]
I tried multiple tunable settings found on this forum [...]

So long as you're not terminating iperf sessions on the firewall, about the only tunables/sysctls that'll help you are RSS: net.isr.bindthreads, net.isr.maxthreads, net.inet.rss.bits, net.inet.rss.enabled. Have a look at "netstat -Q" (IIRC) to check. I believe OPNsense sets most other necessary sysctls reasonably. But I believe RSS is mainly good for throughput.  How does CPU utilization look while running the tests?

If you weren't using a Deciso device, I'd recommend looking at the ice devices, e.g. "dmesg | grep ice" and "pciconf -lvV ice0" for any oddities (mine really wanted to set up as x4 v3; you want x4 v4 or x8 v3 at a minimum).

That speed does look suspicious, like a per-flow/session shaper.

[l1lz]

The speed is so ridiculously close to exactly 1 Gbit/s that I would believe the negotiated link speed is at 1 GBit/s for one or both interfaces.

That was also my first thought, but they negotiate 25G, and if I use parallel streams it can get up to 5 Gb/s. But a single flow seems capped at around 1 Gb/s. I also changed settings about error corrections and flow control, but it has no effect.

Code Select Expand

ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
ice0: link state changed to UP
ice1: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
ice1: link state changed to UP


So long as you're not terminating iperf sessions on the firewall, about the only tunables/sysctls that'll help you are RSS: net.isr.bindthreads, net.isr.maxthreads, net.inet.rss.bits, net.inet.rss.enabled. Have a look at "netstat -Q" (IIRC) to check. I believe OPNsense sets most other necessary sysctls reasonably. But I believe RSS is mainly good for throughput.  How does CPU utilization look while running the tests?

I just reverted all the tunables to the default config that came with the device and I enabled this:

• net.isr.bindthreads = 1
• net.isr.maxthreads = -1
• net.inet.rss.enabled = 1
• net.inet.rss.bits = 4

After reboot, results are the same. The command netstat -Q shows a lot of workstreams now. If I use iperf with the option -P 16, I get up to 4.3 Gbit/s and a single stream is around 1.5Gbit/s. During transfer with 16 streams, some CPU cores are indeed used at 100%, global usage is: CPU:  0.0% user,  0.0% nice, 23.6% system, 30.0% interrupt, 46.4% idle.

I noticed an error regarding the ice interfaces while running sysctl command, do you think it's related?

Code Select Expand

root@OPNsense:~ # sysctl -a | grep rss
ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice1: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice2: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice3: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 8:8 9:9 10:10 11:11 12:12 13:13 14:14 15:15
net.inet.rss.enabled: 1
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 16
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 32
net.inet.rss.maxbits: 7
net.inet.rss.mask: 15
net.inet.rss.bits: 4
net.inet.rss.hashalgo: 2
hw.bxe.udp_rss: 0
hw.ix.enable_rss: 1
dev.ax.3.rss_enabled: 1
dev.ax.2.rss_enabled: 1
dev.ax.1.rss_enabled: 1
dev.ax.0.rss_enabled: 1

root@OPNsense:~ # dmesg | grep vectors
igc0: Using MSI-X interrupts with 5 vectors
igc1: Using MSI-X interrupts with 5 vectors
igc2: Using MSI-X interrupts with 5 vectors
igc3: Using MSI-X interrupts with 5 vectors
ice0: Using MSI-X interrupts with 33 vectors
ice1: Using MSI-X interrupts with 33 vectors
ice2: Using MSI-X interrupts with 33 vectors
ice3: Using MSI-X interrupts with 33 vectors
ax0: Using MSI-X interrupts with 16 vectors
ax1: Using MSI-X interrupts with 16 vectors
ax2: Using MSI-X interrupts with 6 vectors
ax3: Using MSI-X interrupts with 6 vectors

As far as I understand, RSS only helps with multiple flows. Is 1Gbit/s the best we can expect from this device for a single flow?

[pfry]

[...]
I noticed an error regarding the ice interfaces while running sysctl command, do you think it's related?

I do not, but of course I could be wrong. I don't recall the error on my machine (offline, so I can't look), but I believe I had had both virtualization and SR-IOV disabled in the BIOS (regular PC, bare metal FreeBSD install).

Speaking of that, I quoted the wrong pciconf command: should be "-lvc [interface]".

How about some more "netstat"s: "-m", "-i", "-sp ip" - anything look odd? (I have an anomalous Idrop stat on ixl interfaces and lots of unaccountable Oerrs on one bridge.)

[...] Is 1Gbit/s the best we can expect from this device for a single flow?

It shouldn't be, but I don't have similar hardware to test.

[Greg_E]

Assuming you want fast opt1 to opt2 file transfers, have you tried copying a large file from one computer to the other?

[AdSchellevis]

@l1lz just to be sure, you are testing with the firewall in between iperf3 machines and with our default config. running iperf3 on your firewall is a common testing issue as it influences your measurements (a lot).

our default test setup (which reaches reaches the advertised speeds), is setup like:

[client] --> [firewall] --> [server]


The client uses an iperf3 command like:

Code Select Expand

iperf3 -c <other-host> -P 8 -Z -t 5 -M 1500


Somethings to consider when numbers don't match:

* ice needs firmware loaded, which is in our default config, the tunable is called "ice_ddp_load"
* make sure to disable the default reply-to rule (Firewall: Settings: Advanced), when a gateway is accidentally configured it may send you packets in the wrong direction.

For regular routing in my experience rss doesn't really help much as most network cards already distribute the load properly over different cores. When results are less than expected, always check `top -CHIPS` as it helps figuring out if flows are distributed as one would expect.

Best regards,

Ad

[l1lz]

our default test setup (which reaches reaches the advertised speeds), is setup like:

[client] --> [firewall] --> [server]

I confirm that we are not using the firewall's iperf server but two machines connecting to the firewall as described in your message. There was a switch initially but I just did the test again without any switch and the result is about the same (around 5 Gbit/s with parallel threads).

Code Select Expand

root@PROXMOX4:~# iperf3 -c 10.1.1.12 -p 5201 -P 8 -Z -t 5 -M 1500
Connecting to host 10.1.1.12, port 5201

[...]

[SUM]   0.00-5.00   sec  2.79 GBytes  4.79 Gbits/sec  873             sender
[SUM]   0.00-5.02   sec  2.77 GBytes  4.75 Gbits/sec                  receiver

iperf Done.


I checked "Disable reply-to on WAN rules" but it changed nothing. The tunable `ice_ddp_load` is set to `YES`.

How about some more "netstat"s: "-m", "-i", "-sp ip" - anything look odd?

I'm definitely not an expert, but I don't have packets drops or errors here. Same for the pciconf command:

Code Select Expand

root@OPNsense:~ # pciconf -lvc ice0
ice0@pci0:6:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1593 subvendor=0x8086 subdevice=0x0005
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller E810-C for SFP'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 512 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x8000]
    cap 10[a0] = PCI-Express 2 endpoint max data 512(512) FLR RO
                 max read 512
                 link x16(x16) speed 8.0(16.0)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 000e[148] = ARI 1
    ecap 0003[150] = Serial 1 000100ffff000000
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0100, VF RID Stride 0x0001
                     VF Device ID 0x1889
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
    ecap 0019[1d0] = PCIe Sec 1 lane errors 0
    ecap 0025[200] = Data Link Feature 1
    ecap 0026[210] = Physical Layer 16.0 GT/s 1
    ecap 0027[250] = Lane Margining at Receiver 1


[User65192]

Hello,

we experience the same Problem with two DEC4280. We tried all the things mentioned in this thread and more. At the start we were below 500 MBit/s, but we can reach now 1 GBit/s.

The test iperf3 setup is also:

client -> firewall -> server

But with only the 10GBit interfaces. To troubleshoot the issue even more we tried a selection of SFP+ modules including fiber, DAC and RJ45.

The speed seems to be normal with the RJ45 (igc0 - igc3) port on the firewall if connected via a 1G switch.