[solved] Throughput very slow

[ascii]

Hello togehter,

i'm currently evaluating opnsense.

My testsetup is a Dell PowerEdge R220 with a Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz (4 cores) and 4GB of RAM
2 onborad nic from broadcom and 2 PCIe from intel.

setup is
1st laptop -> switch -> LAN opnsense -> WAN opnsense -> switch -> 2nd laptop
switches are Cisco cat4500.
on lan and wan site a static IP with NAT
firewall rules just standard.
no plugins installed.

i'm testing with iperf from 1st to 2nd laptop
on broadcom i get about 360Mbit
on the intel nics abour 400Mbit

iperf3 -c 192.168.178.48 -t 90
Connecting to host 192.168.178.48, port 5201
[  5] local 10.68.133.5 port 33592 connected to 192.168.178.48 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  45.0 MBytes   378 Mbits/sec   50   26.9 KBytes
......
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-90.00  sec  3.81 GBytes   363 Mbits/sec  3992             sender
[  5]   0.00-90.00  sec  3.81 GBytes   363 Mbits/sec                  receiver
iperf Done.

all NICs are up at 1Gbit full


i already tried tuneables with
kern.ipc.nmbclusters="131072"   also to 1000000
hw.bge.tso_enable=0
hw.pci.enable_msix=0

didn't made a difference.

any suggestions?

[bartjsmit]

What throughput do you get without OPNsense? I.e. replace the firewall with a cable.

Bart...

[ascii]

ok so i did same or a lot of tests

1st laptop is a lenovo x230t with kali linux
2nd is a lenovo x280 with win10

laptops directly connected

Code: [Select]

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-90.00  sec  9.78 GBytes   934 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  9.78 GBytes   933 Mbits/sec                  receiver
laptops connected via switch

Code: [Select]

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-90.00  sec  9.78 GBytes   933 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  9.78 GBytes   933 Mbits/sec                  receiver
now with the dell and opnsense
i did upgrade the bios und broadcom nic earlier. sadly the guys who where suppost to that did not

broadcom to broadcom

Code: [Select]

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-90.00  sec  7.02 GBytes   670 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  7.02 GBytes   670 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  5.89 GBytes   563 Mbits/sec    1             sender
[  5]   0.00-90.00  sec  5.89 GBytes   563 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  5.91 GBytes   564 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  5.91 GBytes   564 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  5.91 GBytes   564 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  5.91 GBytes   564 Mbits/sec                  receiver

broadcom to intel

Code: [Select]

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-90.00  sec  7.95 GBytes   759 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  7.95 GBytes   759 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  6.07 GBytes   579 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  6.07 GBytes   579 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  7.98 GBytes   762 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  7.98 GBytes   762 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  7.97 GBytes   761 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  7.97 GBytes   761 Mbits/sec                  receiver

intel to intel

Code: [Select]

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-90.00  sec  8.88 GBytes   848 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  8.88 GBytes   848 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  6.26 GBytes   598 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  6.26 GBytes   598 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  6.26 GBytes   598 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  6.26 GBytes   598 Mbits/sec                  receiver

[  5]   0.00-90.00  sec  8.89 GBytes   848 Mbits/sec    0             sender
[  5]   0.00-90.00  sec  8.89 GBytes   848 Mbits/sec                  receiver

after that i disabled the firewall option on opnsense and just used it as a router.
same results.
i used top on the cli but i couldn't see any process eating up cpu's

i attached the full results

[bartjsmit]

If there is no CPU spike then the next most likely candidate is I/O. What IOPS are you getting from your disks? Can you try OPNsense in live mode from a USB disk?

Also have a look at jumbo frames to reduce packet overhead.

Bart...

[ascii]

i will test it next week.
currently there is a normal Hard drive inside. needed to get rid of the SAS Controller in order to free up space for the dual Intel nic. i will test with a SSD and from the USB live.

where can i look at the jumbo frames? i'm not So familiar with BSD.

[bartjsmit]

You can set it in the web interface:

Interfaces, LAN/WAN, MTU

Bart...

[ascii]

i'm not near the server today so i can't replace the disk with a SSD. will do that tomorrow and also run it from the flash drive.

i did same testes on the IOPS. at least what i could find on the internet.

Code: [Select]

diskinfo -t /dev/ada0
/dev/ada0
        512             # sectorsize
        320072933376    # mediasize in bytes (298G)
        625142448       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        620181          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        HGST HTS725032A7E630    # Disk descr.
        TF645AY121ZYVT  # Disk ident.
        No              # TRIM/UNMAP support
        7200            # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Seek times:
        Full stroke:      250 iter in   5.960445 sec =   23.842 msec
        Half stroke:      250 iter in   4.695310 sec =   18.781 msec
        Quarter stroke:   500 iter in   7.120242 sec =   14.240 msec
        Short forward:    400 iter in   2.276236 sec =    5.691 msec
        Short backward:   400 iter in   2.852166 sec =    7.130 msec
        Seq outer:       2048 iter in   0.159691 sec =    0.078 msec
        Seq inner:       2048 iter in   0.179786 sec =    0.088 msec

Transfer rates:
        outside:       102400 kbytes in   0.990192 sec =   103414 kbytes/sec
        middle:        102400 kbytes in   1.187860 sec =    86205 kbytes/sec
        inside:        102400 kbytes in   1.767360 sec =    57940 kbytes/sec
amd with bonnie++, but to be honest i can't read/understand the result.

Code: [Select]

./bonnie++ -d /tmp -r 4096 -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
OPNsense.localdo 8G  237k  99 73.1m   7 30.3m   4  374k  98 67.0m   4 182.0   2
Latency             34621us     243ms    1721ms     120ms     284ms    4059ms
Version  1.98       ------Sequential Create------ --------Random Create--------
OPNsense.localdomai -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 15118.130375  28 +++++ +++ +++++ +++
Latency             83292us      33us      36us     159ms      15us      26us
1.98,1.98,OPNsense.localdomain,1,1582516591,8G,,8192,5,237,99,74894,7,31053,4,374,98,68619,4,182.0,2,16,,,,,+++++,+++,+++++,+++,+++++,+++,15118,28,+++++,+++,+++++,+++,34621us,243ms,1721ms,120ms,284ms,4059ms,83292us,33us,36us,159ms,15us,26us
i also looked at jumbo frames.
i set the MTU up to 9.000
now i get an ~20-30% of my iperf tries speeds at ~ 933 Mbits and the rest of the time ~ 670 Mbits

Code: [Select]

# iperf3 --client 192.168.178.5 -t 5
Connecting to host 192.168.178.5, port 5201
[  5] local 10.68.133.5 port 59666 connected to 192.168.178.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   111 MBytes   933 Mbits/sec    0   69.3 KBytes       
[  5]   1.00-2.00   sec   111 MBytes   932 Mbits/sec    0   69.3 KBytes       
[  5]   2.00-3.00   sec   111 MBytes   934 Mbits/sec    0   69.3 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   932 Mbits/sec    0   69.3 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec    0   69.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec   556 MBytes   933 Mbits/sec    0             sender
[  5]   0.00-5.00   sec   556 MBytes   932 Mbits/sec                  receiver

iperf Done.
# iperf3 --client 192.168.178.5 -t 5
Connecting to host 192.168.178.5, port 5201
[  5] local 10.68.133.5 port 59670 connected to 192.168.178.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  80.2 MBytes   673 Mbits/sec    0   69.3 KBytes       
[  5]   1.00-2.00   sec  79.7 MBytes   669 Mbits/sec    0   69.3 KBytes       
[  5]   2.00-3.00   sec  80.0 MBytes   671 Mbits/sec    0   69.3 KBytes       
[  5]   3.00-4.00   sec  79.8 MBytes   670 Mbits/sec    0   69.3 KBytes       
[  5]   4.00-5.00   sec  79.2 MBytes   665 Mbits/sec    0   69.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec   399 MBytes   669 Mbits/sec    0             sender
[  5]   0.00-5.00   sec   399 MBytes   669 Mbits/sec                  receiver

iperf Done.
i can leave with the ~930 Mbits.
the iperf server is still at 1500MTU since i can't access it. But in the final setup it would be WAN with MTU of 1500 anyways.

now the only question is way does it drop off to ~ 670Mbits quite often.
i also disabled the Hardware Vlan filtering. that did't change anythink

[bartjsmit]

now the only question is way does it drop off to ~ 670Mbits quite often.

Typically the high throughput is served from cache and the lower value is when the firewall needs to use disk. Testing from SSD or flash is the next obvious step.

Bart...

[ascii]

i moved to a SSD and tested USB drive.
Looks very good.
i also had the PowerID enabled. After disabeling that it worked perfect.
Thank you for your support.