OPNsense Forum
English Forums => Hardware and Performance => Topic started by: ascii on February 20, 2020, 03:14:23 pm
-
Hello togehter,
i'm currently evaluating opnsense.
My testsetup is a Dell PowerEdge R220 with a Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz (4 cores) and 4GB of RAM
2 onborad nic from broadcom and 2 PCIe from intel.
setup is
1st laptop -> switch -> LAN opnsense -> WAN opnsense -> switch -> 2nd laptop
switches are Cisco cat4500.
on lan and wan site a static IP with NAT
firewall rules just standard.
no plugins installed.
i'm testing with iperf from 1st to 2nd laptop
on broadcom i get about 360Mbit
on the intel nics abour 400Mbit
iperf3 -c 192.168.178.48 -t 90
Connecting to host 192.168.178.48, port 5201
[ 5] local 10.68.133.5 port 33592 connected to 192.168.178.48 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 45.0 MBytes 378 Mbits/sec 50 26.9 KBytes
......
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-90.00 sec 3.81 GBytes 363 Mbits/sec 3992 sender
[ 5] 0.00-90.00 sec 3.81 GBytes 363 Mbits/sec receiver
iperf Done.
all NICs are up at 1Gbit full
i already tried tuneables with
kern.ipc.nmbclusters="131072" also to 1000000
hw.bge.tso_enable=0
hw.pci.enable_msix=0
didn't made a difference.
any suggestions?
-
What throughput do you get without OPNsense? I.e. replace the firewall with a cable.
Bart...
-
ok so i did same or a lot of tests
1st laptop is a lenovo x230t with kali linux
2nd is a lenovo x280 with win10
laptops directly connected
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-90.00 sec 9.78 GBytes 934 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 9.78 GBytes 933 Mbits/sec receiver
laptops connected via switch
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-90.00 sec 9.78 GBytes 933 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 9.78 GBytes 933 Mbits/sec receiver
now with the dell and opnsense
i did upgrade the bios und broadcom nic earlier. sadly the guys who where suppost to that did not :(
broadcom to broadcom
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-90.00 sec 7.02 GBytes 670 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 7.02 GBytes 670 Mbits/sec receiver
[ 5] 0.00-90.00 sec 5.89 GBytes 563 Mbits/sec 1 sender
[ 5] 0.00-90.00 sec 5.89 GBytes 563 Mbits/sec receiver
[ 5] 0.00-90.00 sec 5.91 GBytes 564 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 5.91 GBytes 564 Mbits/sec receiver
[ 5] 0.00-90.00 sec 5.91 GBytes 564 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 5.91 GBytes 564 Mbits/sec receiver
broadcom to intel
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-90.00 sec 7.95 GBytes 759 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 7.95 GBytes 759 Mbits/sec receiver
[ 5] 0.00-90.00 sec 6.07 GBytes 579 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 6.07 GBytes 579 Mbits/sec receiver
[ 5] 0.00-90.00 sec 7.98 GBytes 762 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 7.98 GBytes 762 Mbits/sec receiver
[ 5] 0.00-90.00 sec 7.97 GBytes 761 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 7.97 GBytes 761 Mbits/sec receiver
intel to intel
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-90.00 sec 8.88 GBytes 848 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 8.88 GBytes 848 Mbits/sec receiver
[ 5] 0.00-90.00 sec 6.26 GBytes 598 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 6.26 GBytes 598 Mbits/sec receiver
[ 5] 0.00-90.00 sec 6.26 GBytes 598 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 6.26 GBytes 598 Mbits/sec receiver
[ 5] 0.00-90.00 sec 8.89 GBytes 848 Mbits/sec 0 sender
[ 5] 0.00-90.00 sec 8.89 GBytes 848 Mbits/sec receiver
after that i disabled the firewall option on opnsense and just used it as a router.
same results.
i used top on the cli but i couldn't see any process eating up cpu's
i attached the full results
-
If there is no CPU spike then the next most likely candidate is I/O. What IOPS are you getting from your disks? Can you try OPNsense in live mode from a USB disk?
Also have a look at jumbo frames to reduce packet overhead.
Bart...
-
i will test it next week.
currently there is a normal Hard drive inside. needed to get rid of the SAS Controller in order to free up space for the dual Intel nic. i will test with a SSD and from the USB live.
where can i look at the jumbo frames? i'm not So familiar with BSD.
-
You can set it in the web interface:
Interfaces, LAN/WAN, MTU
Bart...
-
i'm not near the server today so i can't replace the disk with a SSD. will do that tomorrow and also run it from the flash drive.
i did same testes on the IOPS. at least what i could find on the internet.
diskinfo -t /dev/ada0
/dev/ada0
512 # sectorsize
320072933376 # mediasize in bytes (298G)
625142448 # mediasize in sectors
4096 # stripesize
0 # stripeoffset
620181 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
HGST HTS725032A7E630 # Disk descr.
TF645AY121ZYVT # Disk ident.
No # TRIM/UNMAP support
7200 # Rotation rate in RPM
Not_Zoned # Zone Mode
Seek times:
Full stroke: 250 iter in 5.960445 sec = 23.842 msec
Half stroke: 250 iter in 4.695310 sec = 18.781 msec
Quarter stroke: 500 iter in 7.120242 sec = 14.240 msec
Short forward: 400 iter in 2.276236 sec = 5.691 msec
Short backward: 400 iter in 2.852166 sec = 7.130 msec
Seq outer: 2048 iter in 0.159691 sec = 0.078 msec
Seq inner: 2048 iter in 0.179786 sec = 0.088 msec
Transfer rates:
outside: 102400 kbytes in 0.990192 sec = 103414 kbytes/sec
middle: 102400 kbytes in 1.187860 sec = 86205 kbytes/sec
inside: 102400 kbytes in 1.767360 sec = 57940 kbytes/sec
amd with bonnie++, but to be honest i can't read/understand the result.
./bonnie++ -d /tmp -r 4096 -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
OPNsense.localdo 8G 237k 99 73.1m 7 30.3m 4 374k 98 67.0m 4 182.0 2
Latency 34621us 243ms 1721ms 120ms 284ms 4059ms
Version 1.98 ------Sequential Create------ --------Random Create--------
OPNsense.localdomai -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ 15118.130375 28 +++++ +++ +++++ +++
Latency 83292us 33us 36us 159ms 15us 26us
1.98,1.98,OPNsense.localdomain,1,1582516591,8G,,8192,5,237,99,74894,7,31053,4,374,98,68619,4,182.0,2,16,,,,,+++++,+++,+++++,+++,+++++,+++,15118,28,+++++,+++,+++++,+++,34621us,243ms,1721ms,120ms,284ms,4059ms,83292us,33us,36us,159ms,15us,26us
i also looked at jumbo frames.
i set the MTU up to 9.000
now i get an ~20-30% of my iperf tries speeds at ~ 933 Mbits and the rest of the time ~ 670 Mbits
# iperf3 --client 192.168.178.5 -t 5
Connecting to host 192.168.178.5, port 5201
[ 5] local 10.68.133.5 port 59666 connected to 192.168.178.5 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec 0 69.3 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 932 Mbits/sec 0 69.3 KBytes
[ 5] 2.00-3.00 sec 111 MBytes 934 Mbits/sec 0 69.3 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 932 Mbits/sec 0 69.3 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 0 69.3 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 556 MBytes 933 Mbits/sec 0 sender
[ 5] 0.00-5.00 sec 556 MBytes 932 Mbits/sec receiver
iperf Done.
# iperf3 --client 192.168.178.5 -t 5
Connecting to host 192.168.178.5, port 5201
[ 5] local 10.68.133.5 port 59670 connected to 192.168.178.5 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 80.2 MBytes 673 Mbits/sec 0 69.3 KBytes
[ 5] 1.00-2.00 sec 79.7 MBytes 669 Mbits/sec 0 69.3 KBytes
[ 5] 2.00-3.00 sec 80.0 MBytes 671 Mbits/sec 0 69.3 KBytes
[ 5] 3.00-4.00 sec 79.8 MBytes 670 Mbits/sec 0 69.3 KBytes
[ 5] 4.00-5.00 sec 79.2 MBytes 665 Mbits/sec 0 69.3 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 399 MBytes 669 Mbits/sec 0 sender
[ 5] 0.00-5.00 sec 399 MBytes 669 Mbits/sec receiver
iperf Done.
i can leave with the ~930 Mbits.
the iperf server is still at 1500MTU since i can't access it. But in the final setup it would be WAN with MTU of 1500 anyways.
now the only question is way does it drop off to ~ 670Mbits quite often.
i also disabled the Hardware Vlan filtering. that did't change anythink
-
now the only question is way does it drop off to ~ 670Mbits quite often.
Typically the high throughput is served from cache and the lower value is when the firewall needs to use disk. Testing from SSD or flash is the next obvious step.
Bart...
-
i moved to a SSD and tested USB drive.
Looks very good.
i also had the PowerID enabled. After disabeling that it worked perfect.
Thank you for your support.