Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
QuoteWhat's the correct way to add a permanent static ARP entry besides running "arp -s"?static_arp_pairs ?
https://man.freebsd.org/cgi/man.cgi?rc.conf(5)
Hi Berserker, thanks for being patient with my not so helpful suggestions, glad you got it solved! Would you consider adding 'Solved' or any indicator to your topic title, to help others?
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 172.13.160.1 UGS ix1
dns.google c-24-99-70-1.hsd1. UGHS igb0
dns.google 172.13.160.1 UGHS ix1
10.0.10.0/24 link#9 U ix0_vlan
opnsense link#9 UHS lo0
10.0.20.0/24 link#10 U ix0_vlan
10.0.20.1 link#10 UHS lo0
10.0.30.0/25 link#12 U ix0_vlan
10.0.30.1 link#12 UHS lo0
10.0.40.0/28 link#17 U wg1
10.0.40.1 link#17 UHS lo0
10.0.40.2 link#17 UHS wg1
10.0.60.0/28 link#13 U ix0_vlan
10.0.60.1 link#13 UHS lo0
10.0.70.0/26 link#14 U ix0_vlan
10.0.70.1 link#14 UHS lo0
10.0.80.0/29 link#15 U ix0_vlan
10.0.80.1 link#15 UHS lo0
10.0.90.0/27 link#16 U ix0_vlan
10.0.90.1 link#16 UHS lo0
10.0.250.0/24 link#11 U ix0_vlan
10.0.250.1 link#11 UHS lo0
wg_ip_adr wg_gateway UGHS wg2
10.68.7.116 link#18 UHS wg2
10.68.7.117 link#18 UH lo0
24.99.70.0/23 link#3 U igb0
c-24-99-70-247.hsd link#3 UHS lo0
localhost link#5 UH lo0
172.13.160.0/22 link#2 U ix1
172.13.163.219 link#2 UHS lo0
192.168.1.0/24 172.13.160.1 UGS ix1
192.168.1.2 link#2 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
localhost link#5 UHS lo0
fe80::%lo0/64 link#5 U lo0
fe80::1%lo0 link#5 UHS lo0
ix0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,NOMAP>
ether b4:96:91:21:c9:74
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: ATT_WAN (wan)
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,NOMAP>
ether b4:96:91:21:c9:76
inet wan_ipaddr netmask 0xfffffc00 broadcast 172.13.163.255
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: COMCAST_WAN (opt8)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether ac:1f:6b:21:c2:28
inet 24.99.70.247 netmask 0xfffffe00 broadcast 255.255.255.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether ac:1f:6b:21:c2:29
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
groups: enc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
syncpeer: 0.0.0.0 maxupd: 128 defer: off
syncok: 1
groups: pfsync
ix0_vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN10_LAN (lan)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255
groups: vlan
vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN20_IoT (opt1)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.20.1 netmask 0xffffff00 broadcast 10.0.20.255
groups: vlan
vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan250: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN250_WORK (opt7)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.250.1 netmask 0xffffff00 broadcast 10.0.250.255
groups: vlan
vlan: 250 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN30_MGMT (opt2)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.30.1 netmask 0xffffff80 broadcast 10.0.30.127
groups: vlan
vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan60: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN60_PROXMOX (opt3)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.60.1 netmask 0xfffffff0 broadcast 10.0.60.15
groups: vlan
vlan: 60 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan70: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN70_INTERNET (opt4)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.70.1 netmask 0xffffffc0 broadcast 10.0.70.63
groups: vlan
vlan: 70 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan80: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN80_IPMI (opt5)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.80.1 netmask 0xfffffff8 broadcast 10.0.80.7
groups: vlan
vlan: 80 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN90_SERVICES (opt6)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.90.1 netmask 0xffffffe0 broadcast 10.0.90.31
groups: vlan
vlan: 90 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
options=80000<LINKSTATE>
inet 10.0.40.1 netmask 0xfffffff0
groups: wg wireguard
nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
wg2: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
options=80000<LINKSTATE>
inet wg_addr netmask 0xffffffff
groups: wg wireguard
nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
QuoteFor reference, 192.168.1.2 is an alias IPv4 address attached to the WAN interfacewith /32 mask i believe?
root@opnsense:~ # arp -a | grep 192.
? (192.168.1.2) at b4:96:91:21:c9:76 on ix1 permanent [ethernet]
root@opnsense:~ # arp -s 192.168.1.1 MAC_ADDR
arp: cannot intuit interface index and type for 192.168.1.1
root@opnsense:~ #
And is this with a DEC3840? If you're seeing 7-9Gbps average, where does the 14.6Gbps firewall throughput number come from?
Yes, the results are from the DEC3840.
Total firewall throughput is the maximum the firewall can handle and is calculated by saturating the firewall with small packets (full system utilisation) multiplied with a standard package size of 1500 bytes. So in this case we measured a peak performance of 1200KPPS multiplied with 1500 bytes, leaves us 14.4Gbps.
By the way this is with the firewall enabled, routing only performance is higher (disable the firewall).
If you run iperf3 from a OPNsense interface to a client, your only limiting factor is the single core performance of your CPU(s).
If you run iperf3 from client 1 to client 2 and have OPNsense in the middle, it has to do a lot of work routing the Packets with pf(4), which uses lots of CPU time.
Afaik iperf3 usually only creates one tcp stream, which isn't really a real world load on a firewall.
You could try to run multiple parallel streams with the -P flag:Quote-P, --parallel n
number of parallel client streams to run. Note that iperf3 is single threaded, so if you are CPU bound, this will not yield higher throughput.
All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.
Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.
Cheers,
Franco
Well, Deciso are the ones that posted the numbers, are they not? If the numbers are saying that "14.6Gbps" is a total of 7Gbps coming into the box, and 7Gbps going out of the box from source to destination, then that is *incredibly* misleading especially for a 1000 euro+ piece of equipment. If I'm misinterpreting those numbers and I should, realistically, see a full near 10Gb inter-VLAN routing performance from this box (given I can get this on other routers, my switching hardware and clients aren't the problem), then there's something either wrong with my config or there's some tuning that needs to be involved.
Or perhaps is there a special version of OPNsense that these are supposed to run pre-tuned to properly achieve the advertised performance numbers?
Not sure what the issue is with you specifically setup, however we did notice:Current version / new test
- When running IPsec on the same box leads to a performance penalty.
- With the current kernel the scheduling is not optimal, resulting in a somewhat fluctuating throughput. This is resolved in the new Freebsd 13.1 kernel that has been released as beta https://forum.opnsense.org/index.php?topic=28505.0, so feel free to test this as well.
- And obviously the online documentation has a typo where the total firewall throughput was also mentioned as port-port throughput. Since these are max 10Gbps ports, one cannot route more traffic than that. This has been corrected. Peak (see below is about 9.3 Gbps, we now list slightly below that number).
- Testing is done with spectre/meltdown mitigation disabled (default config for our firewalls), see also https://docs.opnsense.org/troubleshooting/hardening.html
Now I just retested the performance with IPerf on the current kernel (using OPNsense® Business Edition 22.4 / should be the same as current 22.1 version) where the traffic flows through the firewall:
Test Server Port 1 --> Firewall Port 1 --> Firewall Port 2 --> Test Server Port 2
In optimal situation this results in 9.3Gbps:
# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[ 5] local 10.0.0.20 port 44956 connected to 192.168.10.20 port 5201
[ 7] local 10.0.0.20 port 44958 connected to 192.168.10.20 port 5201
[ 9] local 10.0.0.20 port 44960 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44962 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44964 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44966 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44968 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44970 connected to 192.168.10.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 46.1 MBytes 387 Mbits/sec 13 249 KBytes
[ 7] 0.00-1.00 sec 313 MBytes 2.63 Gbits/sec 404 1.20 MBytes
[ 9] 0.00-1.00 sec 38.2 MBytes 320 Mbits/sec 30 226 KBytes
[ 11] 0.00-1.00 sec 43.9 MBytes 368 Mbits/sec 26 245 KBytes
[ 13] 0.00-1.00 sec 31.3 MBytes 262 Mbits/sec 14 192 KBytes
[ 15] 0.00-1.00 sec 41.0 MBytes 344 Mbits/sec 1 253 KBytes
[ 17] 0.00-1.00 sec 422 MBytes 3.54 Gbits/sec 137 1.81 MBytes
[ 19] 0.00-1.00 sec 50.2 MBytes 421 Mbits/sec 21 265 KBytes
[SUM] 0.00-1.00 sec 986 MBytes 8.27 Gbits/sec 646
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 60.5 MBytes 507 Mbits/sec 0 386 KBytes
[ 7] 1.00-2.00 sec 230 MBytes 1.93 Gbits/sec 14 975 KBytes
[ 9] 1.00-2.00 sec 54.5 MBytes 458 Mbits/sec 0 351 KBytes
[ 11] 1.00-2.00 sec 57.9 MBytes 486 Mbits/sec 0 374 KBytes
[ 13] 1.00-2.00 sec 42.7 MBytes 358 Mbits/sec 1 240 KBytes
[ 15] 1.00-2.00 sec 59.7 MBytes 501 Mbits/sec 0 379 KBytes
[ 17] 1.00-2.00 sec 403 MBytes 3.38 Gbits/sec 58 1.43 MBytes
[ 19] 1.00-2.00 sec 76.6 MBytes 643 Mbits/sec 0 427 KBytes
[SUM] 1.00-2.00 sec 985 MBytes 8.26 Gbits/sec 73
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 65.8 MBytes 552 Mbits/sec 12 387 KBytes
[ 7] 2.00-3.00 sec 171 MBytes 1.43 Gbits/sec 2 794 KBytes
[ 9] 2.00-3.00 sec 65.3 MBytes 548 Mbits/sec 6 353 KBytes
[ 11] 2.00-3.00 sec 66.3 MBytes 556 Mbits/sec 13 367 KBytes
[ 13] 2.00-3.00 sec 55.8 MBytes 468 Mbits/sec 0 366 KBytes
[ 15] 2.00-3.00 sec 82.1 MBytes 689 Mbits/sec 0 511 KBytes
[ 17] 2.00-3.00 sec 323 MBytes 2.71 Gbits/sec 103 698 KBytes
[ 19] 2.00-3.00 sec 164 MBytes 1.37 Gbits/sec 6 527 KBytes
[SUM] 2.00-3.00 sec 993 MBytes 8.33 Gbits/sec 142
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 63.1 MBytes 530 Mbits/sec 42 386 KBytes
[ 7] 3.00-4.00 sec 140 MBytes 1.17 Gbits/sec 1 663 KBytes
[ 9] 3.00-4.00 sec 65.8 MBytes 552 Mbits/sec 5 348 KBytes
[ 11] 3.00-4.00 sec 80.7 MBytes 677 Mbits/sec 0 499 KBytes
[ 13] 3.00-4.00 sec 80.5 MBytes 675 Mbits/sec 0 499 KBytes
[ 15] 3.00-4.00 sec 72.7 MBytes 610 Mbits/sec 15 345 KBytes
[ 17] 3.00-4.00 sec 276 MBytes 2.32 Gbits/sec 89 542 KBytes
[ 19] 3.00-4.00 sec 202 MBytes 1.69 Gbits/sec 111 392 KBytes
[SUM] 3.00-4.00 sec 981 MBytes 8.23 Gbits/sec 263
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 57.4 MBytes 481 Mbits/sec 60 275 KBytes
[ 7] 4.00-5.00 sec 129 MBytes 1.08 Gbits/sec 0 790 KBytes
[ 9] 4.00-5.00 sec 73.7 MBytes 619 Mbits/sec 0 475 KBytes
[ 11] 4.00-5.00 sec 77.9 MBytes 654 Mbits/sec 12 459 KBytes
[ 13] 4.00-5.00 sec 98.1 MBytes 823 Mbits/sec 11 442 KBytes
[ 15] 4.00-5.00 sec 64.3 MBytes 539 Mbits/sec 3 337 KBytes
[ 17] 4.00-5.00 sec 257 MBytes 2.15 Gbits/sec 242 291 KBytes
[ 19] 4.00-5.00 sec 217 MBytes 1.82 Gbits/sec 101 415 KBytes
[SUM] 4.00-5.00 sec 974 MBytes 8.17 Gbits/sec 429
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 50.2 MBytes 421 Mbits/sec 8 283 KBytes
[ 7] 5.00-6.00 sec 149 MBytes 1.25 Gbits/sec 0 918 KBytes
[ 9] 5.00-6.00 sec 96.8 MBytes 812 Mbits/sec 0 604 KBytes
[ 11] 5.00-6.00 sec 85.0 MBytes 713 Mbits/sec 11 421 KBytes
[ 13] 5.00-6.00 sec 70.0 MBytes 587 Mbits/sec 4 421 KBytes
[ 15] 5.00-6.00 sec 48.1 MBytes 404 Mbits/sec 25 245 KBytes
[ 17] 5.00-6.00 sec 216 MBytes 1.81 Gbits/sec 76 449 KBytes
[ 19] 5.00-6.00 sec 258 MBytes 2.16 Gbits/sec 96 421 KBytes
[SUM] 5.00-6.00 sec 973 MBytes 8.16 Gbits/sec 220
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 46.4 MBytes 389 Mbits/sec 23 307 KBytes
[ 7] 6.00-7.00 sec 145 MBytes 1.22 Gbits/sec 1 770 KBytes
[ 9] 6.00-7.00 sec 98.8 MBytes 828 Mbits/sec 22 549 KBytes
[ 11] 6.00-7.00 sec 68.8 MBytes 577 Mbits/sec 10 419 KBytes
[ 13] 6.00-7.00 sec 92.5 MBytes 776 Mbits/sec 0 556 KBytes
[ 15] 6.00-7.00 sec 44.4 MBytes 372 Mbits/sec 7 290 KBytes
[ 17] 6.00-7.00 sec 255 MBytes 2.14 Gbits/sec 65 598 KBytes
[ 19] 6.00-7.00 sec 228 MBytes 1.92 Gbits/sec 61 566 KBytes
[SUM] 6.00-7.00 sec 979 MBytes 8.22 Gbits/sec 189
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 58.6 MBytes 491 Mbits/sec 6 324 KBytes
[ 7] 7.00-8.00 sec 139 MBytes 1.16 Gbits/sec 33 650 KBytes
[ 9] 7.00-8.00 sec 87.5 MBytes 734 Mbits/sec 15 516 KBytes
[ 11] 7.00-8.00 sec 56.4 MBytes 473 Mbits/sec 9 305 KBytes
[ 13] 7.00-8.00 sec 102 MBytes 860 Mbits/sec 2 504 KBytes
[ 15] 7.00-8.00 sec 53.7 MBytes 451 Mbits/sec 15 309 KBytes
[ 17] 7.00-8.00 sec 218 MBytes 1.83 Gbits/sec 127 444 KBytes
[ 19] 7.00-8.00 sec 265 MBytes 2.22 Gbits/sec 134 503 KBytes
[SUM] 7.00-8.00 sec 981 MBytes 8.23 Gbits/sec 341
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 66.5 MBytes 557 Mbits/sec 0 449 KBytes
[ 7] 8.00-9.00 sec 108 MBytes 902 Mbits/sec 6 565 KBytes
[ 9] 8.00-9.00 sec 104 MBytes 870 Mbits/sec 17 450 KBytes
[ 11] 8.00-9.00 sec 65.7 MBytes 551 Mbits/sec 0 433 KBytes
[ 13] 8.00-9.00 sec 102 MBytes 860 Mbits/sec 6 442 KBytes
[ 15] 8.00-9.00 sec 53.9 MBytes 452 Mbits/sec 34 220 KBytes
[ 17] 8.00-9.00 sec 230 MBytes 1.93 Gbits/sec 54 441 KBytes
[ 19] 8.00-9.00 sec 251 MBytes 2.10 Gbits/sec 156 457 KBytes
[SUM] 8.00-9.00 sec 981 MBytes 8.23 Gbits/sec 273
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 91.7 MBytes 769 Mbits/sec 2 404 KBytes
[ 7] 9.00-10.00 sec 115 MBytes 965 Mbits/sec 0 692 KBytes
[ 9] 9.00-10.00 sec 93.8 MBytes 786 Mbits/sec 0 581 KBytes
[ 11] 9.00-10.00 sec 75.2 MBytes 631 Mbits/sec 11 408 KBytes
[ 13] 9.00-10.00 sec 75.0 MBytes 629 Mbits/sec 20 290 KBytes
[ 15] 9.00-10.00 sec 47.8 MBytes 401 Mbits/sec 0 341 KBytes
[ 17] 9.00-10.00 sec 236 MBytes 1.98 Gbits/sec 35 338 KBytes
[ 19] 9.00-10.00 sec 246 MBytes 2.07 Gbits/sec 62 345 KBytes
[SUM] 9.00-10.00 sec 981 MBytes 8.23 Gbits/sec 130
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 606 MBytes 509 Mbits/sec 166 sender
[ 5] 0.00-10.00 sec 604 MBytes 506 Mbits/sec receiver
[ 7] 0.00-10.00 sec 1.60 GBytes 1.37 Gbits/sec 461 sender
[ 7] 0.00-10.00 sec 1.60 GBytes 1.37 Gbits/sec receiver
[ 9] 0.00-10.00 sec 778 MBytes 653 Mbits/sec 95 sender
[ 9] 0.00-10.00 sec 776 MBytes 650 Mbits/sec receiver
[ 11] 0.00-10.00 sec 678 MBytes 569 Mbits/sec 92 sender
[ 11] 0.00-10.00 sec 675 MBytes 566 Mbits/sec receiver
[ 13] 0.00-10.00 sec 751 MBytes 630 Mbits/sec 58 sender
[ 13] 0.00-10.00 sec 747 MBytes 627 Mbits/sec receiver
[ 15] 0.00-10.00 sec 568 MBytes 476 Mbits/sec 100 sender
[ 15] 0.00-10.00 sec 565 MBytes 474 Mbits/sec receiver
[ 17] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 986 sender
[ 17] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec 748 sender
[ 19] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec receiver
[SUM] 0.00-10.00 sec 9.58 GBytes 8.23 Gbits/sec 2706 sender
[SUM] 0.00-10.00 sec 9.56 GBytes 8.21 Gbits/sec receiver
iperf Done.
root@perftest1:/opt/OPNsense_perftest# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[ 5] local 10.0.0.20 port 44974 connected to 192.168.10.20 port 5201
[ 7] local 10.0.0.20 port 44976 connected to 192.168.10.20 port 5201
[ 9] local 10.0.0.20 port 44978 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44980 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44982 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44984 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44986 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44988 connected to 192.168.10.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 189 MBytes 1.59 Gbits/sec 17 449 KBytes
[ 7] 0.00-1.00 sec 91.0 MBytes 764 Mbits/sec 4 361 KBytes
[ 9] 0.00-1.00 sec 108 MBytes 905 Mbits/sec 4 392 KBytes
[ 11] 0.00-1.00 sec 164 MBytes 1.38 Gbits/sec 4 532 KBytes
[ 13] 0.00-1.00 sec 127 MBytes 1.07 Gbits/sec 5 436 KBytes
[ 15] 0.00-1.00 sec 150 MBytes 1.26 Gbits/sec 7 445 KBytes
[ 17] 0.00-1.00 sec 125 MBytes 1.05 Gbits/sec 2 382 KBytes
[ 19] 0.00-1.00 sec 178 MBytes 1.49 Gbits/sec 14 603 KBytes
[SUM] 0.00-1.00 sec 1.11 GBytes 9.50 Gbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 165 MBytes 1.38 Gbits/sec 0 506 KBytes
[ 7] 1.00-2.00 sec 110 MBytes 921 Mbits/sec 0 528 KBytes
[ 9] 1.00-2.00 sec 115 MBytes 967 Mbits/sec 0 558 KBytes
[ 11] 1.00-2.00 sec 142 MBytes 1.19 Gbits/sec 0 698 KBytes
[ 13] 1.00-2.00 sec 121 MBytes 1.01 Gbits/sec 0 602 KBytes
[ 15] 1.00-2.00 sec 156 MBytes 1.31 Gbits/sec 0 567 KBytes
[ 17] 1.00-2.00 sec 154 MBytes 1.29 Gbits/sec 0 520 KBytes
[ 19] 1.00-2.00 sec 158 MBytes 1.33 Gbits/sec 0 632 KBytes
[SUM] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 160 MBytes 1.34 Gbits/sec 0 520 KBytes
[ 7] 2.00-3.00 sec 109 MBytes 918 Mbits/sec 0 662 KBytes
[ 9] 2.00-3.00 sec 113 MBytes 951 Mbits/sec 0 688 KBytes
[ 11] 2.00-3.00 sec 140 MBytes 1.17 Gbits/sec 0 827 KBytes
[ 13] 2.00-3.00 sec 120 MBytes 1.01 Gbits/sec 0 731 KBytes
[ 15] 2.00-3.00 sec 157 MBytes 1.32 Gbits/sec 0 621 KBytes
[ 17] 2.00-3.00 sec 159 MBytes 1.33 Gbits/sec 0 612 KBytes
[ 19] 2.00-3.00 sec 158 MBytes 1.33 Gbits/sec 0 674 KBytes
[SUM] 2.00-3.00 sec 1.09 GBytes 9.36 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 168 MBytes 1.41 Gbits/sec 0 544 KBytes
[ 7] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 773 KBytes
[ 9] 3.00-4.00 sec 110 MBytes 923 Mbits/sec 0 796 KBytes
[ 11] 3.00-4.00 sec 136 MBytes 1.14 Gbits/sec 0 940 KBytes
[ 13] 3.00-4.00 sec 118 MBytes 986 Mbits/sec 0 836 KBytes
[ 15] 3.00-4.00 sec 160 MBytes 1.34 Gbits/sec 0 686 KBytes
[ 17] 3.00-4.00 sec 158 MBytes 1.32 Gbits/sec 0 675 KBytes
[ 19] 3.00-4.00 sec 159 MBytes 1.33 Gbits/sec 0 707 KBytes
[SUM] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 161 MBytes 1.35 Gbits/sec 0 556 KBytes
[ 7] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 873 KBytes
[ 9] 4.00-5.00 sec 114 MBytes 954 Mbits/sec 0 891 KBytes
[ 11] 4.00-5.00 sec 135 MBytes 1.13 Gbits/sec 0 1.01 MBytes
[ 13] 4.00-5.00 sec 118 MBytes 986 Mbits/sec 12 928 KBytes
[ 15] 4.00-5.00 sec 159 MBytes 1.33 Gbits/sec 0 694 KBytes
[ 17] 4.00-5.00 sec 160 MBytes 1.34 Gbits/sec 0 685 KBytes
[ 19] 4.00-5.00 sec 159 MBytes 1.33 Gbits/sec 0 716 KBytes
[SUM] 4.00-5.00 sec 1.09 GBytes 9.37 Gbits/sec 12
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 562 KBytes
[ 7] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 0 968 KBytes
[ 9] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 0 986 KBytes
[ 11] 5.00-6.00 sec 140 MBytes 1.17 Gbits/sec 0 1.10 MBytes
[ 13] 5.00-6.00 sec 129 MBytes 1.08 Gbits/sec 0 1018 KBytes
[ 15] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 711 KBytes
[ 17] 5.00-6.00 sec 146 MBytes 1.23 Gbits/sec 0 700 KBytes
[ 19] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 732 KBytes
[SUM] 5.00-6.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 567 KBytes
[ 7] 6.00-7.00 sec 139 MBytes 1.16 Gbits/sec 0 1.04 MBytes
[ 9] 6.00-7.00 sec 136 MBytes 1.14 Gbits/sec 0 1.06 MBytes
[ 11] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 1.17 MBytes
[ 13] 6.00-7.00 sec 138 MBytes 1.15 Gbits/sec 0 1.09 MBytes
[ 15] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 719 KBytes
[ 17] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 708 KBytes
[ 19] 6.00-7.00 sec 140 MBytes 1.17 Gbits/sec 0 739 KBytes
[SUM] 6.00-7.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 567 KBytes
[ 7] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 1.13 MBytes
[ 9] 7.00-8.00 sec 138 MBytes 1.15 Gbits/sec 0 1.14 MBytes
[ 11] 7.00-8.00 sec 142 MBytes 1.20 Gbits/sec 0 1.18 MBytes
[ 13] 7.00-8.00 sec 138 MBytes 1.15 Gbits/sec 0 1.17 MBytes
[ 15] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 724 KBytes
[ 17] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 712 KBytes
[ 19] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 742 KBytes
[SUM] 7.00-8.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 188 MBytes 1.58 Gbits/sec 183 563 KBytes
[ 7] 8.00-9.00 sec 132 MBytes 1.11 Gbits/sec 285 645 KBytes
[ 9] 8.00-9.00 sec 138 MBytes 1.15 Gbits/sec 270 694 KBytes
[ 11] 8.00-9.00 sec 109 MBytes 912 Mbits/sec 251 488 KBytes
[ 13] 8.00-9.00 sec 104 MBytes 870 Mbits/sec 257 477 KBytes
[ 15] 8.00-9.00 sec 150 MBytes 1.26 Gbits/sec 96 507 KBytes
[ 17] 8.00-9.00 sec 142 MBytes 1.20 Gbits/sec 254 453 KBytes
[ 19] 8.00-9.00 sec 154 MBytes 1.29 Gbits/sec 150 515 KBytes
[SUM] 8.00-9.00 sec 1.09 GBytes 9.37 Gbits/sec 1746
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 122 MBytes 1.02 Gbits/sec 0 565 KBytes
[ 7] 9.00-10.00 sec 148 MBytes 1.24 Gbits/sec 0 792 KBytes
[ 9] 9.00-10.00 sec 142 MBytes 1.20 Gbits/sec 0 823 KBytes
[ 11] 9.00-10.00 sec 98.8 MBytes 828 Mbits/sec 0 586 KBytes
[ 13] 9.00-10.00 sec 91.2 MBytes 765 Mbits/sec 0 565 KBytes
[ 15] 9.00-10.00 sec 149 MBytes 1.25 Gbits/sec 0 602 KBytes
[ 17] 9.00-10.00 sec 194 MBytes 1.63 Gbits/sec 0 636 KBytes
[ 19] 9.00-10.00 sec 120 MBytes 1.01 Gbits/sec 0 548 KBytes
[SUM] 9.00-10.00 sec 1.04 GBytes 8.93 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.55 GBytes 1.33 Gbits/sec 200 sender
[ 5] 0.00-10.00 sec 1.54 GBytes 1.33 Gbits/sec receiver
[ 7] 0.00-10.00 sec 1.19 GBytes 1.02 Gbits/sec 289 sender
[ 7] 0.00-10.00 sec 1.19 GBytes 1.02 Gbits/sec receiver
[ 9] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec 274 sender
[ 9] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec receiver
[ 11] 0.00-10.00 sec 1.32 GBytes 1.13 Gbits/sec 255 sender
[ 11] 0.00-10.00 sec 1.31 GBytes 1.13 Gbits/sec receiver
[ 13] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec 274 sender
[ 13] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec receiver
[ 15] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 103 sender
[ 15] 0.00-10.00 sec 1.47 GBytes 1.27 Gbits/sec receiver
[ 17] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 256 sender
[ 17] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 164 sender
[ 19] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec receiver
[SUM] 0.00-10.00 sec 10.9 GBytes 9.34 Gbits/sec 1815 sender
[SUM] 0.00-10.00 sec 10.9 GBytes 9.32 Gbits/sec receiver
iperf Done.
Retesting a couple of times does show a spread with an average of about 7-9Gbps.
With the new FreeBSD 13.1 kernel the performance averages at about 8.7Gbps (standard MTU) and fluctuates a lot less. So while a bit lower than our peak, it will likely result in higher throughput on average.
Hope this clears things up for everyone.
All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.
Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.
Cheers,
Franco
I don't think there are significant changes on 22.1.x and I'm not sure why people sometimes wonder if there are... release notes state what is being worked on and source repositories have annotated changes too.
As a general principle: performance gain is nice, but stability is much nicer still.
Cheers,
Franco