Messages

What's the correct way to add a permanent static ARP entry besides running "arp -s"?

static_arp_pairs ?
https://man.freebsd.org/cgi/man.cgi?rc.conf(5)

I have that added in /etc/rc.conf but it does not work. Is it in the wrong file?

Gonna open this back up. I was successfully able to add the static ARP entry, but it disappears after a few, maybe 10-20 minutes. I do see the entry says "permanent" but something is removing it, not sure what.

What's the correct way to add a permanent static ARP entry besides running "arp -s"?

Hi Berserker, thanks for being patient with my not so helpful suggestions, glad you got it solved! Would you consider adding 'Solved' or any indicator to your topic title, to help others?

Done thanks.

well that was added by the system when I set the secondary IP, how do I get it to not add it?

I'm an idiot. I had a static route added from before when I was testing some other things. I removed it, and was properly able to add the static ARP entry now. Thank you!

netstat

Code Select Expand

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.13.160.1       UGS         ix1
dns.google         c-24-99-70-1.hsd1. UGHS       igb0
dns.google         172.13.160.1       UGHS        ix1
10.0.10.0/24       link#9             U      ix0_vlan
opnsense           link#9             UHS         lo0
10.0.20.0/24       link#10            U      ix0_vlan
10.0.20.1          link#10            UHS         lo0
10.0.30.0/25       link#12            U      ix0_vlan
10.0.30.1          link#12            UHS         lo0
10.0.40.0/28       link#17            U           wg1
10.0.40.1          link#17            UHS         lo0
10.0.40.2          link#17            UHS         wg1
10.0.60.0/28       link#13            U      ix0_vlan
10.0.60.1          link#13            UHS         lo0
10.0.70.0/26       link#14            U      ix0_vlan
10.0.70.1          link#14            UHS         lo0
10.0.80.0/29       link#15            U      ix0_vlan
10.0.80.1          link#15            UHS         lo0
10.0.90.0/27       link#16            U      ix0_vlan
10.0.90.1          link#16            UHS         lo0
10.0.250.0/24      link#11            U      ix0_vlan
10.0.250.1         link#11            UHS         lo0
wg_ip_adr          wg_gateway         UGHS        wg2
10.68.7.116        link#18            UHS         wg2
10.68.7.117        link#18            UH          lo0
24.99.70.0/23      link#3             U          igb0
c-24-99-70-247.hsd link#3             UHS         lo0
localhost          link#5             UH          lo0
172.13.160.0/22    link#2             U           ix1
172.13.163.219     link#2             UHS         lo0
192.168.1.0/24     172.13.160.1       UGS         ix1
192.168.1.2        link#2             UHS         lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
localhost          link#5             UHS         lo0
fe80::%lo0/64      link#5             U           lo0
fe80::1%lo0        link#5             UHS         lo0



ifconfig

Code Select Expand

ix0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,NOMAP>
ether b4:96:91:21:c9:74
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: ATT_WAN (wan)
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,NOMAP>
ether b4:96:91:21:c9:76
inet wan_ipaddr netmask 0xfffffc00 broadcast 172.13.163.255
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: COMCAST_WAN (opt8)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether ac:1f:6b:21:c2:28
inet 24.99.70.247 netmask 0xfffffe00 broadcast 255.255.255.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether ac:1f:6b:21:c2:29
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
groups: enc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
syncpeer: 0.0.0.0 maxupd: 128 defer: off
syncok: 1
groups: pfsync
ix0_vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN10_LAN (lan)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255
groups: vlan
vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN20_IoT (opt1)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.20.1 netmask 0xffffff00 broadcast 10.0.20.255
groups: vlan
vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan250: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN250_WORK (opt7)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.250.1 netmask 0xffffff00 broadcast 10.0.250.255
groups: vlan
vlan: 250 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN30_MGMT (opt2)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.30.1 netmask 0xffffff80 broadcast 10.0.30.127
groups: vlan
vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan60: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN60_PROXMOX (opt3)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.60.1 netmask 0xfffffff0 broadcast 10.0.60.15
groups: vlan
vlan: 60 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan70: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN70_INTERNET (opt4)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.70.1 netmask 0xffffffc0 broadcast 10.0.70.63
groups: vlan
vlan: 70 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan80: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN80_IPMI (opt5)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.80.1 netmask 0xfffffff8 broadcast 10.0.80.7
groups: vlan
vlan: 80 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix0_vlan90: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLAN90_SERVICES (opt6)
options=4000000<NOMAP>
ether b4:96:91:21:c9:74
inet 10.0.90.1 netmask 0xffffffe0 broadcast 10.0.90.31
groups: vlan
vlan: 90 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
options=80000<LINKSTATE>
inet 10.0.40.1 netmask 0xfffffff0
groups: wg wireguard
nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
wg2: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
options=80000<LINKSTATE>
inet wg_addr netmask 0xffffffff
groups: wg wireguard
nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>


bump

For reference, 192.168.1.2 is an alias IPv4 address attached to the WAN interface

with /32 mask i believe?

I assumed this was for the ip space, which the ONT's is on a /24, so I added it as a /24

I want a static ARP for an IP that the opnsense system does not know about, so I'm not sure if static ARP under DHCP lease is the answer here, especially because it wouldn't be on any LAN interfaces. If that's still the way to do it, then I can try but I have my doubts here

I haven't! I didn't realize it did. Do I just add it as if I were adding a device to the WoL function with the right interface, IP and MAC?

Edit: Just tried and it doesn't seem to be adding the entry.

I have an ONT that allowed communication over a local IP space but loses connection every X minutes if the device is not ARP'd. To get around this, a static ARP entry is required for its MAC, but I'm unable to add it for that space

Code Select Expand


root@opnsense:~ # arp -a | grep 192.
? (192.168.1.2) at b4:96:91:21:c9:76 on ix1 permanent [ethernet]
root@opnsense:~ # arp -s 192.168.1.1 MAC_ADDR
arp: cannot intuit interface index and type for 192.168.1.1
root@opnsense:~ #


For reference, 192.168.1.2 is an alias IPv4 address attached to the WAN interface in the interface settings. 192.168.1.1 is the address of the ONT in question.

Any hints?

And is this with a DEC3840? If you're seeing 7-9Gbps average, where does the 14.6Gbps firewall throughput number come from?

Yes, the results are from the DEC3840.
Total firewall throughput is the maximum the firewall can handle and is calculated by saturating the firewall with small packets (full system utilisation) multiplied with a standard package size of 1500 bytes. So in this case we measured a peak performance of 1200KPPS multiplied with 1500 bytes, leaves us 14.4Gbps.

By the way this is with the firewall enabled, routing only performance is higher (disable the firewall).

So...would a 21.x config restored to 22.x cause slow performance? Is there anything to try for troubleshooting purposes when it comes to solving these issues? I can confirm layer 2 is not an issue, seeing full 10Gb across Layer 2, and like I mentioned, from the firewall itself out over Layer 3 to a client device is seeing full 10Gb.

If you run iperf3 from a OPNsense interface to a client, your only limiting factor is the single core performance of your CPU(s).

If you run iperf3 from client 1 to client 2 and have OPNsense in the middle, it has to do a lot of work routing the Packets with pf(4), which uses lots of CPU time.

Afaik iperf3 usually only creates one tcp stream, which isn't really a real world load on a firewall.
You could try to run multiple parallel streams with the -P flag:

-P, --parallel n
              number of parallel client streams to run. Note that iperf3 is single threaded, so if you are CPU bound, this will not yield higher throughput.

I mentioned in my post that these results were "fully-threaded" as in, running 4 or 8 parallel streams to take advantage of the multi-core performance. The numbers posted by Deciso were tested using IMIX which should give you *worse* performance than iperf3, so something is off with my setup or these numbers are not correct.

[Current version / new test]

All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.

Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.


Cheers,
Franco

Well, Deciso are the ones that posted the numbers, are they not? If the numbers are saying that "14.6Gbps" is a total of 7Gbps coming into the box, and 7Gbps going out of the box from source to destination, then that is *incredibly* misleading especially for a 1000 euro+ piece of equipment. If I'm misinterpreting those numbers and I should, realistically, see a full near 10Gb inter-VLAN routing performance from this box (given I can get this on other routers, my switching hardware and clients aren't the problem), then there's something either wrong with my config or there's some tuning that needs to be involved.

Or perhaps is there a special version of OPNsense that these are supposed to run pre-tuned to properly achieve the advertised performance numbers?

Not sure what the issue is with you specifically setup, however we did notice:

• When running IPsec on the same box leads to a performance penalty.
  
• With the current kernel the scheduling is not optimal, resulting in a somewhat fluctuating throughput. This is resolved in the new Freebsd 13.1 kernel that has been released as beta https://forum.opnsense.org/index.php?topic=28505.0, so feel free to test this as well.
• And obviously the online documentation has a typo where the total firewall throughput was also mentioned as port-port throughput. Since these are max 10Gbps ports, one cannot route more traffic than that. This has been corrected. Peak (see below is about 9.3 Gbps, we now list slightly below that number).
• Testing is done with spectre/meltdown mitigation disabled (default config for our firewalls), see also https://docs.opnsense.org/troubleshooting/hardening.html

Current version / new test
Now I just retested the performance with IPerf on the current kernel (using OPNsense® Business Edition 22.4 / should be the same as current 22.1 version) where the traffic flows through the firewall:

Test Server Port 1 --> Firewall Port 1 --> Firewall Port 2 --> Test Server Port 2

In optimal situation this results in 9.3Gbps:

# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[  5] local 10.0.0.20 port 44956 connected to 192.168.10.20 port 5201
[  7] local 10.0.0.20 port 44958 connected to 192.168.10.20 port 5201
[  9] local 10.0.0.20 port 44960 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44962 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44964 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44966 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44968 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44970 connected to 192.168.10.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  46.1 MBytes   387 Mbits/sec   13    249 KBytes       
[  7]   0.00-1.00   sec   313 MBytes  2.63 Gbits/sec  404   1.20 MBytes       
[  9]   0.00-1.00   sec  38.2 MBytes   320 Mbits/sec   30    226 KBytes       
[ 11]   0.00-1.00   sec  43.9 MBytes   368 Mbits/sec   26    245 KBytes       
[ 13]   0.00-1.00   sec  31.3 MBytes   262 Mbits/sec   14    192 KBytes       
[ 15]   0.00-1.00   sec  41.0 MBytes   344 Mbits/sec    1    253 KBytes       
[ 17]   0.00-1.00   sec   422 MBytes  3.54 Gbits/sec  137   1.81 MBytes       
[ 19]   0.00-1.00   sec  50.2 MBytes   421 Mbits/sec   21    265 KBytes       
[SUM]   0.00-1.00   sec   986 MBytes  8.27 Gbits/sec  646             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  60.5 MBytes   507 Mbits/sec    0    386 KBytes       
[  7]   1.00-2.00   sec   230 MBytes  1.93 Gbits/sec   14    975 KBytes       
[  9]   1.00-2.00   sec  54.5 MBytes   458 Mbits/sec    0    351 KBytes       
[ 11]   1.00-2.00   sec  57.9 MBytes   486 Mbits/sec    0    374 KBytes       
[ 13]   1.00-2.00   sec  42.7 MBytes   358 Mbits/sec    1    240 KBytes       
[ 15]   1.00-2.00   sec  59.7 MBytes   501 Mbits/sec    0    379 KBytes       
[ 17]   1.00-2.00   sec   403 MBytes  3.38 Gbits/sec   58   1.43 MBytes       
[ 19]   1.00-2.00   sec  76.6 MBytes   643 Mbits/sec    0    427 KBytes       
[SUM]   1.00-2.00   sec   985 MBytes  8.26 Gbits/sec   73             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  65.8 MBytes   552 Mbits/sec   12    387 KBytes       
[  7]   2.00-3.00   sec   171 MBytes  1.43 Gbits/sec    2    794 KBytes       
[  9]   2.00-3.00   sec  65.3 MBytes   548 Mbits/sec    6    353 KBytes       
[ 11]   2.00-3.00   sec  66.3 MBytes   556 Mbits/sec   13    367 KBytes       
[ 13]   2.00-3.00   sec  55.8 MBytes   468 Mbits/sec    0    366 KBytes       
[ 15]   2.00-3.00   sec  82.1 MBytes   689 Mbits/sec    0    511 KBytes       
[ 17]   2.00-3.00   sec   323 MBytes  2.71 Gbits/sec  103    698 KBytes       
[ 19]   2.00-3.00   sec   164 MBytes  1.37 Gbits/sec    6    527 KBytes       
[SUM]   2.00-3.00   sec   993 MBytes  8.33 Gbits/sec  142             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  63.1 MBytes   530 Mbits/sec   42    386 KBytes       
[  7]   3.00-4.00   sec   140 MBytes  1.17 Gbits/sec    1    663 KBytes       
[  9]   3.00-4.00   sec  65.8 MBytes   552 Mbits/sec    5    348 KBytes       
[ 11]   3.00-4.00   sec  80.7 MBytes   677 Mbits/sec    0    499 KBytes       
[ 13]   3.00-4.00   sec  80.5 MBytes   675 Mbits/sec    0    499 KBytes       
[ 15]   3.00-4.00   sec  72.7 MBytes   610 Mbits/sec   15    345 KBytes       
[ 17]   3.00-4.00   sec   276 MBytes  2.32 Gbits/sec   89    542 KBytes       
[ 19]   3.00-4.00   sec   202 MBytes  1.69 Gbits/sec  111    392 KBytes       
[SUM]   3.00-4.00   sec   981 MBytes  8.23 Gbits/sec  263             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  57.4 MBytes   481 Mbits/sec   60    275 KBytes       
[  7]   4.00-5.00   sec   129 MBytes  1.08 Gbits/sec    0    790 KBytes       
[  9]   4.00-5.00   sec  73.7 MBytes   619 Mbits/sec    0    475 KBytes       
[ 11]   4.00-5.00   sec  77.9 MBytes   654 Mbits/sec   12    459 KBytes       
[ 13]   4.00-5.00   sec  98.1 MBytes   823 Mbits/sec   11    442 KBytes       
[ 15]   4.00-5.00   sec  64.3 MBytes   539 Mbits/sec    3    337 KBytes       
[ 17]   4.00-5.00   sec   257 MBytes  2.15 Gbits/sec  242    291 KBytes       
[ 19]   4.00-5.00   sec   217 MBytes  1.82 Gbits/sec  101    415 KBytes       
[SUM]   4.00-5.00   sec   974 MBytes  8.17 Gbits/sec  429             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  50.2 MBytes   421 Mbits/sec    8    283 KBytes       
[  7]   5.00-6.00   sec   149 MBytes  1.25 Gbits/sec    0    918 KBytes       
[  9]   5.00-6.00   sec  96.8 MBytes   812 Mbits/sec    0    604 KBytes       
[ 11]   5.00-6.00   sec  85.0 MBytes   713 Mbits/sec   11    421 KBytes       
[ 13]   5.00-6.00   sec  70.0 MBytes   587 Mbits/sec    4    421 KBytes       
[ 15]   5.00-6.00   sec  48.1 MBytes   404 Mbits/sec   25    245 KBytes       
[ 17]   5.00-6.00   sec   216 MBytes  1.81 Gbits/sec   76    449 KBytes       
[ 19]   5.00-6.00   sec   258 MBytes  2.16 Gbits/sec   96    421 KBytes       
[SUM]   5.00-6.00   sec   973 MBytes  8.16 Gbits/sec  220             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  46.4 MBytes   389 Mbits/sec   23    307 KBytes       
[  7]   6.00-7.00   sec   145 MBytes  1.22 Gbits/sec    1    770 KBytes       
[  9]   6.00-7.00   sec  98.8 MBytes   828 Mbits/sec   22    549 KBytes       
[ 11]   6.00-7.00   sec  68.8 MBytes   577 Mbits/sec   10    419 KBytes       
[ 13]   6.00-7.00   sec  92.5 MBytes   776 Mbits/sec    0    556 KBytes       
[ 15]   6.00-7.00   sec  44.4 MBytes   372 Mbits/sec    7    290 KBytes       
[ 17]   6.00-7.00   sec   255 MBytes  2.14 Gbits/sec   65    598 KBytes       
[ 19]   6.00-7.00   sec   228 MBytes  1.92 Gbits/sec   61    566 KBytes       
[SUM]   6.00-7.00   sec   979 MBytes  8.22 Gbits/sec  189             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  58.6 MBytes   491 Mbits/sec    6    324 KBytes       
[  7]   7.00-8.00   sec   139 MBytes  1.16 Gbits/sec   33    650 KBytes       
[  9]   7.00-8.00   sec  87.5 MBytes   734 Mbits/sec   15    516 KBytes       
[ 11]   7.00-8.00   sec  56.4 MBytes   473 Mbits/sec    9    305 KBytes       
[ 13]   7.00-8.00   sec   102 MBytes   860 Mbits/sec    2    504 KBytes       
[ 15]   7.00-8.00   sec  53.7 MBytes   451 Mbits/sec   15    309 KBytes       
[ 17]   7.00-8.00   sec   218 MBytes  1.83 Gbits/sec  127    444 KBytes       
[ 19]   7.00-8.00   sec   265 MBytes  2.22 Gbits/sec  134    503 KBytes       
[SUM]   7.00-8.00   sec   981 MBytes  8.23 Gbits/sec  341             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  66.5 MBytes   557 Mbits/sec    0    449 KBytes       
[  7]   8.00-9.00   sec   108 MBytes   902 Mbits/sec    6    565 KBytes       
[  9]   8.00-9.00   sec   104 MBytes   870 Mbits/sec   17    450 KBytes       
[ 11]   8.00-9.00   sec  65.7 MBytes   551 Mbits/sec    0    433 KBytes       
[ 13]   8.00-9.00   sec   102 MBytes   860 Mbits/sec    6    442 KBytes       
[ 15]   8.00-9.00   sec  53.9 MBytes   452 Mbits/sec   34    220 KBytes       
[ 17]   8.00-9.00   sec   230 MBytes  1.93 Gbits/sec   54    441 KBytes       
[ 19]   8.00-9.00   sec   251 MBytes  2.10 Gbits/sec  156    457 KBytes       
[SUM]   8.00-9.00   sec   981 MBytes  8.23 Gbits/sec  273             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  91.7 MBytes   769 Mbits/sec    2    404 KBytes       
[  7]   9.00-10.00  sec   115 MBytes   965 Mbits/sec    0    692 KBytes       
[  9]   9.00-10.00  sec  93.8 MBytes   786 Mbits/sec    0    581 KBytes       
[ 11]   9.00-10.00  sec  75.2 MBytes   631 Mbits/sec   11    408 KBytes       
[ 13]   9.00-10.00  sec  75.0 MBytes   629 Mbits/sec   20    290 KBytes       
[ 15]   9.00-10.00  sec  47.8 MBytes   401 Mbits/sec    0    341 KBytes       
[ 17]   9.00-10.00  sec   236 MBytes  1.98 Gbits/sec   35    338 KBytes       
[ 19]   9.00-10.00  sec   246 MBytes  2.07 Gbits/sec   62    345 KBytes       
[SUM]   9.00-10.00  sec   981 MBytes  8.23 Gbits/sec  130             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   606 MBytes   509 Mbits/sec  166             sender
[  5]   0.00-10.00  sec   604 MBytes   506 Mbits/sec                  receiver
[  7]   0.00-10.00  sec  1.60 GBytes  1.37 Gbits/sec  461             sender
[  7]   0.00-10.00  sec  1.60 GBytes  1.37 Gbits/sec                  receiver
[  9]   0.00-10.00  sec   778 MBytes   653 Mbits/sec   95             sender
[  9]   0.00-10.00  sec   776 MBytes   650 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   678 MBytes   569 Mbits/sec   92             sender
[ 11]   0.00-10.00  sec   675 MBytes   566 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec   751 MBytes   630 Mbits/sec   58             sender
[ 13]   0.00-10.00  sec   747 MBytes   627 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec   568 MBytes   476 Mbits/sec  100             sender
[ 15]   0.00-10.00  sec   565 MBytes   474 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec  986             sender
[ 17]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.91 GBytes  1.64 Gbits/sec  748             sender
[ 19]   0.00-10.00  sec  1.91 GBytes  1.64 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  9.58 GBytes  8.23 Gbits/sec  2706             sender
[SUM]   0.00-10.00  sec  9.56 GBytes  8.21 Gbits/sec                  receiver

iperf Done.
root@perftest1:/opt/OPNsense_perftest# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[  5] local 10.0.0.20 port 44974 connected to 192.168.10.20 port 5201
[  7] local 10.0.0.20 port 44976 connected to 192.168.10.20 port 5201
[  9] local 10.0.0.20 port 44978 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44980 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44982 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44984 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44986 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44988 connected to 192.168.10.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   189 MBytes  1.59 Gbits/sec   17    449 KBytes       
[  7]   0.00-1.00   sec  91.0 MBytes   764 Mbits/sec    4    361 KBytes       
[  9]   0.00-1.00   sec   108 MBytes   905 Mbits/sec    4    392 KBytes       
[ 11]   0.00-1.00   sec   164 MBytes  1.38 Gbits/sec    4    532 KBytes       
[ 13]   0.00-1.00   sec   127 MBytes  1.07 Gbits/sec    5    436 KBytes       
[ 15]   0.00-1.00   sec   150 MBytes  1.26 Gbits/sec    7    445 KBytes       
[ 17]   0.00-1.00   sec   125 MBytes  1.05 Gbits/sec    2    382 KBytes       
[ 19]   0.00-1.00   sec   178 MBytes  1.49 Gbits/sec   14    603 KBytes       
[SUM]   0.00-1.00   sec  1.11 GBytes  9.50 Gbits/sec   57             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   165 MBytes  1.38 Gbits/sec    0    506 KBytes       
[  7]   1.00-2.00   sec   110 MBytes   921 Mbits/sec    0    528 KBytes       
[  9]   1.00-2.00   sec   115 MBytes   967 Mbits/sec    0    558 KBytes       
[ 11]   1.00-2.00   sec   142 MBytes  1.19 Gbits/sec    0    698 KBytes       
[ 13]   1.00-2.00   sec   121 MBytes  1.01 Gbits/sec    0    602 KBytes       
[ 15]   1.00-2.00   sec   156 MBytes  1.31 Gbits/sec    0    567 KBytes       
[ 17]   1.00-2.00   sec   154 MBytes  1.29 Gbits/sec    0    520 KBytes       
[ 19]   1.00-2.00   sec   158 MBytes  1.33 Gbits/sec    0    632 KBytes       
[SUM]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   160 MBytes  1.34 Gbits/sec    0    520 KBytes       
[  7]   2.00-3.00   sec   109 MBytes   918 Mbits/sec    0    662 KBytes       
[  9]   2.00-3.00   sec   113 MBytes   951 Mbits/sec    0    688 KBytes       
[ 11]   2.00-3.00   sec   140 MBytes  1.17 Gbits/sec    0    827 KBytes       
[ 13]   2.00-3.00   sec   120 MBytes  1.01 Gbits/sec    0    731 KBytes       
[ 15]   2.00-3.00   sec   157 MBytes  1.32 Gbits/sec    0    621 KBytes       
[ 17]   2.00-3.00   sec   159 MBytes  1.33 Gbits/sec    0    612 KBytes       
[ 19]   2.00-3.00   sec   158 MBytes  1.33 Gbits/sec    0    674 KBytes       
[SUM]   2.00-3.00   sec  1.09 GBytes  9.36 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   168 MBytes  1.41 Gbits/sec    0    544 KBytes       
[  7]   3.00-4.00   sec   112 MBytes   944 Mbits/sec    0    773 KBytes       
[  9]   3.00-4.00   sec   110 MBytes   923 Mbits/sec    0    796 KBytes       
[ 11]   3.00-4.00   sec   136 MBytes  1.14 Gbits/sec    0    940 KBytes       
[ 13]   3.00-4.00   sec   118 MBytes   986 Mbits/sec    0    836 KBytes       
[ 15]   3.00-4.00   sec   160 MBytes  1.34 Gbits/sec    0    686 KBytes       
[ 17]   3.00-4.00   sec   158 MBytes  1.32 Gbits/sec    0    675 KBytes       
[ 19]   3.00-4.00   sec   159 MBytes  1.33 Gbits/sec    0    707 KBytes       
[SUM]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   161 MBytes  1.35 Gbits/sec    0    556 KBytes       
[  7]   4.00-5.00   sec   112 MBytes   944 Mbits/sec    0    873 KBytes       
[  9]   4.00-5.00   sec   114 MBytes   954 Mbits/sec    0    891 KBytes       
[ 11]   4.00-5.00   sec   135 MBytes  1.13 Gbits/sec    0   1.01 MBytes       
[ 13]   4.00-5.00   sec   118 MBytes   986 Mbits/sec   12    928 KBytes       
[ 15]   4.00-5.00   sec   159 MBytes  1.33 Gbits/sec    0    694 KBytes       
[ 17]   4.00-5.00   sec   160 MBytes  1.34 Gbits/sec    0    685 KBytes       
[ 19]   4.00-5.00   sec   159 MBytes  1.33 Gbits/sec    0    716 KBytes       
[SUM]   4.00-5.00   sec  1.09 GBytes  9.37 Gbits/sec   12             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   150 MBytes  1.26 Gbits/sec    0    562 KBytes       
[  7]   5.00-6.00   sec   126 MBytes  1.06 Gbits/sec    0    968 KBytes       
[  9]   5.00-6.00   sec   126 MBytes  1.06 Gbits/sec    0    986 KBytes       
[ 11]   5.00-6.00   sec   140 MBytes  1.17 Gbits/sec    0   1.10 MBytes       
[ 13]   5.00-6.00   sec   129 MBytes  1.08 Gbits/sec    0   1018 KBytes       
[ 15]   5.00-6.00   sec   150 MBytes  1.26 Gbits/sec    0    711 KBytes       
[ 17]   5.00-6.00   sec   146 MBytes  1.23 Gbits/sec    0    700 KBytes       
[ 19]   5.00-6.00   sec   150 MBytes  1.26 Gbits/sec    0    732 KBytes       
[SUM]   5.00-6.00   sec  1.09 GBytes  9.37 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   141 MBytes  1.18 Gbits/sec    0    567 KBytes       
[  7]   6.00-7.00   sec   139 MBytes  1.16 Gbits/sec    0   1.04 MBytes       
[  9]   6.00-7.00   sec   136 MBytes  1.14 Gbits/sec    0   1.06 MBytes       
[ 11]   6.00-7.00   sec   141 MBytes  1.18 Gbits/sec    0   1.17 MBytes       
[ 13]   6.00-7.00   sec   138 MBytes  1.15 Gbits/sec    0   1.09 MBytes       
[ 15]   6.00-7.00   sec   141 MBytes  1.18 Gbits/sec    0    719 KBytes       
[ 17]   6.00-7.00   sec   141 MBytes  1.18 Gbits/sec    0    708 KBytes       
[ 19]   6.00-7.00   sec   140 MBytes  1.17 Gbits/sec    0    739 KBytes       
[SUM]   6.00-7.00   sec  1.09 GBytes  9.37 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    567 KBytes       
[  7]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0   1.13 MBytes       
[  9]   7.00-8.00   sec   138 MBytes  1.15 Gbits/sec    0   1.14 MBytes       
[ 11]   7.00-8.00   sec   142 MBytes  1.20 Gbits/sec    0   1.18 MBytes       
[ 13]   7.00-8.00   sec   138 MBytes  1.15 Gbits/sec    0   1.17 MBytes       
[ 15]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    724 KBytes       
[ 17]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    712 KBytes       
[ 19]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    742 KBytes       
[SUM]   7.00-8.00   sec  1.09 GBytes  9.37 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   188 MBytes  1.58 Gbits/sec  183    563 KBytes       
[  7]   8.00-9.00   sec   132 MBytes  1.11 Gbits/sec  285    645 KBytes       
[  9]   8.00-9.00   sec   138 MBytes  1.15 Gbits/sec  270    694 KBytes       
[ 11]   8.00-9.00   sec   109 MBytes   912 Mbits/sec  251    488 KBytes       
[ 13]   8.00-9.00   sec   104 MBytes   870 Mbits/sec  257    477 KBytes       
[ 15]   8.00-9.00   sec   150 MBytes  1.26 Gbits/sec   96    507 KBytes       
[ 17]   8.00-9.00   sec   142 MBytes  1.20 Gbits/sec  254    453 KBytes       
[ 19]   8.00-9.00   sec   154 MBytes  1.29 Gbits/sec  150    515 KBytes       
[SUM]   8.00-9.00   sec  1.09 GBytes  9.37 Gbits/sec  1746             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   122 MBytes  1.02 Gbits/sec    0    565 KBytes       
[  7]   9.00-10.00  sec   148 MBytes  1.24 Gbits/sec    0    792 KBytes       
[  9]   9.00-10.00  sec   142 MBytes  1.20 Gbits/sec    0    823 KBytes       
[ 11]   9.00-10.00  sec  98.8 MBytes   828 Mbits/sec    0    586 KBytes       
[ 13]   9.00-10.00  sec  91.2 MBytes   765 Mbits/sec    0    565 KBytes       
[ 15]   9.00-10.00  sec   149 MBytes  1.25 Gbits/sec    0    602 KBytes       
[ 17]   9.00-10.00  sec   194 MBytes  1.63 Gbits/sec    0    636 KBytes       
[ 19]   9.00-10.00  sec   120 MBytes  1.01 Gbits/sec    0    548 KBytes       
[SUM]   9.00-10.00  sec  1.04 GBytes  8.93 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.55 GBytes  1.33 Gbits/sec  200             sender
[  5]   0.00-10.00  sec  1.54 GBytes  1.33 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.19 GBytes  1.02 Gbits/sec  289             sender
[  7]   0.00-10.00  sec  1.19 GBytes  1.02 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  1.21 GBytes  1.04 Gbits/sec  274             sender
[  9]   0.00-10.00  sec  1.21 GBytes  1.04 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.32 GBytes  1.13 Gbits/sec  255             sender
[ 11]   0.00-10.00  sec  1.31 GBytes  1.13 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  1.17 GBytes  1.01 Gbits/sec  274             sender
[ 13]   0.00-10.00  sec  1.17 GBytes  1.01 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec  103             sender
[ 15]   0.00-10.00  sec  1.47 GBytes  1.27 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec  256             sender
[ 17]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec  164             sender
[ 19]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.9 GBytes  9.34 Gbits/sec  1815             sender
[SUM]   0.00-10.00  sec  10.9 GBytes  9.32 Gbits/sec                  receiver

iperf Done.

Retesting a couple of times does show a spread with an average of about 7-9Gbps.

With the new FreeBSD 13.1 kernel the performance averages at about 8.7Gbps (standard MTU) and fluctuates a lot less. So while a bit lower than our peak, it will likely result in higher throughput on average.

Hope this clears things up for everyone.

And is this with a DEC3840? If you're seeing 7-9Gbps average, where does the 14.6Gbps firewall throughput number come from?

All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.

Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.


Cheers,
Franco

Well, Deciso are the ones that posted the numbers, are they not? If the numbers are saying that "14.6Gbps" is a total of 7Gbps coming into the box, and 7Gbps going out of the box from source to destination, then that is *incredibly* misleading especially for a 1000 euro+ piece of equipment. If I'm misinterpreting those numbers and I should, realistically, see a full near 10Gb inter-VLAN routing performance from this box (given I can get this on other routers, my switching hardware and clients aren't the problem), then there's something either wrong with my config or there's some tuning that needs to be involved.

Or perhaps is there a special version of OPNsense that these are supposed to run pre-tuned to properly achieve the advertised performance numbers?

I don't think there are significant changes on 22.1.x and I'm not sure why people sometimes wonder if there are... release notes state what is being worked on and source repositories have annotated changes too.

As a general principle: performance gain is nice, but stability is much nicer still.


Cheers,
Franco

All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.

Or am I looking at the specifications of this firewall incorrectly. It mentions 14.6Gbps throughput, does that include inbound *and* outbound or should I reasonably expect full near 10Gb routing performance for traffic going both in and out of the box?