PC Engines APU2 1Gbit traffic not achievable

[FrenchFries]

When testing on the same VLAN (so OPNsense does nothing):

Code Select Expand

iperf3 -R -c 10.90.70.250
Connecting to host 10.90.70.250, port 5201
Reverse mode, remote host 10.90.70.250 is sending
[  5] local 10.90.70.110 port 42160 connected to 10.90.70.250 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  93.5 MBytes   784 Mbits/sec                 
[  5]   1.00-2.00   sec  93.6 MBytes   785 Mbits/sec                 
[  5]   2.00-3.00   sec  93.6 MBytes   786 Mbits/sec                 
[  5]   3.00-4.00   sec  94.2 MBytes   790 Mbits/sec                 
[  5]   4.00-5.00   sec  95.8 MBytes   803 Mbits/sec                 
[  5]   5.00-6.00   sec  95.1 MBytes   798 Mbits/sec                 
[  5]   6.00-7.00   sec  95.8 MBytes   803 Mbits/sec                 
[  5]   7.00-8.00   sec  96.1 MBytes   806 Mbits/sec                 
[  5]   8.00-9.00   sec  95.9 MBytes   805 Mbits/sec                 
[  5]   9.00-10.00  sec  96.1 MBytes   806 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   950 MBytes   797 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   950 MBytes   797 Mbits/sec                  receiver

This is close to the speed of inter-VLAN routing with OPNsense.
So OPNsense is very efficient in inter-VLAN routing.

And just to confirm, speed with direct link is close to 1Gb/s:

Code Select Expand

iperf3 -R -p 5206 -c bouygues.iperf.fr
Connecting to host bouygues.iperf.fr, port 5206
Reverse mode, remote host bouygues.iperf.fr is sending
[  5] local 192.168.1.158 port 58658 connected to 89.84.1.222 port 5206
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   111 MBytes   930 Mbits/sec                 
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec                 
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   4.00-5.00   sec   112 MBytes   942 Mbits/sec                 
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec                 
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec                 
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec                  receiver

[FrenchFries]

OK, I get it. When connecting to the Internet, data is going through WAN on a different network interface. However it is not clear why it is SO much slower than inter-VLAN routing with OPNsense firewall.

[realex]

Good evening!

I have an apu2 with 4 NICs and 1000 MBit/s from my ISP (Cable-Modem - bridge mode - opnsense/apu2). I get ~ 650 MBit/s, which gives a gap of ~ 250 MBit/s i pay, but i don't get...

My thoughts to have apu2 + opnsense = 1000 MBit/s:

• use a switch with link aggregation for - let's say - 2 ports for WAN and another 2 ports for LAN
• Have this two trunks each in a different vlan on that switch, VLAN 1 = LAN, VLAN 2 = WAN
• create a lagg  "lagg0" with 2 ports (igb2 + igb3) on opnsens' side and assign the lagg to "WAN"
• do the same for "lagg1" with igb0 + igb1, assign to "LAN"
• connect the bridge-mode-cable-modem to another port at the switch with configured VLAN 2 (WAN)
• connect all LAN CLients to ports with VLAN 1 (LAN)

Would this solve the problem or do i have an error in reasoning?

Maybe someone tested in this direction?



Alex

[spi39492]

I struggle the same issue. My setup is opnsense 21.1 virtualized as XEN HVM. I'Ve got virtual switches for different vlans. Opnsense just uses plain interfaces. Only vlan is for pppoe Internet uplink.

iperf gives me something like 19 Gbps domU (= virtual machine) to domU in the same vlan. Through opnsense this goes down as to ~700 Mbps. LAN-device on a physical switch to a domU is something around ~300 Mbps. Internet with one single session is somehow limited to around ~250 Mbps.

I checked some of the suggestions from https://bsdrp.net/documentation/technical_docs/performance, namely
net.inet6.ip6.redirect = 0
net.inet.ip.redirect was already 0
hw.igb.rx_process_limit = -1
hw.igb.tx_process_limit = -1
kern.random.harvest.mask = 351

It seems to tune the performance a bit - but still need to investigate that further.
Internet goes up to ~400 Mbps.
LAN - domU ~510 Mbps
domU - domU stays around ~700 Mbps.

Found some other BSD relating optimization stuff. need to look into it.
https://calomel.org/network_performance.html
https://calomel.org/freebsd_network_tuning.html

[spi39492]

Found some other BSD relating optimization stuff. need to look into it.
https://calomel.org/network_performance.html
https://calomel.org/freebsd_network_tuning.html

Playing around with

Code Select Expand

net.inet.ip.ifq.maxlen
kern.ipc.maxsockbuf
net.inet.tcp.recvbuf_inc
net.inet.tcp.recvbuf_max
net.inet.tcp.recvspace
net.inet.tcp.sendbuf_inc
net.inet.tcp.sendbuf_max
net.inet.tcp.sendspace
net.inet.tcp.tso

doesn't seem to change anything. Not sure exactly, if all settings get set correctly, sysctl -a doesn't show them all.

[spi39492]

A quick check with openwrt 19.07 as domU on same server gives LAN - domU 890 Mbps. So almost Gbit wire speed.

With a slight modification - e1000 NICs as virtualized network adapters I get
- LAN - domU wire speed of 940 Mbps
- domU - domU 7700 Mbps (7.7 Gbps)

[thowe]

I think you are mixing two things in this thread:

This thread is about the optimization of APU-based hardware devices, which can only do 1GBit/s when specifically optimized on FreeBSD.

The other issue could be performance problems of 21.1 on XEN based virtualization at best. There are already more participants here in the forum with this observation.

I would rather not discuss the XEN issue in this APU thread, as you are more likely to meet users who are also concerned.

[Ricardo]

I think you are mixing two things in this thread:

This thread is about the optimization of APU-based hardware devices, which can only do 1GBit/s when specifically optimized on FreeBSD.

The other issue could be performance problems of 21.1 on XEN based virtualization at best. There are already more participants here in the forum with this observation.

I would rather not discuss the XEN issue in this APU thread, as you are more likely to meet users who are also concerned.

Thanks, I was about to ask the same thing (dont mix 2 different things in 1 thread)

[spi39492]

I think you are mixing two things in this thread:

This thread is about the optimization of APU-based hardware devices, which can only do 1GBit/s when specifically optimized on FreeBSD.

The other issue could be performance problems of 21.1 on XEN based virtualization at best. There are already more participants here in the forum with this observation.

I would rather not discuss the XEN issue in this APU thread, as you are more likely to meet users who are also concerned.

Understood that this is specifically on APU-based boards. I observe also performnce issues and couldn't find anything somehow related for Xen. That's why I am interested in your observations - I'd give the performance tuning tips a try.

[thowe]

Start with e.g. these (from this thread):

net.inet6.ip6.redirect = 0
net.inet.ip.redirect = 0
hw.igb.rx_process_limit = -1 (these are hardware dependent and will probably not match your NIC in the VM)
hw.igb.tx_process_limit = -1 (these are hardware dependent and will probably not match your NIC in the VM)

[spi39492]

Start with e.g. these (from this thread):

net.inet6.ip6.redirect = 0
net.inet.ip.redirect = 0
hw.igb.rx_process_limit = -1 (these are hardware dependent and will probably not match your NIC in the VM)
hw.igb.tx_process_limit = -1 (these are hardware dependent and will probably not match your NIC in the VM)

Thx - have these. Helped me to increase the speed (as mentioned in one of my posts). But still far away from Gbit.

[mater]

Switch to a Odroid H2+, it achieves Gigabit with no issue.
I gonna sell my APU Board

[Patrick M. Hausen]

Switch to a Odroid H2+, it achieves Gigabit with no issue.

Realtek LAN? Sorry, I'll pass. Yes, I see the problem with the apu, but Odroid is not the solution, IMHO.
Protectli looks good ... a bit difficult to find in Europe, though.

[Ricardo]

Switch to a Odroid H2+, it achieves Gigabit with no issue.

Realtek LAN? Sorry, I'll pass. Yes, I see the problem with the apu, but Odroid is not the solution, IMHO.
Protectli looks good ... a bit difficult to find in Europe, though.

I am no fan of realtek either (that lame company has still bad reputation in 2021 when that really came from the 90s mid 00s), but just because it has a shiny intel logo stick on it, doesnt mean it will have superb performance. Just looking at the fancy i225-V (B1 was broken design and promised B2 is still faulty, now even newer B3 fixed it really) fiasco with 2,5Gbit is literally broken and the workaround was to switch back to 1Gbit max. While the black sheep realtek RTL8125BG just works.---

[rmayr]

I have been struggling with performance on the APU4. While in initial testing, I was able to get around 700MBit/s with 2 iperf3 streams, with my fully configured firewall rule set (but minimal rules for the actual path I am testing), I am now down to around 250MBit/s and can't get it higher.

Settings from this thread, from https://www.reddit.com/r/homelab/comments/fciqid/hardware_for_my_home_pfsense_router/fjfl8ic/, and from https://teklager.se/en/knowledge-base/opnsense-performance-optimization/ have all been applied, and I am not sure when the performance drop occurred.

What is the best way to debug what's going on here? This is quite frustrating, as I know the hardware to be capable of full GBit/s routing.