PC Engines APU2 1Gbit traffic not achievable

[Ricardo]

Thanks. Has someone experiences with this?
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856#c11

In the last post in that topic, the freebsd maintaner Eugene Grosbein closed the bug with "Closed Works As Intended", so they dont acknowledge this is a bug, and seems will never be fixed. You have to forget Pcengines APU2 for PPPoE WAN and 1Gbit. Unless PCengines release a new PCB design with a much stronger (2Ghz+) embedded APU.

[telefonmann]

Yes, I've set "net.isr.maxthreads" and "net.isr.numthreads" to the number of cores (4) and net.isr.dispatch to "deferred". This led to a a slight performance increase (~10%). I will now try to offload PPPoE stuff from the firewall to the modem (my modem has this option) and see what happens.

[telefonmann]

Just for testing today I deactivated my PPPoE interface in OPNsense and - guess what - the performance on the other two (non-PPPoE) interfaces DOUBLED instantly after a reboot (sorry to say that after reactivating PPPoE the "boost" went away - even without rebooting).
So it looks like you can achieve 1GBit speed only if you don't use PPPoE - regardless of the FW OS used (forget about the others, they will have the same problem).
I bought a DrayTek 165 (VDSL2+ 35b) modem now which is capable of handling the whole PPPoE stuff on its own. This way the OPNsense will only get IP traffic and it should finally work.

[iam]

Yes, I've set "net.isr.maxthreads" and "net.isr.numthreads" to the number of cores (4) and net.isr.dispatch to "deferred". This led to a a slight performance increase (~10%). I will now try to offload PPPoE stuff from the firewall to the modem (my modem has this option) and see what happens.

I can recommend these settings. Before the upgrade to 20.7 I had measured always values between 250 and 280 MBit/s. After the update I measured values between 180 and 220 MBit/s only. Now I measure values up to 313 MBit/s. Our contract says 300MBit/s, so that's really nice :)

[dave]

I recommend these setting to

[rcmcronny]

Hi,

Just for testing today I deactivated my PPPoE interface in OPNsense and - guess what - the performance on the other two (non-PPPoE)
[...]
I bought a DrayTek 165 (VDSL2+ 35b) modem now which is capable of handling the whole PPPoE stuff on its own. This way the OPNsense will only get IP traffic and it should finally work.

I have a vigor 130, this should also work with that, could you please give a link on what to change in the config. Actually i do vlan tagging on the vigor only. Do the pppoe stuff at the modem with vlan tagging is perhaps a better method, would like to dive in, if it is a better setup for me.

Thanks,
Ronny

[dave]

I wanted to try the same thing but it looks like the EU and US version of the Vigor 130 are actually different.
The US version can act as a router (enabling it to handle PPPoE auth and encapsulation itself), whereas the EU version doesn't have this functionality.

[telefonmann]

@rcmcronny, http://www.draytektr.com/documents/product/619F295B-B506-2CF1-D9115EA3B629181F.pdf
First set operation mode to "router mode", after this the device will reboot. Then just use the wizard (pages 7 ff. in the manual)

[FrenchFries]

I am using an older APU1C model with only two cores.
I understand only one core is being used for routing.

Just a remark, testing network bandwidth with Ikoula Speedtest is not the right methodology, as it is very unaccurate.
Using Ikoula speedtest, my result was 440 Mbits/s downstream.

But the accurate speed measured with iperf3 is 571 Mbits/s

Code Select Expand

iperf3 -p 9222 -c bouygues.iperf.fr
Connecting to host bouygues.iperf.fr, port 9222
[  5] local 10.90.20.1 port 60560 connected to 89.84.1.222 port 9222
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  69.1 MBytes   579 Mbits/sec  268    462 KBytes       
[  5]   1.00-2.00   sec  68.8 MBytes   577 Mbits/sec    0    567 KBytes       
[  5]   2.00-3.00   sec  67.5 MBytes   566 Mbits/sec    2    461 KBytes       
[  5]   3.00-4.00   sec  68.8 MBytes   577 Mbits/sec    0    563 KBytes       
[  5]   4.00-5.00   sec  67.5 MBytes   566 Mbits/sec    0    648 KBytes       
[  5]   5.00-6.00   sec  67.5 MBytes   566 Mbits/sec    2    544 KBytes       
[  5]   6.00-7.00   sec  70.0 MBytes   587 Mbits/sec    0    632 KBytes       
[  5]   7.00-8.00   sec  67.5 MBytes   566 Mbits/sec    2    533 KBytes       
[  5]   8.00-9.00   sec  67.5 MBytes   566 Mbits/sec    0    621 KBytes       
[  5]   9.00-10.00  sec  68.8 MBytes   577 Mbits/sec    2    510 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   683 MBytes   573 Mbits/sec  276             sender
[  5]   0.00-10.00  sec   680 MBytes   571 Mbits/sec                  receiver

So I can confirm that the APU1 with older core and older NIC can achieve 571 Mbits downstream.
This is OPNsense latest version 20.7.
I am connecting from a GNU/Linux laptop using an RJ-45 wire and IPv4.
OPNsense is connected to the fiber router with and RJ-45 wire and IPv4 with NAT.

iperf3 also has an option to using multiple connection streams, which is -P 2 for two cores :

Code Select Expand

iperf3 -P 2 -p 9222 -c bouygues.iperf.fr
Connecting to host bouygues.iperf.fr, port 9222
[  5] local 10.90.20.1 port 38612 connected to 89.84.1.222 port 9222
[  7] local 10.90.20.1 port 38614 connected to 89.84.1.222 port 9222
^[[A[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  25.4 MBytes   213 Mbits/sec   12    229 KBytes       
[  7]   0.00-1.00   sec  46.0 MBytes   386 Mbits/sec   64    318 KBytes       
[SUM]   0.00-1.00   sec  71.3 MBytes   598 Mbits/sec   76             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  31.0 MBytes   260 Mbits/sec    0    314 KBytes       
[  7]   1.00-2.00   sec  37.2 MBytes   312 Mbits/sec    2    279 KBytes       
[SUM]   1.00-2.00   sec  68.2 MBytes   572 Mbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  37.0 MBytes   311 Mbits/sec    1    290 KBytes       
[  7]   2.00-3.00   sec  32.3 MBytes   271 Mbits/sec    1    263 KBytes       
[SUM]   2.00-3.00   sec  69.3 MBytes   582 Mbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  34.8 MBytes   292 Mbits/sec    1    263 KBytes       
[  7]   3.00-4.00   sec  31.9 MBytes   268 Mbits/sec    1    245 KBytes       
[SUM]   3.00-4.00   sec  66.7 MBytes   560 Mbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  34.2 MBytes   287 Mbits/sec    0    348 KBytes       
[  7]   4.00-5.00   sec  33.4 MBytes   280 Mbits/sec    1    239 KBytes       
[SUM]   4.00-5.00   sec  67.6 MBytes   567 Mbits/sec    1             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  39.6 MBytes   333 Mbits/sec    1    307 KBytes       
[  7]   5.00-6.00   sec  28.5 MBytes   239 Mbits/sec    2    226 KBytes       
[SUM]   5.00-6.00   sec  68.1 MBytes   571 Mbits/sec    3             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  39.3 MBytes   330 Mbits/sec    0    389 KBytes       
[  7]   6.00-7.00   sec  30.0 MBytes   251 Mbits/sec    0    311 KBytes       
[SUM]   6.00-7.00   sec  69.3 MBytes   581 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  36.5 MBytes   306 Mbits/sec    1    355 KBytes       
[  7]   7.00-8.00   sec  30.9 MBytes   259 Mbits/sec    1    305 KBytes       
[SUM]   7.00-8.00   sec  67.4 MBytes   565 Mbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  36.7 MBytes   308 Mbits/sec    1    329 KBytes       
[  7]   8.00-9.00   sec  32.0 MBytes   268 Mbits/sec    1    293 KBytes       
[SUM]   8.00-9.00   sec  68.7 MBytes   577 Mbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  35.2 MBytes   295 Mbits/sec    1    305 KBytes       
[  7]   9.00-10.00  sec  31.9 MBytes   268 Mbits/sec    1    279 KBytes       
[SUM]   9.00-10.00  sec  67.1 MBytes   563 Mbits/sec    2             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   350 MBytes   293 Mbits/sec   18             sender
[  5]   0.00-10.01  sec   348 MBytes   292 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   334 MBytes   280 Mbits/sec   74             sender
[  7]   0.00-10.01  sec   331 MBytes   278 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec   684 MBytes   574 Mbits/sec   92             sender
[SUM]   0.00-10.01  sec   679 MBytes   569 Mbits/sec                  receiver

But this gave me the same results (probably a limitation of my hardware NICs ?).

IMHO, you should make a test using iperf3 for accurate results.
iperf3 should be running on client, not directly on OPNsense of course.

Edit : iperf3 should be used with -R option to ask the server to send information, otherwize you are testing upload speed. My upload speed is around 600Mbits/s, so I need to retest downloading with -R option.

[chemlud]

I wanted to try the same thing but it looks like the EU and US version of the Vigor 130 are actually different.
The US version can act as a router (enabling it to handle PPPoE auth and encapsulation itself), whereas the EU version doesn't have this functionality.

Nope, the "EU-version" of the Vigor 130 (if there is such a thing) can act as router or modem (bridged mode), but as it is most often used as modem, it comes pre-configured in modem mode. Let the sense do the PPPoE and VLAN is the preferred configuration.

[FrenchFries]

Here is a more accurate downloading speedtest with iperf3 and one thread.
I used -P option to test downloading speed, not uploading :

Code Select Expand

iperf3 -R -P 1 -c bouygues.iperf.fr
Connecting to host bouygues.iperf.fr, port 5201
Reverse mode, remote host bouygues.iperf.fr is sending
[  5] local 10.90.20.1 port 39286 connected to 89.84.1.222 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  53.7 MBytes   450 Mbits/sec                 
[  5]   1.00-2.00   sec  52.8 MBytes   443 Mbits/sec                 
[  5]   2.00-3.00   sec  55.3 MBytes   464 Mbits/sec                 
[  5]   3.00-4.00   sec  61.4 MBytes   515 Mbits/sec                 
[  5]   4.00-5.00   sec  54.3 MBytes   456 Mbits/sec                 
[  5]   5.00-6.00   sec  53.0 MBytes   445 Mbits/sec                 
[  5]   6.00-7.00   sec  53.8 MBytes   451 Mbits/sec                 
[  5]   7.00-8.00   sec  48.3 MBytes   405 Mbits/sec                 
[  5]   8.00-9.00   sec  54.6 MBytes   458 Mbits/sec                 
[  5]   9.00-10.00  sec  54.4 MBytes   457 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   550 MBytes   461 Mbits/sec  31826             sender
[  5]   0.00-10.00  sec   542 MBytes   455 Mbits/sec                  receiver


However, with two threads, I have the same results :

iperf3 -R -P 2 -c bouygues.iperf.fr
Connecting to host bouygues.iperf.fr, port 5201
Reverse mode, remote host bouygues.iperf.fr is sending
[  5] local 10.90.20.1 port 40064 connected to 89.84.1.222 port 5201
[  7] local 10.90.20.1 port 40066 connected to 89.84.1.222 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  28.8 MBytes   241 Mbits/sec                 
[  7]   0.00-1.00   sec  25.7 MBytes   216 Mbits/sec                 
[SUM]   0.00-1.00   sec  54.5 MBytes   457 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  27.5 MBytes   231 Mbits/sec                 
[  7]   1.00-2.00   sec  27.6 MBytes   232 Mbits/sec                 
[SUM]   1.00-2.00   sec  55.1 MBytes   462 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  23.6 MBytes   198 Mbits/sec                 
[  7]   2.00-3.00   sec  29.4 MBytes   246 Mbits/sec                 
[SUM]   2.00-3.00   sec  53.0 MBytes   444 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  23.8 MBytes   200 Mbits/sec                 
[  7]   3.00-4.00   sec  26.9 MBytes   226 Mbits/sec                 
[SUM]   3.00-4.00   sec  50.7 MBytes   426 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  27.3 MBytes   229 Mbits/sec                 
[  7]   4.00-5.00   sec  23.7 MBytes   199 Mbits/sec                 
[SUM]   4.00-5.00   sec  51.0 MBytes   428 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  19.8 MBytes   166 Mbits/sec                 
[  7]   5.00-6.00   sec  30.2 MBytes   253 Mbits/sec                 
[SUM]   5.00-6.00   sec  50.0 MBytes   419 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  16.9 MBytes   142 Mbits/sec                 
[  7]   6.00-7.00   sec  34.6 MBytes   290 Mbits/sec                 
[SUM]   6.00-7.00   sec  51.5 MBytes   432 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  16.9 MBytes   142 Mbits/sec                 
[  7]   7.00-8.00   sec  34.0 MBytes   285 Mbits/sec                 
[SUM]   7.00-8.00   sec  50.8 MBytes   426 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  15.8 MBytes   133 Mbits/sec                 
[  7]   8.00-9.00   sec  38.1 MBytes   320 Mbits/sec                 
[SUM]   8.00-9.00   sec  53.9 MBytes   452 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  14.2 MBytes   119 Mbits/sec                 
[  7]   9.00-10.00  sec  40.1 MBytes   336 Mbits/sec                 
[SUM]   9.00-10.00  sec  54.3 MBytes   455 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   219 MBytes   183 Mbits/sec  21640             sender
[  5]   0.00-10.00  sec   215 MBytes   180 Mbits/sec                  receiver
[  7]   0.00-10.01  sec   315 MBytes   264 Mbits/sec  29816             sender
[  7]   0.00-10.00  sec   310 MBytes   260 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec   534 MBytes   447 Mbits/sec  51456             sender
[SUM]   0.00-10.00  sec   525 MBytes   440 Mbits/sec                  receiver

I also testing on local server (connected on different VLAN with different subnets, so OPNsense is acting as NAT):

iperf3 -R -c 10.90.70.250
Connecting to host 10.90.70.250, port 5201
Reverse mode, remote host 10.90.70.250 is sending
[  5] local 10.90.20.1 port 54348 connected to 10.90.70.250 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  85.7 MBytes   719 Mbits/sec                 
[  5]   1.00-2.00   sec  86.8 MBytes   728 Mbits/sec                 
[  5]   2.00-3.00   sec  86.3 MBytes   724 Mbits/sec                 
[  5]   3.00-4.00   sec  85.8 MBytes   720 Mbits/sec                 
[  5]   4.00-5.00   sec  84.9 MBytes   712 Mbits/sec                 
[  5]   5.00-6.00   sec  81.9 MBytes   687 Mbits/sec                 
[  5]   6.00-7.00   sec  88.6 MBytes   744 Mbits/sec                 
[  5]   7.00-8.00   sec  87.6 MBytes   735 Mbits/sec                 
[  5]   8.00-9.00   sec  85.7 MBytes   719 Mbits/sec                 
[  5]   9.00-10.00  sec  87.3 MBytes   732 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   864 MBytes   724 Mbits/sec   19             sender
[  5]   0.00-10.00  sec   861 MBytes   722 Mbits/sec                  receiver

Here I can achieve 722 Mbits, which is pretty good for an older APU1c platform.
Same results with 2 threads.

Two remarks:

1) I cannot explain why iperf3 is so much faster on a local iperf3 server with NAT.

2) OPNsense does not seem to support multiple core routing, as speed is not  higher with two treads.
I even tested with two clients and there is roughly the same speed.

Do I miss something in my OPNsense settings?
I would expect speed to be higher with two iperf3 threads.
Or is pf single threaded on OPNsense?

[dave]

Nope, the "EU-version" of the Vigor 130 (if there is such a thing) can act as router or modem (bridged mode), but as it is most often used as modem, it comes pre-configured in modem mode. Let the sense do the PPPoE and VLAN is the preferred configuration.

I emailed Draytek:

Regarding the authentication, the DrayTek UK Vigor 130 was designed to support bridge mode out of the box. You can consider the Vigor 2762 series that can handle PPP authentication.

They didn't outright say no I guess.

[hushcoden]

I emailed Draytek:

Regarding the authentication, the DrayTek UK Vigor 130 was designed to support bridge mode out of the box. You can consider the Vigor 2762 series that can handle PPP authentication.

They didn't outright say no I guess.

Have you tried not to use the BT firmware but the alternative one?

[dave]

Nope.  Have you?

[FrenchFries]

Any idea why speed is higher with a local iperf3 server on a different subnet.
I tested with IPv6 (no NAT) to make sure no NAT was used, the same difference applies:

Client : Linux laptop
Server : bouygues.iperf.fr
Firewall : APU1c OPNsense 20.7 latest.
WAN connected to Gig fiber.
Same results for IPv4 and IPv6, so NAT is not the issue.

iperf3 -6 -R -p 5206 -c bouygues.iperf.fr
Connecting to host bouygues.iperf.fr, port 5206
Reverse mode, remote host bouygues.iperf.fr is sending
[  5] local 2a01:e0a:2ed:6231:b11b:ac7c:1c41:b3f7 port 53940 connected to 2001:860:deff:1000::2 port 5206
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  54.9 MBytes   461 Mbits/sec                 
[  5]   1.00-2.00   sec  59.2 MBytes   497 Mbits/sec                 
[  5]   2.00-3.00   sec  55.5 MBytes   466 Mbits/sec                 
[  5]   3.00-4.00   sec  53.1 MBytes   446 Mbits/sec                 
[  5]   4.00-5.00   sec  52.6 MBytes   442 Mbits/sec                 
[  5]   5.00-6.00   sec  55.4 MBytes   465 Mbits/sec                 
[  5]   6.00-7.00   sec  53.0 MBytes   445 Mbits/sec                 
[  5]   7.00-8.00   sec  51.9 MBytes   435 Mbits/sec                 
[  5]   8.00-9.00   sec  49.0 MBytes   411 Mbits/sec                 
[  5]   9.00-10.00  sec  58.2 MBytes   488 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   549 MBytes   460 Mbits/sec  40855             sender
[  5]   0.00-10.00  sec   543 MBytes   455 Mbits/sec                  receiver

Client : Linux laptop
Server : Another APU1c running Debian Linux on a separate isolated VLAN (firewall is routing).
Firewall : APU1c OPNsense 20.7 latest routing between VLANs.

Code Select Expand

iperf3 -R -c 10.90.70.250
Connecting to host 10.90.70.250, port 5201
Reverse mode, remote host 10.90.70.250 is sending
[  5] local 10.90.20.1 port 56430 connected to 10.90.70.250 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  87.1 MBytes   731 Mbits/sec                 
[  5]   1.00-2.00   sec  87.9 MBytes   737 Mbits/sec                 
[  5]   2.00-3.00   sec  82.2 MBytes   689 Mbits/sec                 
[  5]   3.00-4.00   sec  83.5 MBytes   701 Mbits/sec                 
[  5]   4.00-5.00   sec  88.2 MBytes   740 Mbits/sec                 
[  5]   5.00-6.00   sec  87.2 MBytes   731 Mbits/sec                 
[  5]   6.00-7.00   sec  87.7 MBytes   736 Mbits/sec                 
[  5]   7.00-8.00   sec  82.8 MBytes   695 Mbits/sec                 
[  5]   8.00-9.00   sec  88.4 MBytes   741 Mbits/sec                 
[  5]   9.00-10.00  sec  90.4 MBytes   758 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   869 MBytes   728 Mbits/sec  577             sender
[  5]   0.00-10.00  sec   865 MBytes   726 Mbits/sec                  receiver

This is not clear to me why there is such a different.
Why is routing between VLANs wit firewall so much faster than routing with IPv6 and gigabyte fiber?
To confirm: my VLANs are not communicating directly on switch.