PPPoE performance with current setup

[mattuz]

Hello everyone I have a Lenovo Tiny m910q with I3-7100T and 4GB of RAM. I installed via pci-e a x4 nics rtl8125 card.
In the next month my local isp will hook up fiber optic with up to 2.5GB speed, and I'm getting ready all the hardware.

I started doing some lcoal test and iperf3 between opnsense and a pc results 2.37Gbit/s that seem ok. But using the same pc with linux and creating a PPPoE server is giving awful performance, I cannot get it past 500Mbit/s even with all the optimization recommanded online.

I tried two different pc for the PPPoE server to make sure it wasn't this side bootlenecking (even on a ryzen 9) but I can't get it past that limit. Looking at opnsense host cpu usage during the test it doesn't go over 30% so I don't really get it...

Unfortunatly I do not have any other scenario to test.. I would like to get your input it what might be the cause of this... I would like to get this Lenovo Tiny (opnsense host) to be 2.5GB capable as soon as the connectivity is hooked up

[pfry]

PPPoE is apparently single-thread on FreeBSD... but that should only halve your performance (or thereabouts) on that dual-core CPU. The CPU load figure may be misleading, but even proving that premise may not be enlightening. You could try OPNsense on a machine with greater single-thread performance, but a 3.4GHz Skabylake shouldn't be too bad. A Zen 5 (e.g. 9600X at 5.4GHz) might double it (on this workload). I haven't used PPPoE (knocking on wood...) myself (even my old DSLs used bridging or routing for static IPs, which I've always had; my only PPP link was ISDN, and 128k was pretty easy to achieve).

Edit: For the heck of it (quoting myself), you might try "netstat" - "-m", "-i", perhaps "-Q", "-T", "-x", "-s" options (most have to be issued separately), and see if anything looks bad. I'm not sure if these will provide useful data for a PPPoE device.

[meyergru]

Did you use the Realtek vendor or the FreeBSD native driver on OpnSense?

[mattuz]

Edit: For the heck of it (quoting myself), you might try "netstat" - "-m", "-i", perhaps "-Q", "-T", "-x", "-s" options (most have to be issued separately), and see if anything looks bad. I'm not sure if these will provide useful data for a PPPoE device.

Thanks I will try these commands and update the thread with the results.

Did you use the Realtek vendor or the FreeBSD native driver on OpnSense?

I'm using os-realtek-re drivers. The card plugged in without those didn't even show the interfaces on the GUI. I had "pci3: <network, ethernet> at device 0.0 (no driver attached"

Looking at netstat -Q it seems CPU 2 is handling major traffic compared to others. (this is during an iperf3 test)

Code Select Expand

=== netstat -Q ===
Configuration:
Setting                        Current        Limit
Thread count                         4            4
Default queue limit                256        10240
Dispatch policy               deferred          n/a
Threads bound to CPUs          enabled          n/a

Protocols:
Name   Proto QLimit Policy Dispatch Flags
ip         1   1000    cpu   hybrid   C--
igmp       2    256 source  default   ---
rtsock     3    256 source  default   ---
arp        4    256 source  default   ---
ether      5    256    cpu   direct   C--
ip6        6   1000    cpu   hybrid   C--
ip_direct     9    256    cpu   hybrid   C--
ip6_direct    10    256    cpu   hybrid   C--

Workstreams:
WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled
   0   0   ip         0    29        0     7688        0   101826   109514
   0   0   igmp       0     0        0        0        0        0        0
   0   0   rtsock     0     0        0        0        0        0        0
   0   0   arp        0     1        0        0        0       33       33
   0   0   ether      0     0     5905        0        0        0     5905
   0   0   ip6        0     1        0        0        0        1        1
   0   0   ip_direct     0     0        0        0        0        0        0
   0   0   ip6_direct     0     0        0        0        0        0        0
   1   1   ip         0    18        0     4310        0    64904    69211
   1   1   igmp       0     0        0        0        0        0        0
   1   1   rtsock     0     0        0        0        0        0        0
   1   1   arp        0     2        0        0        0    10006    10006
   1   1   ether      0     0      718        0        0        0      718
   1   1   ip6        0     1        0        0        0       10       10
   1   1   ip_direct     0     0        0        0        0        0        0
   1   1   ip6_direct     0     0        0        0        0        0        0
   2   2   ip         0    83        0    26970        0   227620   254590
   2   2   igmp       0     0        0        0        0        0        0
   2   2   rtsock     0     3        0        0        0       36       36
   2   2   arp        0     0        0        0        0        0        0
   2   2   ether      0     0   529402        0        0        0   529402
   2   2   ip6        0     1        0        0        0        2        2
   2   2   ip_direct     0     0        0        0        0        0        0
   2   2   ip6_direct     0     0        0        0        0        0        0
   3   3   ip         0    24        0     5346        0   174797   180143
   3   3   igmp       0     0        0        0        0        0        0
   3   3   rtsock     0     0        0        0        0        0        0
   3   3   arp        0     1        0        0        0      106      106
   3   3   ether      0     0    26856        0        0        0    26856
   3   3   ip6        0     0        0       16        0        0       16
   3   3   ip_direct     0     0        0        0        0        0        0
   3   3   ip6_direct     0     0        0        0        0        0        0


[mattuz]

Just a side not: I've been trying the same setup with OpenWRT since based on google suggestions it handles PPPoE better, the results are worse than Opnsense. I get 380-400 Mbit/s. With ip-ip link the results are great as with opnsense (2.3-2.4Gbit/s)
I was starting to think the issue is coming from the PPPoE server but it doesn't make much sense since I'm getting the same performance using my laptop as a server (i7-11th mobile cpu) and my desktop (ryzen 9 3900x).