WAN faster than LAN...

[bpalob]

I do have a weird behavior with my Opnsense installation. Here´s the environment:

SW:  OPNsense 21.7.1-amd64
        FreeBSD 12.1-RELEASE-p19-HBSD
        OpenSSL 1.1.1k 25 Mar 2021

HW:  APU4D4 (4xIntel I211AT), 4GB RAM,  AMD Embedded G series GX-412TC, 1 GHz quad Jaguar
        core with 64 bit and AES-NI support, 32K data + 32K instruction cache per core, shared 2MB
        L2 cache.

NET:  1gbps Cable WAN connection on IGB1, 1gbps LAN interface on IGB0.

When I connect directly to the modem from the PC, I get close to the 1gbps WAN throughput.

Now the weird thing I do not really understand:

-  PC-Opnsense IPerf3 with 4 parallel streams (best restults) I get around 500mbps max.
-  PC-WAN (Oakla) I get about 650mbps

Not only is of course the WAN performance way below expectation, but the LAN side is even more surprising. How can I get 500mbps on the LAN and 650mbps on WAN, considering I have to go though the same LAN to get to these 650mbps?

BTW, I also tried UDP on the LAN, no chance, rarely above 500mbps (various combinations of parallel streams, reverse testing and bandwidth tested).

Any hints on how I can improve the LAN performance, or maybe even getting the WAN speed up?

Thanks.

[sorano]

Now the weird thing I do not really understand:

-  PC-Opnsense IPerf3 with 4 parallel streams (best restults) I get around 500mbps max.
-  PC-WAN (Oakla) I get about 650mbps

Not only is of course the WAN performance way below expectation, but the LAN side is even more surprising. How can I get 500mbps on the LAN and 650mbps on WAN, considering I have to go though the same LAN to get to these 650mbps?

Any hints on how I can improve the LAN performance, or maybe even getting the WAN speed up?

Not really that weird nor surprising...

And you've actually already dropped the answer to your problem:

You do not understand how it works.

Let me ask you a question:

What is the biggest requirement to get good throughput when routing packets?

Then let me ask you another question:

Which main compute resource gets consumed when performing iperf tests?

See what I'm getting at here? And I'm not talking about NIC line speed.

When you perform an iperf test between your client pc and your firewall the firewall has to spend CPU cycles both to process the packets and to handle the iperf data.

While when you perform a speedtest on the Internet the firewall is free to spend all of its power on processing packets.

Simple as that!

So the correct way to test lan performance as you call it, is between two clients on two different interfaces.

Regarding your WAN speed I'd say that it's also CPU related. 1 GHz core speed is likely not enough raw power to route 1 Gbit.