WAN to LAN fast, inside LAN slow

[jds]

It is hard to be sure that this is really an OPNsense issue, but was hoping someone here could help.
My setup is cable modem to mini PC running 18.7.9 OPNsense.  Plugged into this firewall is a gigabit TP-Link
switch, which further connects to a linux box as a server (ethernet), a ubiquiti AC LR AP, a powerline
adapter, and an Obihai VOIP box.  OPNSense has almost everything running out through openvpn.

When I run speedtest-cli on the server, I get a respectable ~120mbps.  When I run anything connected
WiFi (via the AP, android tablet, phone, laptop), I get at least 70mbps from outside. However, when I run
iperf3 in server mode on the server, and check the speed to my laptop, I get only ~10mbps. 
I have rebooted everything (OPNsense, laptop, server, AP) but repeatedly get only ~10mbps.
It has persisted for several days.  The reverse is true with running iperf3 as server on the laptop and checking
speed to the server.

Is it possible that OPNsense is somehow limiting the bandwidth for purely LAN connections?
Is it the switch?  The AP?  What other tests could I run to try and pinpoint the source of the problem?
Any help is appreciated.

[jds]

How can this happen?  The connection from server to outside and laptop to outside uses all the same kit, and then some, as does the connection between laptop and server.  If anything the laptop to server connection should be much faster.  This latter connection avoids the cable modem and mini PC.  I even switched off intrusion detection in the firewall, but this had no influence.

[jds]

OK, nobody has any pointers.  But, I am going to keep up with this.  It seems like the likely culprit was the TP-Link switch, so I replaced it with a netgear switch, but I still have the same problem.  I even switched off the OPNSense box, and got the same iperf xfer speeds between my laptop and server.   This proves it is not OPNSense, so it really doesn't make sense to post here.

[bartjsmit]

I reckon you need to apply Wireshark to this. Run a trace on both the laptop and the server during an iperf run and start digging.

Bart...

[guest19757]

Well, if this helps are not, not sure of relevant tools for Freebsb (e.g. ethtool), are you getting TX/RX flucturations ? E.g. Errors/Underrun/Overrun? Is there differences in TCP vs UDP? At least you can narrow down the protocol?

However, when I run
iperf3 in server mode on the server, and check the speed to my laptop, I get only ~10mbps. 

What's missing here, what connection type? Wired/Wireless?

[jds]

Wireshark is an excellent idea---I need more data.

The strange bit is that the connections (wired/wifi) never change during the different tests, just the speed between different
parts. The laptop is wifi always, and the server is wired always.  They both get fast connections with the outside internet,
but slow connections between each other.  I have not yet checked TCP vs. UDP.  Not sure how to check the TX/RX fluctuations,
and there were no errors reported in iperf.

[guest19757]

This may or may not be relevant but I recalled on PFsense forums that speed testing between Clients and the firewall are unreliable. Opnsense/Pfsense is designed to maximized for routing throughput. Nevertheless, ifconfig output will show any TX/RX problems. With iperf3, try with UDP, at least from there you can narrow it down to TCP or UDP.

Regards

[jds]

I reckon you need to apply Wireshark to this. Run a trace on both the laptop and the server during an iperf run and start digging.

Bart...

I finally got the time to start looking at this wifi connection issue again, and have started looking at wireshark.
It looks like a beast, in both positive and negative senses.  There is a massive amount of documentation, youtube tutorials,
blogs, recommendations, and instructions.  I feel completely overwhelmed by it.

Does anyone have a pointer to a good place to start reading for this particular problem?  Reading what is out there could take weeks.

[xames]

i have similar problems and i ajust mss clamping to 1400 and goes fine, next i buy another nic card, and all goes perfect.

[jds]

Thanks for the suggestion about MSS clamping. I tried that, but no difference.  This was expected, since I have the same performance issues without opnsense even running. 

I have run wireshark while doing the iperf3 test, and have the results, but do not really know what to look for.
I did notice that the packet length from client to server is 1514, but the packets sent from the server to the client
have length of only 66.

[jds]

Nevermind, the server was connected both hardwire and through wireless, which caused the confusion.

More data, I guess.  If I try to use iperf3 to check connection between server and laptop, it fails.  On
the server side I run: iperf3 -s V
On the client (laptop) side, I run:

$ iperf3 -c 192.168.1.4 -u4b 10G -P 2 -n 10G -w 1M -p 5201
Connecting to host 192.168.1.4, port 5201
iperf3: error - unable to read from stream socket: Resource temporarily unavailable

While the server side shows:
$ iperf3 -sV
iperf 3.1.3
Linux jdsden 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Time: Sun, 17 Feb 2019 20:55:30 GMT
Accepted connection from 192.168.1.5, port 56098
      Cookie: jay-XPS-13-9370.1550436930.056092.41
[  5] local 192.168.1.211 port 5201 connected to 192.168.1.5 port 47026
iperf3: the client has unexpectedly closed the connection
iperf 3.1.3
Linux jdsden 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

I have checked the firewall, even though it is the same port as used by TCP, which connects with terribly slow results.

I noticed that the server is responding with the incorrect IP address, for some reason.