New 1G line, achievable direct from modem, but not through opnsense box?

[keanu]

Hi all,


Somewhat tearing my hair out about this. Been at it for about a week now. Upgraded to Virgin 1Gbit, got the new box, whacked it into modem mode and hooked it up to my OPNSense box (I believe it's a Qotom? Either way, J1900, 8GB RAM, MSATA, yada yada.) and ran a speedtest (fast.com though it doesn't seem to make a difference if I use speedtest either) and it comes back as anywhere between 100Mbps - 200Mbps. I tried with my PC hooked directly into the VM Router (SH4) and it's coming back as between 900Mbps - 1Gbps. So it's not the VM modem, it must be the Opnsense box? Strangest part about it all though is that on the old Hitron router, I used to hit roughly 3-400Mbps (On a 350 line), but now it doesn't want to go over 200 at all? I've tried a few fixes posted on the OPNSense forum, (e.g. https://forum.opnsense.org/index.php?topic=9693.0 ) but it hasn't made a difference.


At a total loss here, any help would be greatly appreciated!

TIA

[siga75]

I would start observing the top command output when doing the speed test

Other thing that comes in my mind: packet fragmentation? maybe take a capture of the WAN traffic on the OPNsense box

[mimugmail]

Interfaces : LAN : mss to 1300 for a first test

[keanu]

I would start observing the top command output when doing the speed test

Other thing that comes in my mind: packet fragmentation? maybe take a capture of the WAN traffic on the OPNsense box

Just given it a look with the top command and this is what comes back:
Goes from:
CPU:  0.5% user,  0.0% nice,  0.7% system,  0.4% interrupt, 98.4% idle
to:
CPU:  0.2% user,  0.0% nice, 51.9% system,  0.6% interrupt, 47.3% idle

Speedtest comes back at about 200 for that.

As said in the op though, it used to sit at around 350-400, so i'm not sure why it's dropped to half of it?

How would I go around taking a capture of the WAN traffic on the OPNSense box please?

Interfaces : LAN : mss to 1300 for a first test

Just tried this but hasn't made a difference sadly.

Thank you very much for your replies, I really appreciate the help!

[mimugmail]

Interfaces : Disgnostics : Packet Capture

[keanu]

I'm not entirely sure if this is correct, please let me know if you need me to do anything else for it as this is my first time doing this!

I ran the packet capture then started a speedtest. Please see attached.

Thanks,

[flexibug]

Haven't looked at your capture, do you just NAT and firewall, i,e. No other Service(s) running? My 4-core AMD 1GHz jaguar (APU) maxed out at around 480 Mbit/s with nothing else running using OPNSense 18.xy. Adding IDP/IDS brought it down to less than 140. Now I am using a 4-core Xeon 3.2 GHz (E3-1220v3) that went up to 540 (including IDS/IDP) running OPNSense up to 20.1. After upgrading to 20.7. it went down below 340. Always on the very same 600/60 line with same modem and same NICs (Intel). For the time being I reverted back to 20.1.9.

I've read about even more substantial perfomance degradation somehow related to 20.7. Why don't you give it a try with 20.1.9?

[keanu]

As far as i'm aware, I don't have any IDS running in the background. I've just double checked and the box is unticked so I believe it's off! I do have about 9 VLANs though with about 10-11 rules each - i'm not sure if this would make a massive difference?

I have read that aswell, however the upgrade to 20.7 was done last night in an attempt to fix it - which sadly hasn't worked.

I have had someone else say that the core may not be strong enough so i'm just trying to work out prices for potentially upgrading. Would you say a Ryzen 3 3200G would be strong enough to run my network? Or would I be more looking at a Ryzen 5 2600 or higher?

Thanks for your reply / help! I really appreciate it!

[siga75]

there's really few data in that capture, do

- start the capture
- run speedtest, or download a big file
- stop the capture

I suggest to insert
- 0 as "count"
- "72" as Packet Length, so only the headers are grabbed

[keanu]

there's really few data in that capture, do

- start the capture
- run speedtest, or download a big file
- stop the capture

I suggest to insert
- 0 as "count"
- "72" as Packet Length, so only the headers are grabbed

Got'cha! All done! It's too large for the forum so i've had to whack it onto WorkUpload.

Code Select Expand

https://workupload.com/file/ycKxSmB3fzh

Password for it is "opnsense" (without the quotes)

Hopefully that helps!

[siga75]

does not look so bad, just some retransmissions, which is normal TCP behaviour, for the congestion avoidance

[siga75]

"CPU:  0.2% user,  0.0% nice, 51.9% system,  0.6% interrupt, 47.3% idle"

how many cores do you have? if you have 2 cores it can be a single-threaded process using 100% of that core, it's system usage so that's system calls, typically network and disk I/O

[siga75]

BTW: looks like you have TCP timestamps disabled?

try executing

[root@myfw ~]# sysctl net.inet.tcp.rfc1323
net.inet.tcp.rfc1323: 1

[keanu]

BTW: looks like you have TCP timestamps disabled?

try executing

[root@myfw ~]# sysctl net.inet.tcp.rfc1323
net.inet.tcp.rfc1323: 1

Hi Siga,

That command comes back as:

root@OPNsense:~ # sysctl net.inet.tcp.rfc1323
net.inet.tcp.rfc1323: 1

It's a Intel J1900, so 4c I believe. It has a MSATA drive aswell, so it should be a bit quicker regarding calls.

I ended up biting the bullet earlier and ordering a Ryzen 3 3200G etc to replace it. So we'll see how that one goes once all the bits arrive. Fingers crossed it should fix the problem!

[keanu]

Hi all,

So the new box has certainley helped. It's gone from around 100-200Mbps to roughly 800Mbps. Not running with VMWare in the background anymore either. Does anyone have any ideas why it won't reach the full 1Gbps?

This is the output from top during the speedtest:

last pid:  3742;  load averages:  0.11,  0.31,  0.30                                                  up 0+00:26:00  08:17:37
46 processes:  1 running, 45 sleeping
CPU:  0.0% user,  0.0% nice,  8.0% system, 16.8% interrupt, 75.2% idle
Mem: 139M Active, 1879M Inact, 547M Wired, 324M Buf, 11G Free
Swap: 10G Total, 10G Free

That speedtest came back at roughly 610Mbps (So it can vary pretty wildly between 500-800Mbps now.

Forgot to add, this is the speedtest from the box itself:

Retrieving speedtest.net configuration...
Testing from Virgin Media (86.24.81.72)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Vodafone UK [1.48 km]: 12.771 ms
Testing download speed................................................................................
Download: 775.08 Mbit/s
Testing upload speed...
Upload: 51.14 Mbit/s

TIA!