Messages

Just a quick bump so it doesn't get lost in the forums!

Hi all,

So, this may be a bit of a weird post, and I apologise. First off, I'd like to say that i'm aware that this isn't the norm. Long story short. I was given the opportunity to rack a server in a datacenter under a work rack, all well and good. It started getting a bit dangerous when adding firewall rules on the main firewall and as such, I opted to essentially "Pass through" a static IP to my vmware box, and run a nested copy of opnsense on there. Everything is working fine (per say), got internet access, can access (certain*) ports outside, but then recently i've had a friend ask me to host a gameserver for him. Nomatter what we try, we can't access the server via the static IP. Tried it via a OpenVPN connection direct to the OPNSense box, and it works. The ports are all forwarded, have confirmed in the logs, etc. It'll even show up as "Connecting" on the gameserver, but then time out. Just can't figure out why though?

(*Certain meaning except from current issue)

The way it's setup is as follows:

Datacenter cable drop --> pfsense baremetal --> My ESXI --> OPNSense --> VM

The weird part of it is, whilst looking through the logs of the pfsense box, it's passing everything through as expected. All fine. When I get to the OPNSense box, the ports it's trying to connect on have changed (Sometimes only slightly, e.g. +1 on the port). The PFSense box has a 1:1 NAT setup, blocking to other vLANs and then an allow all rule on it. That's pretty much it. So i'm not entirely sure what's going on after that?

What's even weirder is that certain parts of it (e.g. say portainer) works fine. Portforward it through, and I can access it externally without an issue. I've even tried setting up a diagnostic "Allow all" rule, same issue, so i'm confident it's not a firewall issue, maybe something to do with the NAT? But as far as i'm aware, the nat on the main pfsense box is being bypassed and ignored, exactly how I wanted.

I have the following rules set:

1:1 NAT:

Pfsense baremetal:
Interface: WAN
Source: PublicIP
Internal IP: 10.20.0.254 (IP of my OPNSense)
Destination IP: *

Outbound NAT:

PFSense baremetal:
Interface: "mydedicatedport"
Source: 10.20.0.254 (IP of my OPNSense)
Source Port: TCP/UDP/*
Destination: *
Destination Port: TCP/UDP/*
NAT Address: Public IP Address
NAT Port: *
Static Port: Checked

And on the OPNSense install:
Interface: WAN
Source: 10.20.1.152/32 (IP of server)
Source Port: TCP/UDP/*
Destination: *
Destination Port: TCP/UDP/*
NAT Address: WAN Address
NAT Port: *
Static Port: Checked

And a Port Forward on the OPNSense router of;
Interface: WAN
Protocol: TCP/UDP
Source Address: *
Source Ports: *
Destination Address: WAN Address
Destination Port: Alias inc the following ports: 2456:2459, 4380, 27000:27031, 27036
Target IP (NAT IP): 10.20.1.152
Target Ports (NAT Ports): Same as Dest port Alias

I can't think what else it could be though, I don't have this issue with anything else I port forward through the setup, so i'm a bit stuck onto where to go from here.

Any suggestions are greatly appreciated.

Thanks in advance!

Just wanted to add in my thanks, working on my end and has fixed my flakey gig connection ranging from 400-600Mbps to sit solid at a gig!

Swapped over to an Intel genuine 4 port nic as I thought that the realtek one I was using for the LAN may have been causing issues, however it hasn't made a difference sadly. If anyone has any ideas, i'd greatly appreciate it. Thanks.

800Mb is good in my opinion, for a 1G network

but indeed, with some tuning you could achieve more

the upload/download ratio is huge BTW

also, I would try using http://www.dslreports.com/speedtest so you have bufferbloat information too

The connection is actually between like 1124-1145 or so. Upload is 50-52 or so.

Yeahh the upload isn't fantastic, but there isn't a lot I can do about it with Virgin sadly. I believe it's something to do with the DOCSIS 3.1 they use? Still, gig connection for £62pcm isn't bad!

Perfect, will run that after I make the changes suggested!

Sadly it's still coming out as the following:

18ms
653 megabit/s
18ms
51.6 megabit/s

PowerD is disabled, and everything below it is on Maximum.

Thanks,

Sometimes the servers from speedtest are slower than 1G

This was tested using both fast.com and speedtest.net. Both come out relatively the same.

Hi all,

So the new box has certainley helped. It's gone from around 100-200Mbps to roughly 800Mbps. Not running with VMWare in the background anymore either. Does anyone have any ideas why it won't reach the full 1Gbps?

This is the output from top during the speedtest:

last pid:  3742;  load averages:  0.11,  0.31,  0.30                                                  up 0+00:26:00  08:17:37
46 processes:  1 running, 45 sleeping
CPU:  0.0% user,  0.0% nice,  8.0% system, 16.8% interrupt, 75.2% idle
Mem: 139M Active, 1879M Inact, 547M Wired, 324M Buf, 11G Free
Swap: 10G Total, 10G Free

That speedtest came back at roughly 610Mbps (So it can vary pretty wildly between 500-800Mbps now.

Forgot to add, this is the speedtest from the box itself:

Retrieving speedtest.net configuration...
Testing from Virgin Media (86.24.81.72)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Vodafone UK [1.48 km]: 12.771 ms
Testing download speed................................................................................
Download: 775.08 Mbit/s
Testing upload speed...
Upload: 51.14 Mbit/s

TIA!

BTW: looks like you have TCP timestamps disabled?

try executing

[root@myfw ~]# sysctl net.inet.tcp.rfc1323
net.inet.tcp.rfc1323: 1

Hi Siga,

That command comes back as:

root@OPNsense:~ # sysctl net.inet.tcp.rfc1323
net.inet.tcp.rfc1323: 1

It's a Intel J1900, so 4c I believe. It has a MSATA drive aswell, so it should be a bit quicker regarding calls.

I ended up biting the bullet earlier and ordering a Ryzen 3 3200G etc to replace it. So we'll see how that one goes once all the bits arrive. Fingers crossed it should fix the problem!

there's really few data in that capture, do

- start the capture
- run speedtest, or download a big file
- stop the capture

I suggest to insert
- 0 as "count"
- "72" as Packet Length, so only the headers are grabbed

Got'cha! All done! It's too large for the forum so i've had to whack it onto WorkUpload.

Code Select Expand

https://workupload.com/file/ycKxSmB3fzh

Password for it is "opnsense" (without the quotes)

Hopefully that helps!

As far as i'm aware, I don't have any IDS running in the background. I've just double checked and the box is unticked so I believe it's off! I do have about 9 VLANs though with about 10-11 rules each - i'm not sure if this would make a massive difference?

I have read that aswell, however the upgrade to 20.7 was done last night in an attempt to fix it - which sadly hasn't worked.

I have had someone else say that the core may not be strong enough so i'm just trying to work out prices for potentially upgrading. Would you say a Ryzen 3 3200G would be strong enough to run my network? Or would I be more looking at a Ryzen 5 2600 or higher?

Thanks for your reply / help! I really appreciate it!

I'm not entirely sure if this is correct, please let me know if you need me to do anything else for it as this is my first time doing this!

I ran the packet capture then started a speedtest. Please see attached.

Thanks,

I would start observing the top command output when doing the speed test

Other thing that comes in my mind: packet fragmentation? maybe take a capture of the WAN traffic on the OPNsense box

Just given it a look with the top command and this is what comes back:
Goes from:
CPU:  0.5% user,  0.0% nice,  0.7% system,  0.4% interrupt, 98.4% idle
to:
CPU:  0.2% user,  0.0% nice, 51.9% system,  0.6% interrupt, 47.3% idle

Speedtest comes back at about 200 for that.

As said in the op though, it used to sit at around 350-400, so i'm not sure why it's dropped to half of it?

How would I go around taking a capture of the WAN traffic on the OPNSense box please?

Interfaces : LAN : mss to 1300 for a first test

Just tried this but hasn't made a difference sadly.

Thank you very much for your replies, I really appreciate the help!

Hi all,


Somewhat tearing my hair out about this. Been at it for about a week now. Upgraded to Virgin 1Gbit, got the new box, whacked it into modem mode and hooked it up to my OPNSense box (I believe it's a Qotom? Either way, J1900, 8GB RAM, MSATA, yada yada.) and ran a speedtest (fast.com though it doesn't seem to make a difference if I use speedtest either) and it comes back as anywhere between 100Mbps - 200Mbps. I tried with my PC hooked directly into the VM Router (SH4) and it's coming back as between 900Mbps - 1Gbps. So it's not the VM modem, it must be the Opnsense box? Strangest part about it all though is that on the old Hitron router, I used to hit roughly 3-400Mbps (On a 350 line), but now it doesn't want to go over 200 at all? I've tried a few fixes posted on the OPNSense forum, (e.g. https://forum.opnsense.org/index.php?topic=9693.0 ) but it hasn't made a difference.


At a total loss here, any help would be greatly appreciated!

TIA