Emulation of VSAT speed and latency with OpnSense

[MilenKo]

Hello dear community members. A few weeks ago I was charged with a task to find a way to limit our office Internet to a specific rack we are building with the bandwidth and latency of a slow satellite connection (VSAT). I was playing a lot and read tons of howto's and followed tutorials one after the other, however I had some issues that I could not resolve myself. I've been working as an IT for all my life, however I did never played with OpnSense or used any other software as bandwidth limiting/shaping.

Let me give some more detailed information and what steps I took to make it work. Most likely I have made a simple logic mistake, however I am unable to find it.

The WAN interface of the box is behind our private network and have the following info:

IP: 192.168.9.112
Mask: 24
Gateway: 192.168.9.254

The LAN interface of the box is set to communicate with the newly built rack and has the following details:
IP: 192.168.170.1
Mask: 24
Gateway: N/A

The VSAT speed I am trying to emulate is: Up: 3Mbps/Down 1Mbps with latency ~700ms.

The goal is to allow traffic from any LAN interface to Internet using the bandwidth and latency limiter as well as to allow connections to the new rack LAN from the WAN interface (which is our private network)

Following a few simple tutorials, I created two limiters in Firewall->Traffic Shaper->Limitters. One was called VSATLimitUP having bandwidth of 3Mbps and 350ms latency. The other was called VSATLimitDown having 1Mbps limit with latency 350 (the reason I put 350 instead of 700 was the fact that the ping reply became doubled prior to in/out communication so I split the desired latency in two and set the result to both limiters).

From there everything seemed to be simple, creating a rule in the firewall with the following settings:

Action: Pass
Interface: LAN
Protocol: any
Source: LAN Subnet
Destination: WAN Subnet
Description: LAN to WAN over VSAT
In = VSATLimitUP & Out = VSATLimitDown

Moved the rule above all others and tested to ping. From here I had different results - once I had full LAN speed of <1ms to lan/wan or had a proper latency of 700ms to any IP (lan or wan). It is good if I have a ping to WAN with the latency, however to the LAN I should not be limited and should have <1ms to any device behind 192.168.170.0/24 network.

A similar WAN rule was created:

Action: Pass
Interface: WAN
Protocol: any
Source: WAN Subnet
Destination: LAN Subnet
Description: WAN to LAN over VSAT
In = VSATLimitDown & Out = VSATLimitUP (reversed order of the LAN rule as per most manuals and tutorials)

As a result, I was able to get some traffic being limited, however it did not apply to the limits I set (3Mbps = ~250-300KBps and 1Mbps ~110-120KBps) I get around 50-60K of download speed and is not affected by any change in the limitters.

On the other hand, any attempt to access the LAN network from the WAN (192.168.9.X to 192.168.170.X) is blocked even though on the client remote windows machine I added a route: route add 192.168.170.0 mask 255.255.255.0 192.168.9.112

So far I am able to access the OpnSense over the WAN (I added another rule from any to This Firewall) and ping it, but am unable to ping the second interface of the LAN (192.168.170.1) I feel like I am missing a rule to pass the traffic from 192.168.9.112 to 192.168.170.1 but tried to do that with no limitters and was not able to.

On top of that, looking at our network syslog I noticed that the box is trying to have connections to external network around midnight almost every minute. I went back to the firewall on port 53. I went back to the firewall and stopped any DNS services as well as NTP protocol thinking that the box would stop doing that. The next morning I discovered that there was still the same attempt to go out of the box to the same IP's. I thought that this might be the update attempt, but the box was already updated manually by me, so there is some connection that is still blocked and I need to stop this service as it is flooding the network and slowing down the ASA we have as a firewall.

So any shared thoughts about how to setup the limiters in my case and make the traffic goes both ways from LAN (192.168.170.0/24) to Internet and reverse, but at the same time not limiting the ping to any local LAN IP's is highly appreciated. I know there are tons of howto's out there, however most of them are reverse to my need - reducing the bandwidth and reducing the latency instead of increasing it.

Here are two of the best howto's I found so far and used as a guide but I was not able to fully accomplish the task:
http://www.squidworks.net/2012/08/pfsense-2-0-limiting-users-upload-and-download-speeds-by-limiting-bandwidth/
https://www.reddit.com/r/PFSENSE/comments/3e67dk/flexible_vs_fixed_limiters_troubleshooting_with/

Any info, any spotted mistakes or needed corrections would make my day

[MilenKo]

Btw, I just checked tonight and I am still getting the same attempts for external connection:

Here is the log I got from our firewall:

---------------------------------------------------------
192.168.9.254    Jan 10 11:08:25  Jan       local6    warning 10 2017 11:08:19           %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:202.12.27.33/53 to INSIDE_V109:192.168.9.112/13849; packet length 1097 bytes exceeds configured limit of 512 bytes
192.168.9.254    Jan 10 11:08:25  Jan       local6    warning 10 2017 11:08:19           %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:217.149.76.228/53 to INSIDE_V111:192.168.11.10/57447; packet length 519 bytes exceeds configured limit of 512 bytes
192.168.9.254    Jan 10 11:08:25  Jan       local6    warning 10 2017 11:08:19           %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:202.12.27.33/53 to INSIDE_V109:192.168.9.112/37048; packet length 1097 bytes exceeds configured limit of 512 bytes
192.168.9.254    Jan 10 11:08:26  Jan       local6    warning 10 2017 11:08:20           %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:192.58.128.30/53 to INSIDE_V109:192.168.9.112/25684; packet length 1097 bytes exceeds configured limit of 512 bytes
192.168.9.254    Jan 10 11:08:26  Jan       local6    warning 10 2017 11:08:20           %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:192.58.128.30/53 to INSIDE_V109:192.168.9.112/19010; packet length 1097 bytes exceeds configured limit of 512 bytes
192.168.9.254    Jan 10 11:08:26  Jan       local6    warning 10 2017 11:08:20           %ASA-4-410001: Dropped UDP DNS reply from OUTSIDE:198.97.190.53/53 to INSIDE_V109:192.168.9.112/18967;
------------------------------------------------

Any ideas why the box is trying to connect to the IP's on port 53 (DNS) even after I stopped any DNS related resolving services and the LAN net is using a local IP (192.168.170.1) as its DNS?

[jschellevis]

Hi MilenKo,

Looks like your firewall is complaining about the packet size.. in other words MTU/MSS settings.
You can change this on the interface page.

Cheers,

Jos

[MilenKo]

Hello jschellevis. Looks like the packet size of the DNS query was left as default (512Kb) so the bigger packs get dropped.

I was able to set the proper speed limit of VSAT (256Mbps/128Mbps with latency of 700) and from the LAN network to access the Internet properly.

Right now I am trying to figure out why I am unable to get pass through OpnSense box to the LAN network. In other words there is no access from 192.168.9.0/24 network to 192.168.170.0 network via the OpnSense box.

Trying to figure it out, I even went to disable fully the firewall however I am still unable to ping neither the LAN network, nor the LAN interface of the OpnSense box. From the box itself though I am able to ping a client on both ends.

What am I missing? Should I add a static route from WAN Net to LAN Net via the box itself???

On my PC I issued: route add 192.168.170.0 mask 255.255.255.0 192.168.170.1 (which is the OPNSense LAN IP) but again - I am unable to ping even that interface, so there is still something to be configured on the box to allow the packets to travel both ways (with or without limit)...

[MilenKo]

OK. Seems like the community is really supportive. 63 reads and only one comment. Great. Looks like I will have to find my own way to another alternative that would do the job. Pity.. I wanted to try OpnSense and donate but without being able to use the appliance it won't make much sense...

[weust]

Just a thought, but have you tried placing the box without any special configuration for your VSAT testing, just see if connections are getting through at all?

Also, keep in mind this is a community forum. Not a payed service ;-)
Hooking up with us on IRC can help too. More active people there, but keep in mind it's IRC.