Messages

No, I don't have access to the server logs since I don't own the server side.

I have tried adding "mssfix 1300" or "mssfix 1000" in the "Advanced" box inside the OpenVPN client config.  I then rebooted to make sure everything is fresh.  Unfortunately, that does not get rid of the packet loss issue.

Without manually adding the mssfix option, the default mssfix is 1450 according to the OpenVPN log.

I have also tried to set MSS to 1300 or 1000 inside Interfaces > [VPN Interface], then rebooted to make sure it takes effect.  Unfortunately, this also did not resolve the packet loss issue.

Based on this, I'm not sure MSS is the fix for this problem.  Any other ideas?

oops. sorry. it's probably on server side log only.
do you have access to server logs?
and have you already tried --mssfix?

No, there is nothing like that in the log, at least not at log level 4.

verbose 4 is realy huge )
is there messages like "AEAD Decrypt error: bad packet ID.."?

I have tried using TCP protocol for OpenVPN.  That does not solve the packet loss issue, and makes the maximum download speed (before the gateway shuts down) slower than UDP.

OpenVPN log level is set to 4.  Here is a snippet of what I see if I filter to the word "warning":

2020-09-08T22:51:34   openvpn[80340]: 128.14.134.170:35260 WARNING: Bad encapsulated packet length from peer (5635), which must be > 0 and <= 1626 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...]
2020-09-08T22:37:25   openvpn[51358]: WARNING: 'auth' is used inconsistently, local='auth [null-digest]', remote='auth SHA1'
2020-09-08T22:37:25   openvpn[51358]: WARNING: 'cipher' is used inconsistently, local='cipher AES-256-GCM', remote='cipher AES-256-CBC'
2020-09-08T22:37:25   openvpn[51358]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1550', remote='link-mtu 1558'
2020-09-08T21:37:25   openvpn[51358]: WARNING: 'auth' is used inconsistently, local='auth [null-digest]', remote='auth SHA1'
2020-09-08T21:37:25   openvpn[51358]: WARNING: 'cipher' is used inconsistently, local='cipher AES-256-GCM', remote='cipher AES-256-CBC'
2020-09-08T21:37:25   openvpn[51358]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1550', remote='link-mtu 1558'
2020-09-08T21:23:58   openvpn[80340]: 74.82.47.2:56208 WARNING: Bad encapsulated packet length from peer (5635), which must be > 0 and <= 1626 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...]
2020-09-08T20:37:25   openvpn[51358]: WARNING: 'auth' is used inconsistently, local='auth [null-digest]', remote='auth SHA1'
2020-09-08T20:37:25   openvpn[51358]: WARNING: 'cipher' is used inconsistently, local='cipher AES-256-GCM', remote='cipher AES-256-CBC'
2020-09-08T20:37:25   openvpn[51358]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1550', remote='link-mtu 1558'
2020-09-08T19:38:50   openvpn[80340]: 162.142.125.35:49570 WARNING: Bad encapsulated packet length from peer (18245), which must be > 0 and <= 1626 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...]
2020-09-08T19:38:49   openvpn[80340]: 162.142.125.35:47916 WARNING: Bad encapsulated packet length from peer (5635), which must be > 0 and <= 1626 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...]
2020-09-08T19:37:25   openvpn[51358]: WARNING: 'auth' is used inconsistently, local='auth [null-digest]', remote='auth SHA1'
2020-09-08T19:37:25   openvpn[51358]: WARNING: 'cipher' is used inconsistently, local='cipher AES-256-GCM', remote='cipher AES-256-CBC'
2020-09-08T19:37:25   openvpn[51358]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1550', remote='link-mtu 1558'
2020-09-08T19:31:40   openvpn[80340]: 193.34.131.57:39276 WARNING: Bad encapsulated packet length from peer (5635), which must be > 0 and <= 1626 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...]

Here is what I see if I filter to the word "error":

2020-09-08T19:38:50   openvpn[80340]: 162.142.125.35:33464 SIGUSR1[soft,tls-error] received, client-instance restarting
2020-09-08T19:38:50   openvpn[80340]: 162.142.125.35:33464 Fatal TLS error (check_tls_errors_co), restarting
2020-09-08T19:38:50   openvpn[80340]: 162.142.125.35:33464 TLS Error: tls-crypt unwrapping failed from [AF_INET]162.142.125.35:33464
2020-09-08T19:38:50   openvpn[80340]: 162.142.125.35:33464 tls-crypt unwrap error: packet too short
2020-09-08T19:38:50   openvpn[80340]: 162.142.125.35:32788 SIGUSR1[soft,tls-error] received, client-instance restarting
2020-09-08T19:38:50   openvpn[80340]: 162.142.125.35:32788 Fatal TLS error (check_tls_errors_co), restarting
2020-09-08T19:38:50   openvpn[80340]: 162.142.125.35:32788 TLS ERROR: initial packet local/remote key_method mismatch, local key_method=2, op=P_CONTROL_HARD_RESET_CLIENT_V1
2020-09-08T16:51:10   openvpn[80340]: 192.35.168.193:58604 SIGUSR1[soft,tls-error] received, client-instance restarting
2020-09-08T16:51:10   openvpn[80340]: 192.35.168.193:58604 Fatal TLS error (check_tls_errors_co), restarting
2020-09-08T16:51:10   openvpn[80340]: 192.35.168.193:58604 TLS Error: tls-crypt unwrapping failed from [AF_INET]192.35.168.193:58604
2020-09-08T16:51:10   openvpn[80340]: 192.35.168.193:58604 tls-crypt unwrap error: packet too short
2020-09-08T16:51:10   openvpn[80340]: 192.35.168.193:58230 SIGUSR1[soft,tls-error] received, client-instance restarting
2020-09-08T16:51:10   openvpn[80340]: 192.35.168.193:58230 Fatal TLS error (check_tls_errors_co), restarting
2020-09-08T16:51:10   openvpn[80340]: 192.35.168.193:58230 TLS ERROR: initial packet local/remote key_method mismatch, local key_method=2, op=P_CONTROL_HARD_RESET_CLIENT_V1

what's in the openvpn log?
have you tried tcp?

(1)  Inside Interfaces > [VPN Interface].  Do I need to set this for WAN interface too?
(2)  Inside Firewall > Settings > Normalization.  And this section is confusing to me, and I am not sure how to properly set it.
(3)  Inside VPN > OpenVPN > Clients, where I can try to set MTU and MSS directly in the VPN connection settings.

(3) I think

[Background]

I am having a weird problem that I cannot figure out despite many hours of work. I hope someone here could give some thoughts as to why this is happening.

Background: I am running vSphere ESXi 7.0. I have a VM that is running OPNsense 20.1.9_1 (although I have also tried this with OPNsense 20.7.2 and get the same result). To be very conservative, I assigned 4 CPUs and 6 GB of RAM for this VM, and OPNsense reports that it is nowhere near using that much resources. I have two vswitches, one for the LAN and one for the WAN. On the particular LAN and WAN port associated with the OPNsense VM, I disabled all security (accept promiscuous mode, MAC address changes, and forged transmits) and I set VLAN trunking (0-4094).  I do run a few VLANs, so the VLAN trunking is needed. I don't think the reduced security is needed, but I just set everything to "accept" in case it was causing this problem. I have gigabit fiber on the WAN physical uplink, connected to the ISP gateway. Inside OPNsense Interface > Settings, I set it to disable all hardware offloading (CRC, TSO, LRO, and VLAN filtering).

Problem: Inside OPNsense, I have two gateways: one for the WAN and one through VPN (I use OpenVPN, with UDP protocol). There is no problem on the WAN gateway. I've tested large downloads that are tens of GBs from the internet and am able to get sustained full gigabit speed (around 90 MB/s) with 0% packet loss (as reported by DPINGER inside System > Gateways). So this indicates to me that there is no problem with the VM, the vswitches, or anything.

However, when I use firewall rules to direct traffic through the VPN gateway, I have problems.  If I force the download speed in the download client to be lowish, around 10 MB/s, then there is only a small amount of packet loss and the download proceeds to completion.  However, when I allow the speed to be higher, anything more than 15-20 MB/s, packet loss climbs very quickly (15 to 20% within 30 seconds, and continues to climb) and within 1-2 minutes, the VPN gateway just stops responding. The packet loss will go back down to 0% and VPN will work again if I stop the download. To be clear, the VPN connection is capable of much faster than 20 MB/s -- when I run OPNsense bare metal, I can easily get 50 MB/s on the VPN gateway with 0% packet loss.

Solutions?  So here is what is confusing me. When I run OPNsense in the VM, everything on the WAN works perfectly, at full gigabit speed with 0% packet loss. But when I direct traffic to VPN, I get huge packet loss that shuts down the gateway.

However, if I run OPNsense bare metal, I don't get any packet loss on WAN and VPN gateway. This indicates to me that the problem is not the VPN. There seems to be some weird interaction between using the VPN inside a VM that is causing the problem. I've tried everything, so what could it be?

Someone suggested that this looks like a fragmentation issue, and recommended that I play around with the MTU and MSS settings.  I've played around, but am not sure how to properly set the MTU and MSS.  There are at least three places where you can set these parameters in OPNsense that I can see:

(1)  Inside Interfaces > [VPN Interface].  Do I need to set this for WAN interface too?
(2)  Inside Firewall > Settings > Normalization.  And this section is confusing to me, and I am not sure how to properly set it.
(3)  Inside VPN > OpenVPN > Clients, where I can try to set MTU and MSS directly in the VPN connection settings.

Which of the three places should I set MTU and MSS?  Do I need to reboot every time I change these settings for them to take effect?

My MTU test:  Based on the ping test suggested on the link below, the largest ping that does not fragment is 1472, which suggests my MTU is 1500.  This is using a Windows computer whose traffic is directed through the VPN.  https://kb.netgear.com/19863/Ping-Test-to-determine-Optimal-MTU-Size-on-Router

Is there a way to redirect all LAN traffic to a particular address (e.g. www.google.com) to a particular IP at a specific port (e.g., 192.168.1.20:8000)?

I want to redirect all Google searches to this internal IP address at a non-standard port.  I'm guessing you can do this by making a custom DNS entry in unbound?  But I need the traffic directed to a particular port, and I don't think DNS does that.  I would appreciate any ideas on how to achieve this.

I recently set up a transparent proxy to filter traffic for selected devices on the network. It works great.

But when you try to access a blacklisted page, the proxy gives this message that says "access denied" and to contact your administrator. Is there a way to customize that page to say something else?

Or even silently redirect any blacklisted page access attempts to another website randomly selected from a list?

This never happened before when I was on 18.7.xx and I was running BIND to do DNSBL for many months.  Then over the weekend I got around to upgrading to the current 19.1.7(?) and this has happened maybe 2 to 3 times?  So maybe once every couple of days?

It's not super frequent, but because I run everything through DNSBL, the internet stops working when BIND dies.

I setup Monit last night to check on BIND and restart it whenever the process does not exist, but I can't tell if it's working correctly yet.

Hello,

I'm using BIND to run DNSBL and it has been working great for a while.  But recently (and this coincides with upgrade to the most recent version of OpnSense), I find that BIND sometimes exits due to an error that I do not understand, and I'd have to manually restart it.  I've pasted the BIND log below (in reverse chron order).  Does anyone know why this is happening?

Is there a way to make BIND automatically restart upon exit?  Or at least get a notice that it died?

Thanks!


09-May-2019 07:46:38.754    general: critical: exiting (due to assertion failure)
09-May-2019 07:46:38.754    general: critical: #7 0x0 in ??
09-May-2019 07:46:38.754    general: critical: #6 0x3479b784c36 in ??
09-May-2019 07:46:38.754    general: critical: #5 0x14c5f2204d in ??
09-May-2019 07:46:38.754    general: critical: #4 0x14c5e53ab1 in ??
09-May-2019 07:46:38.754    general: critical: #3 0x14c5e50bd0 in ??
09-May-2019 07:46:38.754    general: critical: #2 0x14c5e4a428 in ??
09-May-2019 07:46:38.754    general: critical: #1 0x14c5f021fa in ??
09-May-2019 07:46:38.754    general: critical: #0 0x14c5d192c0 in ??
09-May-2019 07:46:38.754    general: critical: resolver.c:4895: INSIST(dns_name_issubdomain(&fctx->name, &fctx->domain)) failed, back trace
09-May-2019 07:46:38.754    lame-servers: info: FORMERR resolving 'c-0.19-a7000081.80081.1770.f17.2fc8.210.0.ig6b68dgmvuczv4udl6dz2i2g5.avts.mcafee.com/A/IN': 127.0.0.1#53
09-May-2019 07:46:38.754    resolver: notice: DNS format error from 127.0.0.1#53 resolving c-0.19-a7000081.80081.1770.f17.2fc8.210.0.ig6b68dgmvuczv4udl6dz2i2g5.avts.mcafee.com/A for client 192.168.128.237#58153: non-improving referral

I'm running 18.1.9. 

I've tried running the update check while pkg was not running (pgrep pkg returns nothing).  Same sort of error happens. 

Sometimes, I get this error message instead:  "Timeout while connecting to the selected mirror."

When I run the check, it looks like two instances of pkg are running (two ID's) -- I'm not sure if this is normal.

Hello,

I keep getting this error when I try to check for firmware updates:  "Firmware status check was aborted internally. Please try again."

I've tried checking multiple times and it never works.  Is there any way to manually do this or get around this problem?  My mirror is LeaseWeb (San Francisco), LibreSSL, Production.  I've tried other  servers and get the same error.

Thanks!

Thanks, Fabian.  I think most people use something like Opnsense because we want to take greater control over how our network traffic is flowing, and learn about how networking works.  Same motivation here.  I also want to understand how Tor and VPN interact in Opnsense because that will help me decide whether and how Tor would be useful to me.

I would appreciate any other thoughts on this issue, or suggestions on how I might investigate to determine the answer.

Hello,

I've set up a VPN client and firewall rules to redirect all non-local traffic to go out via the VPN gateway.  DNS is via unbound in resolver mode, and the outgoing interface is the VPN as well.  In effect, all non-local traffic on LAN is going out the VPN.  See attached picture of my firewall rules.

I recently set up Tor just to play around with it.  See attached Tor settings.  When I set by browser to use Sock5 proxy at the Opnsense router:9050 (see attached Firefox settings), I am successfully connecting via the Tor network (confirmed using Tor check).

My question is:  does Tor and VPN interact in this set up?  Is the Tor outgoing traffic bypassing the VPN and going to the internet via WAN?  Or is it going out via the VPN gateway?  It seems like the Tor traffic (from my browser to Opnsense router, then Opnsense router to Tor entry point) is leaving the Opnsense router via the VPN gateway.  But I'm not really sure. 

Similarly, how does DNS lookup work in this Tor setup?  In Firefox, I checked "Proxy DNS when usign SOCKS v5."  Does that mean DNS queries from my browser are all going through Tor?  Does that mean that DNS queries are bypassing unbound?

I would appreciate if someone could educate me on how this works.

Thanks!

Hello,

I'm a bit confused about the pros and cons of using this Cloudflare setup.  Can someone enlighten me?

I thought the reason one might want to use unbound in resolve mode is so that all DNS queries would begin at the root servers, then resolve each step down so that you end up with a response that is for sure accurate.  The downside is that this takes longer because you have to query multiple servers (and I guess the queries are not encrypted).

In forward mode, you're just querying your favorite server (e,g., OpenDNS) without going to the root servers.  The pro is that this is faster.  The con is that you're trusting that the server is not messing with the response (or tracking your queries).  These queries are still not encrypted.

It sounds like this new server 1.1.1.1 is DNS using forward mode, with the same pros and cons, except they encrypt the queries, claim it's the fastest server, and claim to not log your queries.

Is the above summary correct in terms of the pros and cons?

I think I have narrowed down the issues to either a problem with my NAT settings or firewall rules.  Attached are my NAT and LAN firewall settings.  Unbound is set to use AirVPN1-3 as the outgoing network interface.  If I have disable the firewall rule that redirects LAN traffic to VPN_Gateways, then IP Leaks test will show my real IP, and that my DNS servers are the AirVPN exit nodes.  This indicates that the VPN connections are fine since the DNS queries are going out on the VPN interfaces as expected.

When I turn on the firewall rule that redirects the LAN traffic to the VPN_Gateways, connections will time out.  I have tried various settings for that rule (e.g., changing the destination to be "any," or different versions of the private subnets), but nothing seems to work.  It seems to me that there may be some interaction between the firewall rules and Unbound DNS resolution.  It seems like without the firewall redirect rule, the DNS works fine.  But with the firewall redirect rule, DNS is not resolving?

I would really appreciate any help on these settings.  I want to redirect everything on LAN (except a few IP's that is in the Alias Bypass_VPN) to the VPN_Gateways.

Ok, I continue to have issues.  I've now set up everything (I think).  The VPN client connections seem fine, the route table looks reasonable, I have created firewall rules to direct traffic to the load balancing VPN gateway group, I have set up outbound NAT rules (I think this is correct, but am not sure as I never completely understood the NAT page).  I also set the Unbound outgoing network to be the VPN client interfaces.

Using the above set up, I run IP Leak tests and it correctly shows my VPN's IP, and shows that I use DNS servers equal to the VPN's IP.  So everything looks like it's working correct. 

Except, the connections are not consistent.  Sometimes the internet works very quickly.  Sometimes connections time out in the browser.  I don't see any pattern to when connections would time out, nor do I see anything in the logs that would indicate a problem. 

What would cause the set up to work sometimes, but not other times?  The working / not working changes from minute to minute, back and forth.