Messages

Hello

I've been having some issues for quite some time, since 21.7. I'm currently on the most recent version:
OPNsense 22.7.4-amd64
FreeBSD 13.1-RELEASE-p2
OpenSSL 1.1.1q 5 Jul 2022

I have 2 ISP connections, main one being WAN and backup (radio) being WAN2.

WAN is fiber optic but ISP sucks, unfortunately cannot cancel as of now and have to deal with the issues, at least 2 times a week around midnight there is packet loss ~25% so it's not entirely down.... our setup correctly switches to WAN2.

This issue tend to last for 1 or 2 hours, but when WAN starts working correctly now and there is no longer packet loss all the traffic keeps going through WAN2 without switching back to WAN.

I even tried setting up a corn task to reset the WAN interface around 3am.

Allow default gateway switching = OFF
GW GROUP: failover (WAN Tier 1, WAN Tier 2)
FIREWALL LAN Rule: !192.168.0.0/16  Gateway: failover

Anything we are not setting up correctly? Or is this an issue with opnsense?

Hi

I'm trying to find a way for another platform we have read the current gateway statuses and display alerts while using it.

Is there any way to read via API/JSON the gateway statuses? I have 3 ISPs which I would like to monitor without having to login to OPNSense GUI.

Thanks

If you use TCP as transport protocol, then please disregard; TCP will not have fragmentation

If you use UDP as transport protocol, then take a packet capture: Interfaces / Diagnostics / Packet Capture. Select UDP as protocol, select the peer ip address if you know it, but leave port number unset; follow-up fragments do not carry port numbers.

Start the capture and start a file transfer through OpenVPN.

If you see something like
    20:58:29.321685 IP a.b.c.d > u.v.w.x: ip-proto-17

then you have fragmentation issues. 17 is the protocol number for UDP, but no port numbers are displayed because they are missing in any but the initial fragment.

Depending on how big the reassembled packet is, you may also see "bad length x > y" for the initial fragment (where port numbers are shown).

If that is the case, start with something like "mssfix 1300". This is low enough you should not have UDP fragmentation. You can experiment with higher values and find the optimum value that still works without fragmentation. The exact value will also depend on the client's internet connection.

This only helps for TCP connections inside the tunnel, large UDP packets will still be fragmented.

Its setup for UDP, did the test and so far packets are not showing up fragmentation, I'll do more tests for nothing so far.

I just upgraded from OpnSense 19.7 to 20.1.3, ran more tests, same behavious. I did howerver tested with 2 devices now simultaneously and same speed between both. Ran speedtest in my PC and my phone and both maintain the speed at 25-30 mbps even at the same time.

Still, made the changes suggested just in case like adding mssfix and no improvement, also changed from AES-128-CBC to  AES-128-GCM, added RDRAND crypto engine, made sure AESNI is enabled, and adaptive compression, slight improvement from avg 25 mbps to 30 mbps.

You have explicitly disabled mssfix. Can you check if openvpn encapsulated traffic gets fragmented? That should be avoided, as fragment reassembly is rather slow.

I'm sorry but how would I check this? pcaps?

Hola a todos los que preguntan.

Yo logré realizar esto basandome en políticas por IP y no por usuario, pero no por medio de la interface si no por medio de reglas personalizadas desde el shell de OpnSense.

Lo que hice fue lo siguiente:

1. Ingresar por SSH a OpnSense

2. Navegar a /usr/local/etc/squid/pre-auth

3. Crear un archivo custom.conf y agregar las reglas en formato de Squid de las direcciones IPs y nombrandolas por grupos
ej.: acl grupo1 src 192.168.1.2
acl grupo1 src 192.168.1.3
acl grupo2 src 192.168.1.4
acl bloqueo1 dstdomain .facebook.com
acl bloqueo1 dstdomain .google.com
acl bloqueo2 dstdomain .instagram.com
http_access deny grupo1 bloqueo1
http_access deny CONNECT grupo1 bloqueo1
http_access deny grupo2 bloqueo2
http_access deny CONNECT grupo2 bloqueo2

4. Recargar las nuevas reglas a squid ejecutando el comando:
service squid reload

Listo!

Ich spreche nicht wirklich Deutsch, sondern verstehen es nur ein wenig (bereit bald einmal zu lernen: P), so bin ich mit Google Translate.

Ich habe jedoch das gleiche Problem mit einem Intel Atom C2758, der über eine Bandbreite von 200/200 weit unter dem i7 liegt (vorausgesetzt, der von Ihnen erwähnte i7 befindet sich auf dem Server) und über OpenVPN nur 25/25 erreicht.

Versuchte alles und Ihre CPU sollte überhaupt kein Problem sein.

Haben Sie versucht, Ihre Server- und Client-Puffereinstellungen zu optimieren?

Thank you for reply

I think cpu should be enough for 100.

Openssl tests for AES-128 and AES-256 EITHER CBC and Gcb throw about 130 mbps.

Cpu does have AES-NI support and single thread usage when transferring data at the apparent max speed of 25 mbps is ~25%, therefore I'm not even using 50% of a single core/thread.

I'm currently getting very low throughput on OpenVPN. ISP connection is 200/200 however over OpenVPN it goes down to 25/25 mbps aprox.

According to a lot of research online this GW server+CPU should be more than enough to achieve at least 100 mbps, but I would like to get as near as possible to the 200 mbps available bandwidth, if not possible well at least get closer to 100.

Server specs are:
OPNsense 19.7.8-amd64
FreeBSD 11.2-RELEASE-p16-HBSD
OpenSSL 1.0.2t 10 Sep 2019
CPU: Intel(R) Atom(TM) CPU C2758 @ 2.40GHz
RAM: 8GB
SSD: 256GB Samsung EVO


OpenVPN settings:
Encryption: AES-128-CBC
No Hardware Crypt acceleration
No compression

Custom server settings
fast-io;
sndbuf 0;
rcvbuf 0;
push "sndbuf 524288";
push "rcvbuf 524288";

Custom client settings
fast-io
fragment 0
mssfix 0
sndbuf 524288
rcvbuf 524288


Any suggestions?

Nevermind, classic case of finding the solution after asking for help. Was unaware of the localhost workaround.

EDIT: resolved.

Hi,

So far multi-wan has been working OK in my setup (OPNsense 19.7.8-amd64) if a link goes down default gateway switching does is thing.

We currently have 3 ISPs and 1 LAN port in the gateway.

Currently this is the setup:
ISP1 (100/100 DHCP, default gw)
ISP2 (10/10 Static, deprecated, to be removed)
ISP3 (200/200 Static, new one)

Anyway, I have a VPN connection working through the default gateway but it has a dynamic IP and I'm attempting the VPN to be able to receive connections through a the new service (non-default/non-active) gateway but it doesn't work.

When I try to connect to the VPN via ISP3 I ran packet captures and incoming traffic as expected comes from a the non-default gateway ISP3, but the outgoing packets are trying to go via the default gateway (ISP1) instead of the same WAN port it came from (ISP3).

Am I missing something?

Thanks

I see.... I do have an additional rule for dual wan fail over, which now that you mention it it could be causing to allow the traffic that should be blocked.

If that's so.... so should I go about it? should I just disable the load balancing and allow opnsense to switch the default gateway and let it decide? Thanks.

[(now this is what I'm not sure if I need one even for a non transparent proxy)]

Hi

I'm a little confused about SSL and Squid Proxy.

So I've deployed Squid and trying to implement category based rules along with some custom domain rules, however it doesn't seem to be blocking correctly the domains blacklisted.

I do NOT need transparent mode, I'm perfectly OK in setup devices manually with proxy settings.

Squid settings are:
General:
Proxy enabled
Use via header
X-Forwarded-For header handling Append client's IP

Forward proxy, all default except:
Enable SSL inspection: yes
CA to use: none (now this is what I'm not sure if I need one even for a non transparent proxy)

Let me know, thanks!
Sergio M.

So with the wan balancing/fail over setup finally now my only remaining configuration is per client proxy settings.

What I want is everybody to go through the cache proxy. Regular users to have Facebook/Youtube/... blocked, some devices with some things unblocked, other devices other sites unblocked, so on...

For what I understand everybody has the same privileges if using the proxy. Is it possible to have different settings for other devices?

Thanks.

got it thanks!!

@proxykid
are you using 19.7.3?

OPNsense 19.7.2-amd64
FreeBSD 11.2-RELEASE-p12-HBSD
OpenSSL 1.0.2s 28 May 2019

For the traceroute point:
Could be related to a problem I had.
https://forum.opnsense.org/index.php?topic=13832.0

Apparently it could be it indeed, I guess if that is the issue I would need a work around on the meanwhile....