Messages

24.10.1-amd64 same issue:

  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[Hardware: Dell R720]

Hello!

If you have a chance please review https://www.academia.edu/33882347/Suricata_Extreme_Performance_Tuning

I had to disabled most of the intel prefetching options in the BIOS and reduce the TX and RX queues for the nics.  Once I did that I could run IDS/IPS without having any speed issues.

Note that when you start/stop Suricata it will cause dpinger to output errors like you listed.

hi @klamath

OPNsense 21.7.3_3-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021

Hardware: Dell R720
CPU 1   Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz   Model 62 Stepping 4   2600 MHz 8core
CPU 2   Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz   Model 62 Stepping 4   2600 MHz 8core

Ram : DDR-3   64.00 GB   Presence Detected   Dual Rank   1866 MHz

Ethernet:
NIC Slot 6   Intel(R) Ethernet Converged Network Adapter X540-T2 (WAN,DMZ)
Integrated NIC 1   Intel(R) GbE 4P I350-t rNDC (LAN,MANAGEMENT)

When Suricata is enabled with IDS/IPS protection the max WAN speed is capped at around 650-670Mbps, with IPS mode disabled I can achieve full 827Mb/s down.

I can't say that the ethernet cards we use are not compatible with suricata IPS running on freebsd, because you have witnessed that it works properly in the previous kernel.

At the same time, when I follow the dpinger service, the situation is as follows:

2021-11-12T02:35:16   dpinger[78904]   send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr    
2021-11-11T13:01:05   dpinger[62032]   WAN_GWv4_ X: sendto error: 55   
2021-11-11T02:35:29   dpinger[72741]   GATEWAY ALARM: WAN_GWv4_ (Addr: XAlarm: 0 RTT: 13002us RTTd: 125us Loss: 0%)   
2021-11-11T02:35:29   dpinger[62032]   WAN_GWv4_ X.255.0.37: Clear latency 13002us stddev 125us loss 0%   
2021-11-11T02:35:17   dpinger[38016]   GATEWAY ALARM: WAN_GWv4_ (Addr: X.255.0.37 Alarm: 1 RTT: 12983us RTTd: 102us Loss: 25%)   
2021-11-11T02:35:17   dpinger[62032]   WAN_GWv4_ X.255.0.37: Alarm latency 12983us stddev 102us loss 25%   
2021-11-11T02:35:14   dpinger[62032]   send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr X.255.0.37 bind_addr X.255.0.38 identifier "WAN_GWv4_ "   
2021-11-10T17:00:24   dpinger[89102]   WAN_GWv4_ X.255.0.37: sendto error: 55



It would be great if we could find a solution and suggestion for this problem, thank you for your valuable information sharing.

That looks ok, I am wondering if you can include the logs when IDS fails, It seems that it is running successfully.

are you running Suricata and Sensei on the same interface?  It seems that Suricata is crashing and that is causing your gateway monitoring to flap, can you include logs from Suricata?

I am trying to setup a site to site VPN.  I created a new VLAN and gateway on the remote VPN since the inside networks are overlapping.

The connection establishes, I can ping from the Opnsense firewall the remote VPN host, however I cannot connect from the "Inside" Vlan.
I am not sure if the return traffic is hairpinning back to the local LAN and not back out the openVPN interface.


Side A (Client):

LAN:192.168.1.0/24
Tunnel: 10.80.80.0/24
Remote Network: 10.81.81.0/24

Note: I am using Gateway groups, HA WAN

Side B (Server)

LAN: 192.168.1.0/24 (not used)
Vlan99: 10.81.81.0/24 (used for VPN)
Tunnel: 10.80.80.0/24
Local Network: 10.81.81.0/24 (Vlan99)


Ping From firewall to remote host:
root@cerberus:~ # ping 10.81.81.10
PING 10.81.81.10 (10.81.81.10): 56 data bytes
64 bytes from 10.81.81.10: icmp_seq=0 ttl=63 time=81.705 ms
64 bytes from 10.81.81.10: icmp_seq=1 ttl=63 time=72.062 ms

SSH/WEB from Side A to Side B:

2021-08-16T19:49:16   filterlog[17007]   116,,,fae559338f65e11c53669fc3642c93c2,ovpnc4,match,pass,out,4,0x0,,63,0,0,DF,6,tcp,60,192.168.1.19,10.81.81.10,58012,22,0,S,256715406,,29200,,mss;sackOK;TS;nop;wscale
2021-08-16T19:49:12   filterlog[17007]   116,,,fae559338f65e11c53669fc3642c93c2,ovpnc4,match,pass,out,4,0x0,,127,0,0,DF,6,tcp,48,192.168.1.24,10.81.81.10,51943,443,0,S,749930554,,64240,,mss;nop;nop;sackOK
2021-08-16T19:49:12   filterlog[17007]   116,,,fae559338f65e11c53669fc3642c93c2,ovpnc4,match,pass,out,4,0x0,,127,0,0,DF,6,tcp,48,192.168.1.24,10.81.81.10,50996,443,0,S,313488011,,64240,,mss;nop;nop;sackOK


SSH/WEB from Side B to Side A (return traffic)

2021-08-17T00:48:43   filterlog[27813]   77,,,0,em0_vlan99,match,pass,out,4,0x0,,126,0,0,DF,6,tcp,48,192.168.1.24,10.81.81.10,59967,443,0,S,1496152610,,64240,,mss;nop;nop;sackOK,fae559338f65e11c53669fc3642c93c2
2021-08-17T00:47:39   filterlog[27813]   77,,,0,em0_vlan99,match,pass,out,4,0x0,,62,0,0,DF,6,tcp,60,192.168.1.19,10.81.81.10,57662,22,0,S,1969582485,,29200,,mss;sackOK;TS;nop;wscale,fae559338f65e11c53669fc3642c93c2
2021-08-17T00:46:33   filterlog[27813]   77,,,0,em0_vlan99,match,pass,out,4,0x0,,62,0,0,DF,6,tcp,60,192.168.1.19,10.81.81.10,57662,22,0,S,1969582485,,29200,,mss;sackOK;TS;nop;wscale,fae559338f65e11c53669fc3642c93c2


Rules:

Side A:

Inside:
IPv4 *    *    *    10.81.81.0/24    *    *
OpenVPN
IPv4 *    *    *    10.81.81.0/24    *    *    *

Side B:

Vlan99:
IPv4 *    *    *    *    *    *    *
OpenVPN:
IPv4 *    *    *    *    *    *    *


I haven't had a chance to run a remote tcpdump, I did run it last night on side A and can see the VPN traffic flow out, but I don't think im seeing return traffic hit:

00:00:00.126673 rule 116/0(match): pass out on ovpnc4: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
   192.168.1.19.42478 > 10.81.81.10.22: Flags , cksum 0x6481 (correct), seq 4135526895, win 29200, options [mss 1420,sackOK,TS val 3650534517 ecr 0,nop,wscale 7], length 0

So some things that might help, decrease ring size of the network interface that is being monitored. Disable HT, Enable/Disable some of these bios settings [1], make sure the system is set to performance mode for CPU Freq scaling.


https://www.academia.edu/33882347/Suricata_Extreme_Performance_Tuning

Having the same issue, I found that if I disable and enable any firewall rule and apply it the problem is fixed.  There must be something that isnt reloading the HA rules on a gateway failure.

Howdy!

I am setting up a Site to Site VPN using OpenVPN.  I created a new OpenVPN endpoint and assigned it a 10.69.69.0/24 network.  I would like to get a 1:1 NAT going so I can give selective access across the VPN without exposing my internal network to the remote VPN.  I am having some issues setting the NAT 1:1 to work correctly, I have attached my 1:1 rule along with my FW rule, looking to see what else I might need to get this to work right.

Thanks!

I ended up creating two default routes to the monitoring destinations and removing the DNS IPs from Opnsense, it seems to be working as I want now.

Tim

Attached NAT rules.

Hello,

I have been running a multi-wan failover for a few months now.  Last week i decided to make the leap into DoT and got that setup with Unbound + Adguard plugin.  I setup Unbound to listen on port 5153 and set Adguard to point to Unbound as the upstream DNS resolver.  I setup a portward to redirect all DNS traffic to the local gateway of whatever subnet the client is on. 

I noticed that whatever I did i was always getting redirected to the primary remote health checker for the multiwan setup.  IE I set Cloudflare to be my unbound DoT resolver, but when having DNS per interface listed in System-> Settings -> General it would not respect any portforwards nor unbound DNS upstream.

If I remove the DNS resolvers from opnsense's WAN interfaces, unbound starts to work, nowever dpinger seems to use the primary WAN to send requests out and not the backup WAN's monitoring interface.

Any help would be appreciated!


Thanks,
Tim

Could it be DNS blackhole?  Can you set static IP on phone and use upstream DNS?

Seems that Freenas is having the same issues around iflib: https://jira.ixsystems.com/plugins/servlet/mobile#issue/NAS-107593

They posted a work around that may or may not help people here:

sysctl net.iflib.min_tx_latency=1

I got it sorted, the "main" rules didnt show up in the GUI, ended up finding this and creating a policy and whitelisting rules 11, 16 and 1206 worked!

Thank you @Fright

For the request buffering I checked on nginx website and applied that change, once i applied that change everything started to pass the remote connectivity checks.  I did some more digging with WAF, here is a snippit of my logs, how can i track down the rules triggering this?

==> /var/log/nginx/mail.xxx.com,exchange.ad.xxx.com,autodiscover.xxx.com,_autodiscover.xxx.com.error.log <==
2021/02/09 14:40:26 [error] 51224#100230: *38 NAXSI_EXLOG: ip=168.61.212.41&server=autodiscover.xxx.com&uri=%2FAutodiscover%2FAutodiscover.xml&id=16&zone=BODY&var_name=&content=, client: 168.61.212.41, server: mail.xxx.com, request: "POST /Autodiscover/Autodiscover.xml HTTP/1.1", host: "autodiscover.xxx.com"
2021/02/09 14:40:26 [error] 51224#100230: *38 NAXSI_FMT: ip=168.61.212.41&server=autodiscover.xxx.com&uri=/Autodiscover/Autodiscover.xml&vers=1.3&total_processed=1&total_blocked=1&config=block&zone0=BODY&id0=16&var_name0=, client: 168.61.212.41, server: mail.xxx.com, request: "POST /Autodiscover/Autodiscover.xml HTTP/1.1", host: "autodiscover.xxx.com"


2021/02/09 14:40:26 [error] 51224#100230: *39 NAXSI_EXLOG: ip=168.61.212.41&server=autodiscover.xxx.com&uri=%2FAutodiscover%2FAutodiscover.xml&id=11&zone=BODY&var_name=&content=, client: 168.61.212.41, server: mail.xxx.com, request: "POST /Autodiscover/Autodiscover.xml HTTP/1.1", host: "autodiscover.xxx.com"
2021/02/09 14:40:26 [error] 51224#100230: *39 NAXSI_FMT: ip=168.61.212.41&server=autodiscover.xxx.com&uri=/Autodiscover/Autodiscover.xml&vers=1.3&total_processed=2&total_blocked=2&config=block&zone0=BODY&id0=11&var_name0=, client: 168.61.212.41, server: mail.xxx.com, request: "POST /Autodiscover/Autodiscover.xml HTTP/1.1", host: "autodiscover.xxx.com"


Thanks for sticking with me on all this!  I appreciate it greatly!