Messages

For the request buffering I checked on nginx website and applied that change, once i applied that change everything started to pass the remote connectivity checks.  I did some more digging with WAF, here is a snippit of my logs, how can i track down the rules triggering this?

==> /var/log/nginx/mail.xxx.com,exchange.ad.xxx.com,autodiscover.xxx.com,_autodiscover.xxx.com.error.log <==
2021/02/09 14:40:26 [error] 51224#100230: *38 NAXSI_EXLOG: ip=168.61.212.41&server=autodiscover.xxx.com&uri=%2FAutodiscover%2FAutodiscover.xml&id=16&zone=BODY&var_name=&content=, client: 168.61.212.41, server: mail.xxx.com, request: "POST /Autodiscover/Autodiscover.xml HTTP/1.1", host: "autodiscover.xxx.com"
2021/02/09 14:40:26 [error] 51224#100230: *38 NAXSI_FMT: ip=168.61.212.41&server=autodiscover.xxx.com&uri=/Autodiscover/Autodiscover.xml&vers=1.3&total_processed=1&total_blocked=1&config=block&zone0=BODY&id0=16&var_name0=, client: 168.61.212.41, server: mail.xxx.com, request: "POST /Autodiscover/Autodiscover.xml HTTP/1.1", host: "autodiscover.xxx.com"


2021/02/09 14:40:26 [error] 51224#100230: *39 NAXSI_EXLOG: ip=168.61.212.41&server=autodiscover.xxx.com&uri=%2FAutodiscover%2FAutodiscover.xml&id=11&zone=BODY&var_name=&content=, client: 168.61.212.41, server: mail.xxx.com, request: "POST /Autodiscover/Autodiscover.xml HTTP/1.1", host: "autodiscover.xxx.com"
2021/02/09 14:40:26 [error] 51224#100230: *39 NAXSI_FMT: ip=168.61.212.41&server=autodiscover.xxx.com&uri=/Autodiscover/Autodiscover.xml&vers=1.3&total_processed=2&total_blocked=2&config=block&zone0=BODY&id0=11&var_name0=, client: 168.61.212.41, server: mail.xxx.com, request: "POST /Autodiscover/Autodiscover.xml HTTP/1.1", host: "autodiscover.xxx.com"


Thanks for sticking with me on all this!  I appreciate it greatly!

Do you have any recommended rules that should be disabled in WAF?  It seems that just enabling WAF without any rules selected makes the remote connectivity tester to fail.  I don't see rules blocking in waf_denied.access.log

Code Select Expand


168.61.212.41 - - [09/Feb/2021:12:22:14 -0600] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 405 150 "-" "Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4615; Pro; MS Connectivity Analyzer)" "-"
168.61.212.41 - - [09/Feb/2021:12:22:14 -0600] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 405 175 "-" "Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4615; Pro; MS Connectivity Analyzer)" "-"
168.61.212.41 - - [09/Feb/2021:12:22:15 -0600] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 405 150 "-" "Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4615; Pro; MS Connectivity Analyzer)" "-"
168.61.212.41 - - [09/Feb/2021:12:22:15 -0600] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 405 175 "-" "Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4615; Pro; MS Connectivity Analyzer)" "-"


I got it working!  I had to turn off response and request buffering, and set the Maximum Body Size to 2G.

Perfect,

I got to this part of the remote test and it is still failing:

Code Select Expand


Attempting to ping the MAPI Mail Store endpoint with identity: xxxxx-9188-4474-9811-0ef5db77cf19@xxx.com:6001.
The attempt to ping the endpoint failed.
Additional Details
An RPC error was thrown by the RPC Runtime process. Error 1818 CallCancelled
RPC Status: 1818 CallCancelled
Timestamp: 2/9/2021 5:33:20 PM
Generating Component: 14 (WinHttp)
Status: 1818
Detection Location: 1390 (HTTP2ClientVirtualConnection__ClientOpenInternal10)
Flags: 0
Parameters:
30000



Perfect, I was trying to stage all the changes before applying the config.  For matching up the UUID for the directory, what should I look for as I have other services besides exchange:

Code Select Expand


root@cerberus:~ # grep include /usr/local/etc/nginx/nginx.conf
include       mime.types;
js_include /usr/local/opnsense/scripts/nginx/ngx_functions.js;
include http_post/*.conf;
# include nginx_web.conf;
include opnsense_http_vhost_plugins/*.conf;
    #include tls.conf;
      include       fastcgi_params;
      include        fastcgi_params;
    include b21a09c6-db5d-4ce0-bfe0-dd7e31d89811_pre/*.conf;
    include 7b884db8-eafa-43d0-bdae-ec4a66a97cad_post/*.conf;
    include b21a09c6-db5d-4ce0-bfe0-dd7e31d89811_post/*.conf;
    #include tls.conf;
      include       fastcgi_params;
      include        fastcgi_params;
    include 7b844599-dbd4-4b45-ad56-e22dd094d6d5_pre/*.conf;
    include 016848f7-b0de-401b-b961-b7bfeed575ab_post/*.conf;
    include 7b844599-dbd4-4b45-ad56-e22dd094d6d5_post/*.conf;
    #include tls.conf;
      include       fastcgi_params;
      include        fastcgi_params;
    include f94e4a2e-e9ea-419b-a5b8-763890eaa89b_pre/*.conf;
    include d9acf7d4-f0b5-4530-b574-ad4b28375e18_post/*.conf;
    include f94e4a2e-e9ea-419b-a5b8-763890eaa89b_post/*.conf;
    #include tls.conf;
      include       fastcgi_params;
      include        fastcgi_params;
    include f857a060-bf3e-4d6d-af2a-5073ee117b2d_pre/*.conf;
    include cc8de2e8-3e31-4994-bd0c-5712a874fb04_post/*.conf;
    include f857a060-bf3e-4d6d-af2a-5073ee117b2d_post/*.conf;
    #include tls.conf;
      include       fastcgi_params;
      include        fastcgi_params;
    include 040f7fe9-396e-4b8e-8e4c-19a3eff357c4_pre/*.conf;
    include 5eeb9519-02e7-4dee-9368-f5dd50f5779d_post/*.conf;
    include 040f7fe9-396e-4b8e-8e4c-19a3eff357c4_post/*.conf;
    #include tls.conf;
      include       fastcgi_params;
      include        fastcgi_params;
    include fcdd1729-8503-4055-80e3-cf74112ca928_pre/*.conf;
    include 8498a998-7bbf-4401-80aa-7498170d3a34_post/*.conf;
    include fcdd1729-8503-4055-80e3-cf74112ca928_post/*.conf;
    include opnsense_stream_vhost_plugins/*.conf;



I just blew away the installed package and reinstalled, same issue still, this is the last few lines of the running config.

root@cerberus:~ # tail -n5 /usr/local/etc/nginx/nginx.conf
    include opnsense_stream_vhost_plugins/*.conf;

}
# mail {
# }

Yes I did, nginx tries to start when I hit apply:

Code Select Expand


root@cerberus:~ # grep -n -A1 -B1 _post /usr/local/opnsense/service/templates/OPNsense/Nginx/location.conf
208-{% endif %}{# honeypot #}
209:    include {{ location['@uuid'] }}_post/*.conf;
210-}


I do see some warnings appear with nginx starts but it seems that is around SSL cert joining?

[09-Feb-2021 09:49:16 America/Chicago] PHP Warning:  Invalid argument supplied for foreach() in /usr/local/opnsense/scripts/nginx/setup.php on line 184

Howdy!

I got some time to try this, hitting a snag on the location config file not generating:

/usr/local/opnsense/service/templates/OPNsense/Nginx/location.conf

Code Select Expand


{% endif %}{# honeypot #}
    include {{ location['@uuid'] }}_post/*.conf;
}



root@cerberus:~ # grep include /usr/local/etc/nginx/nginx.conf
include       mime.types;
js_include /usr/local/opnsense/scripts/nginx/ngx_functions.js;
include http_post/*.conf;
# include nginx_web.conf;
include opnsense_http_vhost_plugins/*.conf;
    include opnsense_stream_vhost_plugins/*.conf;

Running version 20.7.8 Business Edition 
Kernel 21.1 (FreeBSD cerberus.underworld.local 12.1-RELEASE-p12-HBSD FreeBSD 12.1-RELEASE-p12-HBSD #0  3c6040c7243(stable/21.1)-dirty: Mon Jan 25 12:27:52 CET 2021     root@sensey:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64)

Hardware: https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E300-9D-8CN8TP.cfm
Ram: 64 GB ECC

When Suricata is enabled with IDS/IPS protection the max WAN speed is capped at around 650-670Mbps, with IPS mode disabled I can achieve full 1Gb/s down.

I did not see any of this issue with 20.7.5-NEXT, but since I upgraded to 20.7.8 and the -next kernels got removed I have no way to restore my performance back to satisfactory levels.

Thank you for the response, the reason I quoted you I was trying to pin-point what you are wanting us to test for and what artifacts we can transmit back for follow up.   I am not trying to finger point or assign blame to anyone or any project, I am trying to see if there is some tooling that we the users can consume and send back on what we are seeing on our end to cut down on the "we didn't see that in our testing" back and forth as it doesn't help resolve the issues at hand.

If at the end of the day this level of back and forth is helpful and there is no standardized troubleshooting tooling we can use so be it, but I would like to work on getting this issue resolved.

That being said what are the next steps I and the other people having issues with the 21.1 kernel with IDS/IPS enabled provide you with narrowing down the issue?

@klamath I'm not sure why your quoting my message, but as said if it's purely about netmap, one could try to use bridge to pinpoint issues if they exists for their setup. It's definitely not the case that there are issues in all releases, we test the hardware we provide on periodic bases and haven't seen a lot of (major) issues ourselves.

Quite some reports about performance are related to too optimistic assumptions (e.g. expecting 1Gbps IPS on an apu board for example) or drivers which aren't very well supported (we ship what's being offered upstream, if support isn't great in FreeBSD for netmap, it highly likely isn't great on our end). IPS needs quite some computing power and isn't comparable to normal routing/firewall functions at all in terms of requirements.

When it comes to testing,  we tend to offer test kernels and release candidates on periodic bases. To help catching issues up front, please do test, document behaviour when experiencing issues, and try to track them to FreeBSD bug reports if they exists. When there are fixes available upstream, we often assess if we can backport them into our system. Quite some fixes have been merged in the last versions for various drivers (with quite some help from the Sensei people as Franco mentioned), I haven't seen side affects in terms of performance myself, but that doesn't mean they don't exist for some drivers.


Best regards,

Ad

Is there any tooling that opnsense can provide or recommend the users experiencing slowness with IDS/IPS can consume to help close loop this issue?  I started using opnsense a few months ago and it seems that every minor and major release has some regression with netmap/iflib.  I think having some tooling the users can use to provide meaningful feedback to the dev team to chase down these issues would be useful to all.

I haven't seen performance issues on my setup between these versions, but since there have been quite some fixes around netmap in different kernels, it's often a good idea to check if it's Suricata causing issues or netmap.

There is a simple "bridge" tool for netmap available in the kernel source directory, if people want to check netmap behaviour on their hardware they can always build the tool and create a bridge between the card and the host-stack to rule out certain driver issues.

To install it, you need the kernel source directory in place (/sur/src), you can use the build tools (https://github.com/opnsense/tools) to checkout all sources on your machine.

When the sources are in place, you can build the tools using the following commands (on amd64):

Code Select Expand


cd /usr/src/tools/tools/netmap/
make bridge



Next make sure netmap isn't used (no Suricata or Sensei) and create  a bridge between the physical connection and the host stack, assuming the interface in question is called vmx2, the command would look like:

Code Select Expand


/usr/obj/usr/src/amd64.amd64/tools/tools/netmap/bridge -i netmap:vmx2 -i netmap:vmx2


Wait a few seconds and start the test again with OPNsense in between. When netmap isn't interfering the test with or without bridge should show roughly the same numbers.

Best regards,

Ad

I upgraded the kernel in my 20.7 install to 21.1 kernel and notice a speed drop when IPS is enabled.

IDS only:
Speedtest by Ookla

Server: ZochNet - Lincoln, TX (id = 20875)
ISP: Grande Communications
Latency:    22.87 ms   (0.73 ms jitter)
Download:   926.23 Mbps (data used: 1.0 GB)
Upload:    46.38 Mbps (data used: 66.3 MB)
Packet Loss:     0.0%

IDS/IPS:
Speedtest by Ookla

Server: ZochNet - Lincoln, TX (id = 20875)
ISP: Grande Communications
Latency:    19.63 ms   (1.04 ms jitter)
Download:   666.98 Mbps (data used: 626.3 MB)
Upload:    45.06 Mbps (data used: 67.3 MB)
Packet Loss:     0.0%

I did not have these speed issues when using the 20.7.5-next kernel branch with my hardware.

I am running the business version of opnsense, I recently did the upgrade from 20.7.5 to 20.7.8 and allowed the kernel to get upgraded, after the upgrade my setup was running at a capped speed with IDS/IPS enabled.  Having the -next kernel branch available would have saved me another upgrade to 21.1 tonight.   

Any other version than 20.7.8_4 does not have the signature fingerprint for 21.1.. I'm not sure why you are not on this version. ;)

There is the "-i" parameter to do insecure upgrades, but it's nicer to not use it and use the latest 20.7 release with:

# opnsense-update -kr 21.1
# opnsense-shell reboot


Cheers,
Franco

I will need to force an upgrade to 21.1 it seems to get the fixed kernel.

root@cerberus:~ # opnsense-update -zkr 21.1
Fetching kernel-21.1-amd64.txz: ... failed, no signature found
root@cerberus:~ # opnsense-update -kr 21.1
Fetching kernel-21.1-amd64.txz: ............ failed, signature invalid

Can I upgrade the kernel on 20.7 to 21.1 to get the netmap and intel driver fixes without upgrading the OS to 21.1?