[SOLVED] Intrusion Detection issues with IPS Enabled

[IvanP]

Hi all,

Since a few days we've a Deciso OPNsense firewall (Dual A10 QC SSD rack) in use in front of our webservers. Before this OPNsense firewall we had 2 PFsense firewall's.
The OPNsense firewalls are configured in HA (CARP).

We want to use the Intrusion Detection service of OPNsense. So to do this I enabled IDS and IPS.
Then I checked the 'Rules'-tab and saw some active rules without activating any ruleset (no problem in my opinion). Before enabling any ruleset we did some tests. We came to the conclusion that most of the websites are working fine, except our PRTG page (monitoring: https://www.paessler.com/prtg) and also creating a new email with Roundcube webmail was not working. Those pages keep saying 'Connecting' in Google Chrome and the circle inside the tab keeps rotating.
On the 'Alert'-tab was no information at all. To be sure that no rule can be the problem, I disabled all the rules and did a retry, unfortunately the same result.
I can solve the issues of the pages not loading by disabling IPS and only let IDS enabled. Then I can also enable all rules and the page load normally. Off course we want to use IPS to auto block some bad traffic

Is there a solution to enable IPS and let the webpages function properly?

P.s. all the hardware acceleration settings are disabled:
 - hardware checksum offload
 - hardware TCP segmentation offload
 - hardware large receive offload
 - VLAN Hardware Filtering

Versions:
OPNsense 16.1.13-amd64   
FreeBSD 10.2-RELEASE-p14   
OpenSSL 1.0.2h 3 May 2016

[AdSchellevis]

Hi IvanP,

On which interface (or interfaces) is IPS enabled?

Best regards,

Ad

[IvanP]

Hi Ad,

IPS is only enabled on the WAN interface. This WAN interface is a VLAN on a LAGG of 2 physical interfaces.
I attached 3 screenshots of the configuration to make it more clear.

Ivan

[AdSchellevis]

Hi Ivan,

IPS uses netmap, which relies heavily on driver support.
The odd thing is that not all your traffic seems to drop, only parts of it. you could try to do some packet capture on the wan interface first to see what it tries todo for that part of the traffic.

If you have time, can you test some different scenarios to see when it goes wrong? (backup your config first  )

step 1) the most basic setup, one device using wan on a single port, no vlan, no lagg, no carp
step 2) extend your setup with carp, like you have now, but without vlan/lagg and test again
step 3) then add your vlan to this setup and test again
step 4) finally add lagg to the setup


Best regards,

Ad

[IvanP]

Hi Ad,

Thanks for the quick replies.

It's unfortunately not possible for us to test the steps you mention. Because the device is in a production environment at the moment with live traffic, so it's impossible to reconfigure vlan, lagg and interface settings. Also physical access to the device is difficult (it's in a datacenter 150km away).

Is there a possibility to enable extra logging, some sort of debug, for IPS and see what it's doing?

Kind regards,

Ivan

[AdSchellevis]

Hi Ivan,

Did some testing here at our office and it looks like netmap/suricata doesn't handle vlan's correctly, which could very well be your issue.
I have no lagg setup available at the moment, so we have to go step by step here I guess.

The setup which should work, but at the moment cannot be configured in the user interface, is to listen on the physical interface (emX) in promisc mode.

I've added an issue to our tracker https://github.com/opnsense/core/issues/935

Only disadvantages are you cannot enable IPS on one of the vlans and you must listen to all traffic because the physical interface itself doesn't have an address which it can listen for.

Best regards,

Ad

[IvanP]

Hi Ad,

Thanks again for your great assistance!

I was rechecking our config and I think I have the possibility to remove vlan 2 from lagg0 (WAN). Because vlan2 is the only vlan on the WAN LAGG (lagg0). So I can change the switch ports from trunk- to access-ports and map the WAN port assignment in OPNsense (Interfaces: Assignments) directly to lagg0 (WAN). In this way the vlan-tagging will be done by the switch and not OPNsense. For OPNsense it will look like a normal lagg-port.

Do you think I will bypass the vlan problem on netmap/Suricata with the above config changes?

I think I'll be able to apply this change upcoming weekend.

Kind regards,

Ivan

[AdSchellevis]

Hi Ivan,

I'm not sure if you won't run into issues with the lagg interface, I haven't tested that scenario on my end.

I think I can provide you with some files before the weekend to patch your machine with the proposed changes, but its advisable to make sure that you are onside and can use the console to the box.

Best regards,

Ad

[IvanP]

Hi Ad,

For now it's not necessary to provide us the patched files, we'll wait for the official release.

I'll will try the change I mentioned before and give it another try with IPS this weekend. I'll keep you posted about my findings.

Again many thanks for your assistance and quick reply's!

Kind regards,

Ivan

[IvanP]

Hi Ad,

I did the changes I mentioned before. So there is no vlan on the WAN interface. The WAN interface is just assigned to LAGG0.

I did a complete reboot of both firewalls after I disabled all IDS/IPS settings. After the reboot I completely reconfigured Intrusion Prevention.. Unfortunately the PRTG-webpage is still not working, all other websites we're hosting are working fine with IPS enabled.

A conclusion for now is that getting rid of the VLAN config on the WAN-interface is not the solution.

Is there an option to exclude for example the internal or external IP? Or when a certain NAT-rule is applied that there will be no IPS processing for that traffic? (I now this is for example possible with a Fortinet Firewall).

[AdSchellevis]

Hi Ivan,

You cannot skip IPS processing when using netmap, because it captures the traffic before it hits the normal driver.
Given that the traffic is not blocked (no alerts), excluding things from within suricata probably won't make a difference.

The changes to capture on the actual device (emX) are in GitHub already (https://github.com/opnsense/core/commit/3bacc74549cb5e8ea4e50087bb9ed175e1104a05), they probably will be available in stable within a few weeks.

It still is odd that only some hosts are not accessible, but I recommend waiting for the option to enable promisc mode on emX for your device.

Best regards,

Ad

[IvanP]

Hi Ad,

Thanks again for the clear answers.
We'll wait for the changes to become stable.

Kind regards,

Ivan

[franco]

Merge target for this is 16.1.15 after an extra round of testing in 16.1.14-devel.