OPNsense Forum

English Forums => Intrusion Detection and Prevention => Topic started by: GaardenZwerch on October 11, 2019, 03:58:38 pm

Title: No Internet access when IPS is on
Post by: GaardenZwerch on October 11, 2019, 03:58:38 pm
Hi,

I have been testing IDS, and my results are good so far. I see proper alerts for the rules I have enabled, so far so good.
When I turn on IPS, however, the internal networks don't have Internet access anymore.
I attach a screenshot of the flags and values I have set.

All internal networks are in VLANS attached to ixl0 (intel NIC with 10Gbps SFP modules).
 

I have successfully set fc to 0 on all interfaces (sysctl dev.ixl.0.fc=0, and so on)

What did I do wrong?


Thanks
Frank
Title: Re: No Internet access when IPS is on
Post by: mimugmail on October 11, 2019, 04:07:38 pm
Is this latest OPNsense? There were problems mit X710 cards recently .. do you have latest firmware on the NIC?
Title: Re: No Internet access when IPS is on
Post by: GaardenZwerch on October 11, 2019, 04:13:32 pm
Hi Michael,
OPNsense is 19.7.3
How do I check (or event update) the NIC firmware?

Thanks a lot,

Frank
Title: Re: No Internet access when IPS is on
Post by: GaardenZwerch on October 11, 2019, 04:40:51 pm
Re,
I have now updated to latest (19.7.5).

Here's what pciconv -lvs has to say about my NIC:

Code: [Select]
ixl0@pci0:1:0:0: class=0x020000 card=0x00088086 chip=0x15728086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller X710 for 10GbE SFP+'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 129 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
                 link x8(x8) speed 8.0(8.0) ASPM L1(L1)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 0cf5dbfffffefd3c
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0110, VF RID Stride 0x0001
                     VF Device ID 0x154c
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1
    ecap 0019[1d0] = PCIe Sec 1 lane errors 0xff
Title: Re: No Internet access when IPS is on
Post by: mimugmail on October 11, 2019, 05:30:29 pm
Nic firmware is listed via sysctl -a | grep ixl
Title: Re: No Internet access when IPS is on
Post by: GaardenZwerch on October 11, 2019, 06:21:49 pm
Code: [Select]
dev.ixl.0.fw_version: fw 6.0.48442 api 1.7 nvm 6.01 etid 800035b0 oem 1.262.0
dev.ixl.0.%pnpinfo: vendor=0x8086 device=0x1572 subvendor=0x8086 subdevice=0x0008 class=0x020000
dev.ixl.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.PEG0.PEGP
dev.ixl.0.%driver: ixl
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k

is what I get there.

Thanks
Title: Re: No Internet access when IPS is on
Post by: mimugmail on October 11, 2019, 07:51:31 pm
Hm, should be ok, but I didnt test yet if it still breaks the nic
Title: Re: No Internet access when IPS is on
Post by: Pocket_Sevens on October 13, 2019, 03:38:56 am
I’ve noticed Suricata seems to have problems with VLANS. My AP has 2 SSID’s: one set with a VLAN and one without.  When I run Suricata with IPS on, I have connectivity on the SSID without a VLAN but lose connectivity on the other SSID (with the VLAN).  No matter my settings, I still lose connectivity on my VLAN connected SSID.
Title: Re: No Internet access when IPS is on
Post by: mimugmail on October 13, 2019, 06:42:02 am
Then maybe it's better to use WAN interface?
Title: Re: No Internet access when IPS is on
Post by: GaardenZwerch on October 14, 2019, 10:43:03 am
Then maybe it's better to use WAN interface?

The trouble is that the logs are less useful, also for identifying false positives.
And: I manage my sites from the outside, so if I sabotage the WAN link, I cannot undo this without going there physically. Yes, WAN is ixl too  ::)

I’ve noticed Suricata seems to have problems with VLANS.

Can anybody confirm this? The GUI seems to be clear that you need promiscuous 'on' and run suricata on the physical NIC, but I have seen ppl state the opposite here in the forum.

I will try this in a lab, but with igb interfaces.

Title: Re: No Internet access when IPS is on
Post by: Pocket_Sevens on October 14, 2019, 05:55:20 pm
Quote
Can anybody confirm this? The GUI seems to be clear that you need promiscuous 'on' and run suricata on the physical NIC, but I have seen ppl state the opposite here in the forum.

I will try this in a lab, but with igb interfaces.

Hi GaardenZwerch.  Here is my setup:

Main SSID from AP pointing to LAN (no VLAN).
IOT SSID from AP pointing to LAN with VLAN (to separate IOT from the rest of the LAN)

I have set IPS with WAN/Main LAN with and without promiscuous mode on and off.  I have set IPS with WAN/Main LAN/IOT LAN with and without promiscuous mode on and off. 

No matter what I do, I seem to run into the same issue:  Main LAN has connectivity; IOT LAN does not.  (Note, if I select IOT LAN (with our without promiscuous mode) and hit Apply, I do get connectivity on that LAN for a while but then it loses access later.  Only selecting the Main LAN causes immediate disconnect on the IOT LAN).

If I turn IPS off, everything works (since it's only in detection mode).

Let me know if you want me to post or DM any of my setup to help in testing.
Title: Re: No Internet access when IPS is on
Post by: Cajuba on October 14, 2019, 06:41:18 pm

Can anybody confirm this? The GUI seems to be clear that you need promiscuous 'on' and run suricata on the physical NIC, but I have seen ppl state the opposite here in the forum.


Yes, I can confirm this. 
Following the GUI‘s instructions makes VLANs unusable. My workaround is to put all devices I want to be protected by IPS into seperate VLANs / subnets and turn IPS on on these interfaces. My native non-VLAN subnet remains  “unprotected“.
Title: Re: No Internet access when IPS is on
Post by: Pocket_Sevens on October 14, 2019, 07:06:39 pm
Yes, I can confirm this. 
Following the GUI‘s instructions makes VLANs unusable. My workaround is to put all devices I want to be protected by IPS into seperate VLANs / subnets and turn IPS on on these interfaces. My native non-VLAN subnet remains  “unprotected“.

Is this with promiscuous mode turned on or off?
Title: Re: No Internet access when IPS is on
Post by: Cajuba on October 14, 2019, 08:06:10 pm

Is this with promiscuous mode turned on or off?

It‘s turned on.
Title: Re: No Internet access when IPS is on
Post by: dcol on October 15, 2019, 11:53:32 pm
IDS will lose internet if you restart the suricata service. You have to reboot to fix.
But updating the rules with a download does not affect the connection.

I always wondered why it did this?
Title: Re: No Internet access when IPS is on
Post by: GaardenZwerch on October 16, 2019, 04:13:55 pm
OK,
my tests (both Lab and Production) confirm this.
I run suricata on each VLAN and leave promiscuous mode on, and IPS works. I have tested with igb and ixl interfaces.

Thanks,
Frank
Title: Re: No Internet access when IPS is on
Post by: Cajuba on October 18, 2019, 11:30:41 am
Meanwhile, I am a bit confused...  :-\

As I wrote in my previous posts I had to run IPS on my VLAN Interfaces, but not on the physical interface. Otherwise I would not get DHCP leases on my VLAN Subnets an I could not connect to the internet.

Then the world turned upside down...  :o
A few days ago I had to perform several reboots after some issues with power supply. After that I was not able to get a DHCP lease with the exact config that used to work before.  So I played around a bit.  After configuring IPS running on the physical LAN interface, but not on the VLAN interfaces anymore I immediately got DHCP Leases on all of my VLAN Subnets. This seems to be stable so far.

I have no idea why the system's behaviour changed after the reboots. From my point of view this seems to be quite strange...
Title: Re: No Internet access when IPS is on
Post by: Pocket_Sevens on October 18, 2019, 03:01:26 pm
OK,
my tests (both Lab and Production) confirm this.
I run suricata on each VLAN and leave promiscuous mode on, and IPS works. I have tested with igb and ixl interfaces.

Thanks,
Frank

Hi Frank.

Thanks for testing this.  Just to clarify: was this on the VLANs only or also the physical LAN interface?
Title: Re: No Internet access when IPS is on
Post by: Pocket_Sevens on October 18, 2019, 03:03:51 pm
Meanwhile, I am a bit confused...  :-\

As I wrote in my previous posts I had to run IPS on my VLAN Interfaces, but not on the physical interface. Otherwise I would not get DHCP leases on my VLAN Subnets an I could not connect to the internet.

Then the world turned upside down...  :o
A few days ago I had to perform several reboots after some issues with power supply. After that I was not able to get a DHCP lease with the exact config that used to work before.  So I played around a bit.  After configuring IPS running on the physical LAN interface, but not on the VLAN interfaces anymore I immediately got DHCP Leases on all of my VLAN Subnets. This seems to be stable so far.

I have no idea why the system's behaviour changed after the reboots. From my point of view this seems to be quite strange...

Hey Cajuba.  Did you upgrade to 19.7.5_5 per chance?
Title: Re: No Internet access when IPS is on
Post by: GaardenZwerch on October 21, 2019, 03:29:21 pm
OK,
my tests (both Lab and Production) confirm this.
I run suricata on each VLAN and leave promiscuous mode on, and IPS works. I have tested with igb and ixl interfaces.

Thanks,
Frank

Hi Frank.

Thanks for testing this.  Just to clarify: was this on the VLANs only or also the physical LAN interface?


Hi,

IPS only on the VLANs, not on the physical NIC. Promiscuous mode ON.

Best regards,
Title: Re: No Internet access when IPS is on
Post by: Cajuba on October 21, 2019, 04:32:50 pm
Hey Cajuba.  Did you upgrade to 19.7.5_5 per chance?

Yes, my device is running on 19.7.5_5