No Internet access when IPS is on

[GaardenZwerch]

Hi,

I have been testing IDS, and my results are good so far. I see proper alerts for the rules I have enabled, so far so good.
When I turn on IPS, however, the internal networks don't have Internet access anymore.
I attach a screenshot of the flags and values I have set.

All internal networks are in VLANS attached to ixl0 (intel NIC with 10Gbps SFP modules).
 

I have successfully set fc to 0 on all interfaces (sysctl dev.ixl.0.fc=0, and so on)

What did I do wrong?


Thanks
Frank

[mimugmail]

Is this latest OPNsense? There were problems mit X710 cards recently .. do you have latest firmware on the NIC?

[GaardenZwerch]

Hi Michael,
OPNsense is 19.7.3
How do I check (or event update) the NIC firmware?

Thanks a lot,

Frank

[GaardenZwerch]

Re,
I have now updated to latest (19.7.5).

Here's what pciconv -lvs has to say about my NIC:

Code: [Select]

ixl0@pci0:1:0:0: class=0x020000 card=0x00088086 chip=0x15728086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller X710 for 10GbE SFP+'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 129 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
                 link x8(x8) speed 8.0(8.0) ASPM L1(L1)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 0cf5dbfffffefd3c
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0110, VF RID Stride 0x0001
                     VF Device ID 0x154c
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1
    ecap 0019[1d0] = PCIe Sec 1 lane errors 0xff


[mimugmail]

Nic firmware is listed via sysctl -a | grep ixl

[GaardenZwerch]

Code: [Select]

dev.ixl.0.fw_version: fw 6.0.48442 api 1.7 nvm 6.01 etid 800035b0 oem 1.262.0
dev.ixl.0.%pnpinfo: vendor=0x8086 device=0x1572 subvendor=0x8086 subdevice=0x0008 class=0x020000
dev.ixl.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.PEG0.PEGP
dev.ixl.0.%driver: ixl
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.9.9-k

is what I get there.

Thanks

[mimugmail]

Hm, should be ok, but I didnt test yet if it still breaks the nic

[Pocket_Sevens]

I’ve noticed Suricata seems to have problems with VLANS. My AP has 2 SSID’s: one set with a VLAN and one without.  When I run Suricata with IPS on, I have connectivity on the SSID without a VLAN but lose connectivity on the other SSID (with the VLAN).  No matter my settings, I still lose connectivity on my VLAN connected SSID.

[mimugmail]

Then maybe it's better to use WAN interface?

[GaardenZwerch]

Then maybe it's better to use WAN interface?

The trouble is that the logs are less useful, also for identifying false positives.
And: I manage my sites from the outside, so if I sabotage the WAN link, I cannot undo this without going there physically. Yes, WAN is ixl too 

I’ve noticed Suricata seems to have problems with VLANS.

Can anybody confirm this? The GUI seems to be clear that you need promiscuous 'on' and run suricata on the physical NIC, but I have seen ppl state the opposite here in the forum.

I will try this in a lab, but with igb interfaces.

[Pocket_Sevens]

Can anybody confirm this? The GUI seems to be clear that you need promiscuous 'on' and run suricata on the physical NIC, but I have seen ppl state the opposite here in the forum.

I will try this in a lab, but with igb interfaces.

Hi GaardenZwerch.  Here is my setup:

Main SSID from AP pointing to LAN (no VLAN).
IOT SSID from AP pointing to LAN with VLAN (to separate IOT from the rest of the LAN)

I have set IPS with WAN/Main LAN with and without promiscuous mode on and off.  I have set IPS with WAN/Main LAN/IOT LAN with and without promiscuous mode on and off. 

No matter what I do, I seem to run into the same issue:  Main LAN has connectivity; IOT LAN does not.  (Note, if I select IOT LAN (with our without promiscuous mode) and hit Apply, I do get connectivity on that LAN for a while but then it loses access later.  Only selecting the Main LAN causes immediate disconnect on the IOT LAN).

If I turn IPS off, everything works (since it's only in detection mode).

Let me know if you want me to post or DM any of my setup to help in testing.

[Cajuba]

Can anybody confirm this? The GUI seems to be clear that you need promiscuous 'on' and run suricata on the physical NIC, but I have seen ppl state the opposite here in the forum.

Yes, I can confirm this. 
Following the GUI‘s instructions makes VLANs unusable. My workaround is to put all devices I want to be protected by IPS into seperate VLANs / subnets and turn IPS on on these interfaces. My native non-VLAN subnet remains  “unprotected“.

[Pocket_Sevens]

Yes, I can confirm this. 
Following the GUI‘s instructions makes VLANs unusable. My workaround is to put all devices I want to be protected by IPS into seperate VLANs / subnets and turn IPS on on these interfaces. My native non-VLAN subnet remains  “unprotected“.

Is this with promiscuous mode turned on or off?

[Cajuba]

Is this with promiscuous mode turned on or off?

It‘s turned on.

[dcol]

IDS will lose internet if you restart the suricata service. You have to reboot to fix.
But updating the rules with a download does not affect the connection.

I always wondered why it did this?