Opnsense 21.7.5 to 21.7.6 Suricata 6.0.4 killing WAN internet connection

[patan32]

Describe the bug
After updating opnsense to version OPNsense 21.7.6-amd64 and enabling suricata 6.0.4 version stops my internet connectivity. The service will start as normal with nothing in the logs and after few hours it will kill my internet connection and in the wan interface it will remove the ip and replace it with .dhcp. As soon as i restart suricata service my internet comes back and i can see my public ip address (nothing will show in the suricata and system logs).

I am using all the rule sets in suricata and created policies. I have been running the same system with version 21.7.5 which had no issues and with the same rule set. After upgrading to version 21.7.6 the problem appeared and it will kill my internet connection. I run my internet connection directly from ISP ONT on VLAN 10. On suricata i have WAN selected for interface which is a VLAN 10 and i have enabled Promiscuous mode. I have not changed anything on my config side.

I knew the problem started after upgrading to latest version. I re-imaged the box to 21.7.1 and using the manual Flavour settings under updates i put this code "21.7/MINT/21.7.5/OpenSSL/" to upgrade to version 21.7.5 and restoring my config. I enabled suricata and i haven't had any issues. This tells me the issue is with suricata killing internet connection without any logs. I am not sure what has changed with the new version of suricata. I tested on Sophos SG430 Rev 1 hardware and the issue is the same.

Relevant log files
Suricata shows no faults or issues in the logs when it drops internet connection. System logs show no issues.

I did see this though in the system logs, but i also see this error when i am running version Suricata 6.0.3_3.

2021-12-12T13:45:06 kernel 906.106372 [ 853] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
2021-12-12T13:45:06 kernel 906.015935 [ 853] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
2021-12-12T13:45:06 kernel 905.926248 [ 853] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048

Environment
Software version used and hardware type if relevant.
e.g.:
Hardware is : Sophos SG330 Rev 1
Currently running OPNsense 21.7.5-amd64FreeBSD 12.1-RELEASE-p21-HBSDOpenSSL 1.1.1l 24 Aug 2021
CPU type | Intel(R) Core(TM) i5-4570S CPU @ 2.90GHz (4 cores)
'I210 Gigabit Network Connection'

ix0@pci0:1:0:0: class=0x020000 card=0x02031374 chip=0x10c68086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82598EB 10-Gigabit AF Dual Port Network Connection'
class = network
ix1@pci0:1:0:1: class=0x020000 card=0x02031374 chip=0x10c68086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82598EB 10-Gigabit AF Dual Port Network Connection'
class = network
igb0@pci0:2:0:0: class=0x020000 card=0x30e015bb chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
igb1@pci0:3:0:0: class=0x020000 card=0x30e015bb chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
igb2@pci0:4:0:0: class=0x020000 card=0x30e015bb chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
igb3@pci0:5:0:0: class=0x020000 card=0x30e015bb chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
igb4@pci0:6:0:0: class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
igb5@pci0:7:0:0: class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
igb6@pci0:8:0:0: class=0x020000 card=0x000015bb chip=0x15218086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'I350 Gigabit Network Connection'
class = network
igb7@pci0:8:0:1: class=0x020000 card=0x000015bb chip=0x15218086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'I350 Gigabit Network Connection'
class = network
igb8@pci0:9:0:0: class=0x020000 card=0x0000ffff chip=0x15228086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'I350 Gigabit Fiber Network Connection'
class = network
igb9@pci0:9:0:1: class=0x020000 card=0x0000ffff chip=0x15228086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'I350 Gigabit Fiber Network Connection'
class = network

https://github.com/opnsense/plugins/issues/2706

[mimugmail]

Can you post output of console when it stalls? (serial cable or display port)

[AdSchellevis]

Officially IPS doesn't support virtual interfaces, such as VLAN's (https://docs.opnsense.org/manual/ips.html#choosing-an-interface). In 21.7.6 we added the new netmap api which seems to enable emulated mode in these cases.

I've seen a similar setup yesterday where someone added a bunch of vlan's to IPS, which starts but will not detect anything in previous versions of suricata (for a functional setup you need to capture the parent in promisc mode).

In 21.7.7 we will temporary revert the new api, which will then be back in 22.1, I haven't tried if it works for vlan's in 22.1, maybe it does, maybe it doesn't, if it doesn't we may have to tighten validations at some point from preventing people to choose these type of setups.

Best regards,

Ad

[wuwzy]

The following is my solution, just temporarily. It seems that this is a common problem, please solve it head-on.

https://forum.opnsense.org/index.php?topic=25877.0

[AdSchellevis]

The following is my solution, just temporarily. It seems that this is a common problem, please solve it head-on.

https://forum.opnsense.org/index.php?topic=25877.0

I think I already explained what we are going to do in the next release (revert, but bring it back in 22.1), enabling IPS on VLAN's will be pointless though even if it starts, it is highly unlikely it will detect/block anything.

[guest31184]

In my system, I have VLANs, but the IPS runs on the physical device only (I have WAN and LAN=igb1) and several VLANs which are assigned to LAN. I still had the broken connections (on LAN side) with 21.7.6.
Am I wrong assuming that this should be working - or is this also a configuration where IPS would not work (I got several alarms (I hope also blocks) on IPS mode)?

[AdSchellevis]

On the physical interface it should work in principle, provided the hardware is properly supported. You can always try the beta for 22.1 by the way, maybe if there's a driver issue a newer version of the kernel might show different behaviour.

Best regards,

Ad

[patan32]

Officially IPS doesn't support virtual interfaces, such as VLAN's (https://docs.opnsense.org/manual/ips.html#choosing-an-interface). In 21.7.6 we added the new netmap api which seems to enable emulated mode in these cases.

I've seen a similar setup yesterday where someone added a bunch of vlan's to IPS, which starts but will not detect anything in previous versions of suricata (for a functional setup you need to capture the parent in promisc mode).

In 21.7.7 we will temporary revert the new api, which will then be back in 22.1, I haven't tried if it works for vlan's in 22.1, maybe it does, maybe it doesn't, if it doesn't we may have to tighten validations at some point from preventing people to choose these type of setups.

Best regards,

Ad

I have even enabled it on physical interface and still kills my internet using the new suricata package. I enabled igb0 with  Promiscuous mode enabled and it was dropping lots of packets on the Errors Out in the interfaces. Doesn't matter what interfaces i choose when using 21.7.6 it kills my internet and the config works on 21.7.5. Clearly i have a supported hardware with supported NIC drivers from Freebsd.

How do i upgrade to 22.1? Is this still in DEV?

[patan32]

This is my current setup with 21.7.5.

https://ibb.co/ZgNmvY0

Why does Suricata only show one physical interface? It really doesn't matter but my main capturing interface is igb0. Shouldn't i be seeing ix0 interface too which is on LAN side?

https://ibb.co/z6S7ZwC

[AdSchellevis]

How do i upgrade to 22.1? Is this still in DEV?

Update to 21.7.6 then switch to development (type in firmware settings)

[patan32]

On the physical interface it should work in principle, provided the hardware is properly supported. You can always try the beta for 22.1 by the way, maybe if there's a driver issue a newer version of the kernel might show different behaviour.

Best regards,

Ad

Why am i getting lots of errors on the interface WAN where IPS is enabled on physical interface. I didn't have this problem when i had IPS enabled on VLAN interface. Is this normal?

https://ibb.co/vvjNbjB

Doesn't seem to make sense now why have so much dropped packets.