Suricata not starting on one WAN interface

[srijan]

Hello Everyone,

I have the following setup. I have two WAN interfaces (one PPPoE and another Static) and have configured a LAN interface and a DMZ interface.

I am currently running OPNsense 16.7.2-i386. The hardware used has the following configuration:

hw.model: Intel(R) Celeron(R) M processor          600MHz
hw.machine: i386
hw.ncpu: 1

real memory  = 536870912 (512 MB)
avail memory = 481464320 (459 MB)

The issue that I am is facing is rather strange. If I select only one PPPoE WAN interface for Intrusion Detection, Suricata starts up all file and I can see it up.

Here are the logs:
23/8/2016 -- 18:53:04 - <Notice> - This is Suricata version 3.1.1 RELEASE
23/8/2016 -- 18:54:31 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started.

Now if I enable only the Static WAN interface, Suricata fails to start and gives an out of memory error. Specific logs are as below:

23/8/2016 -- 18:58:44 - <Notice> - This is Suricata version 3.1.1 RELEASE
23/8/2016 -- 19:00:15 - <Error> - [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't register em2 with netmap: Cannot allocate memory
23/8/2016 -- 19:00:15 - <Error> - [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't register em2 with netmap: Cannot allocate memory
23/8/2016 -- 19:00:15 - <Error> - [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-em2" closed on initialization.
23/8/2016 -- 19:00:15 - <Error> - [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...


Hardware CRC, Hardware TSO and Hardware LRO are disabled for all interfaces.

I know RAM is less, but I fail to understand if it starts with one WAN interface why it cannot start with the other WAN interface. My ultimate goal is to have Intrusion Detection on both the WAN interfaces.

Any ideas?


Thanks and Regards,
-=Srijan Nandi

[srijan]

Sorry, forgot to add. I have a 4GB swap space and when I start suricata on the Static WAN Interface, I see RAM being eaten, but swap is still not utilised.

m.swap_enabled: 1
vm.disable_swapspace_pageouts: 0
vm.defer_swapspace_pageouts: 0
vm.swap_idle_enabled: 0
vm.stats.vm.v_swappgsout: 153221
vm.stats.vm.v_swappgsin: 69916
vm.stats.vm.v_swapout: 26527
vm.stats.vm.v_swapin: 21265
vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
vm.nswapdev: 1
vm.swap_async_max: 4
vm.swap_maxpages: 1894560
vm.swap_reserved: 591814656
vm.swap_total: 4294967296

[srijan]

In system logs, I see the following:

Aug 23 19:00:15 OPNsense kernel: 015.175801 [ 518] netmap_obj_malloc         no more netmap_buf objects
Aug 23 19:00:15 OPNsense kernel: 015.184088 [ 683] netmap_new_bufs           no more buffers after 3874 of 4096
Aug 23 19:00:15 OPNsense kernel: 015.193470 [1423] netmap_mem_rings_create   Cannot allocate buffers for rx_ring
Aug 23 19:00:15 OPNsense kernel: 015.235613 [ 518] netmap_obj_malloc         no more netmap_buf objects
Aug 23 19:00:15 OPNsense kernel: 015.243970 [ 683] netmap_new_bufs           no more buffers after 3874 of 4096
Aug 23 19:00:15 OPNsense kernel: 015.253353 [1423] netmap_mem_rings_create   Cannot allocate buffers for rx_ring

Is there a way to increase 'netmap_new_bufs'?

[franco]

Hi Srijan,

Netmap / Suricata IPS mode is quite greedy, upon startup it will grab the whole contiguous buffer space it can find, but that also means it needs to find it, probably starts in RAM space and then fails naturally as it would have to resort to SWAP space. It wouldn't perform well in SWAP, if at all. I do not know.

I'm quite surprised that Suricata works for this low amount of RAM in any case. The fix, unfortunately, is more RAM either in the form of new hardware or RAM extension. In fact, we recommend at least 1GB for all Suricata / Squid usage.

https://docs.opnsense.org/manual/hardware.html#hardware-requirements

Minimal / 512 MB stages: "you can run all standard features, expect for the ones that require disk writes, e.g. a caching proxy (cache) or intrusion detection and prevention (alert database)."


Cheers,
Franco

[srijan]

Thanks, Franco. My guess was that. My current hardware has built-in RAM, so maybe I have to get a new hardware for Opnsense.

I thought there could be a way to tweak the netmap_buf, via a kernel configuration and increase the size.

Though, I have already in the MBUF by adding a line in loader.conf, kern.ipc.nmbclusters="131072". I though this will increase the netmap_buf as well.

Thank you so much for the quick reply.

Regards,
-=Srijan Nandi

[franco]

Hi Srijan,

There are some more tweaks for netmap buffer/ring size/num, but I really do not know how it all ties together, the documentation is sparse. Maybe this helps...

% sysctl dev.netmap
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size: 2048
dev.netmap.buf_curr_num: 0
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 0
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 0
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 0
dev.netmap.ring_size: 73728
dev.netmap.priv_if_num: 1
dev.netmap.priv_if_size: 1024
dev.netmap.if_curr_num: 0
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 0
dev.netmap.if_size: 1024
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.admode: 0
dev.netmap.mmap_unreg: 0
dev.netmap.fwd: 0
dev.netmap.flags: 0
dev.netmap.adaptive_io: 0
dev.netmap.txsync_retry: 2
dev.netmap.no_pendintr: 1
dev.netmap.mitigate: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.ixl_crcstrip: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
dev.netmap.ix_crcstrip: 0


Cheers,
Franco

[srijan]

Hello Franco,

I narrowed it down the one of these that need a change, not sure which one though.

dev.netmap.buf_num: 163840

dev.netmap.ring_num: 200

dev.netmap.buf_size: 2048

After reading the Man Pages, here is what I have got:

"netmap supports raw packet I/O through a port, which can be connected to a physical interface (NIC), to the host stack, or to a VALE switch). Ports use preallocated circular queues of buffers (rings) residing in an mmapped region. There is one ring for each transmit/receive queue   of a NIC or virtual port. An additional ring pair connects to the host   stack.

Sizes and number of objects (netmap_if, netmap_ring, buffers) for the global memory region. The only parameter worth modifying is dev.netmap.buf_num as it impacts the total amount of memory used by netmap."

I will try to tweak these values and see if this resolves the issue.

Thanks and Regards,
-=Srijan Nandi

[srijan]

Hello Franco,

I changed the following parameters to:

dev.netmap.buf_num=200000
dev.netmap.ring_num=800
dev.netmap.buf_size=3096

But, still the same results. Therefore, it is very much clear that Intrusion Prevention requires minimum 1GB RAM.

Sorry for the botheration.

Thanks and Regards,
-=Srijan Nandi

[franco]

Hi Srijan,

Sorry to hear that this couldn't be fixed. In any way,thanks for checking back.


Cheers,
Franco