Optimization up and drops decrease with Autofp and BPF for IDS mode Suricata

[jonny5]

One:
Running IDS mode via PCAP, with buffer modifications

Two:
I've modified the suricata service file to achieve this, not a permanent fix, wanting to discuss and see if it helps others before attempting to see how to integrate this improvement

The whole of my /usr/local/etc/rc.d/suricata:

Code Select Expand

#                Default: root
# suricata_pidfile (str):    Pidfile to store pid of suricata process
#                Default: /var/run/suricata.pid

. /etc/rc.subr

name="suricata"
rcvar=suricata_enable

start_precmd="suricata_prestart"
command="/usr/local/bin/suricata"

load_rc_config $name

[ -z "$suricata_enable" ]    && suricata_enable="NO"
[ -z "$suricata_conf" ]        && suricata_conf="/usr/local/etc/suricata/suricata.yaml"
# [ -z "$suricata_flags" ]    && suricata_flags="-D"
suricata_flags="-D --pcap-buffer-size=2147483647 -F /usr/local/etc/suricata/capture-filter.bpf"
[ -z "$suricata_divertport" ]    && suricata_divertport="8000"
[ -z "$suricata_netmap" ]    && suricata_netmap="NO"
[ -z "$suricata_user" ]        && suricata_user="root"
[ -z "$suricata_pidfile" ]    && suricata_pidfile="/var/run/suricata.pid"

if [ -n "$suricata_interface" ]; then
    for interface in $suricata_interface; do
        suricata_flags="$suricata_flags --pcap=$interface"
    done
elif [ "$suricata_netmap" != "NO" ]; then
    suricata_flags="$suricata_flags --netmap"
else
    suricata_flags="$suricata_flags -d $suricata_divertport"
    info "Inline Mode on divert port $suricata_divertport (suricata_interface not defined)"
fi

pidfile=$suricata_pidfile
suricata_flags="$suricata_flags --pidfile $pidfile"

[ -n "$suricata_conf" ]    && suricata_flags="$suricata_flags -c $suricata_conf"

suricata_prestart()
{
    if ! run_rc_command status > /dev/null; then
        rm -f "$pidfile"
    fi
}
run_rc_command "$1"
Three:
I've found that if I watch the interfaces/trunks that have the VLANs I get less drops and I believe this is because of fewer "interfaces", I update/modify this in my custom.yaml in /usr/local/etc/suricata/conf.d/custom.yaml it gets loaded after the suricata.yaml and is a permanent element of OPNSense that can be manually changed

Important elements of my custom.yaml:

Code Select Expand

host-mode: auto
runmode: autofp
default-packet-size: 1584
max-pending-packets: 25000
defrag:
  memcap: 2gb
  hash-size: 65536
  trackers: 65535
  max-frags: 65535
  prealloc: yes
  timeout: 60
flow:
  memcap: 6gb
  hash-size: 65536
  prealloc: 25600
  emergency-recovery: 30
vlan:
  use-for-tracking: true
stream:
  memcap: 6gb
  # memcap-policy: ignore
  checksum-validation: no
  prealloc-sessions: 262144
  #midstream: false
  #midstream-policy: ignore
  inline: no
  bypass: yes
  # midstream-policy: bypass
  reassembly:
    memcap: 1gb
    # memcap-policy: bypass
    depth: 2mb
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    # randomize-chunk-range: 10
    # raw: yes
    segment-prealloc: 4096
    # check-overlap-different-data: true
mpm-algo: hs
spm-algo: auto
threading:
  set-cpu-affinity: yes
  detect-thread-ratio: 0.75
pcap:
  - interface: ix0
    buffer-size: 2147483647
  - interface: ix1
    buffer-size: 2147483647
  - interface: lagg0
    buffer-size: 2147483647
With PCAP in Autofp mode and threading:set-cpu-affinity: yes and threading:detect-thread-ratio: 0.75 then at least one or more cores is free to IRQ/etc. and with the BPF I have a three stage filter

BPF Filter lightening the load on the CPUs:
1st group - the VLANs I do not want, including the not element as we are excluding all of this:

Code Select Expand

not ((ether[12:2] = 0x8100 and (ether[14:2] & 0x0fff = 78 or ether[14:2] & 0x0fff = 234 or ether[14:2] & 0x0fff = 345)) or 2nd group - following the above line (no return/new-line):

Code Select Expand

(vlan and ((net 192.168.15.0/24 and 192.168.16.0/24) or (net 192.168.15.0/24 and 192.168.89.0/24)) or (net ipv6subnet1 and ipv6subnet2) or (net ipv6subnet1 and ipv6subnet3) or (broadcast or multicast) or (net 192.168.230/24 or 192.168.240.0/24 or 224.0.0.0/4 or ff00::/8 or fe80::/10) or (((icmp or icmp6) or (port 0 or 67 or 68 or 123 or 137 or 138 or 139 or 445 or 514 or 853 or 1900 or 2055 or 2056 or 3000 or 5044 or 5140 or 5142 or 5143 or 5353 or 5355 or 6060 or 8006 or 8291 or 9001 or 9090 or 9200 or 9600)) and (net 192.168.0.0/16 or ipv6subnet::/56)) or (host 192.168.33.44 or 192.168.55.66) or (host ipv6host1 or ipv6host2) or (src host publicip and ((dst port 853) or (src port 00000) or (src portrange 00000-00000 or 00000-00000))) or (dst host publicip and ((src port 853) or (dst port 00000) or (dst portrange 00000-00000 or 00000-00000)))) or 3rd group - following the above line (no return/new-line) is the final and third part (effectively the same as above, only we are say it is not vlan tagged):

Code Select Expand

(ether[12:2] != 0x8100 and ((net 192.168.15.0/24 and 192.168.16.0/24) or (net 192.168.32.0/24 and 192.168.89.0/24)) or (net ipv6subnet1 and ipv6subnet2) or (net ipv6subnet2 and ipv6subnet3) or (broadcast or multicast) or (net 192.168.230/24 or 192.168.240.0/24 or 224.0.0.0/4 or ff00::/8 or fe80::/10) or (((icmp or icmp6) or (port 0 or 67 or 68 or 123 or 137 or 138 or 139 or 445 or 514 or 853 or 1900 or 2055 or 2056 or 3000 or 5044 or 5140 or 5142 or 5143 or 5353 or 5355 or 6060 or 8006 or 8291 or 9001 or 9090 or 9200 or 9600)) and (net 192.168.0.0/16 or ipv6subnet::/56)) or (host 192.168.33.44 or 192.168.55.66) or (host ipv6host1 or ipv6host2) or (src host publicip and ((dst port 853) or (src port 00000) or (src portrange 00000-00000 or 00000-00000))) or (dst host publicip and ((src port 853) or (dst port 00000) or (dst portrange 00000-00000 or 00000-00000)))))
^ Note about the above bit of BPF, it seems with the Code block when you copy misses the space at the end, the first and second group have a space after the "or" and before the next group, be sure to add it back

Extra edit: The "vlan" element of BPF shifts the inspection point for the host and port elements in the packet, it does not filter in or out VLAN packets, and might be the one place for a logic upgrade to this

[jonny5]

If you have enabled stats in your custom.yaml for the eve.json output, this works quite well to watch your drop rate:

Code Select Expand

tail -f -n 100000 /var/log/suricata/eve.json | jq -c 'select(.event_type=="stats") | select(.stats.capture.kernel_packets > 0) | {timestamp: .timestamp, drop_pct: (.stats.capture.kernel_drops / .stats.capture.kernel_packets * 100)}' ;