Optimization up and drops decrease with Autofp and BPF for IDS mode Suricata

[jonny5]

One:
Running IDS mode via PCAP, with buffer modifications

Two:
I've modified the suricata service file to achieve this, not a permanent fix, wanting to discuss and see if it helps others before attempting to see how to integrate this improvement

The whole of my /usr/local/etc/rc.d/suricata:

Code Select Expand

#                Default: root
# suricata_pidfile (str):    Pidfile to store pid of suricata process
#                Default: /var/run/suricata.pid

. /etc/rc.subr

name="suricata"
rcvar=suricata_enable

start_precmd="suricata_prestart"
command="/usr/local/bin/suricata"

load_rc_config $name

[ -z "$suricata_enable" ]    && suricata_enable="NO"
[ -z "$suricata_conf" ]        && suricata_conf="/usr/local/etc/suricata/suricata.yaml"
# [ -z "$suricata_flags" ]    && suricata_flags="-D"
suricata_flags="-D --pcap-buffer-size=2147483647 -F /usr/local/etc/suricata/capture-filter.bpf"
[ -z "$suricata_divertport" ]    && suricata_divertport="8000"
[ -z "$suricata_netmap" ]    && suricata_netmap="NO"
[ -z "$suricata_user" ]        && suricata_user="root"
[ -z "$suricata_pidfile" ]    && suricata_pidfile="/var/run/suricata.pid"

if [ -n "$suricata_interface" ]; then
    for interface in $suricata_interface; do
        suricata_flags="$suricata_flags --pcap=$interface"
    done
elif [ "$suricata_netmap" != "NO" ]; then
    suricata_flags="$suricata_flags --netmap"
else
    suricata_flags="$suricata_flags -d $suricata_divertport"
    info "Inline Mode on divert port $suricata_divertport (suricata_interface not defined)"
fi

pidfile=$suricata_pidfile
suricata_flags="$suricata_flags --pidfile $pidfile"

[ -n "$suricata_conf" ]    && suricata_flags="$suricata_flags -c $suricata_conf"

suricata_prestart()
{
    if ! run_rc_command status > /dev/null; then
        rm -f "$pidfile"
    fi
}
run_rc_command "$1"
Three:
I've found that if I watch the interfaces/trunks that have the VLANs I get less drops and I believe this is because of fewer "interfaces", I update/modify this in my custom.yaml in /usr/local/etc/suricata/conf.d/custom.yaml it gets loaded after the suricata.yaml and is a permanent element of OPNSense that can be manually changed

Important elements of my custom.yaml:

Code Select Expand

host-mode: auto
runmode: autofp
default-packet-size: 1584
max-pending-packets: 25000
defrag:
  memcap: 2gb
  hash-size: 65536
  trackers: 65535
  max-frags: 65535
  prealloc: yes
  timeout: 60
flow:
  memcap: 6gb
  hash-size: 65536
  prealloc: 25600
  emergency-recovery: 30
vlan:
  use-for-tracking: true
stream:
  memcap: 6gb
  # memcap-policy: ignore
  checksum-validation: no
  prealloc-sessions: 262144
  #midstream: false
  #midstream-policy: ignore
  inline: no
  bypass: yes
  # midstream-policy: bypass
  reassembly:
    memcap: 1gb
    # memcap-policy: bypass
    depth: 2mb
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    # randomize-chunk-range: 10
    # raw: yes
    segment-prealloc: 4096
    # check-overlap-different-data: true
mpm-algo: hs
spm-algo: auto
threading:
  set-cpu-affinity: yes
  detect-thread-ratio: 0.75
pcap:
  - interface: ix0
    buffer-size: 2147483647
  - interface: ix1
    buffer-size: 2147483647
  - interface: lagg0
    buffer-size: 2147483647
With PCAP in Autofp mode and threading:set-cpu-affinity: yes and threading:detect-thread-ratio: 0.75 then at least one or more cores is free to IRQ/etc. and with the BPF I have a two stage filter (was three, the author learned more about 'vlan' and how it works in BPF, post has since been updated)

The file we will be creating is "/usr/local/etc/suricata/capture-filter.bpf" and it gets used/called with the suricata service file modification listed above

BPF Filter lightening the load on the CPUs:
1st group - the subnets and hosts and port and host combos I do not want, including the not element as we are excluding all of this:

Code Select Expand

not ((ip or ip6) and ((net 192.168.15.0/24 and 192.168.16.0/24) or (net 192.168.32.0/24 and 192.168.89.0/24) or (net ipv6subnet1 and ipv6subnet2) or (net ipv6subnet2 and ipv6subnet3) or (broadcast or multicast) or (net 192.168.230/24 or 192.168.240.0/24 or 224.0.0.0/4 or ff00::/8 or fe80::/10) or (((icmp or icmp6) or (port 0 or 67 or 68 or 123 or 137 or 138 or 139 or 445 or 514 or 853 or 1900 or 2055 or 2056 or 3000 or 5044 or 5140 or 5142 or 5143 or 5353 or 5355 or 6060 or 8006 or 8291 or 9001 or 9090 or 9200 or 9600)) and (net 192.168.0.0/16 or ipv6subnet::/56)) or (host 192.168.33.44 or 192.168.55.66) or (host ipv6host1 or ipv6host2) or (src host publicip and ((dst port 853) or (src port 00000) or (src portrange 00000-00000) or (src portrange 00000-00000))) or (dst host publicip and ((src port 853) or (dst port 00000) or (dst portrange 00000-00000) or (dst portrange 00000-00000))))) or 2nd group - the VLANs I do not want, and a repeat of the exclusions from above, following the above line (no return/new-line):

Code Select Expand

((vlan 200 or vlan 300) or (vlan and ((net 192.168.15.0/24 and 192.168.16.0/24) or (net 192.168.15.0/24 and 192.168.89.0/24)) or (net ipv6subnet1 and ipv6subnet2) or (net ipv6subnet1 and ipv6subnet3) or (broadcast or multicast) or (net 192.168.230/24 or 192.168.240.0/24 or 224.0.0.0/4 or ff00::/8 or fe80::/10) or (((icmp or icmp6) or (port 0 or 67 or 68 or 123 or 137 or 138 or 139 or 445 or 514 or 853 or 1900 or 2055 or 2056 or 3000 or 5044 or 5140 or 5142 or 5143 or 5353 or 5355 or 6060 or 8006 or 8291 or 9001 or 9090 or 9200 or 9600)) and (net 192.168.0.0/16 or ipv6subnet::/56)) or (host 192.168.33.44 or 192.168.55.66) or (host ipv6host1 or ipv6host2) or (src host publicip and ((dst port 853) or (src port 00000) or (src portrange 00000-00000) or (src portrange 00000-00000))) or (dst host publicip and ((src port 853) or (dst port 00000) or (dst portrange 00000-00000) or (dst portrange 00000-00000))))))
^ Note about the above bit of BPF, it seems with the Code block when you copy misses the space at the end, the first group has a space after the "or" and before the next group, be sure to add it back

Extra edit: Thought I had really dialed it in, my understanding of vlan was a little off, and it works best at the end as it masks all the work after it to shift all subsequent packet payload byte offsets forward by 4 bytes. The original 3 groups are now just two and it appears to work even better. Still in testing to be fair.

2nd Extra edit: A few more lessons in BPF later, portranges stand alone, and using primitives here (ip ip6 instead of ether[] syntax) is a more optimized method.

[jonny5]

If you have enabled stats in your custom.yaml for the eve.json output, this works quite well to watch your drop rate:

Code Select Expand

tail -f -n 100000 /var/log/suricata/eve.json | jq -c 'select(.event_type=="stats") | select(.stats.capture.kernel_packets > 0) | {timestamp: .timestamp, drop_pct: (.stats.capture.kernel_drops / .stats.capture.kernel_packets * 100)}' ;