Messages

Extra,

Shortly after running well for a couple days, I attempted to add back the zerocopy tuning feature and even rebooted, it has not presented an issue at all. Running Suricata in IDS/PCAP mode.

I've also see Unbound suddenly stop working, I also wrote a cronned script that checks if it is active and restarts it if necessary

If you have enabled stats in your custom.yaml for the eve.json output, this works quite well to watch your drop rate:

Code Select Expand

tail -f -n 100000 /var/log/suricata/eve.json | jq -c 'select(.event_type=="stats") | select(.stats.capture.kernel_packets > 0) | {timestamp: .timestamp, drop_pct: (.stats.capture.kernel_drops / .stats.capture.kernel_packets * 100)}' ;

[One:]

One:
Running IDS mode via PCAP, with buffer modifications

Two:
I've modified the suricata service file to achieve this, not a permanent fix, wanting to discuss and see if it helps others before attempting to see how to integrate this improvement

The whole of my /usr/local/etc/rc.d/suricata:

Code Select Expand

#                Default: root
# suricata_pidfile (str):    Pidfile to store pid of suricata process
#                Default: /var/run/suricata.pid

. /etc/rc.subr

name="suricata"
rcvar=suricata_enable

start_precmd="suricata_prestart"
command="/usr/local/bin/suricata"

load_rc_config $name

[ -z "$suricata_enable" ]    && suricata_enable="NO"
[ -z "$suricata_conf" ]        && suricata_conf="/usr/local/etc/suricata/suricata.yaml"
# [ -z "$suricata_flags" ]    && suricata_flags="-D"
suricata_flags="-D --pcap-buffer-size=2147483647 -F /usr/local/etc/suricata/capture-filter.bpf"
[ -z "$suricata_divertport" ]    && suricata_divertport="8000"
[ -z "$suricata_netmap" ]    && suricata_netmap="NO"
[ -z "$suricata_user" ]        && suricata_user="root"
[ -z "$suricata_pidfile" ]    && suricata_pidfile="/var/run/suricata.pid"

if [ -n "$suricata_interface" ]; then
    for interface in $suricata_interface; do
        suricata_flags="$suricata_flags --pcap=$interface"
    done
elif [ "$suricata_netmap" != "NO" ]; then
    suricata_flags="$suricata_flags --netmap"
else
    suricata_flags="$suricata_flags -d $suricata_divertport"
    info "Inline Mode on divert port $suricata_divertport (suricata_interface not defined)"
fi

pidfile=$suricata_pidfile
suricata_flags="$suricata_flags --pidfile $pidfile"

[ -n "$suricata_conf" ]    && suricata_flags="$suricata_flags -c $suricata_conf"

suricata_prestart()
{
    if ! run_rc_command status > /dev/null; then
        rm -f "$pidfile"
    fi
}
run_rc_command "$1"
Three:
I've found that if I watch the interfaces/trunks that have the VLANs I get less drops and I believe this is because of fewer "interfaces", I update/modify this in my custom.yaml in /usr/local/etc/suricata/conf.d/custom.yaml it gets loaded after the suricata.yaml and is a permanent element of OPNSense that can be manually changed

Important elements of my custom.yaml:

Code Select Expand

host-mode: auto
runmode: autofp
default-packet-size: 1584
max-pending-packets: 25000
defrag:
  memcap: 2gb
  hash-size: 65536
  trackers: 65535
  max-frags: 65535
  prealloc: yes
  timeout: 60
flow:
  memcap: 6gb
  hash-size: 65536
  prealloc: 25600
  emergency-recovery: 30
vlan:
  use-for-tracking: true
stream:
  memcap: 6gb
  # memcap-policy: ignore
  checksum-validation: no
  prealloc-sessions: 262144
  #midstream: false
  #midstream-policy: ignore
  inline: no
  bypass: yes
  # midstream-policy: bypass
  reassembly:
    memcap: 1gb
    # memcap-policy: bypass
    depth: 2mb
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    # randomize-chunk-range: 10
    # raw: yes
    segment-prealloc: 4096
    # check-overlap-different-data: true
mpm-algo: hs
spm-algo: auto
threading:
  set-cpu-affinity: yes
  detect-thread-ratio: 0.75
pcap:
  - interface: ix0
    buffer-size: 2147483647
  - interface: ix1
    buffer-size: 2147483647
  - interface: lagg0
    buffer-size: 2147483647
With PCAP in Autofp mode and threading:set-cpu-affinity: yes and threading:detect-thread-ratio: 0.75 then at least one or more cores is free to IRQ/etc. and with the BPF I have a two stage filter (was three, the author learned more about 'vlan' and how it works in BPF, post has since been updated)

The file we will be creating is "/usr/local/etc/suricata/capture-filter.bpf" and it gets used/called with the suricata service file modification listed above

BPF Filter lightening the load on the CPUs:
1st group - the subnets and hosts and port and host combos I do not want, including the not element as we are excluding all of this:

Code Select Expand

not ((ip or ip6) and ((net 192.168.15.0/24 and 192.168.16.0/24) or (net 192.168.32.0/24 and 192.168.89.0/24) or (net ipv6subnet1 and ipv6subnet2) or (net ipv6subnet2 and ipv6subnet3) or (broadcast or multicast) or (net 192.168.230/24 or 192.168.240.0/24 or 224.0.0.0/4 or ff00::/8 or fe80::/10) or (((icmp or icmp6) or (port 0 or 67 or 68 or 123 or 137 or 138 or 139 or 445 or 514 or 853 or 1900 or 2055 or 2056 or 3000 or 5044 or 5140 or 5142 or 5143 or 5353 or 5355 or 6060 or 8006 or 8291 or 9001 or 9090 or 9200 or 9600)) and (net 192.168.0.0/16 or ipv6subnet::/56)) or (host 192.168.33.44 or 192.168.55.66) or (host ipv6host1 or ipv6host2) or (src host publicip and ((dst port 853) or (src port 00000) or (src portrange 00000-00000) or (src portrange 00000-00000))) or (dst host publicip and ((src port 853) or (dst port 00000) or (dst portrange 00000-00000) or (dst portrange 00000-00000))))) or 2nd group - the VLANs I do not want, and a repeat of the exclusions from above, following the above line (no return/new-line):

Code Select Expand

((vlan 200 or vlan 300) or (vlan and ((net 192.168.15.0/24 and 192.168.16.0/24) or (net 192.168.15.0/24 and 192.168.89.0/24)) or (net ipv6subnet1 and ipv6subnet2) or (net ipv6subnet1 and ipv6subnet3) or (broadcast or multicast) or (net 192.168.230/24 or 192.168.240.0/24 or 224.0.0.0/4 or ff00::/8 or fe80::/10) or (((icmp or icmp6) or (port 0 or 67 or 68 or 123 or 137 or 138 or 139 or 445 or 514 or 853 or 1900 or 2055 or 2056 or 3000 or 5044 or 5140 or 5142 or 5143 or 5353 or 5355 or 6060 or 8006 or 8291 or 9001 or 9090 or 9200 or 9600)) and (net 192.168.0.0/16 or ipv6subnet::/56)) or (host 192.168.33.44 or 192.168.55.66) or (host ipv6host1 or ipv6host2) or (src host publicip and ((dst port 853) or (src port 00000) or (src portrange 00000-00000) or (src portrange 00000-00000))) or (dst host publicip and ((src port 853) or (dst port 00000) or (dst portrange 00000-00000) or (dst portrange 00000-00000))))))
^ Note about the above bit of BPF, it seems with the Code block when you copy misses the space at the end, the first group has a space after the "or" and before the next group, be sure to add it back

Extra edit: Thought I had really dialed it in, my understanding of vlan was a little off, and it works best at the end as it masks all the work after it to shift all subsequent packet payload byte offsets forward by 4 bytes. The original 3 groups are now just two and it appears to work even better. Still in testing to be fair.

2nd Extra edit: A few more lessons in BPF later, portranges stand alone, and using primitives here (ip ip6 instead of ether[] syntax) is a more optimized method.

In https://opnsense/ui/firewall/source_nat#:

Since I believe our Firewall Rule Migration, when you create a Source NAT Firewall rule, and try to select a port alias you have created that is a numeric port/portrange the UI will tell you that you must use a port name word to select that port.

The error message:

Code Select Expand

Please specify a valid port number (1-65535). A service name is also possible (afs3-fileserver, aol, auth, avt-profile-1, cvsup, domain, ftp, hbci, http, https, igmpv3lite, imap, imaps, ipsec-msft, ipsec-nat-t, isakmp, l2f, ldap, microsoft-ds, ms-streaming, ms-wbt-server, msnp, nat-stun-port, netbios-dgm, netbios-ns, netbios-ssn, nntp, ntp, openvpn, pop3, pop3s, pptp, radius, radius-acct, rfb, sip, smtp, snmp, snmptrap, ssh, submission, telnet, teredo, tftp, urd, wins).
Effectively to enter a natural port/portrange you have to type the numeric values in manually.

I would think that since the alias is a numeric value it would work here, but you have to select "Single port or range" and then enter the numeric value manually?

Extra:
Other than this, it would appear quite an improvement to move into 27 and now 27.2, also love the fix to the Service status, looks great!!

[FE80::/8]

Happened across a possible typo in documentation specifying FE80::/8 as the link-local subnet for IPv6 at https://docs.opnsense.org/manual/ipv6.html#router-advertisement-ra and according to everything I can find, the IPv6 Link-Local unicast subnet is FE80::/10.

I have not checked any working code to see if this value made its way into development or production code, but wanted to share the finding as it would appear to be a technical inaccuracy so we can start by updating the documentation.

Extra:
There is a "/8" but it is found in FF00::/8 which is IPv6 Multicast and not IPv6 Link-Local unicast (FE80 != FF00).

For starters, if you hit the "advanced" toggle at the top, it shows all editable fields normally - this is okay and more or less required now to see and edit all fields correctly.

Second, without hitting the advanced toggle on a rule edit you will not be able to edit the Source Port, and the inner-menu toggles to open or hide additional Source/Destination/etc. options is inconsistent (you end up hiding Destination elements by trying to show more elements).

For now, it would appear the best method to use the editor is to click the toggle for advanced when editing a Source NAT (Destination NAT does not appear to have this issue).

Everything appears to be working well, just a GUI bug it would seem on the menu interface around editing Source NAT rules.

Just used the migration assistant to move Outbound to Source NAT, seems to have worked!

It did take me I think a little too long to realized the check mark that shows up is the 'activate/go' button, but that's I believe a me problem.

Came here to say I figured this out as well, took a second to be able to decompress the crash file and once I saw "zerocopy" in the list of commands/features, I was like oh yeah, I enabled that myself and it is a possibly useful but debated feature.

Disabled ("1" -> "0" for zerocopy) and everything worked - extra, but I had also uninstalled all microcode and realtek plug-ins, I have since re-installed the plug-ins, everything is working well.

Thank you OPNSense Community and Franco!! ^_^

Already feeling bad about making this post, but, if someone else has this happen - hope you find this and we can figure it out together.

My OPNSense upgraded to 26.1.10 w/no issue, and I have CrowdSec installed and the Firewall Blocker feature enabled (this means there are two PF/Alias lists that CrowdSec manages updates for).

Instead of configuring my Firewall to use the "CrowdSec managed lists" aliases, I have made a new alias/list where I included the CrowdSec lists (IPv4 and IPv6), SpamHaus list, and a personally kept list as my combined blocklist. This is my blocklist for Inbound and Outbound traffic - works great!

Some time ago, I realized I could have "Uptime Kuma" check an alias/list via API to see how recently it has been updated and let me know if it starts aging too much. After a while I found that the 25 minute marker was good for watching this combined blocklist that I had created - more or less, it near always updates within 25 minutes.

For those wondering, it is a JSON check:
URL: https://opnsense.domain.tld/api/firewall/alias/getItem/<UUIDOfList>
Check: (($millis() - $toMillis(alias.last_updated)) / 60000)

Here is the issue:
Yesterday there was an alert that my combined blocklist hadn't updated, and it was getting into its 4th hour. After restarting the PF service, restarting the CrowdSec Firewall Blocker, and trying to modify my combined blocklist (basically open to edit and hit save) in the Alias list, my combined blocklist would never update. The CrowdSec managed lists were updating (even though you cannot see an edit date, you can tell their count had gone down as it didn't add up to the total sum of my combined blocklist), but my combined blocklist held its aging modify/update time and high count.

Once I restarted the OPNSense, things returned to normal and now the next day, continues to do so. No idea what happened there, but, I would recommend you watch your lists and make sure you track the last update date in case they get locked too. If it happens again (this has only happened once so far), I hope to be able to do more to figure out what the hang up is - would love ideas to check!

Similar problem, my Zenarmor appears to be crashing constantly - high CPU utilization.

Code Select Expand

pid 14461 (eastpect), jid 0, uid 0: exited on signal 3 (core dumped)
'eastpect' appears to be Zen Armor - I've turned Zen off and it seemed to reset my network stack, but also appears to have significantly lowered my CPU utilization.

Edit:

Check out this thread - Zenarmor Forum about OPNSense 26.1.3 and Zenarmor CPU utilization

Summary - After doing maintainance on the DB, I switched from Passive Mode to Blocking Routed L3 with Native Netmap and it appears to be working w/out coredump now.

Saw a different error specifically saying pkg-static wouldn't let go of the db, ran the kill command, it said it didn't find anything to kill, re-ran the update, it worked.

Rebooted the router, everything is coming up nicely, thank you OPNSense and Franco!!

In short, no.

Use the OPNSense Alert view to view them or send the logs to a data-lake or something like Elasticsearch/Opensearch/Greylog.

My OPNSense complained about an "unknown error" but I cannot find the log of it yet. Any pointers to reviewing logs for upgrades welcomed!!

My upgrade did work however, and I have modded my OPNSense a little out-of-band.

Can the OPNSense affected also if hacker got access to LAN?

Internal Firewall rules with separate zones/interfaces for Wifi/Client/DMZ/Core/etc. Would advise using VLANs if you can, otherwise subneting with /24s is a good idea.

From what I've read, you might also want to turn on MAC-Address filters on your WAPs and/or OPNSense's DHCP, good luck!

I have MAC filtering enabled. NO SSH and open ports. How to create VLAN or subnetting?

VLANs:
https://docs.opnsense.org/manual/how-tos/vlan_and_lagg.html
https://www.youtube.com/watch?v=9hJyWaQ2x28

Subnetting, well, that is its own thing. Behind your WAN, is your LAN, and you can have multiple interfaces that feed various parts of your network. Generally your router is the point where the different legs interconnect, and you can subnet within a /16 several /24 cidrs.

192.168.0.0/16 = "usable": 192.168.0.1 - 192.168.255.254

192.168.34.0/24 = "usable": 192.168.34.1 - 192.168.34.254

Depending on how familiar you are with this, yes, you will have to assign a gateway IP to the interface you are creating for a subnet - still considered a "usable" IP.

There are subnet calculators out there, or you can make your own in Excel - it all breaks down to binary math and for the "length" of the CIDR is how much of the binary IP value the IP has as neighbors.

/24 = 255.255.255.0 and for the 192.168.34.56 it means that any other IP with a 192.168.34.XXX is a neighbor, and this works like a mask or filter.
In binary:
11111111.11111111.11111111.00000000 masked onto 11000000.10101000.00100010.00111000 and then 11000000.10101000.00100010.00000001 through 11000000.10101000.00100010.11111110 is your neighbor

It is strongly advised to not try to network at smaller than a /24, but if you are filtering/making a group for a process, you can segment much smaller, just know that at a VLAN/Interface actual network group you will want to use /24 for IPv4 and likely /64 for IPv6. If you are lucky you can get a /56 IPv6 from your ISP and you can use /64s out of it for the various interfaces.

Since a VLAN is a "device" you can make a VLAN and have it have its own subnet, you do not need N ethernet/fiber cards to have N subnets (N = Integer number).

Read the docs, make sure you setup your firewall rules to allow devices within an Interface "In", and what the get to "In" to (any == local and WAN).

Would also advise, unless you have found "the hack", it can be often best to interpret an anomaly/issue as a mis-configuration.