Messages

Just a quick update - since disabling IDS/IPS in my last post, the firewall has not crashed again as of this reply.

For readability:

Code Select Expand

db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0xfc0f40
curthread    = 0xfffffe0138c28720: pid 3489 tid 102014 critnest 1 "pfctl"
curpcb       = 0xfffffe0138c28c30
fpcurthread  = 0xfffffe0138c28720: pid 3489 "pfctl"
idlethread   = 0xfffffe00207933a0: tid 100003 "idle: cpu0"
self         = 0xffffffff82c10000
curpmap      = 0xfffffe011668f518
tssp         = 0xffffffff82c10384
rsp0         = 0xfffffe0118fea000
kcr3         = 0x351ae2000
ucr3         = 0x16fe6d000
scr3         = 0x16fe6d000
gs32p        = 0xffffffff82c10404
ldt          = 0xffffffff82c10444
tss          = 0xffffffff82c10434
curvnet      = 0xfffff80001202dc0
db:0:kdb.enter.default>  bt
Tracing pid 3489 tid 102014 td 0xfffffe0138c28720
kdb_enter() at kdb_enter+0x37/frame 0xfffffe0118fe93c0
vpanic() at vpanic+0x1b0/frame 0xfffffe0118fe9410
panic() at panic+0x43/frame 0xfffffe0118fe9470
trap_fatal() at trap_fatal+0x385/frame 0xfffffe0118fe94d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0118fe9530
calltrap() at calltrap+0x8/frame 0xfffffe0118fe9530
--- trap 0xc, rip = 0xffffffff80debe14, rsp = 0xfffffe0118fe9600, rbp = 0xfffffe0118fe9620 ---
rn_walktree() at rn_walktree+0x64/frame 0xfffffe0118fe9620
pfr_get_addrs() at pfr_get_addrs+0x219/frame 0xfffffe0118fe9680
pfioctl() at pfioctl+0x23be/frame 0xfffffe0118fe9b50
devfs_ioctl() at devfs_ioctl+0xc6/frame 0xfffffe0118fe9ba0
vn_ioctl() at vn_ioctl+0x1a4/frame 0xfffffe0118fe9cb0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe0118fe9cd0
kern_ioctl() at kern_ioctl+0x25b/frame 0xfffffe0118fe9d40
sys_ioctl() at sys_ioctl+0xf1/frame 0xfffffe0118fe9e00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe0118fe9f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0118fe9f30
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8012446da, rsp = 0x7fffffffdc38, rbp = 0x7fffffffe0d0 ---

I haven't seen this before but if it doesn't happen on 22.1 it should be easy to find the bad commit.

This is new for 22.7, right?


Cheers,
Franco

Yeah, never had this problem on 22.1 before. I disabled IPS/IDS entirely and it seems to have greatly helped the stability - it was crashing multiple times a day today and yesterday and since turning off Intrustion Detection in services, it hasn't crashed again (yet).

Looks like the panic was caused by "pfctl".  You doing packet inspection of any kind? Perhaps chocking session states?

Just the defaults... IDS was enabled in IPS mode but with no rules downloaded. I did not modify any of those settings from the base install.

I managed to retrieve these crash dumps. Briefly going through them, I'm starting to suspect overheating or other hardware issues?

I'd love to, but I can't even SSH into it when it happens.

Every so often, up to multiple times per day, my firewall appliance locks up and requires a hard reboot to restore services and internet connectivity.

So far, I have been unable to find any logs or crash dumps that would help me isolate the issue outside of one time, which I did submit via the web interface.

I have no idea where to start. Can someone point me in the right direction to troubleshoot this issue? At this point I'm not sure if it's hardware or software.

I'm running it on a KingNovy fanless PC with 6x Intel I225-V, a Celeron N5105, 16 GB of RAM, and a 256 GB NVMe drive.

Got it.  So more or less this is an "expected" issue right now.

Sorry, edited my post for accuracy.  The leases themselves appear, and sometimes the MAC address, but no host names.

Under DHCPv6, even with the option checked to add DHCP leases to DNS, it does not show host entries under DHCPv6 > Leases, and when my DNS server queries OPNsense for my domain, no IPv6 addresses are returned.

Running version 18.7.10_3.  Just set it up last night, so I am not sure if the issue occurred on previous versions.

I talked with the author, there will come many more rules, so I think using this list is only for high end hardware. Probably abuse.ch will stop this ruleset.

How much more "high end" do you need to get with an edge firewall than an x86 box with 8-16GB of RAM?  Again, it doesn't even get to 100% RAM usage before crashing out.
There is clearly an issue with the rule set itself here, and it's clear now that abuse.ch is not interested in fixing it.  RAM is clearly not the issue here.

You can only check If you double the size and see if it happens again.

Unfortunately I don't have UDIMMs around that I can do that with... quite expensive for a home user :P

I have 8GB of RAM in it already and the RAM usage never goes to 100%.  Last time, it went to around 25-30% before crashing.  I don't think it's an issue with the amount of RAM.

This also never happened before whatever change was made to the rule set that introduced the HTML parsing error just recently.  Since then, I have not been able to enable this ruleset.  Considering Suricata's version was not changed in OPNsense I'm inclined to believe there's still an issue in the rule set itself but... either way, it was not crashing Suricata a couple of weeks ago and it is now.

Enabling this rule set still causes RAM usage to grow until Suricata crashes... there is nothing in the Suricata log and the only entry in the general log is "kernel: pid (suricata), uid 0: exited on signal 6 (core dumped)."

Disabling the rule set remains the only way to keep Suricata from crashing.  I've tried reporting the issue to abuse.ch as well but haven't really gotten any response except "fixed," which it isn't...

Please let me know if there's anything I can provide to help narrow down the cause of the issue.

Just to let you guys know, we are back to stalking and harassment now from the user called "htilonom" who was allegedly causing the deletion of our Wikipedia page and involved in creating opnsense.com in the first place [1]

This topic now especially caught the spotlight:

https://www.reddit.com/r/OPNscammed/comments/93pmyk/opnsense_called_out_for_changing_roadmap_a_day/

I am sure this is just coincidental and there is no reason why anyone would focus on this. Especially not Netgate, its owner Jim Thompson, pfSense mastermind, right?

https://twitter.com/gonzopancho/status/1024493145415929856

Consider this a setup and tainted discussion from the start. At this point these attempts are petty at best as if they haven't learned from the opnsense.com debacle and I guess the downwards trend will continue. ;)


[1] https://forum.opnsense.org/index.php?topic=6466.0

Oh brother... what a bunch of mooks.  This kind of behavior just further cements my unwillingness to recommend pfsense in any sort of professional capacity.

I'm still waiting for an answer from them on how one can "steal code" from a supposedly open-source project, lol...

EDIT: Whoops, quoted wrong post... fixed now.

Can you watch the memory Bar in the Dashboard shortly after enabling Suricata?

Yes.  I refresh it several times over the course of a couple of minutes and watch the memory usage grow to about 2.8-3GB before it crashes.  Once it crashes, memory usage goes back to a more normal number like 750MB-1GB depending on usage.

So you get an error in the logs? If not perhaps the ruleset is too big for you system?

The only thing that shows up in the logs is Suricata crashing.

Despite abuse.ch's claims to the contrary on Twitter, the issue still isn't fixed.  Enabling abuse.ch/urlhaus rules still results in Suricata crashing.