OPNsense Forum

Archive => 17.1 Legacy Series => Topic started by: Taomyn on February 02, 2017, 07:26:16 pm

Title: [SOLVED] Firewall hanging every day -- turnstile_broadcast() crash
Post by: Taomyn on February 02, 2017, 07:26:16 pm
Everything was more or less fine after upgrading from 16.7.14 to 17.1 until today when during the day I was unable to access my systems externally. On returning home I discovered the firewall had completely locked up, no ping or GUI. When I looked at the console directly via VGA, I saw this instead of any kind of login prompt:

Code: [Select]
I'll put a screen shot here, or I would if I could figure this forum, nothing works for me :-(
I connected a keyboard and nothing worked so I had to press and hold the power button, then restart it. I checked my Console Options and they appear fine:

Code: [Select]
I'll put a screen shot here, or I would if I could figure this forum, nothing works for me :-(
Any ideas?

I also submitted an error report through the firewall in case that helps.
Title: Re: Firewall hang and odd console
Post by: franco on February 02, 2017, 09:30:04 pm
Try setting the "vt" driver and primary console to "efi". FreeBSD flipped their the default on 11.0, and the old defaults that worked for EFI are not working so well anymore (we kept our settings to minimise impact, ironic).


Cheers,
Franco
Title: Re: Firewall hang and odd console
Post by: Taomyn on February 02, 2017, 10:08:30 pm
Try setting the "vt" driver and primary console to "efi". FreeBSD flipped their the default on 11.0, and the old defaults that worked for EFI are not working so well anymore (we kept our settings to minimise impact, ironic).


Cheers,
Franco

Thanks, that did the trick after I rebooted - if it hangs again, hopefully I can gather more info.
Title: Re: Firewall hang and odd console
Post by: Taomyn on February 03, 2017, 06:23:35 pm
I'm afraid it hung again today, exactly the same way although this time I could see some information - see screen shot, but connecting a keyboard still made no difference and I was forced to kill the power. I also submitted another report through the GUI once the firewall restarted, and I then disabled a few plugins I don't currently use (I was going to play with them later), and see how it goes. All though IPS/IDS is still broken for me, I left that on and will disable it should the problem repeat.
Title: Re: Firewall hang and odd console
Post by: Taomyn on February 05, 2017, 10:14:31 am
It's happened again - have disabled Intrusion Detection this time, though not really working anyway.


Report submitted via GUI also.
Title: Re: Firewall hanging every day
Post by: Taomyn on February 06, 2017, 07:11:21 pm
Happened again today, changed the thread title to highlight this as the main issue.

Is there some way to schedule just a firewall reboot? I'm thinking of doing this to at least give me back some kind of stability.
Title: Re: Firewall hanging every day
Post by: franco on February 07, 2017, 09:15:42 am
We're having the same issue multiple times: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

I'm going to ask FreeBSD for help on this report. Can you post your hardware specs? This is surely something related to a distinct set of conditions.


Cheers,
Franco
Title: Re: Firewall hanging every day
Post by: Taomyn on February 07, 2017, 01:41:46 pm
Is there a good way to get the hardware specs as the firewall sees it, either in the GUI or via the command-line.


It's a while since I built the box and I don't recall every specification of it - but I will get what I can when I get home.
Title: Re: Firewall hanging every day
Post by: franco on February 07, 2017, 04:31:27 pm
You can generate a crash report under and drop me a note in the description. But hardware eval on problematic system has show hardware as new as 2013, so no sure anymore. :/
Title: Re: Firewall hanging every day
Post by: Taomyn on February 07, 2017, 05:20:39 pm
Well my box is only about two years old, so fairly new.


I did submit 3 reports so far, if you can search for any with the email address starting "ferd352" and ending "@gmail.com" (I use plus-addressing) they will be mine.
Title: Re: Firewall hanging every day
Post by: Taomyn on February 09, 2017, 11:32:12 am
I just upgraded to 17.1.1 and wanted to know if there is any chance my problem is likely fixed by the updates? I ask because then I will disable my daily reboot of the firewall and see how it goes.
Title: Re: Firewall hanging every day
Post by: franco on February 09, 2017, 11:35:26 am
No particular change for this. Still need to discuss with FreeBSD. Sorry. ;(
Title: Re: Firewall hanging every day
Post by: Taomyn on February 09, 2017, 11:38:52 am
No worries, thought I would ask, so I'll leave my reboot in place for tomorrow morning then disable it and see what happens at the weekend, at least then I'll be close by to reboot it manually.
Title: Re: Firewall hanging every day
Post by: abraxxa on February 09, 2017, 09:02:34 pm
Just to let you know that I'm experiencing the same hangs where the firewall doesn't reply to any request, neither ping nor https.
Pressing the power button of my mini itx case does reboot it though.
Title: Re: Firewall hanging every day
Post by: Taomyn on February 10, 2017, 08:46:22 am
Just to let you know that I'm experiencing the same hangs where the firewall doesn't reply to any request, neither ping nor https.
Pressing the power button of my mini itx case does reboot it though.

And the same error on the console as I reported above?

Have you submitted a report through the firewall as well? It would be good to get more data points sent in.Go to System, Firmware, Reporter.
Title: Re: Firewall hanging every day
Post by: abraxxa on February 10, 2017, 05:25:16 pm
The reporter doesn't report any errors after the reboot.
I haven't had the possibility to check the console, but for the last three days it was stable.
Updated to 17.1.1 yesterday too.
Title: Re: Firewall hanging every day
Post by: abraxxa on February 10, 2017, 08:44:59 pm
I also switched the console to 'vt', might this be a workaround/fix for the hang?
Title: Re: Firewall hanging every day
Post by: Taomyn on February 11, 2017, 12:20:06 pm
I also switched the console to 'vt', might this be a workaround/fix for the hang?


No, at least it wasn't for me - since updating to 17.1.1 I'm testing mine without a scheduled reboot to see if it's fixed. 1day 4hrs so far....
Title: Re: Firewall hanging every day
Post by: Taomyn on February 11, 2017, 01:53:31 pm
Well so much for that test - it just hung again so will re-enable the daily reboots.


Report submitted as always.
Title: Re: [17.1.1] Firewall hanging every day
Post by: mistergefahrensucher on February 13, 2017, 07:35:21 pm
Same issue on my box. lexcom 3i270d
Installed fresh one week ago.

Died always after one day.

Intel(R) Atom(TM) CPU N270 @ 1.60GHz
2 CPUs: 1 package(s) x 1 core(s) x 2 HTT thread. 1GBRam
NanoBSD form 4GB-CF

Title: Re: [17.1.1] Firewall hanging every day
Post by: Taomyn on February 14, 2017, 10:14:00 am
Died always after one day.


Did you submit a report through the firewall? The "Report" page is a little mis-leading as it will look like there is nothing to report, but hit the button anyway as it will say there is and fill in the details.


The more info we send in, hopefully the sooner we can get this fixed.
Title: Re: [17.1.1] Firewall hanging every day
Post by: mistergefahrensucher on February 15, 2017, 08:04:23 am
Sorry can't provide you with chrashlogs. because the firewall will not boot anymore.
So i switched back to my old CF-Card with a booting pfsense 2.3.1
At the moment i have not the time for investigating.
My intension was to go to opnsense because pfsense will not support i386 and NANOBSD in the comming release.
Have seen some other posts where also users have problems with BSD 11.0 on Atom Hardware.
At the moment i will stay on pfsense 2.3.1 with BSD 10.3.


Title: Re: [17.1.1] Firewall hanging every day
Post by: franco on February 16, 2017, 07:51:58 pm
So far not much luck here. It's a corruption of the VM lists somehow, a locked object returns without a lock. The code does not expect this to happen at all and on 10.3 apparently it did not.

It's somewhere in the amd64 code, so maybe i386 has better luck running smoothly. Can somebody confirm/deny?
Title: Re: [17.1.1] Firewall hanging every day
Post by: franco on February 17, 2017, 08:26:45 am
Regarding the crashes in turnstile_broadcast(), a FreeBSD developer provided a test kernel:

# opnsense-update -kr 17.1.1-rwdebug
# /usr/local/etc/rc.reboot

It still panics, but will print vital debug information when doing so.

Running it and providing the output will hopefully help resolve this.


Thank you all,
Franco
Title: Re: [SOLVED] Firewall hanging every day -- turnstile_broadcast() crash
Post by: angel2040 on March 13, 2017, 12:07:50 am
I missed something.   How is this solved?  I'm hitting this exact issue.
Title: Re: [SOLVED] Firewall hanging every day -- turnstile_broadcast() crash
Post by: djGrrr on March 13, 2017, 01:34:23 am
Are you running OPNsense 17.1.2?
Title: Re: [SOLVED] Firewall hanging every day -- turnstile_broadcast() crash
Post by: angel2040 on March 13, 2017, 07:09:05 am
Are you running OPNsense 17.1.2?

I am now. 
Title: Re: [SOLVED] Firewall hanging every day -- turnstile_broadcast() crash
Post by: franco on March 14, 2017, 02:17:44 pm
The issue is fixed in 17.1.2 and FreeBSD is still asking for users to reproduce this with the debug kernel posted above in order to get it fixed there... I would just like to pass this along, but I can understand our users' stance as well.


Cheers,
Franco