Potential firewall problem

[mszeliga]

Hi

I have now several times experienced a situation where the firewall crashes and becomes a simple router.

I have experienced this behaviour on running virtualized and on real hardware and both in 15.7 and 16.1.
I do not know what happens (yet) but when the firewall crashes the result should be no traffic passing instead of all traffic passing.

After this happens the firewall keep acting as a simple router even after a reboot, only restoring earlier configuration may fix the problem.

I am running with 10 interfaces (on the hardware) and 8 VLAN interfaces on virtual,

I will try to dig the logfiles out of the crashed virtualized tomorrow (9. march).

regrds
Maciej

[temporaryuser]

Hi Maciej,

I have now several times experienced a situation where the firewall crashes and becomes a simple router.
<snip> the result should be no traffic passing instead of all traffic passing.
<snip> After this happens the firewall keep acting as a simple router even after a reboot, only restoring earlier configuration may fix the problem.

Wow. If that turns out to be true, it would be really shocking. The whole intranet with all its network segments wide open and fully exposed to the internet, just because the firewall crashed... 

Could you please specify what you mean with "when the firewall crashes"? Do you mean that the whole operating system (OS) hangs/reboots/freezes, etc. or do you mean that the packet filter of the OS goes crazy while the OS continues to run normally?

Sincerely worried
temporaryuser

[mszeliga]

It seems that the filter stops filtering, everything else is working.

The problem is that you really can't see it: webinterface is running and it is possible to change settings but no changes will fix the problem, the only visible symptom is that I've  got a handfull of log entries about IGMP packets from 10.0.0.0 and then the logging stops (the other possibility has been continous log entries of this type).
After that all packets seems to be forwarded... just like it would be thru a simple router.

A restore of a earlier configuration may fix the problem but I had to go 5 backups back before it worked.

One important clarification is needed: this has only happened on configuration changes and not during normal operation.

I am looking for the logs now, if opnsense team want the whole vm I could upload that.

regards
Maciej

[franco]

Hi Maciej,

That would indeed be helpful. Can you PM me the download link?


Cheers,
Franco

[temporaryuser]

Hi Franco & Maciej,

That would indeed be helpful. Can you PM me the download link?

Is there any update on this serious matter?

Regards
temporaryuser

[franco]

Today is firmware upgrade day, I will have more time on Friday to give a response here.

I would wish that "serious problem" is not going to be a synonym for yet uncategorised behaviour that could range from virtualisation layer issues, bugs in OPNsense or things that predate the fork, solar flares, or something else entirely.

For me this is not a business and all free time, I have a day job and the courtesy of debugging complex setups is given out of care and respect for the project and the people behind it.

[mszeliga]

Hi Franco & Maciej,

That would indeed be helpful. Can you PM me the download link?

Is there any update on this serious matter?

Regards
temporaryuser

I SHOULD probably write that I only had this problem during configuration.

Sorry, I have been quite busy at work.

Regards
Maciej

[packet loss]

mszeliga first off thank you for reporting this issue. You created this post over a month ago with the subject "There is a serious problem!". Unfortunately you haven't had the time to provide any new information since then so the OPNsense team can troubleshoot this. I feel at this point it's appropriate to change the subject title to more closely resemble the problem you are experiencing. Maybe another OPNsense user is experiencing this same issue and could provide some additional helpful information. Again thank you for reporting the issue and hopefully in the future when you have the time this can be resolved.

[mszeliga]

mszeliga first off thank you for reporting this issue. You created this post over a month ago with the subject "There is a serious problem!". Unfortunately you haven't had the time to provide any new information since then so the OPNsense team can troubleshoot this. I feel at this point it's appropriate to change the subject title to more closely resemble the problem you are experiencing. Maybe another OPNsense user is experiencing this same issue and could provide some additional helpful information. Again thank you for reporting the issue and hopefully in the future when you have the time this can be resolved.

I've sent a PM to Franco with a link to the whole VM (as Franco asked me) some days after I posted the first message.
I did see this at least 3 times (once on physical hardware and 2 times on virtual hardware).
My definition of "serious" regarding a firewall is a when the firewall stops acting as a firewall, it may crash, it may burn but it does NOT just drop all it's filtering and pass all traffic.

Now if someone here claims that this was due to misconfiguration then my claim is that the rule validation was broken! (but then why does it not work with a restored, previous working, config ?)

We may of course also just hope that the problem has disappeared in 1.6.8, I am going to install it soon.

I am however open to suggestions for a better title...

[packet loss]

mszeliga I had already assumed you provided a link to your VM to the OPNsense team to help troubleshoot the issue. Troubleshooting an entire system could easily take days especially a complex one. The OPNsense developers are doing their best to fix any issues that have been reported. Hopefully upgrading to 16.1.8 or 16.1.9 will solve your issue. Thank you for the feedback.