update issue (related to configd)

[gratis]

Greetings. My recent experience, in case it is helpful, troubleshooting an issue that developed after an update...

About 3 days ago I did an update, and noticed while watching the updates download something about pfSense3, which caught my attention, and might ultimately point to an answer... Regardless, since that update, I have experienced very erratic behavior.

Besides the configd lockup issue, which is easily identifiable and fixable on the Lobby page after the most recent update, a few other things have happened. There are frequent lock-ups of the DNS Resolver, even though the status shows active. Restarting the Resolver doesn't help, and sometimes it doesn't restart. In addition, at times the network becomes completely non-responsive, and pages time-out and won't reload, internal and external pings timeout, etc.

I disabled all non-essential services, like the IPS/IDS and Proxy Server, even though no changes have been made and they have been working fine for months. However, the issue persisted. I finally downloaded a fresh 16.1 iso and reinstalled. It worked fine, so I started manually re-configuring the services, and all was well. Then I did an update, and continued configuring services, at which point the same erratic behavior started again. Makes me wonder if the "Serious Issue" post before this one is somehow related.

So, now I have reinstalled a fresh 16.1 again, and will reconfigure services again, but will NOT update until at least after the configd lockup issue is resolved, as it is likely related. Will report back if my non-updated configuration has issues over the next few days. Hopefully this helps...

[phoenix]

You mention a couple of times that you reinstall 16.1 then upgraded (when your problem occurred), the current version is 16.1.6. My question would be, which exact version were you installing and which version were you upgrading to. What happens if your do a clean install 16.1.6, does that run OK? What hardware are you running this on?

[gratis]

Good question, thanks.

I downloaded an i386 ISO from the website, which was released in January. It's plain 16.1-i386, FreeBSD 10.2-RELEASE-p11, with no updates. The hardware is an old IBM ThinkCentre, which has been running either pfSense or OPNSense for the last 3 years without issue. Granted, it is very dated, but it works for my current needs.

I had installed the original release (15.1?) then have been updating via the web interface since. When these issues started, I downloaded 16.1 release and upgraded from there, and when upgrading the issues return. I have not attempted to install from 16.1.6 or anything newer, I just know that 16.1 with no updates works, and when upgrading to 16.1.6 it breaks.

Using 16.1 release works for me, and I'm simply sharing my experience, in case it helps to pin-point the issue...

[franco]

Hi there,

There are a couple of things we need to untangle here for clarity.

(1) DNS resolver has been reworked, especially for proper DHCP lease registration behaviour. Unbound 1.5.7 also broke briefly to to an inevitable upstream bug that was fixed in FreeBSD really just today with 1.5.8.

(2) Service start stop was reworked to offer plugins abilities to register services easily in the GUI. If there are bugs, we'll need to find a way to describe / reproduce them.

(3) Configd can crash, but leaves the system in a functional state so far as there is no GUI activity requiring system reconfiguration. Lockups of networking may trace back to IPS on Realtek hardware or other things out of our control.

(4) Without knowing which exact version changed things for you WRT to the above is not going to help as much as you will expect. We do provide fixes for a majority of users. The system is too complex to catch all use cases all the time. Unless these specifics are know, the chances of uncovering issues is not increasing, it's decreasing.

So any further help here is appreciated.


Cheers,
Franco

[gratis]

Ok, so based on your answers, my issues could be a combination of things.

1) DNS thus far, in plain 16.1, works fine. None of the lockups experienced while using 16.1.6, so not sure the issue is Unbound, unless the version changes between 16.1 release and 16.1.6.

2) Can't comment. Haven't had an issue restarting services on 16.1 release.

3) I am using Realtec NICs for LAN/LAN2, but network was locking up on 16.1.6 without IPS enabled. And, the IPS has been working with the current NICs under both pfSense and OPNSense up until the most recent updates. So, not sure the Realtek NICs are an issue in my case.

Again, my issues could be a combination of things. I don't need assistance troubleshooting anything, as it all seems to work in 16.1 with no updates. Just weighing in and letting you guys know that based on my experience, there seems to be a regression introduced in the most recent updates, possibly involving imported pfSense code, in case that helps point to a solution for some of the issues being reported.

Consider my situation resolved, will report back with other issues or insights. Thanks for your help.

[franco]

What you are referring to is an update of the php-pfSense package, which is a module we've been wanting to get rid of ever since 15.1. The recent update 0.3 included only the removal of three functions, there was no code imported whatsoever. An upcoming iteration will remove this module for good.

(1) Yes, the version of unbound changed from 1.5.5 to 1.5.7. Unbound 1.5.8 is out for FreeBSD ports since today. If you are willing to try this as a preview to see if your issues go away I'm all for providing you with the snapshot and the necessary steps to update as well as options to revert to an older unbound version if you so desire to see if your experienced DNS instability was caused by unbound or not.

(2) Ok, thanks, I was being overly cautions of these changes, but what you describe could just be exclusive to (1).

(3) What is your network setup? Does this effect your WAN link only or all network transitions through the box?

[gratis]

I would be willing to try a snapshot, to alleviate any concerns about updating software going forward, yes.

Concerning the network setup, it's simply 2 LANs, one running a trusted set of machines, and another running an untrusted network and an access point. When the network locks up, all traffic stops, I can't ping the LAN interface from either network, nor anything on the WAN. The only thing possible at that point, is a reboot of the router.

[franco]

Thanks, that will help. Regarding the unbound snapshot... are you using amd64/OpenSSL or a different combination? I need to know to build the correct package.

[gratis]

Using i386/OpenSSL.

[franco]

Okay, you can move from/to these unbound versions with the following commands from the console:

(For everyone else reading this: all of these are i386/OpenSSL only)

1.5.5

# pkg add -f https://pkg.opnsense.org/FreeBSD:10:i386/15.7/latest/All/unbound-1.5.5.txz

1.5.7_1

# pkg install -f unbound

1.5.8

# pkg add -f https://pkg.opnsense.org/snapshots/unbound-1.5.8.txz

[franco]

This might sound like a stupid question, but regarding configd lockups: SSH is enabled in your setup?

[gratis]

Actually, from my experience, when troubleshooting, no question should be considered stupid... However, no, SSH is not enabled.

In addition, in case it is helpful, I have been running the firewall on 16.1 this whole time. On two occasions DNS has stopped working. And, on both occasions, when attempting to log-in to the web interface, after entering credentials, the web page hangs and doesn't respond.

When this happens I can still ping external addresses (8.8.8.8 or 208.67.220.220), but there is no DNS resolution, nor a functioning web interface. So, apparently there is some issue going back to 16.1, at least on the i386 branch.

Now, I will update to the latest version, upgrade Unbound using the links you sent previously, and follow-up if there are issues. Thanks.

[franco]

Thanks for checking back. Small note: 16.1.8 now has Unbound 1.5.8 as the default.

[gratis]

Excellent, and easy. Updated, will let you know if there are any more glitches. Thanks.