Not rebooting properly during console upgrade

[Taomyn]

Thought I would report this as it's now the 3rd time in a row where upgrading the firewall from the console seems to get stuck during the reboot.


I've waited well over 10 minutes, but always end up having to press Ctrl-C, then choosing "Reboot" from the menu.


Nothing seems to be harmed afterwards, I just never had this with v16.

[franco]

One of the services is stuck, if it's not already exited. What's in /etc/rc.conf and /etc/rc.conf.d?

[Taomyn]

Code: [Select]

root@bart:~ # ls /etc/rc.conf.d
acme_http_challenge     flowd_aggregate         netflow
captiveportal           haproxy                 squid
flowd                   ipfw                    suricata
root@bart:~ # ls /etc/rc.conf
ls: /etc/rc.conf: No such file or directory

[franco]

Ok, need the internals... cat /etc/rc.conf.d/*

(make sure to scrub output if there is something private in there)

[Taomyn]

Code: [Select]

acme_http_challenge_enable=YES
acme_http_challenge_conf="/var/etc/lighttpd-acme-challenge.conf"
acme_http_challenge_pidfile="/var/run/lighttpd-acme-challenge.pid"
acme_http_challenge_opnsense_bootup_run="/usr/local/opnsense/scripts/OPNsense/AcmeClient/setup.sh"
captiveportal_enable="NO"
#
# Automatic generated configuration for netflow.
# Do not edit this file manually.
#
flowd_enable="YES"
#
# Automatic generated configuration for netflow.
# Do not edit this file manually.
#
flowd_aggregate_enable="YES"
haproxy_enable=YES
haproxy_opnsense_bootup_run="/usr/local/opnsense/scripts/OPNsense/HAProxy/setup.sh"
haproxy_pidfile="/var/run/haproxy.pid"
haproxy_config="/usr/local/etc/haproxy.conf"
# haproxy_flags=""
firewall_enable="NO"
firewall_script="/usr/local/etc/rc.ipfw"
dummynet_enable="YES"
#
# Automatic generated configuration for netflow.
# Do not edit this file manually.
#
netflow_enable="YES"


0'
.?1'
..
0'squid¦¦suricata_enable="YES"
suricata_opnsense_bootup_run="/usr/local/opnsense/scripts/suricata/setup.sh"


# IPS mode, switch to netmapsuricata_netmap=YES

[franco]

It could be HAproxy blocking?

Squid has a directory, I forgot...

# cat /etc/rc.conf.d/squid/*

[Taomyn]

Code: [Select]

root@bart:~ # cat /etc/rc.conf.d/squid/*
squid_enable=NO
Would I be right that the last PID mentioned in the screen shot is the one for the process it's waiting for? If so, I'll try to remember if it happens next update to look up the process from it.

[franco]

I think the addition of hooking into /etc/rc.shutdown caused this, added in 17.1-RC1:

https://github.com/opnsense/changelog/blob/922038/doc/17.1/17.1.r1#L43

SSH should still run, good idea to look for the pid. Theoretically, however, it isn't there anymore and it waits in vain. In that case, find the /var/run/*.pid file that has the actual PID to reveal the service name.


Thanks,
Franco

[Taomyn]

Ok, would that also explain why earlier I tried twice to reboot from the console menu, and only when I went to the shell and typed "reboot" did it actually reboot?

I'm was not able to see the main screen at the time to see if it showed anything, but the SSH session did not show any PIDs it was waiting for.

[akron]

Hi,

I am experiencing something similar, have an HA Setup with HA proxy and the master Firewall does not reboot or power off without going to the console and entering reboot or poweroff.

Backup Firewall reboots without problems when is not being used but If I failover the master one to the backup firewall, backup firewall does not reboot with the same behaviour.

Could this be a bug in HA config or similar?

Cheers
 

[akron]

Ok, would that also explain why earlier I tried twice to reboot from the console menu, and only when I went to the shell and typed "reboot" did it actually reboot?

I'm was not able to see the main screen at the time to see if it showed anything, but the SSH session did not show any PIDs it was waiting for.

Did you discover the PID causing this issue?

I am still facing the problem I suspect HAproxy is the one, in theory should fail-over without problem to the Backup FW

Cheers

[Taomyn]

Going by my reply in another thread, yes haproxy was the culprit:

I upgraded from 17.1.6-amd64 to 17.1.7 via the console option the following happened:


1. The upgrade could not reboot as it was waiting for a process, which when I killed simply killed my external connection. The process was "haproxy". When I arrived home I was unable to ssh to the box, my password was refused, used the console directly, root/no password and issued  a "reboot".

[akron]

Going by my reply in another thread, yes haproxy was the culprit:

I upgraded from 17.1.6-amd64 to 17.1.7 via the console option the following happened:


1. The upgrade could not reboot as it was waiting for a process, which when I killed simply killed my external connection. The process was "haproxy". When I arrived home I was unable to ssh to the box, my password was refused, used the console directly, root/no password and issued  a "reboot".

Ok cool, did you found any work around for this?

Because HA works well if I cut the network immediately fails-over to the backup firewall, but if I restart the main Firewall hangs over and does not failover causing interruption of service.

Cheers