Update to 23.1.8 got stuck

[pp]

Upgrade from 23.1.7_3 to 23.1.8 appeared to be stuck at:

Code: [Select]

***GOT REQUEST TO UPDATE***
Currently running OPNsense 23.1.7_3 at Thu May 25 15:04:32 CEST 2023
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking for upgrades (82 candidates): .......... done
Processing candidates (82 candidates): .... done
The following 33 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
py39-tzdata: 2023.3_1

Installed packages to be UPGRADED:
ca_root_nss: 3.89 -> 3.89.1
crowdsec: 1.4.6_2 -> 1.5.1
crowdsec-firewall-bouncer: 0.0.23.r2_12 -> 0.0.27
curl: 8.0.1 -> 8.1.0
dhcp6c: 20200512_1 -> 20230523
easy-rsa: 3.1.2 -> 3.1.3
lighttpd: 1.4.69 -> 1.4.70
mpd5: 5.9_14 -> 5.9_16
nss: 3.89 -> 3.89.1
openvpn: 2.6.3 -> 2.6.4
opnsense: 23.1.7_3 -> 23.1.8
opnsense-update: 23.1.6 -> 23.1.8
os-crowdsec: 1.0.4 -> 1.0.5
php81: 8.1.18 -> 8.1.19
php81-ctype: 8.1.18 -> 8.1.19
php81-curl: 8.1.18 -> 8.1.19
php81-dom: 8.1.18 -> 8.1.19
php81-filter: 8.1.18 -> 8.1.19
php81-gettext: 8.1.18 -> 8.1.19
php81-ldap: 8.1.18 -> 8.1.19
php81-mbstring: 8.1.18 -> 8.1.19
php81-pdo: 8.1.18 -> 8.1.19
php81-session: 8.1.18 -> 8.1.19
php81-simplexml: 8.1.18 -> 8.1.19
php81-sockets: 8.1.18 -> 8.1.19
php81-sqlite3: 8.1.18 -> 8.1.19
php81-xml: 8.1.18 -> 8.1.19
php81-zlib: 8.1.18 -> 8.1.19
py39-numpy: 1.24.1_1,1 -> 1.24.1_4,1
py39-pandas: 1.5.3_1,1 -> 2.0.1_1,1
py39-requests: 2.29.0 -> 2.30.0
suricata: 6.0.11_1 -> 6.0.12

Number of packages to be installed: 1
Number of packages to be upgraded: 32

The process will require 31 MiB more space.
71 MiB to be downloaded.
[1/33] Fetching php81-sqlite3-8.1.19.pkg: ... done
[2/33] Fetching php81-sockets-8.1.19.pkg: ..... done
[3/33] Fetching lighttpd-1.4.70.pkg: .......... done
[4/33] Fetching opnsense-update-23.1.8.pkg: ..... done
[5/33] Fetching os-crowdsec-1.0.5.pkg: ... done
[6/33] Fetching nss-3.89.1.pkg: .......... done
[7/33] Fetching py39-numpy-1.24.1_4,1.pkg: .......... done
[8/33] Fetching easy-rsa-3.1.3.pkg: ....... done
[9/33] Fetching crowdsec-1.5.1.pkg: .......... done
[10/33] Fetching openvpn-2.6.4.pkg: .......... done
[11/33] Fetching php81-filter-8.1.19.pkg: ... done
[12/33] Fetching php81-8.1.19.pkg: .......... done
[13/33] Fetching py39-pandas-2.0.1_1,1.pkg: .......... done
[14/33] Fetching dhcp6c-20230523.pkg: ......... done
[15/33] Fetching py39-requests-2.30.0.pkg: .......... done
[16/33] Fetching crowdsec-firewall-bouncer-0.0.27.pkg: .......... done
[17/33] Fetching py39-tzdata-2023.3_1.pkg: .......... done
[18/33] Fetching ca_root_nss-3.89.1.pkg: .......... done
[19/33] Fetching php81-ctype-8.1.19.pkg: . done
[20/33] Fetching php81-simplexml-8.1.19.pkg: ... done
[21/33] Fetching php81-session-8.1.19.pkg: ..... done
[22/33] Fetching curl-8.1.0.pkg: .......... done
[23/33] Fetching php81-zlib-8.1.19.pkg: ... done
[24/33] Fetching php81-dom-8.1.19.pkg: ........ done
[25/33] Fetching suricata-6.0.12.pkg: .......... done
[26/33] Fetching mpd5-5.9_16.pkg: .......... done
[27/33] Fetching php81-ldap-8.1.19.pkg: ..... done
[28/33] Fetching php81-xml-8.1.19.pkg: ... done
[29/33] Fetching php81-pdo-8.1.19.pkg: ....... done
[30/33] Fetching php81-curl-8.1.19.pkg: ..... done
[31/33] Fetching php81-mbstring-8.1.19.pkg: .......... done
[32/33] Fetching opnsense-23.1.8.pkg: .......... done
[33/33] Fetching php81-gettext-8.1.19.pkg: . done
Checking integrity... done (0 conflicting)
[1/33] Upgrading py39-numpy from 1.24.1_1,1 to 1.24.1_4,1...
[1/33] Extracting py39-numpy-1.24.1_4,1: .......... done
[2/33] Upgrading php81 from 8.1.18 to 8.1.19...
[2/33] Extracting php81-8.1.19: .......... done
[3/33] Installing py39-tzdata-2023.3_1...
[3/33] Extracting py39-tzdata-2023.3_1: .......... done
[4/33] Upgrading ca_root_nss from 3.89 to 3.89.1...
[4/33] Extracting ca_root_nss-3.89.1: ...... done
[5/33] Upgrading nss from 3.89 to 3.89.1...
[5/33] Extracting nss-3.89.1: .......... done
[6/33] Upgrading easy-rsa from 3.1.2 to 3.1.3...
[6/33] Extracting easy-rsa-3.1.3: .......... done
[7/33] Upgrading py39-pandas from 1.5.3_1,1 to 2.0.1_1,1...
[7/33] Extracting py39-pandas-2.0.1_1,1: .......... done
[8/33] Upgrading crowdsec-firewall-bouncer from 0.0.23.r2_12 to 0.0.27...
[8/33] Extracting crowdsec-firewall-bouncer-0.0.27: ...... done
crowdsec_firewall is running as pid 39371.
Stopping crowdsec_firewall.
Waiting for PIDS: 39371.
[9/33] Upgrading php81-session from 8.1.18 to 8.1.19...
[9/33] Extracting php81-session-8.1.19: .......... done
[10/33] Upgrading curl from 8.0.1 to 8.1.0...
[10/33] Extracting curl-8.1.0: .......... done
[11/33] Upgrading php81-pdo from 8.1.18 to 8.1.19...
[11/33] Extracting php81-pdo-8.1.19: .......... done
[12/33] Upgrading php81-mbstring from 8.1.18 to 8.1.19...
[12/33] Extracting php81-mbstring-8.1.19: .......... done
[13/33] Upgrading php81-sqlite3 from 8.1.18 to 8.1.19...
[13/33] Extracting php81-sqlite3-8.1.19: ......... done
[14/33] Upgrading php81-sockets from 8.1.18 to 8.1.19...
[14/33] Extracting php81-sockets-8.1.19: .......... done
[15/33] Upgrading lighttpd from 1.4.69 to 1.4.70...
===> Creating groups.
Using existing group 'www'.
===> Creating users
Using existing user 'www'.
[15/33] Extracting lighttpd-1.4.70: .......... done
[16/33] Upgrading opnsense-update from 23.1.6 to 23.1.8...
[16/33] Extracting opnsense-update-23.1.8: .......... done
[17/33] Upgrading crowdsec from 1.4.6_2 to 1.5.1...
[17/33] Extracting crowdsec-1.5.1: .......... done
crowdsec is running as pid 66369.
Stopping crowdsec.
Waiting for PIDS: 66369.
[18/33] Upgrading openvpn from 2.6.3 to 2.6.4...
===> Creating groups.
Using existing group 'openvpn'.
===> Creating users
Using existing user 'openvpn'.
[18/33] Extracting openvpn-2.6.4: .......... done
[19/33] Upgrading php81-filter from 8.1.18 to 8.1.19...
[19/33] Extracting php81-filter-8.1.19: ......... done
[20/33] Upgrading dhcp6c from 20200512_1 to 20230523...
[20/33] Extracting dhcp6c-20230523: ........ done
[21/33] Upgrading py39-requests from 2.29.0 to 2.30.0...
[21/33] Extracting py39-requests-2.30.0: .......... done
[22/33] Upgrading php81-ctype from 8.1.18 to 8.1.19...
[22/33] Extracting php81-ctype-8.1.19: ........ done
[23/33] Upgrading php81-simplexml from 8.1.18 to 8.1.19...
[23/33] Extracting php81-simplexml-8.1.19: ......... done
[24/33] Upgrading php81-zlib from 8.1.18 to 8.1.19...
[24/33] Extracting php81-zlib-8.1.19: ........ done
[25/33] Upgrading php81-dom from 8.1.18 to 8.1.19...
[25/33] Extracting php81-dom-8.1.19: .......... done
[26/33] Upgrading suricata from 6.0.11_1 to 6.0.12...
[26/33] Extracting suricata-6.0.12: .......... done
[27/33] Upgrading mpd5 from 5.9_14 to 5.9_16...
[27/33] Extracting mpd5-5.9_16: .......... done
[28/33] Upgrading php81-ldap from 8.1.18 to 8.1.19...
[28/33] Extracting php81-ldap-8.1.19: ........ done
[29/33] Upgrading php81-xml from 8.1.18 to 8.1.19...
[29/33] Extracting php81-xml-8.1.19: ......... done
[30/33] Upgrading php81-curl from 8.1.18 to 8.1.19...
[30/33] Extracting php81-curl-8.1.19: .......... done
[31/33] Upgrading php81-gettext from 8.1.18 to 8.1.19...
[31/33] Extracting php81-gettext-8.1.19: ........ done
[32/33] Upgrading os-crowdsec from 1.0.4 to 1.0.5...
[32/33] Extracting os-crowdsec-1.0.5: .......... done
I checked my crowdsec processes:

Code: [Select]

root@opnsense01:~ # ps aux | grep crowdsec
root    43578   0.0  0.0   13504   2768  -  I    15:05       0:00.00 /bin/sh -c set -- os-crowdsec-1.0.4\n#!/bin/sh\n\n# need to temporarily stop the bouncer to remove all the rules\nservice crowdsec_firewall stop >/dev/nul
root    43757   0.0  0.0   13504   3140  -  I    15:05       0:00.01 /bin/sh /usr/local/etc/rc.d/crowdsec_firewall stop
root    88726   0.0  0.7  914324 121868  -  I    15:04       0:03.12 /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root    90891   0.0  0.2  722256  28864  -  I    15:04       0:00.19 /usr/local/bin/crowdsec-firewall-bouncer -c /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml (crowdsec-firewall-b)

and manually did a 'kill -9 90891' after which the update immediately proceeded and finished successfully.

[franco]

Perhaps it would also hang while stopping for reboot. It's pretty difficult to account for that but for best practice restarts of running services are best avoided during updates (and for core we don't do this at all).


Cheers,
Franco

[svheel]

I had exactly the same issue with the Crowdsec plugin installed, upgrade got stuck at the same point.
Killing the crowdsec-firewall-bouncer was the solution for me too, thanks for the suggestion!

[cbdudley]

Same situation here, the upgrade got stuck at the same point. Killed the firewall-bouncer process and the upgrade completed normally with no further trouble. Thanks to the original poster for the suggestion!

[bucky2780]

same thing happened here... killing the crowdsec process worked a treat...

[OzziGoblin]

Same here, killing Crowdsec continued the update, was a good learning experience as I'm new to OpnSense and Linux 

[Taunt9930]

Any steps we can take to stop this from happening on upgrade? Would unticking enable in the UI before upgrade work?

[franco]

My best guess is this has something to do with the crowdsec software update of a previous update. Stopping might be problematic for reboot now as well perhaps and so would stopping from the GUI. Crowdsec people need to take a closer look here.


Cheers,
Franco

[newsense]

I tried stopping Crowdsec from the GUI first and waited a few minutes. Since the bouncer process wouldn't go away I tried a kill PID, gave it another minute and checked again, then had to go for kill -9

Code: [Select]

root@OPNsense:~ # ps aux |grep 63334
root    56827   0.0  0.0   12648    2116  -  I    09:07      0:00.00 pwait 63334
root    63015   0.0  0.0   12756    2232  -  Is   08:53      0:00.00 daemon: Crowdsec Firewall[63334] (daemon)
root    63334   0.0  0.1  722512   26352  -  I    08:53      0:00.16 /usr/local/bin/crowdsec-firewall-bouncer -c /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml (crowdsec-firewall-b)
root    49270   0.0  0.0   12648    2116  0  I+   08:53      0:00.00 pwait 63334
root    44625   0.0  0.0   12748    2360  1  S+   09:12      0:00.00 grep 63334
root@OPNsense:~ # kill 63334
root@OPNsense:~ # kill 63015
root@OPNsense:~ # ps aux | grep 63334
root    56827   0.0  0.0   12648    2116  -  I    09:07      0:00.00 pwait 63334
root    63334   0.0  0.1  722512   26352  -  S    08:53      0:00.16 /usr/local/bin/crowdsec-firewall-bouncer -c /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml (crowdsec-firewall-b)
root    49270   0.0  0.0   12648    2116  0  I+   08:53      0:00.00 pwait 63334
root    10006   0.0  0.0   12748    2360  1  S+   09:13      0:00.00 grep 63334
root@OPNsense:~ # kill -9 63334
root@OPNsense:~ # ps aux | grep 63334
root    85675   0.0  0.0   12748    2360  1  S+   09:13      0:00.00 grep 63334

On two FWs where Crowdsec was installed but temporarily stopped at the time the upgrade completed without any issue.

[franco]

https://twitter.com/mmetc2/status/1661986481374625792

[mmetc]

Hi, thanks for the report

I could not replicate the issue, downgrading to 1.0.3 - 1.4.6 - 0.0.23.rc2, they all updated and didn't require kill or reboot.

Between 0.0.23.rc2 and 0.0.27 the ip removal is a lot (100x) faster and we did a reworking of the concurrency and signal management so I strongly doubt the issue would happen again.

For 0.0.23.rc2 I could update the plugin to do the kill -9, but I thought that the "service" command would already do that.
I suspect the bouncer could be slow while removing banned ips one by one, so it would be harder to replicate on a fast machine or vm. I'll try with 200k+ decisions.

[Waschl]

Hi,

had the same issue on 1 of 3 firewall updates. Killing the crowdsec bouncer did the job.

Regards

Michael

[SpinningRust]

Exact same issue here, kill -9 of crowdsec-firewall-b resolved the issue for me too. Thank you!

[FredsterNL]

Just joining the fun: I had the same issue as well 

-Tried stopping Crowsec to no avail
- SoftReset of my firewall running on a DEC-750 caused it to become even less responsive
- Applied the magic fixes everything solution: pulled the plug 

After it was back, the missing components could be replaced (including the CrowdSec component and new kernel)

So, apparently Crowsec could not be stopped, causing the upgrade to habg.

Would be nice if a hang is detected, a KILL -9 could be offered?

[franco]

Would be nice if a hang is detected, a KILL -9 could be offered?

Killing major upgrades in progress as a later consequence? Probably not.


Cheers,
Franco