OPNsense Forum

Archive => 23.1 Legacy Series => Topic started by: pp on May 25, 2023, 03:19:38 pm

Title: Update to 23.1.8 got stuck
Post by: pp on May 25, 2023, 03:19:38 pm: Upgrade from 23.1.7_3 to 23.1.8 appeared to be stuck at:

Code: [Select]
***GOT REQUEST TO UPDATE*** Currently running OPNsense 23.1.7_3 at Thu May 25 15:04:32 CEST 2023 Updating OPNsense repository catalogue... OPNsense repository is up to date. All repositories are up to date. Updating OPNsense repository catalogue... OPNsense repository is up to date. All repositories are up to date. Checking for upgrades (82 candidates): .......... done Processing candidates (82 candidates): .... done The following 33 package(s) will be affected (of 0 checked): New packages to be INSTALLED: py39-tzdata: 2023.3_1 Installed packages to be UPGRADED: ca_root_nss: 3.89 -> 3.89.1 crowdsec: 1.4.6_2 -> 1.5.1 crowdsec-firewall-bouncer: 0.0.23.r2_12 -> 0.0.27 curl: 8.0.1 -> 8.1.0 dhcp6c: 20200512_1 -> 20230523 easy-rsa: 3.1.2 -> 3.1.3 lighttpd: 1.4.69 -> 1.4.70 mpd5: 5.9_14 -> 5.9_16 nss: 3.89 -> 3.89.1 openvpn: 2.6.3 -> 2.6.4 opnsense: 23.1.7_3 -> 23.1.8 opnsense-update: 23.1.6 -> 23.1.8 os-crowdsec: 1.0.4 -> 1.0.5 php81: 8.1.18 -> 8.1.19 php81-ctype: 8.1.18 -> 8.1.19 php81-curl: 8.1.18 -> 8.1.19 php81-dom: 8.1.18 -> 8.1.19 php81-filter: 8.1.18 -> 8.1.19 php81-gettext: 8.1.18 -> 8.1.19 php81-ldap: 8.1.18 -> 8.1.19 php81-mbstring: 8.1.18 -> 8.1.19 php81-pdo: 8.1.18 -> 8.1.19 php81-session: 8.1.18 -> 8.1.19 php81-simplexml: 8.1.18 -> 8.1.19 php81-sockets: 8.1.18 -> 8.1.19 php81-sqlite3: 8.1.18 -> 8.1.19 php81-xml: 8.1.18 -> 8.1.19 php81-zlib: 8.1.18 -> 8.1.19 py39-numpy: 1.24.1_1,1 -> 1.24.1_4,1 py39-pandas: 1.5.3_1,1 -> 2.0.1_1,1 py39-requests: 2.29.0 -> 2.30.0 suricata: 6.0.11_1 -> 6.0.12 Number of packages to be installed: 1 Number of packages to be upgraded: 32 The process will require 31 MiB more space. 71 MiB to be downloaded. [1/33] Fetching php81-sqlite3-8.1.19.pkg: ... done [2/33] Fetching php81-sockets-8.1.19.pkg: ..... done [3/33] Fetching lighttpd-1.4.70.pkg: .......... done [4/33] Fetching opnsense-update-23.1.8.pkg: ..... done [5/33] Fetching os-crowdsec-1.0.5.pkg: ... done [6/33] Fetching nss-3.89.1.pkg: .......... done [7/33] Fetching py39-numpy-1.24.1_4,1.pkg: .......... done [8/33] Fetching easy-rsa-3.1.3.pkg: ....... done [9/33] Fetching crowdsec-1.5.1.pkg: .......... done [10/33] Fetching openvpn-2.6.4.pkg: .......... done [11/33] Fetching php81-filter-8.1.19.pkg: ... done [12/33] Fetching php81-8.1.19.pkg: .......... done [13/33] Fetching py39-pandas-2.0.1_1,1.pkg: .......... done [14/33] Fetching dhcp6c-20230523.pkg: ......... done [15/33] Fetching py39-requests-2.30.0.pkg: .......... done [16/33] Fetching crowdsec-firewall-bouncer-0.0.27.pkg: .......... done [17/33] Fetching py39-tzdata-2023.3_1.pkg: .......... done [18/33] Fetching ca_root_nss-3.89.1.pkg: .......... done [19/33] Fetching php81-ctype-8.1.19.pkg: . done [20/33] Fetching php81-simplexml-8.1.19.pkg: ... done [21/33] Fetching php81-session-8.1.19.pkg: ..... done [22/33] Fetching curl-8.1.0.pkg: .......... done [23/33] Fetching php81-zlib-8.1.19.pkg: ... done [24/33] Fetching php81-dom-8.1.19.pkg: ........ done [25/33] Fetching suricata-6.0.12.pkg: .......... done [26/33] Fetching mpd5-5.9_16.pkg: .......... done [27/33] Fetching php81-ldap-8.1.19.pkg: ..... done [28/33] Fetching php81-xml-8.1.19.pkg: ... done [29/33] Fetching php81-pdo-8.1.19.pkg: ....... done [30/33] Fetching php81-curl-8.1.19.pkg: ..... done [31/33] Fetching php81-mbstring-8.1.19.pkg: .......... done [32/33] Fetching opnsense-23.1.8.pkg: .......... done [33/33] Fetching php81-gettext-8.1.19.pkg: . done Checking integrity... done (0 conflicting) [1/33] Upgrading py39-numpy from 1.24.1_1,1 to 1.24.1_4,1... [1/33] Extracting py39-numpy-1.24.1_4,1: .......... done [2/33] Upgrading php81 from 8.1.18 to 8.1.19... [2/33] Extracting php81-8.1.19: .......... done [3/33] Installing py39-tzdata-2023.3_1... [3/33] Extracting py39-tzdata-2023.3_1: .......... done [4/33] Upgrading ca_root_nss from 3.89 to 3.89.1... [4/33] Extracting ca_root_nss-3.89.1: ...... done [5/33] Upgrading nss from 3.89 to 3.89.1... [5/33] Extracting nss-3.89.1: .......... done [6/33] Upgrading easy-rsa from 3.1.2 to 3.1.3... [6/33] Extracting easy-rsa-3.1.3: .......... done [7/33] Upgrading py39-pandas from 1.5.3_1,1 to 2.0.1_1,1... [7/33] Extracting py39-pandas-2.0.1_1,1: .......... done [8/33] Upgrading crowdsec-firewall-bouncer from 0.0.23.r2_12 to 0.0.27... [8/33] Extracting crowdsec-firewall-bouncer-0.0.27: ...... done crowdsec_firewall is running as pid 39371. Stopping crowdsec_firewall. Waiting for PIDS: 39371. [9/33] Upgrading php81-session from 8.1.18 to 8.1.19... [9/33] Extracting php81-session-8.1.19: .......... done [10/33] Upgrading curl from 8.0.1 to 8.1.0... [10/33] Extracting curl-8.1.0: .......... done [11/33] Upgrading php81-pdo from 8.1.18 to 8.1.19... [11/33] Extracting php81-pdo-8.1.19: .......... done [12/33] Upgrading php81-mbstring from 8.1.18 to 8.1.19... [12/33] Extracting php81-mbstring-8.1.19: .......... done [13/33] Upgrading php81-sqlite3 from 8.1.18 to 8.1.19... [13/33] Extracting php81-sqlite3-8.1.19: ......... done [14/33] Upgrading php81-sockets from 8.1.18 to 8.1.19... [14/33] Extracting php81-sockets-8.1.19: .......... done [15/33] Upgrading lighttpd from 1.4.69 to 1.4.70... ===> Creating groups. Using existing group 'www'. ===> Creating users Using existing user 'www'. [15/33] Extracting lighttpd-1.4.70: .......... done [16/33] Upgrading opnsense-update from 23.1.6 to 23.1.8... [16/33] Extracting opnsense-update-23.1.8: .......... done [17/33] Upgrading crowdsec from 1.4.6_2 to 1.5.1... [17/33] Extracting crowdsec-1.5.1: .......... done crowdsec is running as pid 66369. Stopping crowdsec. Waiting for PIDS: 66369. [18/33] Upgrading openvpn from 2.6.3 to 2.6.4... ===> Creating groups. Using existing group 'openvpn'. ===> Creating users Using existing user 'openvpn'. [18/33] Extracting openvpn-2.6.4: .......... done [19/33] Upgrading php81-filter from 8.1.18 to 8.1.19... [19/33] Extracting php81-filter-8.1.19: ......... done [20/33] Upgrading dhcp6c from 20200512_1 to 20230523... [20/33] Extracting dhcp6c-20230523: ........ done [21/33] Upgrading py39-requests from 2.29.0 to 2.30.0... [21/33] Extracting py39-requests-2.30.0: .......... done [22/33] Upgrading php81-ctype from 8.1.18 to 8.1.19... [22/33] Extracting php81-ctype-8.1.19: ........ done [23/33] Upgrading php81-simplexml from 8.1.18 to 8.1.19... [23/33] Extracting php81-simplexml-8.1.19: ......... done [24/33] Upgrading php81-zlib from 8.1.18 to 8.1.19... [24/33] Extracting php81-zlib-8.1.19: ........ done [25/33] Upgrading php81-dom from 8.1.18 to 8.1.19... [25/33] Extracting php81-dom-8.1.19: .......... done [26/33] Upgrading suricata from 6.0.11_1 to 6.0.12... [26/33] Extracting suricata-6.0.12: .......... done [27/33] Upgrading mpd5 from 5.9_14 to 5.9_16... [27/33] Extracting mpd5-5.9_16: .......... done [28/33] Upgrading php81-ldap from 8.1.18 to 8.1.19... [28/33] Extracting php81-ldap-8.1.19: ........ done [29/33] Upgrading php81-xml from 8.1.18 to 8.1.19... [29/33] Extracting php81-xml-8.1.19: ......... done [30/33] Upgrading php81-curl from 8.1.18 to 8.1.19... [30/33] Extracting php81-curl-8.1.19: .......... done [31/33] Upgrading php81-gettext from 8.1.18 to 8.1.19... [31/33] Extracting php81-gettext-8.1.19: ........ done [32/33] Upgrading os-crowdsec from 1.0.4 to 1.0.5... [32/33] Extracting os-crowdsec-1.0.5: .......... done
I checked my crowdsec processes:
Code: [Select]
root@opnsense01:~ # ps aux | grep crowdsec root 43578 0.0 0.0 13504 2768 - I 15:05 0:00.00 /bin/sh -c set -- os-crowdsec-1.0.4\n#!/bin/sh\n\n# need to temporarily stop the bouncer to remove all the rules\nservice crowdsec_firewall stop >/dev/nul root 43757 0.0 0.0 13504 3140 - I 15:05 0:00.01 /bin/sh /usr/local/etc/rc.d/crowdsec_firewall stop root 88726 0.0 0.7 914324 121868 - I 15:04 0:03.12 /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml root 90891 0.0 0.2 722256 28864 - I 15:04 0:00.19 /usr/local/bin/crowdsec-firewall-bouncer -c /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml (crowdsec-firewall-b)
and manually did a 'kill -9 90891' after which the update immediately proceeded and finished successfully.
Title: Re: Update to 23.1.8 got stuck
Post by: franco on May 25, 2023, 03:30:16 pm: Perhaps it would also hang while stopping for reboot. It's pretty difficult to account for that but for best practice restarts of running services are best avoided during updates (and for core we don't do this at all).

Cheers,
Franco
Title: Re: Update to 23.1.8 got stuck
Post by: svheel on May 25, 2023, 06:03:52 pm: I had exactly the same issue with the Crowdsec plugin installed, upgrade got stuck at the same point.
Killing the crowdsec-firewall-bouncer was the solution for me too, thanks for the suggestion!
Title: Re: Update to 23.1.8 got stuck
Post by: cbdudley on May 25, 2023, 11:36:41 pm: Same situation here, the upgrade got stuck at the same point. Killed the firewall-bouncer process and the upgrade completed normally with no further trouble. Thanks to the original poster for the suggestion!
Title: Re: Update to 23.1.8 got stuck
Post by: bucky2780 on May 26, 2023, 01:00:59 am: same thing happened here... killing the crowdsec process worked a treat...
Title: Re: Update to 23.1.8 got stuck
Post by: OzziGoblin on May 26, 2023, 06:54:41 am: Same here, killing Crowdsec continued the update, was a good learning experience as I'm new to OpnSense and Linux ;)
Title: Re: Update to 23.1.8 got stuck
Post by: Taunt9930 on May 26, 2023, 08:23:18 am: Any steps we can take to stop this from happening on upgrade? Would unticking enable in the UI before upgrade work?
Title: Re: Update to 23.1.8 got stuck
Post by: franco on May 26, 2023, 08:28:53 am: My best guess is this has something to do with the crowdsec software update of a previous update. Stopping might be problematic for reboot now as well perhaps and so would stopping from the GUI. Crowdsec people need to take a closer look here.

Cheers,
Franco
Title: Re: Update to 23.1.8 got stuck
Post by: newsense on May 26, 2023, 08:57:22 am: I tried stopping Crowdsec from the GUI first and waited a few minutes. Since the bouncer process wouldn't go away I tried a kill PID, gave it another minute and checked again, then had to go for kill -9
Code: [Select]
root@OPNsense:~ # ps aux |grep 63334 root 56827 0.0 0.0 12648 2116 - I 09:07 0:00.00 pwait 63334 root 63015 0.0 0.0 12756 2232 - Is 08:53 0:00.00 daemon: Crowdsec Firewall[63334] (daemon) root 63334 0.0 0.1 722512 26352 - I 08:53 0:00.16 /usr/local/bin/crowdsec-firewall-bouncer -c /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml (crowdsec-firewall-b) root 49270 0.0 0.0 12648 2116 0 I+ 08:53 0:00.00 pwait 63334 root 44625 0.0 0.0 12748 2360 1 S+ 09:12 0:00.00 grep 63334 root@OPNsense:~ # kill 63334 root@OPNsense:~ # kill 63015 root@OPNsense:~ # ps aux | grep 63334 root 56827 0.0 0.0 12648 2116 - I 09:07 0:00.00 pwait 63334 root 63334 0.0 0.1 722512 26352 - S 08:53 0:00.16 /usr/local/bin/crowdsec-firewall-bouncer -c /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml (crowdsec-firewall-b) root 49270 0.0 0.0 12648 2116 0 I+ 08:53 0:00.00 pwait 63334 root 10006 0.0 0.0 12748 2360 1 S+ 09:13 0:00.00 grep 63334 root@OPNsense:~ # kill -9 63334 root@OPNsense:~ # ps aux | grep 63334 root 85675 0.0 0.0 12748 2360 1 S+ 09:13 0:00.00 grep 63334
On two FWs where Crowdsec was installed but temporarily stopped at the time the upgrade completed without any issue.
Title: Re: Update to 23.1.8 got stuck
Post by: franco on May 26, 2023, 09:30:12 am: https://twitter.com/mmetc2/status/1661986481374625792
Title: Re: Update to 23.1.8 got stuck
Post by: mmetc on May 26, 2023, 09:59:53 am: Hi, thanks for the report

I could not replicate the issue, downgrading to 1.0.3 - 1.4.6 - 0.0.23.rc2, they all updated and didn't require kill or reboot.

Between 0.0.23.rc2 and 0.0.27 the ip removal is a lot (100x) faster and we did a reworking of the concurrency and signal management so I strongly doubt the issue would happen again.

For 0.0.23.rc2 I could update the plugin to do the kill -9, but I thought that the "service" command would already do that.
I suspect the bouncer could be slow while removing banned ips one by one, so it would be harder to replicate on a fast machine or vm. I'll try with 200k+ decisions.
Title: Re: Update to 23.1.8 got stuck
Post by: Waschl on May 26, 2023, 05:48:50 pm: Hi,

had the same issue on 1 of 3 firewall updates. Killing the crowdsec bouncer did the job.

Regards

Michael
Title: Re: Update to 23.1.8 got stuck
Post by: SpinningRust on May 26, 2023, 05:55:25 pm: Exact same issue here, kill -9 of crowdsec-firewall-b resolved the issue for me too. Thank you!
Title: Re: Update to 23.1.8 got stuck
Post by: FredsterNL on May 27, 2023, 03:18:01 am: Just joining the fun: I had the same issue as well :o

-Tried stopping Crowsec to no avail
- SoftReset of my firewall running on a DEC-750 caused it to become even less responsive
- Applied the magic fixes everything solution: pulled the plug ;)

After it was back, the missing components could be replaced (including the CrowdSec component and new kernel)

So, apparently Crowsec could not be stopped, causing the upgrade to habg.

Would be nice if a hang is detected, a KILL -9 could be offered?
Title: Re: Update to 23.1.8 got stuck
Post by: franco on May 28, 2023, 11:15:00 am: Quote from: FredsterNL on May 27, 2023, 03:18:01 am
Would be nice if a hang is detected, a KILL -9 could be offered?

Killing major upgrades in progress as a later consequence? Probably not.

Cheers,
Franco
Title: Re: Update to 23.1.8 got stuck
Post by: nzkiwi68 on May 28, 2023, 10:33:04 pm: I had the same issue, also running CrowdSec.

Stopping CrowdSec from the services page, or inside CrowdSec, or rebooting the firewall didn't work. The firewall not respond to even the reboot command.

Used putty to run SSH session and ran these two commands:

Code: [Select]
pgrep crowdsec pkill -9 crowdsec
Firewall pending reboot then occurred. On restart, re-ran firewall upgrade whoich completed successfully.

Ran firewall health audit to check the upgrade and firewall health - passed.
System > Firmware > Status > Run an Audit - Health
Title: Re: Update to 23.1.8 got stuck
Post by: erlendlk on May 28, 2023, 11:44:09 pm: Pretty much the same story here. Killed the bouncer process and the upgrade finished, seemingly successfully. Also ran the health audit with no problems detected.
Title: Re: Update to 23.1.8 got stuck
Post by: Julien on May 29, 2023, 12:30:14 am: i cannot say i have the same issue.
but it keeps erroring during the update. DNS and everything else is working

i noticed the wireguard has stopped working and works only between 23.1.7_3 and not with 23.1.8

***GOT REQUEST TO CHECK FOR UPDATES***

Currently running OPNsense 23.1.7_3 at Mon May 29 00:00:08 CEST 2023

Fetching changelog information, please wait... fetch: transfer timed out

Updating OPNsense repository catalogue...

Fetching meta.conf: . done

and after changing mirro to AMD

***GOT REQUEST TO CHECK FOR UPDATES***

Currently running OPNsense 23.1.7_3 at Mon May 29 00:00:08 CEST 2023

Fetching changelog information, please wait... fetch: transfer timed out

Updating OPNsense repository catalogue...

Fetching meta.conf: . done

pkg: http://mirror.ams1.nl.leaseweb.net/opnsense/FreeBSD:13:amd64/23.1/latest/packagesite.pkg: Operation timed out

this just happens like this without changing anything.

the only think is the other boxes a re updated to 23.1.8

when i do audit connectivity
its comes back with this error

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 23.1.7_3 at Mon May 29 00:53:18 CEST 2023
Checking connectivity for host: pkg.opnsense.org -> 89.149.222.99
PING 89.149.222.99 (89.149.222.99): 1500 data bytes

--- 89.149.222.99 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv4): https://pkg.opnsense.org/FreeBSD:13:amd64/23.1
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 822 packages processed.
All repositories are up to date.
Checking connectivity for host: pkg.opnsense.org -> 2001:1af8:5300:a010:1::1
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://pkg.opnsense.org/FreeBSD:13:amd64/23.1
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.1/latest/meta.txz: Non-recoverable resolver failure
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.1/latest/packagesite.pkg: Non-recoverable resolver failure
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/23.1/latest/packagesite.txz: Non-recoverable resolver failure
Unable to update repository OPNsense
Error updating repositories!
***DONE***
Title: Re: Update to 23.1.8 got stuck
Post by: cmccallu on June 02, 2023, 06:06:27 am: Seems like 23.1.9 fixed this issue as I was able to successfully upgrade on a environment that was failing and rolled back to the older release!