Dashboard items fail and webui begins to hang

[GuruLee]

I just started to experience issues shortly after upgrading to 24.7.x
Where the dashboard items intermittently do not load and the WebGUI overall hangs until I reboot every few days or so.

I cannot even get firmware status or check for updates when it is in this state, nor can I get past SSH login prompt. As though it does not like my password...

But after forceful reboot, all back to normal for a few days...

I went ahead and forced shutdown of Opnsense (running on Protectli J3710) and then updated to 24.7.10_1

However, the dashboard widget content failures are already starting again and this usually is the preamble to the WebGUI hangs and inability to login to SSH.

Any suggestions?

[GuruLee]

Could these huge firewall alias's be causing it?
https://www.dropbox.com/scl/fi/gfea5wm05wh3di8fp76pc/Screenshot-from-2024-12-04-09-30-19.png?rlkey=oivcadj9k96kosjdpwnr4nzi7&st=vbmxfh1r&dl=0

[meyergru]

I have some aliases with >400000 items in it and that causes no problem. Your CPU is quite old though and with Zenarmor running, it might be overwhelmed...

[GuruLee]

I have some aliases with >400000 items in it and that causes no problem. Your CPU is quite old though and with Zenarmor running, it might be overwhelmed...

You say quite old...I bought it a year ago...it's sufficient and not over utilized.
Its a 1.6GHz with 4 cores....also has 8GB of RAM. Moreover, I see no CPU bottleneck with utilization either.

[GuruLee]

Anyone have any recommendations or solution to my problem?

[GuruLee]

Update:
Approx. 24hrs after upgrading to 24.7.10_1-adm64 the dashboard widgets seem stable and have not failed to load data. Also system overall seems responsive in WebGUI.
I still lost SSH access since upgrading to 24.7.x and I've already re-enabled SSH access and restarted more than once. I get the login prompt, but then it closes out after submitted password...

I do not feel my CPU is causing this:
https://cloud.lcsconsulting.biz/s/YtLg599ZrwmXDPg

Any ideas on how to resolve?

[cookiemonster]

please post in code quotes here what you get on a console when you attempt the connection. Use "ssh -vv" to add verbosity. Should show how far it gets before the ssh connection fails.

[GuruLee]

please post in code quotes here what you get on a console when you attempt the connection. Use "ssh -vv" to add verbosity. Should show how far it gets before the ssh connection fails.

I can login with same account to WebGUI and the account is a member of admins as well...

Code Select Expand


OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 172.16.1.1 is address
debug1: Connecting to 172.16.1.1 [172.16.1.1] port 22.
debug1: Connection established.
debug1: identity file /home/lee/.ssh/id_rsa type -1
debug1: identity file /home/lee/.ssh/id_rsa-cert type -1
debug1: identity file /home/lee/.ssh/id_ecdsa type -1
debug1: identity file /home/lee/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/lee/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/lee/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/lee/.ssh/id_ed25519 type -1
debug1: identity file /home/lee/.ssh/id_ed25519-cert type -1
debug1: identity file /home/lee/.ssh/id_ed25519_sk type -1
debug1: identity file /home/lee/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/lee/.ssh/id_xmss type -1
debug1: identity file /home/lee/.ssh/id_xmss-cert type -1
debug1: identity file /home/lee/.ssh/id_dsa type -1
debug1: identity file /home/lee/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.10
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.9 FreeBSD-openssh-portable-9.9.p1_1,1
debug1: compat_banner: match: OpenSSH_9.9 FreeBSD-openssh-portable-9.9.p1_1,1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 172.16.1.1:22 as 'xxxxxx'
debug1: load_hostkeys: fopen /home/lee/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512,sntrup761x25519-sha512@openssh.com,mlkem768x25519-sha256,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-s,kex-strict-s-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:HgLEf7iqB9gIzUuepvjJkMap3ZzBi3PCsNmvL8Kgon4
debug1: load_hostkeys: fopen /home/lee/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '172.16.1.1' is known and matches the ED25519 host key.
debug1: Found key in /home/lee/.ssh/known_hosts:4
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /home/lee/.ssh/id_rsa
debug1: Will attempt key: /home/lee/.ssh/id_ecdsa
debug1: Will attempt key: /home/lee/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/lee/.ssh/id_ed25519
debug1: Will attempt key: /home/lee/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/lee/.ssh/id_xmss
debug1: Will attempt key: /home/lee/.ssh/id_dsa
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
debug1: kex_input_ext_info: ping@openssh.com (unrecognised)
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /home/lee/.ssh/id_rsa
debug1: Trying private key: /home/lee/.ssh/id_ecdsa
debug1: Trying private key: /home/lee/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/lee/.ssh/id_ed25519
debug1: Trying private key: /home/lee/.ssh/id_ed25519_sk
debug1: Trying private key: /home/lee/.ssh/id_xmss
debug1: Trying private key: /home/lee/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req: entering
debug2: input_userauth_info_req: num_prompts 1
(myusername@172.16.1.1) Password:
debug2: input_userauth_info_req: entering
debug2: input_userauth_info_req: num_prompts 0
Authenticated to 172.16.1.1 ([172.16.1.1]:22) using "keyboard-interactive".
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /home/lee/.ssh/known_hosts for 172.16.1.1 / (none)
debug1: client_input_hostkeys: searching /home/lee/.ssh/known_hosts2 for 172.16.1.1 / (none)
debug1: client_input_hostkeys: hostkeys file /home/lee/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Tue Dec  3 17:54:15 2024 from 192.168.100.90
----------------------------------------------
|      Hello, this is OPNsense 24.7          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:     https://opnsense.org/         |         @@@\\\   ///@@@
| Handbook:    https://docs.opnsense.org/    |       ))))))))   ((((((((
| Forums:      https://forum.opnsense.org/   |         @@@///   \\\@@@
| Code:        https://github.com/opnsense   |        @@@@         @@@@
| Reddit:      https://reddit.com/r/opnsense |         @@@@@@@@@@@@@@@
----------------------------------------------
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: chan_shutdown_read: channel 0: (i0 o1 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug2: channel 0: rcvd close
This account is currently not available.
debug2: channel 0: obuf empty
debug2: chan_shutdown_write: channel 0: (i3 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
Connection to 172.16.1.1 closed.
Transferred: sent 2448, received 3400 bytes, in 0.1 seconds
Bytes per second: sent 31513.8, received 43769.1
debug1: Exit status 1
user@ThinkPad:~$




[meyergru]

The message "This account is currently not available." does usually occur when an account has set /usr/bin/nologin as login shell in /etc/passwd.

[cookiemonster]

Indeed. Fixable by changing it on the UI if available. Otherwise single user mode to change it.

[GuruLee]

The message "This account is currently not available." does usually occur when an account has set /usr/bin/nologin as login shell in /etc/passwd.

I switched it to /bin/csh in the WebGUI and it resolved the SSH issue. Thanks!

[GuruLee]

Indeed. Fixable by changing it on the UI if available. Otherwise single user mode to change it.

Thank you

[GuruLee]

UPDATE: 48 hours after upgrading to 24.7.10_2, the dashboard widgets are still stable and the WebGUI is responsive as normal.

[Red Squirrel]

I get same issue, the widgets all fail one by one within a few seconds of opening home page.  After a while the entire system fails and I have to power cycle it.  Testing it right now but so far I don't feel comfortable deploying this in production. I'm testing something right now, to see if the failure only happen when the webui is left open. Right now it's just sitting at the login screen and I hit refresh once in a while to see if it's still up.   The failures seem to happen overnight, but I'm not positive on an exact time line. I have a continuous ping setup as well, so I might be able to figure it out from there once it stops pinging.

[GuruLee]

Since upgrading to 24.7.10_2 on 12/4/2024, my firewall has been stable with no repeat dashboard widget failures or WebGUI responsiveness issues.

What ver on your on and what do you specs look like, did you check top at CLI?