OPNsense Forum

Archive => 21.1 Legacy Series => Topic started by: gu6884 on March 30, 2021, 10:35:22 pm

Title: [SOLVED] webgui broken after upgrade to 21.1.4
Post by: gu6884 on March 30, 2021, 10:35:22 pm
Hi,

since I've updated, no webgui anymore. Using self-signed certificate
Chrome says "ERR_SSL_PROTOCOL_ERROR"
Tried with Safari as well, but no luck.

Note that with curl it works
Code: [Select]
curl -k https://10.100.1.1

when I try to display the webgui in a browser I get this in /var/log/lighttpd.log
Code: [Select]
Mar 30 22:23:35 router lighttpd[26522]: (mod_openssl.c.3042) SSL: 5 error:0201502D:system library:ioctl:Operation not supported
Mar 30 22:23:35 router lighttpd[26522]: (mod_openssl.c.3042) SSL: 5 error:0201502D:system library:ioctl:Operation not supported
Mar 30 22:23:35 router lighttpd[26522]: (mod_openssl.c.3042) SSL: 5 error:1427D044:SSL routines:construct_stateless_ticket:internal error
Mar 30 22:23:35 router lighttpd[26522]: (mod_openssl.c.3042) SSL: 5 error:0201502D:system library:ioctl:Operation not supported
Mar 30 22:23:35 router lighttpd[26522]: (mod_openssl.c.3059) SSL: -1 5 45 Operation not supported

Checked different threads already. Running this did not help
Code: [Select]
configctl webgui restart renew
Did the system check in the console too, but nothing reported
Code: [Select]
Enter an option: 12

Fetching change log information, please wait... done

This will automatically fetch all available updates and apply them.

Proceed with this action? [y/N]: h

>>> Check installed kernel version
Version 21.1.4 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 21.1.4 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" has 67 dependencies to check.
Checking packages: ..................................................................... done


Any idea what else to do?
Thanks
Title: Re: webgui broken after upgrade to 21.1.4
Post by: kursu on March 31, 2021, 12:29:53 am
I have the same issue,  webui did hang after wireguard removal message in update log
Title: Re: webgui broken after upgrade to 21.1.4
Post by: cranky on March 31, 2021, 10:00:22 am
Can you try to reconfigure the lan and wan via console? I think it asks you to regenerate a cert when you do,maybe that will help?
I personally didn't have any issues, but I'm not using wire guard.
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Weff on March 31, 2021, 11:36:20 am
I had the same kind of issue after upgrade in version 21.1.3
I guess this is due to https hardening.

I rolled back to my last snaptop and define by default the https certificate generated by Opnsense.
You can also try to activate temporary the http mode, regenerate your certificates.

Cheers
Title: Re: webgui broken after upgrade to 21.1.4
Post by: kursu on March 31, 2021, 02:06:52 pm
Factory reset fixed the issue for me.
Title: Re: webgui broken after upgrade to 21.1.4
Post by: karlson2k on March 31, 2021, 03:35:21 pm
Same here. I'm using my self-signed local CA and local certificates.
After upgrade to 21.1.4 completely lost access to Web UI.

Chrome:
Code: [Select]
ERR_SSL_PROTOCOL_ERROR
Firefox:
Just does not load the page

The command
Code: [Select]
configctl webgui restart renew just makes Chrome to warn me about new certificate and then again the same error.

curl -vk https://10.51.51.1:
Code: [Select]
*   Trying 10.51.51.1:443...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.51.51.1 (10.51.51.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1881 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [520 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=opnrouter.intdomain.local; C=NL; ST=Zuid-Holland; L=Middelharnis; O=OPNsense self-signed web certificate
*  start date: Mar 31 12:40:52 2021 GMT
*  expire date: May  2 12:40:52 2022 GMT
*  issuer: CN=opnrouter.intdomain.local; C=NL; ST=Zuid-Holland; L=Middelharnis; O=OPNsense self-signed web certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5571c874bea0)
} [5 bytes data]
> GET / HTTP/2
> Host: 10.51.51.1
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS alert, internal error (592):
{ [2 bytes data]
* OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error, errno 0
* Failed receiving HTTP2 data
* OpenSSL SSL_write: SSL_ERROR_ZERO_RETURN, errno 0
* Failed sending HTTP2 data
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host 10.51.51.1 left intact
curl: (56) OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error, errno 0

ssh and console works, I can see mentioned errors in /var/log/lighttpd.log
Code: [Select]
Mar 31 15:51:03 opnrouter lighttpd[19466]: (mod_openssl.c.3042) SSL: 5 error:0201502D:system library:ioctl:Operation not supported
Mar 31 15:51:03 opnrouter lighttpd[19466]: (mod_openssl.c.3042) SSL: 5 error:0201502D:system library:ioctl:Operation not supported
Mar 31 15:51:03 opnrouter lighttpd[19466]: (mod_openssl.c.3042) SSL: 5 error:1427D044:SSL routines:construct_stateless_ticket:internal error
Mar 31 15:51:03 opnrouter lighttpd[19466]: (mod_openssl.c.3042) SSL: 5 error:0201502D:system library:ioctl:Operation not supported
Mar 31 15:51:03 opnrouter lighttpd[19466]: (mod_openssl.c.3059) SSL: -1 5 45 Operation not supported

Additional (related?) repeated errors in console and in /var/log/system.log:
Code: [Select]
Mar 31 16:02:36 opnrouter kernel: Deprecated code (to be removed in FreeBSD 13): DES cipher via /dev/crypto
Mar 31 16:02:36 opnrouter kernel: Deprecated code (to be removed in FreeBSD 13): 3DES cipher via /dev/crypto
Mar 31 16:02:36 opnrouter kernel: Deprecated code (to be removed in FreeBSD 13): Blowfish cipher via /dev/crypto
Mar 31 16:02:36 opnrouter kernel: Deprecated code (to be removed in FreeBSD 13): CAST128 cipher via /dev/crypto
Mar 31 16:02:36 opnrouter kernel: Deprecated code (to be removed in FreeBSD 13): ARC4 cipher via /dev/crypto
Title: Re: webgui broken after upgrade to 21.1.4
Post by: karlson2k on March 31, 2021, 03:52:40 pm
I've tried the next command as workaround:
Code: [Select]
opnsense-revert -r 21.1.3 openssland it brings back Web UI.

Errors in /var/log/system.log and /var/log/lighttpd.log went away.

But it's clearly a workaround only.
Title: Re: webgui broken after upgrade to 21.1.4
Post by: no_Legend on March 31, 2021, 04:35:04 pm
Same problem in here.
WebGui broken

Cheers Robert
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Fright on March 31, 2021, 05:06:13 pm
if you don't need to use /dev/crypto you can try to delete
<cryptodev_enable> string in config.xml and restart opn
Title: Re: webgui broken after upgrade to 21.1.4
Post by: karlson2k on March 31, 2021, 06:13:04 pm
if you don't need to use /dev/crypto you can try to delete
<cryptodev_enable> string in config.xml and restart opn

I (temporary) solved the problem by
Code: [Select]
opnsense-revert -r 21.1.3 openssl
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Darkopnsense on March 31, 2021, 07:57:03 pm
Hello,

Here we are this morning, update 21.1.4 brought me into the circle of certificate issues.

NO IT DIDN'T AS SIMPLE AS AFIRMED BY FRANCO.
We would not be several to be in the galley.

Until now I knew that there had been an update because I no longer had Internet access. Restarting my device fixed the problem and I noticed that there had been an update.

After reading the threads and trying to resolve in SSH mode, which I gleaned, I am unsuccessful.

# curl -k https://192.168.66.66:48443
empty reply from server
curl: (56) OpenSSL SSL_read: error: 14094438: SSL routines: ssl3_read_bytes: tlsv1 alert internal error, errno 0
# configctl webgui restart renew
okay

Browser //192.168.66.66:48443
ERR_SSL_PROTOCOL_ERROR

I put a back-up machine back into service

Assigning LAN and WAN via the console does not change anything.

Magnificent simplicity.

Regards,
French mother tongue
Title: Re: webgui broken after upgrade to 21.1.4
Post by: franco on March 31, 2021, 08:44:33 pm
> NO IT DIDN'T AS SIMPLE AS AFIRMED BY FRANCO.

So maybe it's a different issue? Let's settle down a bit. The workaround is out there:

# opnsense-revert -r 21.1.3 openssl


Cheers,
Franco
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Fright on April 01, 2021, 10:12:21 am
still think can be related:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254643

may be due to the KTLS for freebsd was merged from master? 1.1.1 does not contain KTLS
Title: Re: webgui broken after upgrade to 21.1.4
Post by: sToRmInG on April 01, 2021, 10:34:14 am
I had similar issues already with 21.1.3 and they are still present in 21.1.4.

The behavior is always the same. After I reboot the OPNsense the Web UI initially works but will eventually stop including unbound.

Luckily SSH is still working and the interfaces are reachable via IP. After I restart all services through the console everything is working as expected once again.
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Fright on April 01, 2021, 10:35:21 am
@sToRmInG
not the same issue if gui restart helps
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Darkopnsense on April 01, 2021, 11:56:59 am
Hello,

Thank you FRANCO however that does not solve the problem.

# opnsense-revert -r 21.1.3 openssl

Browsers //192.168.66.66:48443
ERR_SSL_PROTOCOL_ERROR

Even with
# configctl webgui restart renew

On the other hand if it can help
# curl -k https://192.168.66.66:48443
curl: (35) error: 1408F10B: SSL routines: ssl3_get_record: wrong version number

Regards,
French mother tongue
Title: Re: webgui broken after upgrade to 21.1.4
Post by: pkernstock on April 02, 2021, 01:10:05 am
Hello,

I stumbled about the exact same issue when updating from 21.1.3 to 21.1.4 just a few minutes ago. Self-signed certificates (from the system, nothing customized), no LetsEncrypt, neither reboots nor manual webui restarts changed the situation.

Quote
$ curl -k https://fw.domain.tld/
curl: (56) OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error, errno 0

The workaround as posted earlier works fine:

Quote
root@fw:/var/log # opnsense-revert -r 21.1.3 openssl
Fetching openssl.txz: .... done
Verifying signature with trusted certificate pkg.opnsense.org.20210104... done
openssl-1.1.1k,1: already unlocked
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        openssl: 1.1.1j_1,1

Number of packages to be installed: 1

The process will require 14 MiB more space.
[1/1] Installing openssl-1.1.1j_1,1...
Extracting openssl-1.1.1j_1,1: 100%
root@fw:/var/log # configctl webgui restart
OK
root@fw:/var/log #

Now it works:
Quote
$ curl -k https://fw.domain.tld/
<!doctype html>
[...]

Regards,
Patrik
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Darkopnsense on April 02, 2021, 02:47:06 pm
Hi @pkernstock,

Thank you and I want to believe that it works for sure.

I followed the instructions well

root@Pare-Feu:/home/henri # opnsense-revert -r 21.1.3 openssl
Fetching openssl.txz: ...... done
Verifying signature with trusted certificate pkg.opnsense.org.20210104... done
openssl-1.1.1j_1,1: already unlocked
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
SunnyValley repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        openssl: 1.1.1j_1,1 [unknown-repository]

Number of packages to be installed: 1

The process will require 14 MiB more space.
[1/1] Installing openssl-1.1.1j_1,1...
Extracting openssl-1.1.1j_1,1: 100%
root@Pare-Feu:/home/henri # configctl webgui restart
OK

browsers FAILED
brave : ERR_SSL_PROTOCOL_ERROR
chrome : ERR_SSL_PROTOCOL_ERROR
edge : ERR_SSL_PROTOCOL_ERROR
firefox : SSL_ERROR_RX_RECORD_TOO_LONG
opera : ERR_SSL_PROTOCOL_ERROR
vivaldi : ERR_SSL_PROTOCOL_ERROR

I restarted but ditto

Regards,
French mother tongue
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Fright on April 02, 2021, 04:58:15 pm
@Darkopnsense
Quote
firefox : SSL_ERROR_RX_RECORD_TOO_LONG
Quote
curl: (35) error: 1408F10B: SSL routines: ssl3_get_record: wrong version number
imho there are some problems besides the discussed
can you try with curl -vk?
any clue in /var/log/lighttpd.log?
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Darkopnsense on April 02, 2021, 05:48:05 pm
Hi @fright,

root@Pare-Feu:/home/henri # opnsense-revert -r 21.1.3 openssl
Fetching openssl.txz: ...... done
Verifying signature with trusted certificate pkg.opnsense.org.20210104... done
openssl-1.1.1j_1,1: already unlocked
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
SunnyValley repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        openssl: 1.1.1j_1,1 [unknown-repository]

Number of packages to be installed: 1

The process will require 14 MiB more space.
[1/1] Installing openssl-1.1.1j_1,1...
Extracting openssl-1.1.1j_1,1: 100%
root@Pare-Feu:/home/Stephane # configctl webgui restart
OK
root@Pare-Feu:/home/henri # curl -vk https://192.168.66.66:48443
*   Trying 192.168.66.66:48443...
* Connected to 192.168.66.66 (192.168.66.66) port 48443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /usr/local/etc/ssl/cert.pem
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

I analyze the file lighttpd.log

Regards,
French mother tongue
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Darkopnsense on April 02, 2021, 07:33:25 pm
Good evening everyone,

After analyzing lighttpd.log, I have reset SENSEI and I am currently accessing the interface with different browsers. And this even after restarting, to be sure.

Regards,
mother tongue French
Title: Re: webgui broken after upgrade to 21.1.4
Post by: franco on April 02, 2021, 07:37:57 pm
This is quite the spectacular breakage somewhere up the food chain:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254643


Cheers,
Franco
Title: Re: webgui broken after upgrade to 21.1.4
Post by: franco on April 02, 2021, 07:39:42 pm
PS: does this work too?

# devfs rule apply path crypto hide
# configctl webgui restart
Title: Re: webgui broken after upgrade to 21.1.4
Post by: pkernstock on April 02, 2021, 08:33:11 pm
PS: does this work too?

# devfs rule apply path crypto hide
# configctl webgui restart

Yes, it does:

Quote
root@iefw01:/var/log # opnsense-revert -r 21.1.4 openssl
Fetching openssl.txz: ... done
Verifying signature with trusted certificate pkg.opnsense.org.20210104... done
openssl-1.1.1j_1,1: already unlocked
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        openssl: 1.1.1k,1

Number of packages to be installed: 1

The process will require 14 MiB more space.
[1/1] Installing openssl-1.1.1k,1...
Extracting openssl-1.1.1k,1: 100%
root@iefw01:/var/log # configctl webgui restart
OK
root@iefw01:/var/log # devfs rule apply path crypto hide
root@iefw01:/var/log # configctl webgui restart
OK
root@iefw01:/var/log #

Then:
Quote
$ curl -k https://fw/ | head -n1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2952  100  2952    0     0  38337      0 --:--:-- --:--:-- --:--:-- 38337
<!doctype html>
Title: Re: webgui broken after upgrade to 21.1.4
Post by: Fright on April 02, 2021, 08:50:21 pm
@franco
https://forum.opnsense.org/index.php?topic=22374.msg106411#msg106411
may be related to https://github.com/HardenedBSD/hardenedBSD/commit/aa906e2a4957db700d9e6cc60857e1afe1aecc85#diff-47dbd1172e2a29406be580d23e7933f2dce7cc5de53773849815e37376fa1743 ?

@Darkopnsense
Congrats )
Title: Re: webgui broken after upgrade to 21.1.4
Post by: j_s on April 05, 2021, 09:18:07 am
Welp,  I'm in the same boat.  WebGUI is unavailable after upgrade.

Instead of rolling back the one package as mentioned here, I rolled back the ZFS boot environment. (I always create a new ZFS boot environment before upgrading as a 'just in case' situation)

I'm curious to know what the actual problem is, and I'll wait until the issue is fixed in an official release, then I'll try upgrading again.

Title: Re: webgui broken after upgrade to 21.1.4
Post by: karlson2k on April 05, 2021, 01:19:06 pm
I've tried "crypto hide" solution and it works for me.
As my openssl package was already reverted, I did
Code: [Select]
# devfs rule apply path crypto hide
# opnsense-revert openssl
# configctl webgui restart
WebGUI is still available.


So the simplest workaround for now is:
Code: [Select]
# devfs rule apply path crypto hide
# configctl webgui restart
These two commands should restore the WebGUI access.
Title: Re: webgui broken after upgrade to 21.1.4
Post by: franco on April 05, 2021, 01:41:21 pm
Thanks all so far. The following package should work:

# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/21.1/misc/openssl-1.1.1k,1.txz

We will do a hotfix, but not today as there is nobody in the office to verify the build. So that will probably be tomorrow.

What this means is that /dev/crypto OpenSSL engine support is going to be disabled due to broken patches added in 1.1.1k. I'll leave you to look into who and why...

LibreSSL removed /dev/crytpo support a long time ago, but we still have System: Settings: Miscellaneous
"Use /dev/crypto" non-default settings which broke this for involved users. We ask you to switch this option off now as it is likely being removed from 21.7 to avoid further problems.


Cheers,
Franco
Title: Re: webgui broken after upgrade to 21.1.4
Post by: karlson2k on April 05, 2021, 02:11:14 pm
Thanks all so far. The following package should work:

# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/21.1/misc/openssl-1.1.1k,1.txz
Thanks!

Tested by
Code: [Select]
# devfs rule apply path crypto unhide
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/21.1/misc/openssl-1.1.1k,1.txz
# configctl webgui restart

WebGUI is available!
Title: Re: webgui broken after upgrade to 21.1.4
Post by: franco on April 06, 2021, 11:02:39 am
Thanks a lot for testing @karlson2k

Hotfix went out this morning, update should show up for anyone who still is on the original 21.1.4. Mostly this is to avoid users from below 21.1.4 to trip over the same thing.


Cheers,
Franco
Title: Re: webgui broken after upgrade to 21.1.4
Post by: franco on April 06, 2021, 11:04:15 am
PS: Reverting will now give you the correct OpenSSL binary even when 21.1.4 shows no GUI.

# opnsense-revert -r 21.1.4 openssl
# configctl webgui restart