Topics

1

22.1 Legacy Series / [SOLVED] Update to 22.1.5 didn't finish.

« on: April 10, 2022, 01:32:58 pm »

Hi

I upgraded my local FW to 22.1.5 a couple of days ago and that went fine.  Then I started the same upgrade on my remote FW, and that didn't complete.  It appears to have failed getting base-22.1.5-amd64.txz, and then didn't get the new kernel.  It also hasn't rebooted, yet.  It won't be long before there's a brownout which'll cause a reboot. The end of the update log (full copy attached) is here:

Code: [Select]

.
.
/var/cache/pkg/py38-charset-normalizer-2.0.12.txz
The cleanup will free 59 MiB
Deleting files: .......... done
All done
Nothing to do.
Starting web GUI...done.
Generating RRD graphs...done.
Fetching base-22.1.5-amd64.txz: ..............................pgrep:
Cannot open pidfile `/tmp/opnsense-fetch.pid.8WIINU': No such file or directory
[fetch: transfer timed out] failed, no signature found
***DONE***
How should I go about finishing off this update safely (& remotely) please?

TIA

PeterF

2

21.1 Legacy Series / 21.1.7_1 Upgrade - Fatal error: Uncaught Error:

« on: June 17, 2021, 08:34:39 pm »

Hi

I've just carried out the upgrade from 21.1.6 to 21.1.7_1 on one of my firewalls.

The upgrade log recorded an error which I saw flash past.  I did read up a bit about Phalcon before I started the upgrade.  However, I know nothing about it and hence this error means little to me - but I get the gist of "Fatal error".

The error message, with a bit of context is:

[28/31] Extracting os-nginx-1.23: .......... done
Stopping configd...done
Starting configd.

Fatal error: Uncaught Error: Class 'Phalcon\Validation\Validator' not found in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php:42
Stack trace:
#0 [internal function]: unknown()
#1 [internal function]: Phalcon\Loader->autoLoad('OPNsense\\Base\\V...')
#2 [internal function]: spl_autoload_call('OPNsense\\Base\\V...')
#3 /usr/local/opnsense/mvc/script/run_migrations.php(50): ReflectionClass->__construct('OPNsense\\Base\\V...')
#4 {main}
  thrown in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php on line 42
Keep version OPNsense\Nginx\Nginx (1.20.1)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Nginx: OK
[29/31] Upgrading os-dyndns from 1.24 to 1.24_2...
[29/31] Extracting os-dyndns-1.24_2: .......... done

I didn't get a rebooting notification and initially I thought the system hadn't rebooted.  However, I did get logged out of the console and the system uptime indicates it did a reboot around the time of the upgrade.

If anyone has any ideas of what went wrong  or why, I'd be glad to hear it.  Is it possible that some part of the upgrade process didn't run, or didn't run correctly?  Do I need to do anything to correct this malfunction?

Thanks in advance

PeterF


P.S    In case I've inadvertently omitted some useful information the full log (of the upgrade) is below...

Code: [Select]

***GOT REQUEST TO UPDATE***
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking for upgrades (60 candidates): .......... done
Processing candidates (60 candidates): ..... done
The following 30 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
oniguruma: 6.9.7.1
php73-mbstring: 7.3.28
php73-pecl-psr: 1.1.0
php73-phalcon4: 4.1.2

Installed packages to be UPGRADED:
acme.sh: 2.8.9 -> 2.9.0
curl: 7.76.1 -> 7.77.0
expat: 2.3.0 -> 2.4.1
glib: 2.66.7_1,1 -> 2.66.8,2
isc-dhcp44-relay: 4.4.2_1 -> 4.4.2P1
isc-dhcp44-server: 4.4.2_1 -> 4.4.2P1_1
nspr: 4.30 -> 4.31
nss: 3.65 -> 3.66
openldap-sasl-client: 2.4.58 -> 2.4.59
opnsense: 21.1.6 -> 21.1.7_1
opnsense-lang: 20.1.4 -> 21.1.7
opnsense-update: 21.1.6 -> 21.1.7
os-dyndns: 1.24 -> 1.24_2
os-nginx: 1.22_1 -> 1.23
os-wireguard: 1.6 -> 1.7
pcre2: 10.36 -> 10.37
py37-certifi: 2020.12.5 -> 2021.5.30
py37-chardet: 3.0.4_3,1 -> 4.0.0,1
py37-setuptools: 44.0.0_1 -> 57.0.0
py37-ujson: 3.0.0 -> 4.0.2
py37-yaml: 5.3.1_1 -> 5.4.1
squid: 4.14 -> 4.15
strongswan: 5.9.2_1 -> 5.9.2_2

Installed packages to be REINSTALLED:
os-theme-cicada-1.28 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')
os-theme-rebellion-1.8.7 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')
os-theme-vicuna-1.4 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')

Number of packages to be installed: 4
Number of packages to be upgraded: 23
Number of packages to be reinstalled: 3

The process will require 17 MiB more space.
27 MiB to be downloaded.
[1/30] Fetching strongswan-5.9.2_2.txz: .......... done
[2/30] Fetching squid-4.15.txz: .......... done
[3/30] Fetching py37-yaml-5.4.1.txz: .......... done
[4/30] Fetching py37-ujson-4.0.2.txz: ...... done
[5/30] Fetching py37-setuptools-57.0.0.txz: .......... done
[6/30] Fetching py37-chardet-4.0.0,1.txz: .......... done
[7/30] Fetching py37-certifi-2021.5.30.txz: .......... done
[8/30] Fetching pcre2-10.37.txz: .......... done
[9/30] Fetching os-wireguard-1.7.txz: .. done
[10/30] Fetching os-theme-vicuna-1.4.txz: .......... done
[11/30] Fetching os-theme-rebellion-1.8.7.txz: .......... done
[12/30] Fetching os-theme-cicada-1.28.txz: .......... done
[13/30] Fetching os-nginx-1.23.txz: .......... done
[14/30] Fetching os-dyndns-1.24_2.txz: .... done
[15/30] Fetching opnsense-update-21.1.7.txz: ........ done
[16/30] Fetching opnsense-lang-21.1.7.txz: .......... done
[17/30] Fetching opnsense-21.1.7_1.txz: .......... done
[18/30] Fetching openldap-sasl-client-2.4.59.txz: .......... done
[19/30] Fetching nss-3.66.txz: .......... done
[20/30] Fetching nspr-4.31.txz: .......... done
[21/30] Fetching isc-dhcp44-server-4.4.2P1_1.txz: .......... done
[22/30] Fetching isc-dhcp44-relay-4.4.2P1.txz: .......... done
[23/30] Fetching glib-2.66.8,2.txz: .......... done
[24/30] Fetching expat-2.4.1.txz: .......... done
[25/30] Fetching curl-7.77.0.txz: .......... done
[26/30] Fetching acme.sh-2.9.0.txz: .......... done
[27/30] Fetching php73-phalcon4-4.1.2.txz: .......... done
[28/30] Fetching php73-pecl-psr-1.1.0.txz: .. done
[29/30] Fetching php73-mbstring-7.3.28.txz: .......... done
[30/30] Fetching oniguruma-6.9.7.1.txz: .......... done
Checking integrity... done (1 conflicting)
  - php73-phalcon4-4.1.2 conflicts with php73-phalcon-3.4.5 on /usr/local/etc/php/ext-30-phalcon.ini
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 31 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
php73-phalcon: 3.4.5

New packages to be INSTALLED:
oniguruma: 6.9.7.1
php73-mbstring: 7.3.28
php73-pecl-psr: 1.1.0
php73-phalcon4: 4.1.2

Installed packages to be UPGRADED:
acme.sh: 2.8.9 -> 2.9.0
curl: 7.76.1 -> 7.77.0
expat: 2.3.0 -> 2.4.1
glib: 2.66.7_1,1 -> 2.66.8,2
isc-dhcp44-relay: 4.4.2_1 -> 4.4.2P1
isc-dhcp44-server: 4.4.2_1 -> 4.4.2P1_1
nspr: 4.30 -> 4.31
nss: 3.65 -> 3.66
openldap-sasl-client: 2.4.58 -> 2.4.59
opnsense: 21.1.6 -> 21.1.7_1
opnsense-lang: 20.1.4 -> 21.1.7
opnsense-update: 21.1.6 -> 21.1.7
os-dyndns: 1.24 -> 1.24_2
os-nginx: 1.22_1 -> 1.23
os-wireguard: 1.6 -> 1.7
pcre2: 10.36 -> 10.37
py37-certifi: 2020.12.5 -> 2021.5.30
py37-chardet: 3.0.4_3,1 -> 4.0.0,1
py37-setuptools: 44.0.0_1 -> 57.0.0
py37-ujson: 3.0.0 -> 4.0.2
py37-yaml: 5.3.1_1 -> 5.4.1
squid: 4.14 -> 4.15
strongswan: 5.9.2_1 -> 5.9.2_2

Installed packages to be REINSTALLED:
os-theme-cicada-1.28 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')
os-theme-rebellion-1.8.7 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')
os-theme-vicuna-1.4 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')

Number of packages to be removed: 1
Number of packages to be installed: 4
Number of packages to be upgraded: 23
Number of packages to be reinstalled: 3

The process will require 10 MiB more space.
[1/31] Upgrading py37-setuptools from 44.0.0_1 to 57.0.0...
[1/31] Extracting py37-setuptools-57.0.0: .......... done
[2/31] Upgrading openldap-sasl-client from 2.4.58 to 2.4.59...
[2/31] Extracting openldap-sasl-client-2.4.59: .......... done
[3/31] Upgrading strongswan from 5.9.2_1 to 5.9.2_2...
[3/31] Extracting strongswan-5.9.2_2: .......... done
[4/31] Upgrading squid from 4.14 to 4.15...
===> Creating groups.
Using existing group 'squid'.
===> Creating users
Using existing user 'squid'.
===> Creating homedir(s)
===> Pre-installation configuration for squid-4.15
[4/31] Extracting squid-4.15: .......... done
[5/31] Upgrading py37-ujson from 3.0.0 to 4.0.2...
[5/31] Extracting py37-ujson-4.0.2: ......... done
[6/31] Upgrading opnsense-update from 21.1.6 to 21.1.7...
[6/31] Extracting opnsense-update-21.1.7: .......... done
[7/31] Upgrading opnsense-lang from 20.1.4 to 21.1.7...
[7/31] Extracting opnsense-lang-21.1.7: .......... done
[8/31] Upgrading isc-dhcp44-server from 4.4.2_1 to 4.4.2P1_1...
===> Creating groups.
Using existing group 'dhcpd'.
===> Creating users
Using existing user 'dhcpd'.
[8/31] Extracting isc-dhcp44-server-4.4.2P1_1: .......... done
[9/31] Upgrading isc-dhcp44-relay from 4.4.2_1 to 4.4.2P1...
[9/31] Extracting isc-dhcp44-relay-4.4.2P1: ....... done
[10/31] Deinstalling php73-phalcon-3.4.5...
[10/31] Deleting files for php73-phalcon-3.4.5: ........ done
[11/31] Upgrading nspr from 4.30 to 4.31...
[11/31] Extracting nspr-4.31: .......... done
[12/31] Upgrading curl from 7.76.1 to 7.77.0...
[12/31] Extracting curl-7.77.0: .......... done
[13/31] Installing oniguruma-6.9.7.1...
[13/31] Extracting oniguruma-6.9.7.1: .......... done
[14/31] Installing php73-pecl-psr-1.1.0...
[14/31] Extracting php73-pecl-psr-1.1.0: .......... done
[15/31] Installing php73-mbstring-7.3.28...
[15/31] Extracting php73-mbstring-7.3.28: .......... done
[16/31] Installing php73-phalcon4-4.1.2...
[16/31] Extracting php73-phalcon4-4.1.2: ........ done
[17/31] Upgrading py37-certifi from 2020.12.5 to 2021.5.30...
[17/31] Extracting py37-certifi-2021.5.30: .......... done
[18/31] Upgrading pcre2 from 10.36 to 10.37...
[18/31] Extracting pcre2-10.37: .......... done
[19/31] Upgrading py37-yaml from 5.3.1_1 to 5.4.1...
[19/31] Extracting py37-yaml-5.4.1: .......... done
[20/31] Upgrading py37-chardet from 3.0.4_3,1 to 4.0.0,1...
[20/31] Extracting py37-chardet-4.0.0,1: .......... done
[21/31] Upgrading nss from 3.65 to 3.66...
[21/31] Extracting nss-3.66: .......... done
[22/31] Upgrading glib from 2.66.7_1,1 to 2.66.8,2...
[22/31] Extracting glib-2.66.8,2: .......... done
No schema files found: doing nothing.
[23/31] Upgrading expat from 2.3.0 to 2.4.1...
[23/31] Extracting expat-2.4.1: .......... done
[24/31] Upgrading os-wireguard from 1.6 to 1.7...
[24/31] Extracting os-wireguard-1.7: .......... done
Stopping configd...done
Starting configd.
Keep version OPNsense\Wireguard\General (0.0.1)
Keep version OPNsense\Wireguard\Server (0.0.2)
Keep version OPNsense\Wireguard\Client (0.0.6)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Wireguard: OK
[25/31] Reinstalling os-theme-vicuna-1.4...
[25/31] Extracting os-theme-vicuna-1.4: .......... done
[26/31] Reinstalling os-theme-rebellion-1.8.7...
[26/31] Extracting os-theme-rebellion-1.8.7: .......... done
[27/31] Reinstalling os-theme-cicada-1.28...
[27/31] Extracting os-theme-cicada-1.28: .......... done
[28/31] Upgrading os-nginx from 1.22_1 to 1.23...
[28/31] Extracting os-nginx-1.23: .......... done
Stopping configd...done
Starting configd.

Fatal error: Uncaught Error: Class 'Phalcon\Validation\Validator' not found in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php:42
Stack trace:
#0 [internal function]: unknown()
#1 [internal function]: Phalcon\Loader->autoLoad('OPNsense\\Base\\V...')
#2 [internal function]: spl_autoload_call('OPNsense\\Base\\V...')
#3 /usr/local/opnsense/mvc/script/run_migrations.php(50): ReflectionClass->__construct('OPNsense\\Base\\V...')
#4 {main}
  thrown in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php on line 42
Keep version OPNsense\Nginx\Nginx (1.20.1)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Nginx: OK
[29/31] Upgrading os-dyndns from 1.24 to 1.24_2...
[29/31] Extracting os-dyndns-1.24_2: .......... done
Stopping configd...done
Starting configd.
Reloading plugin configuration
Configuring system logging...done.
[30/31] Upgrading opnsense from 21.1.6 to 21.1.7_1...
[30/31] Extracting opnsense-21.1.7_1: .......... done
Stopping configd...done
Resetting root shell
Updating /etc/shells
Unhooking from /etc/rc
Unhooking from /etc/rc.shutdown
Updating /etc/shells
Registering root shell
Hooking into /etc/rc
Hooking into /etc/rc.shutdown
Starting configd.
>>> Invoking update script 'refresh'
Keep version OPNsense\Monit\Monit (1.0.9)
Keep version OPNsense\Firewall\Alias (1.0.0)
Keep version OPNsense\Firewall\Category (1.0.0)
Keep version OPNsense\OpenVPN\Export (0.0.1)
Keep version OPNsense\CaptivePortal\CaptivePortal (1.0.0)
Keep version OPNsense\Core\Firmware (1.0.0)
Keep version OPNsense\Interfaces\Loopback (1.0.0)
Keep version OPNsense\Interfaces\VxLan (1.0.1)
Keep version OPNsense\Unboundplus\Dnsbl (0.0.1)
Keep version OPNsense\Unboundplus\Miscellaneous (0.0.2)
Keep version OPNsense\Cron\Cron (1.0.2)
Keep version OPNsense\IPsec\IPsec (1.0.0)
Keep version OPNsense\Backup\NextcloudSettings (1.0.0)
Keep version OPNsense\TrafficShaper\TrafficShaper (1.0.3)
Keep version OPNsense\Syslog\Syslog (1.0.0)

3

21.1 Legacy Series / 21.1.3 - Firmware Audit - Health - get "checksum mismatch"

« on: March 12, 2021, 07:07:40 pm »

I updated two firewalls this morning from 21.1.2 to 21.1.3.  One FW in France, one FW in UK.
Both updates appeared to go fine and both systems came back up and got their ipsec VPN connected.

I then ran the audit->health on both systems.
The French system reported everything ok.

The UK system reported checksum errors in 13 files from the python37-3.7.10 package.  The files were:

-rw-r-----  1 root  wheel  3397 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/io.cpython-37.pyc
-rw-r-----  1 root  wheel  29786 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/os.cpython-37.pyc
-rw-r-----  1 root  wheel  10417 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/posixpath.cpython-37.pyc
-rw-r-----  1 root  wheel  16538 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/site.cpython-37.pyc
-rw-r-----  1 root  wheel  4332 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/stat.cpython-37.pyc
-rw-r-----  1 root  wheel  37921 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/threading.cpython-37.pyc
-rw-r-----  1 root  wheel  3587 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/token.cpython-37.pyc
-rw-r-----  1 root  wheel  17819 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/tokenize.cpython-37.pyc
-rw-r-----  1 root  wheel  19610 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/traceback.cpython-37.pyc
-rw-r-----  1 root  wheel  8964 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/types.cpython-37.pyc
-rw-r-----  1 root  wheel  51017 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/typing.cpython-37.pyc
-rw-r-----  1 root  wheel  13824 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/warnings.cpython-37.pyc
-rw-r-----  1 root  wheel  19557 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/weakref.cpython-37.pyc

Note - all files in the same directory, and with the date & time from when I did the update to 21.1.3.

I kept copies of these files, but not the other 513 files in the same directory.

I then reinstalled the python37 package:

python37   3.7.10   111MiB   OPNsense   PSFL   Interpreted object-oriented programming language

see also

https://forum.opnsense.org/index.php?topic=21972.0

I then reran audit->health check again and this time there were no errors.  I got a listing of the files that had problems:

-rw-r--r--  1 root  wheel  3397 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/io.cpython-37.pyc
-rw-r--r--  1 root  wheel  29786 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/os.cpython-37.pyc
-rw-r--r--  1 root  wheel  10417 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/posixpath.cpython-37.pyc
-rw-r--r--  1 root  wheel  16538 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/site.cpython-37.pyc
-rw-r--r--  1 root  wheel  4332 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/stat.cpython-37.pyc
-rw-r--r--  1 root  wheel  37921 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/threading.cpython-37.pyc
-rw-r--r--  1 root  wheel  3587 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/token.cpython-37.pyc
-rw-r--r--  1 root  wheel  17819 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/tokenize.cpython-37.pyc
-rw-r--r--  1 root  wheel  19610 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/traceback.cpython-37.pyc
-rw-r--r--  1 root  wheel  8964 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/types.cpython-37.pyc
-rw-r--r--  1 root  wheel  51017 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/typing.cpython-37.pyc
-rw-r--r--  1 root  wheel  13824 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/warnings.cpython-37.pyc
-rw-r--r--  1 root  wheel  19557 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/weakref.cpython-37.pyc

The file lengths match between these two listings, however, the date stamps differ.  I was puzzled by the differences so decided to look for the "corruptions".  So far, I've analysed only the first file (io.cpython-37.pyc) and it contained 48 corrupted bytes:

!da 5a!1f 1e!1f 1e!da 5a!19 18!1a 19!1b 1a!1c 1b
!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!19 18!1a 19!1b 1a
!1c 1b!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!19 18!1a 19
!1b 1a!1c 1b!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!1e 1d
!da 5a!1c 1b!da 5a!da 5a!1a 19!1d 1c!da 5a!22 20
!23 21!24 22!da 5a!25 23!1f 1e!1f 1e!1f 1e!20 1f

After each "!" there are two bytes.  The first byte is from the "good" file, the second byte is from the corrupted file.  There isn't a consistent corruption between the good byte & the bad.  The majority of errors are the effect of subtracting 1 from the good byte (NB that's not the same as a one-bit error in my book) before storing it.  However, there are a few instances where 2 is subtracted, and a few more where the error is the upper four bits.

I'm not happy about 48 byte errors in a single file, and there's another 12 files to analyse.

I thought about a problem in the package file, so I checked the python37-3.7.10.tgz package file on the mirrors.  There are 24 mirror sites listed.  20 of those would accept my request for a copy of the file.  19 of those files were identical.  The one that differed was from "fourdots.com" and was truncated such that I only got 0.5% of it.  I'm certain that my system didn't install that file because I would have had many more errors, and probably pkg would have barfed, anyway.  So, not a problem with the package.

Franco mentioned "filing system errors" in the reference above.  I would like to pin it down further.
I've wondered about a disk error - it's a small SSD (KINGSTON SMS200S330G mSATA) that's been running for just over 4 years & 4 months.

Does anyone know why the corruped files had today's date, whereas after I had reinstalled the package, the the files had the original dates?

I've just done another check, this time on the French firewall - which didn't produce any errors.  All the files in the relevant directory ( /usr/local/lib/python3.7/__pycache__ ) have exactly the same date+time stamp as the (fresh) UK ones if one compensates for the different timezones.  So I'm beginning to think that the errors happened while the files were being extracted from the package. 

But why?

I'm not expecting definitive answers, but if anyone has any hints or suggestions, I'd be interested to hear them.

Thanks for reading...

PeterF

P.S.  What did we do before the audit->health option appeared???  It certainly proved useful today 

4

18.1 Legacy Series / IPsec (site to site) connection "problems" and an interim solution

« on: May 24, 2018, 04:15:50 pm »

Hi

I have identified a problem with IPsec (site to site) not connecting automatically.

I have two FWs (UK-FW & FR-FW) both running OPNsense 18.1 series:

My tests and investigations described below have been carried out on UK-FW which is up-to-date and has the following overall build:

OPNsense 18.1.8-amd64
FreeBSD 11.1-RELEASE-p10
OpenSSL 1.0.2o 27 Mar 2018

FR-FW also has the same build - i.e. they're both up-to-date at the time of writing.
 
Both firewalls use a domestic ADSL line for their WAN connection. 
I have configured a VPN between the two firewalls. 
Both firewalls use a dynamic dns service to facilitate flexible configuration using FQDN, and not hard-coded IP addresses. 
This VPN uses a PSK and, when it is up, it works well.

The following statement applies to both firewalls.

The ADSL line is connected to a modem which has a PPoE connection to the WAN port on the OPNsense firewall. When the WAN connection comes up, it always has a fresh (& different) public IP address.  This new address is associated quite quickly with its FQDN via the dynamic dns service.

This means that after a reboot of either one or both firewalls, the IPsec tunnel will not be established automatically.  I recognise that the initialisation of the tunnel now occurs at a different time in the boot sequence than when I reported a problem with v17.1.5 (Thanks for fixing that bug  ).  However, I believe that there is a missing component which means that in my case (i.e. neither firewall can have a fixed public IP address) it can never be established automatically. (Please note, there may be something I've missed (in the configuration) and I am open to suggestions and advice  )

Here are my observations and fixes:

• Let's assume that IPsec between UK-FW and FR-FW is working (traffic is flowing).  I will assume in the following scenario that the UK-FW is my local firewall and is described by the details referring to the "Left" side.
  
• This means that the remote firewall (FR-FW) is described by the parameters for the "Right" side.(
• Now consider what happens when FR-FW drops its ADSL connection, and then reacquires it.  After this reconnection, it will have a new public IP address.  FR-FW ensures that this new address is updated to the dynamic dns service.
• When the WAN comes up, FR-FW also appears to update the IPsec configuration file (/usr/local/etc/ipsec.conf) so that its new WAN address is correctly inserted into the configuration lines:
  

Code: [Select]

conn con1
  .
  .
  left = <my WAN ip address>
  leftid = <my WAN ip address>
  .
  .

• The IPsec tunnel will not come up at this point.
• If one looks at the two IPsec configuration files on UK-FW, one will observe discrepancies which will never be detected or fixed.
• In the IPsec configuration file (/usr/local/etc/ipsec.conf), there is a hard-coded IP address for the remote firewall (FR-FW) in the line which starts " rightid = ".  This address is now wrong and there appears to be no automatic fix.
• Also, in the IPsec secrets file (/usr/local/etc/ipsec.secrets), there is also a "stale" (old) IP address associated with the shared key (PSK).

I have identified two ways to bring the IPsec tunnel back up.  They are:

Method 1

• Using the GUI (web console at VPN: IPsec: Tunnel Settings , press the "edit" button for either the phase_1 or phase_2 entry.  Do not change any parameter, but just press the "Save" button.
• When prompted, press the "Apply Changes" button.
• The IPsec tunnel will come up.
• Note:  This action has to be carried out on the system which has not just been assigned a new WAN address.  In the scenario I described above, this would have been carried out on UK-FW

Method 2

• Working on the system which has not just been assigned a new WAN address, one can correct the errors in the two ipsec configuration files.  (This can done manually (with a text editor *1) or scripted (less tedious *2))
• Then one needs to run the commands

Code: [Select]

/usr/local/sbin/ipsec rereadall
/usr/local/sbin/ipsec reload

• The IPsec tunnel will come back up.

Notes *1

• Editing ipsec.conf with a text editor (I used vi) is potentially risky.  It's certainly tedious! 
• Make sure you don't delete or change the two spaces at the beginning of the line.
• Change only the lines with an incorrect IP address; probably the "  leftid = " entry.
• When editing ipsec.secrets, make sure that you change only the IP address at the beginning of the line.
• When you've finished the edit, make sure that the file (ipsec.secrets) is owned by root and has 0600 permissions.

Notes *2

• Having tested my observations and trying a manual fix (see above), I scripted the checking and correction.
• I wrote a shell script (I used bash as I'm more familiar with it) which is defined as a service and can be called from the cron facility (web console at System: Settings: Cron)
• The script is run every minute.
• The script checks IP address for both sides (right and left) in the ipsec.conf file.  It also checks the IP address in the ipsec.secrets file.
• If any error is detected, it makes the necessary correction.
• If, and only if, any change (correction) is made, it then runs the two ipsec commands shown in the code block above.
• My IPsec configurations were simple with only a single Phase_1 and a single Phase_2 defined on each firewall.  Therefore, the script was easy to implement as I didn't have to check (& parse) multiple "conn" sections in ipsec.conf and ipsec.secrets.
• In my experiments and observations, the IPsec tunnel is quickly reestablished.

Results of scripting

• I have installed my script at both ends of my IPsec tunnel - i.e. on UK-FW and FR-FW.
• I can remotely reset the modem on either firewall. The script brings the tunnel back up successfully in each of the three possible scenarios:
  • Modem on UK-FW is reset, so UK-FW gets a new public IP address
  • Modem on FR-FW is reset, so FR-FW gets a new public IP address
  • Both modems are reset at the same time.
  • The IPsec tunnel comes back up within 2 or 3 minutes of the modem being reset. (Remember that the modem has to negotiate a working connection together with PPoE authentication, then the dynamic dns upgrade has to go through and finally remember that the "repair" script runs only once per minute as a cron job.)
• I can remotely reboot either firewall.  The script brings the tunnel back up successfully in each of the three possible scenarios:
  • UK-FW is rebooted, so UK-FW gets a new public IP address
  • FR-FW is rebooted, so FR-FW gets a new public IP address
  • Both firewalls are rebooted at the same time.
  • The IPsec tunnel comes back up within 3 or 4 minutes of the firewall being rebooted.

Feature Request / Bug fix
It would be preferable (particularly for those users with more complex IPsec configurations) if the monitoring of the remote end of each VPN connection could be handled within OPNsense so that the IPsec configuration files could be kept up-to-date (automagically) and this would then mean that IPsec would then have a default "Up" status, rather than a default "Down" status which is what I observe with a stock OPNsense installation at present.

Alternative approach
A long time ago I learned that hard-coding IP addresses into configurations was best avoided if at all possible. So I experimented with /usr/local/etc/inc/plugins.inc.d/ipsec.inc (which appears to be the php code that generates the IPsec configuration files from the user-input data.  I found that the "rightid" parameter has to be a hard-coded IP address.  IPsec will not work with a FQDN here.  Similarly, I couldn't get the secrets file to work with either a FQDN or the special parameter "%any".  My reading of the man page suggested that the "%any" parameter should have worked here, but it's less desirable, probably, for security reasons.

Thanks for reading this long post. 
I hope that it might help someone else solve a problem, and better still prompt a fix within either OPNsense or ipsec.

Peter

5

17.1 Legacy Series / 17.1.5 - Problem with IPsec after a REBOOT

« on: April 26, 2017, 03:58:42 pm »

Hi

I have identified a problem with IPsec after a REBOOT

I have two FWs (UK-FW & FR-FW) both running OPNsense 17.1 series:

My tests and investigations described below have been carried out on UK-FW which is up-to-date and has the following overall build:

OPNsense 17.1.5-amd64
FreeBSD 11.0-RELEASE-p8
OpenSSL 1.0.2k 26 Jan 2017
 
Both firewalls use a domestic ADSL line for their WAN connection. 
I have configured a VPN between the two firewalls. 
Both firewalls use a dynamic dns service to facilitate flexible configuration using FQDN, and not hard-coded IP addresses. 
This VPN uses a PSK, and it works well.

However, when either firewall is rebooted, IPsec reports that

Code: [Select]

charon: 12[IKE] no shared key found for 'aa.bbb.ccc.248' –
and the VPN does not connect.

Now looking in detail at UK-FW since this is running the latest OPNsense build.

The report that there is no shared key turns out to be correct, since although the IPsec configuration is safely stored in /conf/config.xml, the file /usr/local/etc/ipsec.secrets (SECRETS) is empty (it exists, but has zero length).

I can also report that the file /usr/local/etc/ipsec.conf (CONF) is incomplete.  In my case (on UK-FW), the line

Code: [Select]

  rightid = aa.bbb.ccc.ddd
is missing.

The SECRETS and CONF files are produced by function ipsec_configure() in file /usr/local/etc/inc/ipsec.inc.  Correct production of these two files requires a working WAN (public DNS is required).

At boot-up, a start-up script (/usr/local/etc/rc.bootup) is invoked and this calls function ipsec_configure(). 

However, in my two firewalls, the execution of this script (rc.bootup) happens too soon, and before the WAN has come up and is usable.

Using the script /usr/local/etc/rc.bootup as a basis, I carried out some severe pruning to generate a new script which I called rc.ipsec.  This script is shown here:

Code: [Select]

#!/usr/local/bin/php

<?php/*    Copyright (C) 2014-2016 Franco Fichtner <franco@opnsense.org>    Copyright (C) 2004-2009 Scott Ullrich <sullrich@pfsense.org>.    Copyright (C) 2003-2004 Manuel Kasper <mk@neon1.net>.    Copyright (C) 2009 Erik Kristensen    All rights reserved.    Redistribution and use in source and binary forms, with or without    modification, are permitted provided that the following conditions are met:    1. Redistributions of source code must retain the above copyright notice,       this list of conditions and the following disclaimer.    2. Redistributions in binary form must reproduce the above copyright       notice, this list of conditions and the following disclaimer in the       documentation and/or other materials provided with the distribution.    THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY    AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE    AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,    OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF    SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS    INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN    CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)    ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE    POSSIBILITY OF SUCH DAMAGE.*//* looks weird, but means we started PHP successfully */echo "done.\n";echo "Initializing...";flush();$inc_files = array(    'config.inc',    'util.inc',    'interfaces.inc',    'services.inc',);foreach ($inc_files as $inc_file) {    require_once $inc_file;    echo '.';    flush();}echo "done.\n";/* start IPsec tunnels */$ipsec_dynamic_hosts = ipsec_configure(true);/* If there are ipsec dynamic hosts try again to reload the tunnels as rc.newipsecdns does */if ($ipsec_dynamic_hosts) {    ipsec_configure(true);}exit(0);

Running this script (manually) after the system has been rebooted, and when the WAN is up, configures IPsec correctly, and the VPN then comes up very quickly.

The configuration of IPsec definitely needs to take place later in the start-up sequence, but I don’t have enough knowledge of the overall architecture of the start-up process to make a recommendation as to how it should be amended.

Any suggestions please?

Thanks,

Peter