Messages

Hi Franco

Try to poke the update again. If it isn't a persistent problem it should work on a subsequent try.. make sure to account for the reboot that's going to happen once the base and kernel are being picked up.

I tried the update again and as you anticipated (I think) it said there were updates for the base and kernel packages.  These appeared to get installed ok but with no status ouput to the log window.  The system rebooted ok.

After it had rebooted, I did a set of audit tests:
CONNECTIVITY Audit was ok for IPV4 (IPV6 is not available from my ISP)
Then I checked for updates, and it reported all packages were up to date
However, the HEALTH Audit produced some very strange results (file attached) with the final package check reporting

Code Select Expand

.
.
Checking packages: .
zip-3.0_1 has no upstream equivalent
***DONE***for every single package (67).
I also noticed that the list of plugins was truncated to only those that I'd installed.
I checked for updates again, the plugin list was fully populated, and a subsequent HEALTH audit then looked normal.
I'm a bit puzzled by the "failed" HEALTH audit, but everything appears to be working fine.

Thanks for your help

PeterF

[both]

Hi

I thought I'd just tell you that I've been running a VPN between two instances of OPNsense successfully for some years.  Both sites use domestic-grade ADSL and cope with dynamic IP address allocation.  Moreover, I've used IPSEC to provide the VPN as a bridge between the two sites.

IPSEC is more challenging, because the configuration files that OPNsense generates ( /usr/local/etc/ipsec.conf and /usr/local/etc/ipsec.secrets ) both contain hard-coded ip addresses of both sites.  I've got round this by running a check script (every minute) to check VPN status, and the config files.  All discrepancies are fixed with sed and ipsec gets nudged to reload &/or reconnect.

I run the same script at both ends and it's been very successful.
If you want any more detail, then DM me.

PeterF

Hi

I upgraded my local FW to 22.1.5 a couple of days ago and that went fine.  Then I started the same upgrade on my remote FW, and that didn't complete.  It appears to have failed getting base-22.1.5-amd64.txz, and then didn't get the new kernel.  It also hasn't rebooted, yet.  It won't be long before there's a brownout which'll cause a reboot. The end of the update log (full copy attached) is here:

Code Select Expand

.
.
/var/cache/pkg/py38-charset-normalizer-2.0.12.txz
The cleanup will free 59 MiB
Deleting files: .......... done
All done
Nothing to do.
Starting web GUI...done.
Generating RRD graphs...done.
Fetching base-22.1.5-amd64.txz: ..............................pgrep:
Cannot open pidfile `/tmp/opnsense-fetch.pid.8WIINU': No such file or directory
[fetch: transfer timed out] failed, no signature found
***DONE***

How should I go about finishing off this update safely (& remotely) please?

TIA

PeterF

Hi

I experienced very similar problems to that of the OP.  I got my web-console text output back by restarting my browser (Firefox 89.0.1 on Win10).  I did this with the button on the about:profiles page (Firefox only).  I did nothing to clear the cache, or the cookies.

HTH

PeterF

Hi Franco

This is still a bit odd: opnsense-21.1.7_1: missing file /usr/local/www/diag_backup.php

Sorry about that.  I was probably in the process of reinstating the original version when the system checked that directory.  I edit this file so that the default is to backup RRD data (not skip, as supplied). Clearly I was in the process of restoring it (delete mine, rename backup) when the system passed by.  I hadn't spotted it.  Sorry.

As for your shell processes this is normal if you have the console menu unlocked and you probably have a VGA console?

Yes, I do have a VGA console which is working.  For a long while my console didn't work, so I didn't see the shell processes.  I rebuilt the system a few weeks ago (with a vga console), and hadn't noticed the shell processes.

Having more confidence in this update now, I've now updated my remote firewall.  The only "odd" sympton it shows is that if I do a Audit->Health I do not get any output on the web console. (One of my reported omissions when I updated my UK firewall - but that one works as expected now).  The output file in /tmp is generated ok.  However, although my browser was not showing any signs of misbehaving with the web-console of the UK firewall, I though I ought to restart it to see if there was any impact.  I did.  That fixed the lack of screen output for Audit->Health. 

Is this more evidence for Issue 5061, perhaps?

Thanks agian for youur help,

PeterF

[NO]

Hi

Good news to report and I'll keep it short.
It turns out that I can detect NO problem in my web console now.  All the odd artefacts that I reported have been fixed, by restarting my browser (Firefox 89.0.1).

I apologise for not thinking of this before.

Thanks for the help and all the good work that is OPNsense...

PeterF

Something is definitely broken in my web console...

I've attempted to run a health audit (System ->Firmware ->Run an audit ->Health) and all I get is a spinner (in the "Updates" tab, and no output in the screen area below.

However, the health audit is successfully carried out and the text output is generated in "/tmp/pkg_upgrade.progress" which is the output file specified in

/usr/local/opnsense/scripts/firmware/health.sh

The health audit hasn't helped though, because it was unable to find a problem:

Code Select Expand

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 21.1.7_1 (amd64/OpenSSL) at Fri Jun 18 17:32:55 BST 2021
>>> Check installed kernel version
Version 21.1.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 21.1.7 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: ....
opnsense-21.1.7_1: missing file /usr/local/www/diag_backup.php
Checking all packages......... done
>>> Check for core packages consistency
Core package "opnsense" has 67 dependencies to check.
Checking packages: ..................................................................... done
***DONE***

I suspect that this is also related to the fact that I get no traffic graphs.  Something definitely got broken during the update, or by the update error.

PeterF

[System Log - after reboot]

Hi

Thanks for your reply.  Whilst I understand your logic in saying that the error is benign, I can still see some unexplained problems with the system.  These symptons persist after a reboot.

1  Unexplained shells running, started at boot.  If I open a shell (text console, option 8 ) and see who is logged in, I can see another 8 shells running:

Code Select Expand

Last login: Fri Jun 18 10:26:30 2021 from 192.168.110.70
----------------------------------------------
|      Hello, this is OPNsense 21.1          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:      https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:     https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:       https://forum.opnsense.org/  |         @@@///   \\\@@@
| Code:         https://github.com/opnsense  |        @@@@         @@@@
| Twitter:      https://twitter.com/opnsense |         @@@@@@@@@@@@@@@
----------------------------------------------

*** xxxxx.xxxxxxx.xxx: OPNsense 21.1.7_1 (amd64/OpenSSL) ***

LAN (igb2)      -> v4: 192.168.110.100/24
WAN (pppoe0)    -> v4/PPPoE: 84.68.8.135/32
WiFi (igb3)     -> v4: 10.10.10.10/24

HTTPS: SHA256 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
               00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SSH:   SHA256 0000000000000000000000000000/00000000000000 (ECDSA)
SSH:   SHA256 0000000000000000000000000000000000000000000 (ED25519)
SSH:   SHA256 0000000000000000000000000000000000000000000 (RSA)

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 8

root@xxxxx:~ # w
11:12AM  up 53 mins, 9 users, load averages: 0.54, 0.58, 0.50
USER       TTY      FROM                  LOGIN@  IDLE WHAT
root       v5       -                    10:20AM    51 /bin/sh /usr/local/sbin/opnsense-shell
root       v3       -                    10:20AM    51 /bin/sh /usr/local/sbin/opnsense-shell
root       v7       -                    10:20AM    51 /bin/sh /usr/local/sbin/opnsense-shell
root       v6       -                    10:20AM    51 /bin/sh /usr/local/sbin/opnsense-shell
root       v1       -                    10:20AM    51 /bin/sh /usr/local/sbin/opnsense-shell
root       v4       -                    10:20AM    51 /bin/sh /usr/local/sbin/opnsense-shell
root       v2       -                    10:20AM    51 /bin/sh /usr/local/sbin/opnsense-shell
root       v0       -                    10:20AM    51 /bin/sh /usr/local/sbin/opnsense-shell
root       pts/0    picasso.xxx.xxxxxxx  11:12AM     - w
root@stone:~ #

These shells are started at boot time.

2  I also have no real-time traffic graphs in the web console.  If I go to

Code Select Expand

REPORTING -> TRAFFIC I get a screen with the tabs, but nothing else displayed.  There are no axes.  The data select button at the top right says "Nothing Selected" and the drop down button does not work.

Looking for other symptons...

The system log file doesn't seem to display anything untoward after the most recent reboot (see below for a sanitised extract).

Any ideas what starts these shells and why?  Is it time for a clean install?  (I last did a clean install 3 months ago - I don't want to make a habit of it!)

Thanks in advance...

PeterF



System Log - after reboot

Code Select Expand

Jun 18 10:18:50 myopn.mydomain.com syslog-ng[6873]: syslog-ng shutting down; version='3.32.1'
Jun 18 10:20:04 myopn.mydomain.com syslog-ng[22255]: syslog-ng starting up; version='3.32.1'
Jun 18 10:20:04 myopn.mydomain.com opnsense[87833]: plugins_configure loopback_prepare (1)
Jun 18 10:20:04 myopn.mydomain.com opnsense[87833]: plugins_configure loopback_prepare (execute task : loopback_configure_interface(1))
Jun 18 10:20:04 myopn.mydomain.com opnsense[87833]: plugins_configure openvpn_prepare (1)
Jun 18 10:20:04 myopn.mydomain.com opnsense[87833]: plugins_configure openvpn_prepare (execute task : openvpn_prepare(1))
Jun 18 10:20:05 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: The command '/usr/sbin/ngctl msg 'igb1': setautosrc 1' returned exit code '71', the output was 'ngctl: send msg: No such file or directory'
Jun 18 10:20:06 myopn.mydomain.com opnsense[87833]: plugins_configure ipsec_prepare (1)
Jun 18 10:20:06 myopn.mydomain.com opnsense[87833]: plugins_configure ipsec_prepare (execute task : ipsec_configure_vti(1))
Jun 18 10:20:06 myopn.mydomain.com opnsense[87833]: plugins_configure vxlan_prepare (1)
Jun 18 10:20:06 myopn.mydomain.com opnsense[87833]: plugins_configure vxlan_prepare (execute task : vxlan_configure_interface(1))
Jun 18 10:20:12 myopn.mydomain.com opnsense[30434]: /usr/local/etc/rc.newwanip: IP renewal deferred during boot on 'pppoe0'
Jun 18 10:20:15 myopn.mydomain.com opnsense[87833]: plugins_configure earlybootup (1)
Jun 18 10:20:15 myopn.mydomain.com opnsense[87833]: plugins_configure earlybootup (execute task : openssh_configure_do(1))
Jun 18 10:20:15 myopn.mydomain.com opnsense[87833]: plugins_configure earlybootup (execute task : webgui_configure_do(1))
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: ROUTING: entering configure using defaults
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: ROUTING: IPv4 default gateway set to wan
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: ROUTING: setting IPv4 default route to 212.1.1.10
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: ROUTING: keeping current default gateway '212.1.1.10'
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure hosts ()
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure hosts (execute task : dnsmasq_hosts_generate())
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure hosts (execute task : unbound_hosts_generate())
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure dhcp (1)
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))
Jun 18 10:20:16 myopn.mydomain.com dhcpleases[90497]: Could not deliver signal HUP to process because its pidfile does not exist, No such file or directory.
Jun 18 10:20:16 myopn.mydomain.com dhcpleases[90497]: Could not deliver signal HUP to process because its pidfile does not exist, No such file or directory.
Jun 18 10:20:16 myopn.mydomain.com dhcpleases[90497]: Could not deliver signal HUP to process because its pidfile does not exist, No such file or directory.
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on igb2
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure dhcrelay (1)
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure dhcrelay (execute task : dhcpd_dhcrelay_configure(1))
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure dns (1)
Jun 18 10:20:16 myopn.mydomain.com opnsense[87833]: plugins_configure dns (execute task : dnsmasq_configure_do(1))
Jun 18 10:20:17 myopn.mydomain.com opnsense[87833]: plugins_configure dns (execute task : unbound_configure_do(1))
Jun 18 10:20:17 myopn.mydomain.com opnsense[87833]: plugins_configure monitor (1)
Jun 18 10:20:17 myopn.mydomain.com opnsense[87833]: plugins_configure monitor (execute task : dpinger_configure_do(1))
Jun 18 10:20:17 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: The WAN_PPPOE monitor address is empty, skipping.
Jun 18 10:20:24 myopn.mydomain.com opnsense[87833]: plugins_configure vpn (1)
Jun 18 10:20:24 myopn.mydomain.com opnsense[87833]: plugins_configure vpn (execute task : ipsec_configure_do(1))
Jun 18 10:20:25 myopn.mydomain.com opnsense[87833]: plugins_configure vpn (execute task : openvpn_configure_do(1))
Jun 18 10:20:25 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Resyncing OpenVPN instances.
Jun 18 10:20:25 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: OpenVPN server 2 instance started on PID 8408.
Jun 18 10:20:25 myopn.mydomain.com opnsense[87833]: plugins_configure bootup (1)
Jun 18 10:20:25 myopn.mydomain.com opnsense[87833]: plugins_configure bootup (execute task : dyndns_configure_do(1))
Jun 18 10:20:25 myopn.mydomain.com opnsense[62974]: /usr/local/etc/rc.newwanip: IP renewal deferred during boot on 'ovpns2'
Jun 18 10:20:26 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS: updating cache file /var/cache/dyndns_wan_myopn.duckdns.org_0.cache: 84.1.1.1
Jun 18 10:20:26 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS: (Success) IP Address Updated Successfully!
Jun 18 10:20:28 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS: updatedns() starting
Jun 18 10:20:28 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS (myopn.mydomain.com): 84.1.1.1 extracted
Jun 18 10:20:28 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS (myopn.mydomain.com): running dyndns_failover_interface for wan. found pppoe0
Jun 18 10:20:28 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS (myopn.mydomain.com via Custom): _update() starting.
Jun 18 10:20:29 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS (myopn.mydomain.com): _checkStatus() starting.
Jun 18 10:20:29 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS (myopn.mydomain.com): Current Service: custom
Jun 18 10:20:29 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS (myopn.mydomain.com): 84.1.1.1 extracted
Jun 18 10:20:29 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS: updating cache file /var/cache/dyndns_wan_myopn.mydomain.com_2.cache: 84.1.1.1
Jun 18 10:20:29 myopn.mydomain.com opnsense[87833]: /usr/local/etc/rc.bootup: Dynamic DNS: (Success) IP Address Updated Successfully!
Jun 18 10:20:30 myopn.mydomain.com opnsense[87833]: plugins_configure bootup (execute task : ntpd_configure_defer(1))
Jun 18 10:20:30 myopn.mydomain.com opnsense[87833]: plugins_configure bootup (execute task : opendns_configure_do(1))
Jun 18 10:20:30 myopn.mydomain.com opnsense[87833]: plugins_configure bootup (execute task : unbound_configure_do(1))
Jun 18 10:20:32 myopn.mydomain.com syslog-ng[22255]: syslog-ng shutting down; version='3.32.1'
Jun 18 10:20:32 myopn.mydomain.com syslog-ng[56285]: syslog-ng starting up; version='3.32.1'
Jun 18 10:20:32 myopn.mydomain.com opnsense[74970]: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'ovpns2'
Jun 18 10:20:32 myopn.mydomain.com opnsense[74970]: /usr/local/etc/rc.newwanip: Interface '' is disabled or empty, nothing to do.
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'pppoe0'
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: On (IP address: 84.1.1.1) (interface: WAN[wan]) (real interface: pppoe0).
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: plugins_configure hosts ()
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: plugins_configure hosts (execute task : dnsmasq_hosts_generate())
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: plugins_configure hosts (execute task : unbound_hosts_generate())
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 212.1.1.10
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '212.1.1.10'
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: plugins_configure monitor ()
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: plugins_configure monitor (execute task : dpinger_configure_do())
Jun 18 10:20:33 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: The WAN_PPPOE monitor address is empty, skipping.
Jun 18 10:20:35 myopn.mydomain.com opnsense[50433]: plugins_configure vpn (,wan)
Jun 18 10:20:35 myopn.mydomain.com opnsense[50433]: plugins_configure vpn (execute task : ipsec_configure_do(,wan))
Jun 18 10:20:35 myopn.mydomain.com opnsense[50433]: plugins_configure vpn (execute task : openvpn_configure_do(,wan))
Jun 18 10:20:35 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jun 18 10:20:38 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: OpenVPN server 2 instance started on PID 65729.
Jun 18 10:20:38 myopn.mydomain.com opnsense[64320]: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'ovpns2'
Jun 18 10:20:38 myopn.mydomain.com opnsense[64320]: /usr/local/etc/rc.newwanip: Interface '' is disabled or empty, nothing to do.
Jun 18 10:20:40 myopn.mydomain.com opnsense[50433]: plugins_configure newwanip (,wan)
Jun 18 10:20:40 myopn.mydomain.com opnsense[50433]: plugins_configure newwanip (execute task : dyndns_configure_do(,wan))
Jun 18 10:20:41 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS: updating cache file /var/cache/dyndns_wan_myopn.duckdns.org_0.cache: 84.1.1.1
Jun 18 10:20:41 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS: (Success) IP Address Updated Successfully!
Jun 18 10:20:43 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS: updatedns() starting
Jun 18 10:20:43 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS (myopn.mydomain.com): 84.1.1.1 extracted
Jun 18 10:20:43 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS (myopn.mydomain.com): running dyndns_failover_interface for wan. found pppoe0
Jun 18 10:20:43 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS (myopn.mydomain.com via Custom): _update() starting.
Jun 18 10:20:44 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS (myopn.mydomain.com): _checkStatus() starting.
Jun 18 10:20:44 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS (myopn.mydomain.com): Current Service: custom
Jun 18 10:20:44 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS (myopn.mydomain.com): 84.1.1.1 extracted
Jun 18 10:20:44 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS: updating cache file /var/cache/dyndns_wan_myopn.mydomain.com_2.cache: 84.1.1.1
Jun 18 10:20:44 myopn.mydomain.com opnsense[50433]: /usr/local/etc/rc.newwanip: Dynamic DNS: (Success) IP Address Updated Successfully!
Jun 18 10:20:45 myopn.mydomain.com opnsense[50433]: plugins_configure newwanip (execute task : ntpd_configure_defer())
Jun 18 10:20:45 myopn.mydomain.com opnsense[50433]: plugins_configure newwanip (execute task : opendns_configure_do())
Jun 18 10:20:45 myopn.mydomain.com opnsense[50433]: plugins_configure newwanip (execute task : openssh_configure_do(,wan))
Jun 18 10:20:45 myopn.mydomain.com opnsense[50433]: plugins_configure newwanip (execute task : unbound_configure_do(,wan))
Jun 18 10:20:45 myopn.mydomain.com opnsense[50433]: plugins_configure newwanip (execute task : vxlan_configure_interface())
Jun 18 10:20:45 myopn.mydomain.com opnsense[50433]: plugins_configure newwanip (execute task : webgui_configure_do(,wan))
Jun 18 10:20:45 myopn.mydomain.com opnsense[96774]: /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
Jun 18 10:20:45 myopn.mydomain.com opnsense[96774]: /usr/local/etc/rc.routing_configure: ROUTING: IPv4 default gateway set to wan
Jun 18 10:20:45 myopn.mydomain.com opnsense[96774]: /usr/local/etc/rc.routing_configure: ROUTING: setting IPv4 default route to 212.1.1.10
Jun 18 10:20:45 myopn.mydomain.com opnsense[96774]: /usr/local/etc/rc.routing_configure: ROUTING: keeping current default gateway '212.1.1.10'
Jun 18 10:20:45 myopn.mydomain.com opnsense[96774]: plugins_configure monitor (1)
Jun 18 10:20:45 myopn.mydomain.com opnsense[96774]: plugins_configure monitor (execute task : dpinger_configure_do(1))
Jun 18 10:20:45 myopn.mydomain.com opnsense[96774]: /usr/local/etc/rc.routing_configure: The WAN_PPPOE monitor address is empty, skipping.
Jun 18 10:20:52 myopn.mydomain.com syslog-ng[56285]: syslog-ng shutting down; version='3.32.1'
Jun 18 10:20:52 myopn.mydomain.com syslog-ng[76590]: syslog-ng starting up; version='3.32.1'
Jun 18 10:20:52 myopn.mydomain.com opnsense[23999]: /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
Jun 18 10:20:52 myopn.mydomain.com opnsense[23999]: /usr/local/etc/rc.routing_configure: ROUTING: IPv4 default gateway set to wan
Jun 18 10:20:52 myopn.mydomain.com opnsense[23999]: /usr/local/etc/rc.routing_configure: ROUTING: setting IPv4 default route to 212.1.1.10
Jun 18 10:20:52 myopn.mydomain.com opnsense[23999]: /usr/local/etc/rc.routing_configure: ROUTING: keeping current default gateway '212.1.1.10'
Jun 18 10:20:52 myopn.mydomain.com opnsense[23999]: plugins_configure monitor (1)
Jun 18 10:20:52 myopn.mydomain.com opnsense[23999]: plugins_configure monitor (execute task : dpinger_configure_do(1))
Jun 18 10:20:53 myopn.mydomain.com opnsense[23999]: /usr/local/etc/rc.routing_configure: The WAN_PPPOE monitor address is empty, skipping.
Jun 18 10:20:53 myopn.mydomain.com /flowd_aggregate.py[67878]: startup, check database.
Jun 18 10:21:45 myopn.mydomain.com /flowd_aggregate.py[67878]: start watching flowd
Jun 18 10:21:55 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum src_addr_details_086400.sqlite
Jun 18 10:22:12 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum src_addr_000300.sqlite
Jun 18 10:22:12 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum src_addr_003600.sqlite
Jun 18 10:22:12 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum src_addr_086400.sqlite
Jun 18 10:22:15 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum dst_port_000300.sqlite
Jun 18 10:22:15 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum dst_port_003600.sqlite
Jun 18 10:22:15 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum dst_port_086400.sqlite
Jun 18 10:22:18 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum interface_000030.sqlite
Jun 18 10:22:18 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum interface_000300.sqlite
Jun 18 10:22:18 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum interface_003600.sqlite
Jun 18 10:22:18 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum interface_086400.sqlite
Jun 18 10:22:19 myopn.mydomain.com /flowd_aggregate.py[67878]: vacuum done


[Fatal error: Uncaught Error: Class 'Phalcon\Validation\Validator' not found in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php:42Stack trace:#0 [internal function]: unknown()#1 [internal function]: Phalcon\Loader->autoLoad('OPNsense\\Base\\V...')#2 [internal function]: spl_autoload_call('OPNsense\\Base\\V...')#3 /usr/local/opnsense/mvc/script/run_migrations.php(50): ReflectionClass->__construct('OPNsense\\Base\\V...')#4 {main} thrown in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php on line 42]

Hi

I've just carried out the upgrade from 21.1.6 to 21.1.7_1 on one of my firewalls.

The upgrade log recorded an error which I saw flash past.  I did read up a bit about Phalcon before I started the upgrade.  However, I know nothing about it and hence this error means little to me - but I get the gist of "Fatal error".

The error message, with a bit of context is:

[28/31] Extracting os-nginx-1.23: .......... done
Stopping configd...done
Starting configd.

Fatal error: Uncaught Error: Class 'Phalcon\Validation\Validator' not found in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php:42
Stack trace:
#0 [internal function]: unknown()
#1 [internal function]: Phalcon\Loader->autoLoad('OPNsense\\Base\\V...')
#2 [internal function]: spl_autoload_call('OPNsense\\Base\\V...')
#3 /usr/local/opnsense/mvc/script/run_migrations.php(50): ReflectionClass->__construct('OPNsense\\Base\\V...')
#4 {main}
  thrown in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php on line 42
Keep version OPNsense\Nginx\Nginx (1.20.1)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Nginx: OK
[29/31] Upgrading os-dyndns from 1.24 to 1.24_2...
[29/31] Extracting os-dyndns-1.24_2: .......... done

I didn't get a rebooting notification and initially I thought the system hadn't rebooted.  However, I did get logged out of the console and the system uptime indicates it did a reboot around the time of the upgrade.

If anyone has any ideas of what went wrong  or why, I'd be glad to hear it.  Is it possible that some part of the upgrade process didn't run, or didn't run correctly?  Do I need to do anything to correct this malfunction?

Thanks in advance

PeterF


P.S    In case I've inadvertently omitted some useful information the full log (of the upgrade) is below...

Code Select Expand

***GOT REQUEST TO UPDATE***
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking for upgrades (60 candidates): .......... done
Processing candidates (60 candidates): ..... done
The following 30 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
oniguruma: 6.9.7.1
php73-mbstring: 7.3.28
php73-pecl-psr: 1.1.0
php73-phalcon4: 4.1.2

Installed packages to be UPGRADED:
acme.sh: 2.8.9 -> 2.9.0
curl: 7.76.1 -> 7.77.0
expat: 2.3.0 -> 2.4.1
glib: 2.66.7_1,1 -> 2.66.8,2
isc-dhcp44-relay: 4.4.2_1 -> 4.4.2P1
isc-dhcp44-server: 4.4.2_1 -> 4.4.2P1_1
nspr: 4.30 -> 4.31
nss: 3.65 -> 3.66
openldap-sasl-client: 2.4.58 -> 2.4.59
opnsense: 21.1.6 -> 21.1.7_1
opnsense-lang: 20.1.4 -> 21.1.7
opnsense-update: 21.1.6 -> 21.1.7
os-dyndns: 1.24 -> 1.24_2
os-nginx: 1.22_1 -> 1.23
os-wireguard: 1.6 -> 1.7
pcre2: 10.36 -> 10.37
py37-certifi: 2020.12.5 -> 2021.5.30
py37-chardet: 3.0.4_3,1 -> 4.0.0,1
py37-setuptools: 44.0.0_1 -> 57.0.0
py37-ujson: 3.0.0 -> 4.0.2
py37-yaml: 5.3.1_1 -> 5.4.1
squid: 4.14 -> 4.15
strongswan: 5.9.2_1 -> 5.9.2_2

Installed packages to be REINSTALLED:
os-theme-cicada-1.28 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')
os-theme-rebellion-1.8.7 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')
os-theme-vicuna-1.4 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')

Number of packages to be installed: 4
Number of packages to be upgraded: 23
Number of packages to be reinstalled: 3

The process will require 17 MiB more space.
27 MiB to be downloaded.
[1/30] Fetching strongswan-5.9.2_2.txz: .......... done
[2/30] Fetching squid-4.15.txz: .......... done
[3/30] Fetching py37-yaml-5.4.1.txz: .......... done
[4/30] Fetching py37-ujson-4.0.2.txz: ...... done
[5/30] Fetching py37-setuptools-57.0.0.txz: .......... done
[6/30] Fetching py37-chardet-4.0.0,1.txz: .......... done
[7/30] Fetching py37-certifi-2021.5.30.txz: .......... done
[8/30] Fetching pcre2-10.37.txz: .......... done
[9/30] Fetching os-wireguard-1.7.txz: .. done
[10/30] Fetching os-theme-vicuna-1.4.txz: .......... done
[11/30] Fetching os-theme-rebellion-1.8.7.txz: .......... done
[12/30] Fetching os-theme-cicada-1.28.txz: .......... done
[13/30] Fetching os-nginx-1.23.txz: .......... done
[14/30] Fetching os-dyndns-1.24_2.txz: .... done
[15/30] Fetching opnsense-update-21.1.7.txz: ........ done
[16/30] Fetching opnsense-lang-21.1.7.txz: .......... done
[17/30] Fetching opnsense-21.1.7_1.txz: .......... done
[18/30] Fetching openldap-sasl-client-2.4.59.txz: .......... done
[19/30] Fetching nss-3.66.txz: .......... done
[20/30] Fetching nspr-4.31.txz: .......... done
[21/30] Fetching isc-dhcp44-server-4.4.2P1_1.txz: .......... done
[22/30] Fetching isc-dhcp44-relay-4.4.2P1.txz: .......... done
[23/30] Fetching glib-2.66.8,2.txz: .......... done
[24/30] Fetching expat-2.4.1.txz: .......... done
[25/30] Fetching curl-7.77.0.txz: .......... done
[26/30] Fetching acme.sh-2.9.0.txz: .......... done
[27/30] Fetching php73-phalcon4-4.1.2.txz: .......... done
[28/30] Fetching php73-pecl-psr-1.1.0.txz: .. done
[29/30] Fetching php73-mbstring-7.3.28.txz: .......... done
[30/30] Fetching oniguruma-6.9.7.1.txz: .......... done
Checking integrity... done (1 conflicting)
  - php73-phalcon4-4.1.2 conflicts with php73-phalcon-3.4.5 on /usr/local/etc/php/ext-30-phalcon.ini
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 31 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
php73-phalcon: 3.4.5

New packages to be INSTALLED:
oniguruma: 6.9.7.1
php73-mbstring: 7.3.28
php73-pecl-psr: 1.1.0
php73-phalcon4: 4.1.2

Installed packages to be UPGRADED:
acme.sh: 2.8.9 -> 2.9.0
curl: 7.76.1 -> 7.77.0
expat: 2.3.0 -> 2.4.1
glib: 2.66.7_1,1 -> 2.66.8,2
isc-dhcp44-relay: 4.4.2_1 -> 4.4.2P1
isc-dhcp44-server: 4.4.2_1 -> 4.4.2P1_1
nspr: 4.30 -> 4.31
nss: 3.65 -> 3.66
openldap-sasl-client: 2.4.58 -> 2.4.59
opnsense: 21.1.6 -> 21.1.7_1
opnsense-lang: 20.1.4 -> 21.1.7
opnsense-update: 21.1.6 -> 21.1.7
os-dyndns: 1.24 -> 1.24_2
os-nginx: 1.22_1 -> 1.23
os-wireguard: 1.6 -> 1.7
pcre2: 10.36 -> 10.37
py37-certifi: 2020.12.5 -> 2021.5.30
py37-chardet: 3.0.4_3,1 -> 4.0.0,1
py37-setuptools: 44.0.0_1 -> 57.0.0
py37-ujson: 3.0.0 -> 4.0.2
py37-yaml: 5.3.1_1 -> 5.4.1
squid: 4.14 -> 4.15
strongswan: 5.9.2_1 -> 5.9.2_2

Installed packages to be REINSTALLED:
os-theme-cicada-1.28 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')
os-theme-rebellion-1.8.7 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')
os-theme-vicuna-1.4 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:*:*')

Number of packages to be removed: 1
Number of packages to be installed: 4
Number of packages to be upgraded: 23
Number of packages to be reinstalled: 3

The process will require 10 MiB more space.
[1/31] Upgrading py37-setuptools from 44.0.0_1 to 57.0.0...
[1/31] Extracting py37-setuptools-57.0.0: .......... done
[2/31] Upgrading openldap-sasl-client from 2.4.58 to 2.4.59...
[2/31] Extracting openldap-sasl-client-2.4.59: .......... done
[3/31] Upgrading strongswan from 5.9.2_1 to 5.9.2_2...
[3/31] Extracting strongswan-5.9.2_2: .......... done
[4/31] Upgrading squid from 4.14 to 4.15...
===> Creating groups.
Using existing group 'squid'.
===> Creating users
Using existing user 'squid'.
===> Creating homedir(s)
===> Pre-installation configuration for squid-4.15
[4/31] Extracting squid-4.15: .......... done
[5/31] Upgrading py37-ujson from 3.0.0 to 4.0.2...
[5/31] Extracting py37-ujson-4.0.2: ......... done
[6/31] Upgrading opnsense-update from 21.1.6 to 21.1.7...
[6/31] Extracting opnsense-update-21.1.7: .......... done
[7/31] Upgrading opnsense-lang from 20.1.4 to 21.1.7...
[7/31] Extracting opnsense-lang-21.1.7: .......... done
[8/31] Upgrading isc-dhcp44-server from 4.4.2_1 to 4.4.2P1_1...
===> Creating groups.
Using existing group 'dhcpd'.
===> Creating users
Using existing user 'dhcpd'.
[8/31] Extracting isc-dhcp44-server-4.4.2P1_1: .......... done
[9/31] Upgrading isc-dhcp44-relay from 4.4.2_1 to 4.4.2P1...
[9/31] Extracting isc-dhcp44-relay-4.4.2P1: ....... done
[10/31] Deinstalling php73-phalcon-3.4.5...
[10/31] Deleting files for php73-phalcon-3.4.5: ........ done
[11/31] Upgrading nspr from 4.30 to 4.31...
[11/31] Extracting nspr-4.31: .......... done
[12/31] Upgrading curl from 7.76.1 to 7.77.0...
[12/31] Extracting curl-7.77.0: .......... done
[13/31] Installing oniguruma-6.9.7.1...
[13/31] Extracting oniguruma-6.9.7.1: .......... done
[14/31] Installing php73-pecl-psr-1.1.0...
[14/31] Extracting php73-pecl-psr-1.1.0: .......... done
[15/31] Installing php73-mbstring-7.3.28...
[15/31] Extracting php73-mbstring-7.3.28: .......... done
[16/31] Installing php73-phalcon4-4.1.2...
[16/31] Extracting php73-phalcon4-4.1.2: ........ done
[17/31] Upgrading py37-certifi from 2020.12.5 to 2021.5.30...
[17/31] Extracting py37-certifi-2021.5.30: .......... done
[18/31] Upgrading pcre2 from 10.36 to 10.37...
[18/31] Extracting pcre2-10.37: .......... done
[19/31] Upgrading py37-yaml from 5.3.1_1 to 5.4.1...
[19/31] Extracting py37-yaml-5.4.1: .......... done
[20/31] Upgrading py37-chardet from 3.0.4_3,1 to 4.0.0,1...
[20/31] Extracting py37-chardet-4.0.0,1: .......... done
[21/31] Upgrading nss from 3.65 to 3.66...
[21/31] Extracting nss-3.66: .......... done
[22/31] Upgrading glib from 2.66.7_1,1 to 2.66.8,2...
[22/31] Extracting glib-2.66.8,2: .......... done
No schema files found: doing nothing.
[23/31] Upgrading expat from 2.3.0 to 2.4.1...
[23/31] Extracting expat-2.4.1: .......... done
[24/31] Upgrading os-wireguard from 1.6 to 1.7...
[24/31] Extracting os-wireguard-1.7: .......... done
Stopping configd...done
Starting configd.
Keep version OPNsense\Wireguard\General (0.0.1)
Keep version OPNsense\Wireguard\Server (0.0.2)
Keep version OPNsense\Wireguard\Client (0.0.6)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Wireguard: OK
[25/31] Reinstalling os-theme-vicuna-1.4...
[25/31] Extracting os-theme-vicuna-1.4: .......... done
[26/31] Reinstalling os-theme-rebellion-1.8.7...
[26/31] Extracting os-theme-rebellion-1.8.7: .......... done
[27/31] Reinstalling os-theme-cicada-1.28...
[27/31] Extracting os-theme-cicada-1.28: .......... done
[28/31] Upgrading os-nginx from 1.22_1 to 1.23...
[28/31] Extracting os-nginx-1.23: .......... done
Stopping configd...done
Starting configd.

Fatal error: Uncaught Error: Class 'Phalcon\Validation\Validator' not found in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php:42
Stack trace:
#0 [internal function]: unknown()
#1 [internal function]: Phalcon\Loader->autoLoad('OPNsense\\Base\\V...')
#2 [internal function]: spl_autoload_call('OPNsense\\Base\\V...')
#3 /usr/local/opnsense/mvc/script/run_migrations.php(50): ReflectionClass->__construct('OPNsense\\Base\\V...')
#4 {main}
  thrown in /usr/local/opnsense/mvc/app/models/OPNsense/Base/Validators/CallbackValidator.php on line 42
Keep version OPNsense\Nginx\Nginx (1.20.1)
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Nginx: OK
[29/31] Upgrading os-dyndns from 1.24 to 1.24_2...
[29/31] Extracting os-dyndns-1.24_2: .......... done
Stopping configd...done
Starting configd.
Reloading plugin configuration
Configuring system logging...done.
[30/31] Upgrading opnsense from 21.1.6 to 21.1.7_1...
[30/31] Extracting opnsense-21.1.7_1: .......... done
Stopping configd...done
Resetting root shell
Updating /etc/shells
Unhooking from /etc/rc
Unhooking from /etc/rc.shutdown
Updating /etc/shells
Registering root shell
Hooking into /etc/rc
Hooking into /etc/rc.shutdown
Starting configd.
>>> Invoking update script 'refresh'
Keep version OPNsense\Monit\Monit (1.0.9)
Keep version OPNsense\Firewall\Alias (1.0.0)
Keep version OPNsense\Firewall\Category (1.0.0)
Keep version OPNsense\OpenVPN\Export (0.0.1)
Keep version OPNsense\CaptivePortal\CaptivePortal (1.0.0)
Keep version OPNsense\Core\Firmware (1.0.0)
Keep version OPNsense\Interfaces\Loopback (1.0.0)
Keep version OPNsense\Interfaces\VxLan (1.0.1)
Keep version OPNsense\Unboundplus\Dnsbl (0.0.1)
Keep version OPNsense\Unboundplus\Miscellaneous (0.0.2)
Keep version OPNsense\Cron\Cron (1.0.2)
Keep version OPNsense\IPsec\IPsec (1.0.0)
Keep version OPNsense\Backup\NextcloudSettings (1.0.0)
Keep version OPNsense\TrafficShaper\TrafficShaper (1.0.3)
Keep version OPNsense\Syslog\Syslog (1.0.0)

I updated two firewalls this morning from 21.1.2 to 21.1.3.  One FW in France, one FW in UK.
Both updates appeared to go fine and both systems came back up and got their ipsec VPN connected.

I then ran the audit->health on both systems.
The French system reported everything ok.

The UK system reported checksum errors in 13 files from the python37-3.7.10 package.  The files were:

-rw-r-----  1 root  wheel  3397 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/io.cpython-37.pyc
-rw-r-----  1 root  wheel  29786 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/os.cpython-37.pyc
-rw-r-----  1 root  wheel  10417 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/posixpath.cpython-37.pyc
-rw-r-----  1 root  wheel  16538 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/site.cpython-37.pyc
-rw-r-----  1 root  wheel  4332 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/stat.cpython-37.pyc
-rw-r-----  1 root  wheel  37921 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/threading.cpython-37.pyc
-rw-r-----  1 root  wheel  3587 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/token.cpython-37.pyc
-rw-r-----  1 root  wheel  17819 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/tokenize.cpython-37.pyc
-rw-r-----  1 root  wheel  19610 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/traceback.cpython-37.pyc
-rw-r-----  1 root  wheel  8964 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/types.cpython-37.pyc
-rw-r-----  1 root  wheel  51017 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/typing.cpython-37.pyc
-rw-r-----  1 root  wheel  13824 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/warnings.cpython-37.pyc
-rw-r-----  1 root  wheel  19557 Mar 12 11:25 /usr/local/lib/python3.7/__pycache__/weakref.cpython-37.pyc

Note - all files in the same directory, and with the date & time from when I did the update to 21.1.3.

I kept copies of these files, but not the other 513 files in the same directory.

I then reinstalled the python37 package:

python37   3.7.10   111MiB   OPNsense   PSFL   Interpreted object-oriented programming language

see also

https://forum.opnsense.org/index.php?topic=21972.0

I then reran audit->health check again and this time there were no errors.  I got a listing of the files that had problems:

-rw-r--r--  1 root  wheel  3397 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/io.cpython-37.pyc
-rw-r--r--  1 root  wheel  29786 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/os.cpython-37.pyc
-rw-r--r--  1 root  wheel  10417 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/posixpath.cpython-37.pyc
-rw-r--r--  1 root  wheel  16538 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/site.cpython-37.pyc
-rw-r--r--  1 root  wheel  4332 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/stat.cpython-37.pyc
-rw-r--r--  1 root  wheel  37921 Mar  8 23:55 /usr/local/lib/python3.7/__pycache__/threading.cpython-37.pyc
-rw-r--r--  1 root  wheel  3587 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/token.cpython-37.pyc
-rw-r--r--  1 root  wheel  17819 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/tokenize.cpython-37.pyc
-rw-r--r--  1 root  wheel  19610 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/traceback.cpython-37.pyc
-rw-r--r--  1 root  wheel  8964 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/types.cpython-37.pyc
-rw-r--r--  1 root  wheel  51017 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/typing.cpython-37.pyc
-rw-r--r--  1 root  wheel  13824 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/warnings.cpython-37.pyc
-rw-r--r--  1 root  wheel  19557 Mar  8 23:56 /usr/local/lib/python3.7/__pycache__/weakref.cpython-37.pyc

The file lengths match between these two listings, however, the date stamps differ.  I was puzzled by the differences so decided to look for the "corruptions".  So far, I've analysed only the first file (io.cpython-37.pyc) and it contained 48 corrupted bytes:

!da 5a!1f 1e!1f 1e!da 5a!19 18!1a 19!1b 1a!1c 1b
!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!19 18!1a 19!1b 1a
!1c 1b!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!19 18!1a 19
!1b 1a!1c 1b!1e 1d!1f 1e!1f 1e!1f 1e!20 1f!1e 1d
!da 5a!1c 1b!da 5a!da 5a!1a 19!1d 1c!da 5a!22 20
!23 21!24 22!da 5a!25 23!1f 1e!1f 1e!1f 1e!20 1f

After each "!" there are two bytes.  The first byte is from the "good" file, the second byte is from the corrupted file.  There isn't a consistent corruption between the good byte & the bad.  The majority of errors are the effect of subtracting 1 from the good byte (NB that's not the same as a one-bit error in my book) before storing it.  However, there are a few instances where 2 is subtracted, and a few more where the error is the upper four bits.

I'm not happy about 48 byte errors in a single file, and there's another 12 files to analyse.

I thought about a problem in the package file, so I checked the python37-3.7.10.tgz package file on the mirrors.  There are 24 mirror sites listed.  20 of those would accept my request for a copy of the file.  19 of those files were identical.  The one that differed was from "fourdots.com" and was truncated such that I only got 0.5% of it.  I'm certain that my system didn't install that file because I would have had many more errors, and probably pkg would have barfed, anyway.  So, not a problem with the package.

Franco mentioned "filing system errors" in the reference above.  I would like to pin it down further.
I've wondered about a disk error - it's a small SSD (KINGSTON SMS200S330G mSATA) that's been running for just over 4 years & 4 months.

Does anyone know why the corruped files had today's date, whereas after I had reinstalled the package, the the files had the original dates?

I've just done another check, this time on the French firewall - which didn't produce any errors.  All the files in the relevant directory ( /usr/local/lib/python3.7/__pycache__ ) have exactly the same date+time stamp as the (fresh) UK ones if one compensates for the different timezones.  So I'm beginning to think that the errors happened while the files were being extracted from the package. 

But why?

I'm not expecting definitive answers, but if anyone has any hints or suggestions, I'd be interested to hear them.

Thanks for reading...

PeterF

P.S.  What did we do before the audit->health option appeared???  It certainly proved useful today  :)

[many thanks]

Hi mimu

Many thanks for your reply and suggestions.  It was a bit left-field, but it made stop, think and dig a bit further.  What I hadn't said earlier is that what I want to achieve is a bridge between two (private) vlans.  IPsec running on the firewall/router that OPNsense provides actually gives me a simple (almost elegant?) solution and allows me to move whatever data I want, from any device, to where I need it.  For example, the other day I was in the UK and printed a document to a family member in France.  Saved them installing umpteen MB of printer driver to get a single A4 page  :)

So I went and investigated your suggestions.  Zerotier first, because I know absolutely nothing about it.  Looks useful, probably very useful in some scenarios, but it gave me the strong impression that it's a per-device solution.  Fantastic (if you're not put off by the cloud word) for some types of application; but I want to connect anything to everything.  I don't think ZT can offer me that.  Not without installing "client-side" software on almost everything.  Also, they're a business and want to make money; I'm not a business (now) and I can't justify that money.  Finally, one has to address the cloud word.  When I was working, we banned the "just" word.  Engineering is about detail and the "just" word implies it's easy and we haven't thought about the detail (it gets left to someone else).  In my view, "cloud" is almost up there with "just".  I hope this doesn't offend.  I like to know exactly where my data is, and where it's going. I use IPsec (and email, and ...) and I hope I know the risks.  But that's all (as far as I can make it) under my control.  I do use a cloud, but it's my cloud (nextcloud) and it runs on my hardware in my cellar.

Anyway, then I turned to OpenVPN.  I use this already as a "road warrior" and I've therefore used it as a bridge from a single device back to base.  It works very well in this mode - though I find the network delay from Australia back to the UK a little disconcerting!.  I went looking on the OpenVPN site to see how they suggest using it as a bridge.  My quick read of two examples (URLs below in case anyone else is interested) led me to believe that in order to do this, then the solution was more complex than the IPsec solution and involved more hardware in order to do the routing.  I have no objection to having more hardware, but each device needs managing and supporting, and my IT support team (me) is already at full stretch, given the day-time jobs of child-minding the g-kids, gardening, building electronics, ...

Anyway, coming back to technicalities of trying to run a VPN over domestic grade ADSL to two dynamic nodes is really rather wild, but cheap, and very useful when it works.  What any VPN solution is going to need is for each end to "know" the parameters (principally the address) of the other end.  When one end goes down, there is always going to be a lag while it gets itself back up and back in communication with the its peer. 

I've just gone and looked at my IPsec logs to see how long it's been up. It's currently at 1.5 days, and I suspect the previous resync was me testing and prodding and adjusting the script. It now has 2 logs (I like logs for solving problems :) ) - a hiccup log, so I can see quickly when the last problem was, and a minimalist boring log.

So many thanks for taking the time to send me your suggestions. My apologies for the long-winded answer; but I think I'll stay with IPsec for now because it's a good fit for what I need.

Best wishes,

Peter


OpenVPN for site-to-site connections
https://docs.openvpn.net/how-to-tutorialsguides/site-to-site-layer-2-bridging-using-openvpn-access-server/
https://docs.openvpn.net/connecting/site-to-site-routing-explained-in-detail/

Hi bennebartsch

I would hazard a guess that you're logging into your OPNsense box and getting the "Hello, this is OPNsense nn.m welcome screen and the 0-13 menu beneath.  Then I suspect you're selecting option 8 for a Shell.

At that point you're getting the response:

csh: The terminal database could not be opened.
csh: using dumb terminal settings.

If this is the case, then before starting to log in, check to see how your current terminal is defined.  In Unix/Linux type echo $TERM.  In Bitvise (a Windows client), it called "Protocol" on the Terminal tab.  I'm sure there's a similar configuration setting in Putty (but I don't use that). When you find it, I think you'll find that your setting is possibly too esoteric for the limited database in FreeBSD.  Set the variable to "xterm" (a nice vanilla flavour) (*nix: TERM=xterm or TERM="xterm") and try logging in again and selecting option 8.

If this doesn't work, then you'll have to give Franco some more details...

HTH though.

Peter

[First, here's some answers for you:]

Hi Franco

First, here's some answers for you:

What are your GUI-bound settings for: My identifier, Peer identifier

UK-FW		FR-FW
In "VPN: IPsec: Tunnel Settings" My Identifier is set to "Dynamic DNS" with a value of :  "xx1.duckdns.org"		In "VPN: IPsec: Tunnel Settings" My Identifier is set to "Dynamic DNS" with a value of :  "xx2.duckdns.org"

xx1.duckdns.org is the name associated with the public IP address of UK-FW
xx2.duckdns.org is the name associated with the public IP address of FR-FW

Peer identifier is set to "Peer IP address" on both UK-FW and FR-FW

[Do] you use the FQDN on both sides for the remote address?

I guess you mean "Remote gateway" parameter?

If so, then "yes", I use the appropriate FQDN on both sides.
So, on UK-FW the value of the remote gateway is xx2.duckdns.org
and on FR-FW the value of the remote gateway is xx1.duckdns.org

What is your WAN setup... DHCP?

In "Interfaces: [WAN]" the setting for  "IPv4 Configuration Type" is set to "PPPoE"
This is the same on UK-FW and FR-FW.

What interface is your IPsec phase 1 bound to?

In "VPN: IPsec: Tunnel Settings" (Phase 1 settings), the Interface is set to "WAN"
This is the same on UK-FW and FR-FW.

Now a comment (observation?)

In theory IPsec should restart itself and also regenerate an proper up-to-date config on DHCP IP changes. But it sounds like it's not doing this.

• I think I agree with your statement, and the config file is being updated with the "local" (left) data.  I'm not sure how this is done because i haven't dug that deeply into the code.


• However, in the config file on UK-FW, there also exists the (public) IP address for FR-FW.  UK-FW is not checking the public IP address of FR-FW (which it can (only) get from DNS) and ensuring that this address is correct within "  rightid = " in the config file on UK-FW.


• And No 2 above also applies to FR-FW not monitoring the public address of UK-FW for its own ipsec.config file.


• Finally, because I am using a "Mutual PSK" for authentication, you'll also find the public IP address of FR-FW embeded in the ipsec.secrets file of UK-FW.

So, when I wrote my checking and fixing script, I checked all the numerical IP addresses in ipsec.conf associated with left, leftid and rightid parameters and the numerical IP address in ipsec.secrets.  If I found any discrepancy, I fixed it (sed is wonderful :) ) and set a changed flag.  At the end of the script, I used the state of this flag to indicate whether an ipsec rereadall and an ipsec reload were required.

The current version of ipsec appears to only function with numerical IP addresses.  Therefore, it has to be the responsibility of OPNsense to check the validity/accuracy of the data going into ipsec.config and ipsec.secrets.  This is clearly done when the config for Phase_1 or Phase_2 is saved.  I think it just needs to happen regularly and routinely in the background.  I think the Voldermort (*1) installation {p.*e} used to do this, because that was one of the reasons I moved from IPCop to {p.*e} around 2009.  I found that {p.*e} was better than IPCop in restoring an IPsec VPN.  (Of course, it now appears that IPCop has withered on the vine, but I digress. Sorry.)  What I don't have at present, is an installed version of {p.*e) to look at the code.  Sorry.

*1 Voldermort - "He Who Must Not Be Named".

I hope the answers and the numbered comments help explain what I think is "wrong".  If you need any more information, then just ask.

Cheers, and many thanks for the hard work

Peter

[The following statement applies to both firewalls.]

Hi

I have identified a problem with IPsec (site to site) not connecting automatically.

I have two FWs (UK-FW & FR-FW) both running OPNsense 18.1 series:

My tests and investigations described below have been carried out on UK-FW which is up-to-date and has the following overall build:

OPNsense 18.1.8-amd64
FreeBSD 11.1-RELEASE-p10
OpenSSL 1.0.2o 27 Mar 2018

FR-FW also has the same build - i.e. they're both up-to-date at the time of writing.

Both firewalls use a domestic ADSL line for their WAN connection. 
I have configured a VPN between the two firewalls. 
Both firewalls use a dynamic dns service to facilitate flexible configuration using FQDN, and not hard-coded IP addresses. 
This VPN uses a PSK and, when it is up, it works well.

The following statement applies to both firewalls.

The ADSL line is connected to a modem which has a PPoE connection to the WAN port on the OPNsense firewall. When the WAN connection comes up, it always has a fresh (& different) public IP address.  This new address is associated quite quickly with its FQDN via the dynamic dns service.

This means that after a reboot of either one or both firewalls, the IPsec tunnel will not be established automatically.  I recognise that the initialisation of the tunnel now occurs at a different time in the boot sequence than when I reported a problem with v17.1.5 (Thanks for fixing that bug  :) ).  However, I believe that there is a missing component which means that in my case (i.e. neither firewall can have a fixed public IP address) it can never be established automatically. (Please note, there may be something I've missed (in the configuration) and I am open to suggestions and advice  :) )

Here are my observations and fixes:

• Let's assume that IPsec between UK-FW and FR-FW is working (traffic is flowing).  I will assume in the following scenario that the UK-FW is my local firewall and is described by the details referring to the "Left" side.
  
• This means that the remote firewall (FR-FW) is described by the parameters for the "Right" side.(
• Now consider what happens when FR-FW drops its ADSL connection, and then reacquires it.  After this reconnection, it will have a new public IP address.  FR-FW ensures that this new address is updated to the dynamic dns service.
• When the WAN comes up, FR-FW also appears to update the IPsec configuration file (/usr/local/etc/ipsec.conf) so that its new WAN address is correctly inserted into the configuration lines:
  

Code Select Expand

conn con1
  .
  .
  left = <my WAN ip address>
  leftid = <my WAN ip address>
  .
  .

• The IPsec tunnel will not come up at this point.
• If one looks at the two IPsec configuration files on UK-FW, one will observe discrepancies which will never be detected or fixed.
• In the IPsec configuration file (/usr/local/etc/ipsec.conf), there is a hard-coded IP address for the remote firewall (FR-FW) in the line which starts " rightid = ".  This address is now wrong and there appears to be no automatic fix.
• Also, in the IPsec secrets file (/usr/local/etc/ipsec.secrets), there is also a "stale" (old) IP address associated with the shared key (PSK).

I have identified two ways to bring the IPsec tunnel back up.  They are:

Method 1

• Using the GUI (web console at VPN: IPsec: Tunnel Settings , press the "edit" button for either the phase_1 or phase_2 entry.  Do not change any parameter, but just press the "Save" button.
• When prompted, press the "Apply Changes" button.
• The IPsec tunnel will come up.
• Note:  This action has to be carried out on the system which has not just been assigned a new WAN address.  In the scenario I described above, this would have been carried out on UK-FW

Method 2

• Working on the system which has not just been assigned a new WAN address, one can correct the errors in the two ipsec configuration files.  (This can done manually (with a text editor *1) or scripted (less tedious *2))
• Then one needs to run the commands

Code Select Expand

/usr/local/sbin/ipsec rereadall
/usr/local/sbin/ipsec reload

• The IPsec tunnel will come back up.

Notes *1

• Editing ipsec.conf with a text editor (I used vi) is potentially risky.  It's certainly tedious!  ;)
• Make sure you don't delete or change the two spaces at the beginning of the line.
• Change only the lines with an incorrect IP address; probably the "  leftid = " entry.
• When editing ipsec.secrets, make sure that you change only the IP address at the beginning of the line.
• When you've finished the edit, make sure that the file (ipsec.secrets) is owned by root and has 0600 permissions.

Notes *2

• Having tested my observations and trying a manual fix (see above), I scripted the checking and correction.
• I wrote a shell script (I used bash as I'm more familiar with it) which is defined as a service and can be called from the cron facility (web console at System: Settings: Cron)
• The script is run every minute.
• The script checks IP address for both sides (right and left) in the ipsec.conf file.  It also checks the IP address in the ipsec.secrets file.
• If any error is detected, it makes the necessary correction.
• If, and only if, any change (correction) is made, it then runs the two ipsec commands shown in the code block above.
• My IPsec configurations were simple with only a single Phase_1 and a single Phase_2 defined on each firewall.  Therefore, the script was easy to implement as I didn't have to check (& parse) multiple "conn" sections in ipsec.conf and ipsec.secrets.
• In my experiments and observations, the IPsec tunnel is quickly reestablished.

Results of scripting

• I have installed my script at both ends of my IPsec tunnel - i.e. on UK-FW and FR-FW.
• I can remotely reset the modem on either firewall. The script brings the tunnel back up successfully in each of the three possible scenarios:
  
  
  • Modem on UK-FW is reset, so UK-FW gets a new public IP address
  • Modem on FR-FW is reset, so FR-FW gets a new public IP address
  • Both modems are reset at the same time.
  • The IPsec tunnel comes back up within 2 or 3 minutes of the modem being reset. (Remember that the modem has to negotiate a working connection together with PPoE authentication, then the dynamic dns upgrade has to go through and finally remember that the "repair" script runs only once per minute as a cron job.)
• I can remotely reboot either firewall.  The script brings the tunnel back up successfully in each of the three possible scenarios:
  
  
  • UK-FW is rebooted, so UK-FW gets a new public IP address
  • FR-FW is rebooted, so FR-FW gets a new public IP address
  • Both firewalls are rebooted at the same time.
  • The IPsec tunnel comes back up within 3 or 4 minutes of the firewall being rebooted.

Feature Request / Bug fix
It would be preferable (particularly for those users with more complex IPsec configurations) if the monitoring of the remote end of each VPN connection could be handled within OPNsense so that the IPsec configuration files could be kept up-to-date (automagically) and this would then mean that IPsec would then have a default "Up" status, rather than a default "Down" status which is what I observe with a stock OPNsense installation at present.

Alternative approach
A long time ago I learned that hard-coding IP addresses into configurations was best avoided if at all possible. So I experimented with /usr/local/etc/inc/plugins.inc.d/ipsec.inc (which appears to be the php code that generates the IPsec configuration files from the user-input data.  I found that the "rightid" parameter has to be a hard-coded IP address.  IPsec will not work with a FQDN here.  Similarly, I couldn't get the secrets file to work with either a FQDN or the special parameter "%any".  My reading of the man page suggested that the "%any" parameter should have worked here, but it's less desirable, probably, for security reasons.

Thanks for reading this long post. 
I hope that it might help someone else solve a problem, and better still prompt a fix within either OPNsense or ipsec.

Peter

Hi franco

I had to go and lookup XMLRPC, so that's probably a "NO" then  :)
I have no need for IPv6 yet; my ISPs are stuck firmly in V4 land, and my private networks fit easily into V4 as well.
So that's  a "NO" as well then.

Thanks,
Peter