Upgrade OPNsense 24.4.3 to 24.10_1 end in random connection losts,all interfaces

[biricon]

Hello,

we upgrade on friday our own firewall and since that we get random connection losts. The VLANs losts from one to another. We have around 30 VLANs on 3 Hyper-V NICs cables. The cables has the same vlans and we configure on the hyper-v the following:

NIC1: All VLANs (only tagged) used only the VLANs for internal LANs (one unused untagged vlan)
NIC2: reserved for HA (but not used now with a untagged VLAN for that)
NIC3: All VLANs (only tagged) used only for 3 WAN links (one unused untagged vlan)

It starts with the WAN links, that goes down. After that (very short behind) the internal connectios will be lost and no connectivitiy anymore.

After restore from backup the 24.4.3 the problem was gone.

Did anyone knows what was changed in the newer version that we have this kind of issue?

best regards

[biricon]

Today at 6am the firewall did stop working again.

The Health Check say everything fine.

Code: [Select]

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 24.10_7 at Tue Oct 29 07:59:09 CET 2024
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 24.7.6 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 24.7.6 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-OPNBEcore 1.4_2
os-OPNcentral 1.10_1
os-acme-client 4.6
os-clamav 1.8
os-frr 1.41_1
os-haproxy 4.3_1
os-nginx 1.34_2
os-ntopng 1.3
os-redis 1.1_2
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense-business" has 70 dependencies to check.
Checking packages: ....................................................................... done
***DONE***
The VLAN Configuration is on the Hyper-V right now:

LAN all internal VLANs
WAN (extra cable not connected with the internal infrastructor) and 2 VLANs for 2 providers.

We dont use a brige on our installation.

The firewall linux was reachable via console, but all network connectivity was lost. No vlan connection, no ip connectivity anymore.

With the 2FA its not possible to login via console anymore at this state. Only a restart via STRG+ALT+DEL will be possible and than, everything is up and running again.

[cookiemonster]

I suggest you read the release notes again to reaquaint with the changes. You read them first before upgrading, right?
That done, perhaps even if not in the release notes, you could consider setting only tagged VLANs on your interfaces. Not a mix of tagged and untagged, even if not currently used.
p.s. hyper-v is not a great hypervisor for freeBSD but that hasn't been a problem for you so far. I suggest to put in the to-do list to migrate when your infra is stable again.

[biricon]

i find out right now the behavior of the issue that does not exists on the preview versions and yes i read the changelog and search since 2 weeks .. right now i stand up several times in the night an reboot the opnsense. thats realy bad. . .

Right now i have only tagged traffic on the interfaces, but no luck.

So - i can not reproduce the issue:
- i have normaly backups, that live only in his VLANs (thats no problem.)
- i have backups from a IOT lan (for fake by tesla batterys with a PI and fake a smartmeter) for a fronius Installation. This Raspberries makes backups.
If this backups starts the problem starts too. I find out on the traffic on the interface.
After the traffic reaches 600Mbit/s (on the internal lan) the NAT stuff dies on Opensense. If i reach the 1,2Gbis/s the routing between the VLANs dies and than the Opnsense is not reachable anymore from any interface and a console login with 2FA is not possible anymore.

The installation is like describe before, but the the VM specs:
Xeon Gold 6252N (8 cores for opnsense)
8Gbit ram

On the new Dashboad vlans result in a wrong traffic statistic. Because you see the traffic of the Interface AND the traffic of the vlans and it makes not diffrent if the interface (untagged) is enabled or disabled

The script the bring the Interfaces to collaps is a simple backup script, that works in the past without issues:

Code: [Select]

#!/bin/bash
#Festplatte einbinden
mount -t cifs -o user=xxxxxxx,password=xxxxxxx,rw,file_mode=0777,dir_mode=0777 //xxx.xxx.xxx.xxx/Backup /media/nas
#Variablen
BACKUP_PFAD="/media/nas/PiBackups/TBattery03"
BACKUP_ANZAHL="14"
BACKUP_NAME="TBarrery03"
#Backup erstellen
dd if=/dev/sda of=${BACKUP_PFAD}/${BACKUP_NAME}-$(date +%Y%m%d).img bs=1MB
#Komprimieren
cd ${BACKUP_PFAD}
zip ${BACKUP_NAME}-$(date +%Y%m%d) *.img
rm ${BACKUP_NAME}-$(date +%Y%m%d).img
#Alte Sicherung löschen
pushd ${BACKUP_PFAD}; ls -tr ${BACKUP_PFAD}/${BACKUP_NAME}* | head -n -${BACKUP_ANZAHL} | xargs rm; popd
#Festplatte auswerfen
umount /media/nas


[cookiemonster]

That is very interesting as it seems to suggest a hardware glitch that gets triggered with the high transfers.
I do not know hyper-v as I don't use it, so can't help but perhaps you could start focusing on interfaces (NICs)0 both virtual and physical.

[biricon]

Its a Gen 2 VM with its standard hardware, that used before too. Its possible that there is a BSD Hyper-V extensions, that's possible to build in or is already included. But i dont know the specs of the distro that used here.

on the console i dont see that the interface goes down.

Here is the integration article from free bsd:
https://wiki.freebsd.org/HyperV

Did you know whats included on the stock installation?

[cookiemonster]

I do not know but the article you link to is talking about necessaru good bits being included since freeBSD 10 so I would be thinking that OPN will also contain them, no reason not to.
Distro base of OPN is freeBSD. We're on freeBSD 14.x-RELEASE I think as base.

[biricon]

Thats correct, but for FreeBSD 14 i dont find anything about Hyper-V right now. Its very interessting, that the load goes so extremly up, if the traffic is on the interface. But thats only a system load - no process uses mutch CPU at this time.

[cookiemonster]

Easier said than done buy my suggestio if possible, test it on a non-hyperv host.

[biricon]

i dont have a none hyper-v host with the SFP+ connection for testing.

[biricon]

Right now .. its 2am - the firewall stop working again. Without big traffic on the interfaces anymore - backup traffic will stay in its own vlans.

Routing and Firewall Rules works.

No: Webinterface, DHCP, DNS, HAProxy and other services.

i can surf if i use a public dns, static ip and the routing between the vlans works.

a lot of jobs are missing in top and ps -aux

Code: [Select]

root@sense:~ # ps -aux
USER     PID  %CPU %MEM    VSZ   RSS TT  STAT STARTED       TIME COMMAND
root      11 799.0  0.0      0   128  -  RNL  08:17   7864:06.07 [idle]
root       2   1.0  0.0      0   128  -  RL   08:17     56:17.00 [clock]
root   26743   1.0  1.1 120828 93756  0  S+   02:44      0:01.55 /usr/local/bin/php /usr/local/etc/rc.reload_all
root       0   0.0  0.0      0  2640  -  DLs  08:17    261:36.23 [kernel]
root       1   0.0  0.0  11308   656  -  ILs  08:17      0:00.18 /sbin/init
root       3   0.0  0.0      0   144  -  DL   08:17      0:00.00 [crypto]
root       4   0.0  0.0      0    64  -  DL   08:17      0:00.01 [cam]
root       5   0.0  0.0      0  1296  -  DL   08:17      5:36.84 [zfskern]
root       6   0.0  0.0      0    16  -  DL   08:17      0:25.14 [pf purge]
root       7   0.0  0.0      0    16  -  DL   08:17      0:08.33 [rand_harvestq]
root       8   0.0  0.0      0    48  -  DL   08:17     10:21.13 [pagedaemon]
root       9   0.0  0.0      0    16  -  DL   08:17      4:53.15 [vmdaemon]
root      10   0.0  0.0      0    16  -  DL   08:17      0:00.00 [audit]
root      12   0.0  0.0      0    96  -  WL   08:17      1:58.38 [intr]
root      13   0.0  0.0      0    48  -  DL   08:17      0:00.00 [geom]
root      14   0.0  0.0      0    16  -  DL   08:17      0:00.00 [sequencer 00]
root      15   0.0  0.0      0    80  -  DL   08:17      0:01.64 [bufdaemon]
root      16   0.0  0.0      0    16  -  DL   08:17      0:01.48 [vnlru]
root      17   0.0  0.0      0    16  -  DL   08:17      0:00.50 [syncer]
root      29   0.0  0.0      0    16  -  DL   08:17      0:00.01 [aiod1]
root      30   0.0  0.0      0    16  -  DL   08:17      0:00.01 [aiod2]
root      31   0.0  0.0      0    16  -  DL   08:17      0:00.01 [aiod3]
root      32   0.0  0.0      0    16  -  DL   08:17      0:00.02 [aiod4]
root     485   0.0  0.0  12640  1560  -  Is   08:17      0:00.13 /usr/sbin/hv_kvp_daemon
root     487   0.0  0.0  12624  1252  -  Is   08:17      0:00.00 /usr/sbin/hv_vss_daemon
root     544   0.0  0.0  14292  1920  -  Ss   08:18      0:05.47 /sbin/devd
root    7508   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root    8196   0.0  0.0  23444     8  -  IW   -          0:00.00 /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
root    8288   0.0  0.1  65940  8008  -  Ss   08:18     14:29.96 /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
root   10197   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   16479   0.0  0.0  26304  3528  -  S    08:18      0:08.84 /usr/local/bin/python3 /usr/local/sbin/configctl -e -t 0.5 system event config_changed (python3.11)
root   16962   0.0  0.1  27328  5096  -  S    08:18      0:09.77 /usr/local/bin/python3 /usr/local/opnsense/scripts/syslog/lockout_handler (python3.11)
root   17196   0.0  0.1  19724  7120  -  Is   02:45      0:00.02 sshd-session: root [priv] (sshd-session)
sshd   17294   0.0  0.1  19620  6916  -  IC   02:45      0:00.01 sshd-session: root [net] (sshd-session)
root   17402   0.0  0.1  19724  7108  -  I    02:45      0:00.00 sshd-session: root [pam] (sshd-session)
root   19463   0.0  0.0      0   128  -  DL   08:18      0:07.61 [ng_queue]
root   20563   0.0  0.0  12828  2120  -  Is   02:45      0:00.00 /usr/sbin/cron -s
root   23853   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   26286   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   29248   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   34377   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
www    34638   0.0  0.1  51684  9704  -  I    01:09      0:00.00 php-fpm: pool www (php-fpm)
root   37750   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   38435   0.0  0.1  19724  7148  -  Ss   02:46      0:00.02 sshd-session: root [priv] (sshd-session)
root   39482   0.0  0.1  19724  7440  -  S    02:46      0:00.01 sshd-session: root@pts/1 (sshd-session)
root   40370   0.0  0.0  12736     8  -  IWs  -          0:00.00 daemon: /usr/local/bin/samplicate[40499] (daemon)
nobody 40499   0.0  0.0  12636   128  -  I    08:18      0:12.91 /usr/local/bin/samplicate -s 127.0.0.1 -p 2055 127.0.0.1/2056
root   41241   0.0  0.1  19192  4720  -  Ss   08:18      0:00.03 sshd: /usr/local/sbin/sshd [listener] 1 of 10-100 startups (sshd)
root   41487   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   42371   0.0  0.0  51572     8  -  IWs  -          0:00.00 /usr/local/bin/php-cgi
root   42518   0.0  0.0  51572     8  -  IWs  -          0:00.00 /usr/local/bin/php-cgi
root   43352   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   43445   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   43529   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   43816   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   43939   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   44294   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   44525   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   44761   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   44952   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   45192   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   45377   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   45455   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   45596   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   45747   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   45874   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   45953   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46012   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46160   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46187   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46224   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46240   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46300   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46355   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46477   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46492   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46518   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   46536   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   53194   0.0  0.1  19724  7084  -  Is   02:42      0:00.02 sshd-session: root [priv] (sshd-session)
root   54037   0.0  0.1  19724  7380  -  S    02:42      0:00.03 sshd-session: root@pts/0 (sshd-session)
root   60008   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   60291   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   60440   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   73267   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   73903   0.0  0.0  12724     8  -  IWs  -          0:00.00 flowd: monitor (flowd)
_flowd 73973   0.0  0.0  12724  1772  -  Is   08:18      0:21.56 flowd: net (flowd)
root   78678   0.0  0.0  14452  2112  -  S    08:18      0:01.66 /usr/local/sbin/lighttpd -f /var/etc/lighttpd-acme-challenge.conf
root   79717   0.0  0.0  13020  2004  -  Ss   08:18      3:38.38 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root   84399   0.0  0.1  51692  9436  -  Ss   08:18      0:04.08 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
root   84495   0.0  0.0  51684     8  -  IW   -          0:00.00 (php-fpm)
root   84651   0.0  0.0  51684     8  -  IW   -          0:00.00 (php-fpm)
www    84731   0.0  0.0  51684     8  -  IW   -          0:00.00 (php-fpm)
root   86186   0.0  0.1  23732  4356  -  Ss   08:18      0:06.88 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
root   87815   0.0  0.0  50512   632  -  Is   08:18      0:00.14 nginx: master process /usr/local/sbin/nginx
www    87993   0.0  0.0  50512   652  -  I    08:18      0:00.10 nginx: worker process (nginx)
root   95557   0.0  0.0  51572     8  -  IW   -          0:00.00 /usr/local/bin/php-cgi
root   30939   0.0  0.0  12756  1196 v0  Is+  08:18      0:00.00 /usr/libexec/getty Pc ttyv0
root   31334   0.0  0.0  12756  1196 v1  Is+  08:18      0:00.00 /usr/libexec/getty Pc ttyv1
root   31666   0.0  0.0  12756  1200 v2  Is+  08:18      0:00.00 /usr/libexec/getty Pc ttyv2
root   32068   0.0  0.0  12756  1196 v3  Is+  08:18      0:00.00 /usr/libexec/getty Pc ttyv3
root   32335   0.0  0.0  12756  1196 v4  Is+  08:18      0:00.00 /usr/libexec/getty Pc ttyv4
root   32739   0.0  0.0  12756  1196 v5  Is+  08:18      0:00.00 /usr/libexec/getty Pc ttyv5
root   32826   0.0  0.0  12756  1196 v6  Is+  08:18      0:00.00 /usr/libexec/getty Pc ttyv6
root   33023   0.0  0.0  12756  1196 v7  Is+  08:18      0:00.00 /usr/libexec/getty Pc ttyv7
root   54414   0.0  0.0  13284  2236  0  Is+  02:42      0:00.01 /bin/sh /usr/local/sbin/opnsense-shell
root   39613   0.0  0.0  13284  2340  1  Ss   02:46      0:00.01 /bin/sh /usr/local/sbin/opnsense-shell
root   46467   0.0  0.0  13760  3272  1  S    02:46      0:00.01 /bin/csh
root   46647   0.0  0.0  13352  2476  1  R+   02:46      0:00.00 ps -aux
root@sense:~ #
last pid: 29210;  load averages:  0.18,  0.21,  0.18                                                                                                         up 0+18:30:19  02:47:58
82 processes:  1 running, 81 sleeping
CPU:  0.0% user,  0.0% nice,  0.9% system,  0.0% interrupt, 99.1% idle
Mem: 118M Active, 81M Inact, 232K Laundry, 1311M Wired, 56K Buf, 6416M Free
ARC: 364M Total, 117M MFU, 69M MRU, 1681K Anon, 2415K Header, 174M Other
     82M Compressed, 318M Uncompressed, 3.89:1 Ratio
Swap: 8192M Total, 58M Used, 8133M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 8288 root          4  20    0    64M  8024K kqread   6  14:30   0.74% syslog-ng
79717 root          1  20    0    13M  2004K bpf      4   3:39   0.57% filterlog
29210 root          1  20    0    14M  3052K CPU2     2   0:00   0.33% top
16479 root          1  20    0    26M  3528K select   5   0:09   0.02% python3.11
39482 root          1  20    0    19M  7456K select   6   0:00   0.02% sshd-session
  544 root          1  20    0    14M  1920K select   3   0:05   0.01% devd
16962 root          1  20    0    27M  5096K select   5   0:10   0.01% python3.11
26743 root          1  21    0   118M    92M nanslp   4   0:03   0.01% php
86186 root          1  20    0    23M  4356K select   7   0:07   0.01% ntpd
84399 root          1  20    0    50M  9436K kqread   4   0:04   0.00% php-fpm
78678 root          1  20    0    14M  2112K kqread   5   0:02   0.00% lighttpd
73973 _flowd        1  20    0    12M  1772K select   0   0:22   0.00% flowd
40499 nobody        1  20    0    12M   128K sbwait   1   0:13   0.00% samplicate
87815 root          1  20    0    49M   632K pause    1   0:00   0.00% nginx
  485 root          1  20    0    12M  1560K select   5   0:00   0.00% hv_kvp_daemon
87993 www           1  20    0    49M   652K kqread   6   0:00   0.00% nginx
42518 root          1  20    0    50M  8192B wait     2   0:00   0.00% <php-cgi>
42371 root          1  20    0    50M  8192B wait     5   0:00   0.00% <php-cgi>
54037 root          1  20    0    19M  7380K select   1   0:00   0.00% sshd-session
41241 root          1  20    0    19M  4788K select   3   0:00   0.00% sshd
38435 root          1  23    0    19M  7148K select   4   0:00   0.00% sshd-session
53194 root          1  23    0    19M  7084K select   0   0:00   0.00% sshd-session
46467 root          1  20    0    13M  3308K pause    3   0:00   0.00% csh
73903 root          1  20    0    12M  8192B sbwait   5   0:00   0.00% <flowd>
39613 root          1  68    0    13M  2340K wait     3   0:00   0.00% sh
54414 root          1  26    0    13M  2236K wait     3   0:00   0.00% sh
20563 root          1  68    0    13M  2120K nanslp   5   0:00   0.00% cron
34638 www           1  20    0    50M  9704K accept   2   0:00   0.00% php-fpm
  487 root          1  20    0    12M  1252K select   0   0:00   0.00% hv_vss_daemon
32826 root          1  68    0    12M  1196K ttyin    0   0:00   0.00% getty
33023 root          1  68    0    12M  1196K ttyin    1   0:00   0.00% getty
30939 root          1  68    0    12M  1196K ttyin    5   0:00   0.00% getty
29248 root          1  20    0    50M  8192B accept   5   0:00   0.00% <php-cgi>
32068 root          1  68    0    12M  1196K ttyin    6   0:00   0.00% getty
31666 root          1  68    0    12M  1200K ttyin    7   0:00   0.00% getty
32739 root          1  68    0    12M  1196K ttyin    4   0:00   0.00% getty
31334 root          1  68    0    12M  1196K ttyin    7   0:00   0.00% getty
32335 root          1  68    0    12M  1196K ttyin    2   0:00   0.00% getty
 7508 root          1  20    0    50M  8192B accept   1   0:00   0.00% <php-cgi>
 8196 root          1  68    0    23M  8192B wait     2   0:00   0.00% <syslog-ng>
95557 root          1  20    0    50M  8192B accept   5   0:00   0.00% <php-cgi>
26286 root          1  20    0    50M  8192B accept   4   0:00   0.00% <php-cgi>
60291 root          1  20    0    50M  8192B accept   3   0:00   0.00% <php-cgi>
34377 root          1  20    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
23853 root          1  20    0    50M  8192B accept   7   0:00   0.00% <php-cgi>
60008 root          1  20    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
60440 root          1  20    0    50M  8192B accept   5   0:00   0.00% <php-cgi>
84731 www           1  68    0    50M  8192B accept   3   0:00   0.00% <php-fpm>
73267 root          1  20    0    50M  8192B accept   6   0:00   0.00% <php-cgi>
10197 root          1  20    0    50M  8192B accept   6   0:00   0.00% <php-cgi>
84495 root          1  68    0    50M  8192B accept   3   0:00   0.00% <php-fpm>
84651 root          1  68    0    50M  8192B accept   3   0:00   0.00% <php-fpm>
37750 root          1  20    0    50M  8192B accept   4   0:00   0.00% <php-cgi>
41487 root          1  20    0    50M  8192B accept   3   0:00   0.00% <php-cgi>
40370 root          1  68    0    12M  8192B kqread   2   0:00   0.00% <daemon>
45596 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
45192 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
45455 root          1  68    0    50M  8192B accept   5   0:00   0.00% <php-cgi>
46518 root          1  68    0    50M  8192B accept   3   0:00   0.00% <php-cgi>
43445 root          1  68    0    50M  8192B accept   0   0:00   0.00% <php-cgi>
44952 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
45377 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
46300 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
46187 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
43816 root          1  68    0    50M  8192B accept   0   0:00   0.00% <php-cgi>
44761 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
46240 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
43939 root          1  68    0    50M  8192B accept   0   0:00   0.00% <php-cgi>
44525 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
43529 root          1  68    0    50M  8192B accept   1   0:00   0.00% <php-cgi>
46012 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
45874 root          1  68    0    50M  8192B accept   2   0:00   0.00% <php-cgi>
43352 root          1  68    0    50M  8192B accept   1   0:00   0.00% <php-cgi>
45953 root          1  68    0    50M  8192B accept   0   0:00   0.00% <php-cgi>
44294 root          1  68    0    50M  8192B accept   0   0:00   0.00% <php-cgi>
46224 root          1  68    0    50M  8192B accept   1   0:00   0.00% <php-cgi>
46355 root          1  68    0    50M  8192B accept   0   0:00   0.00% <php-cgi>
45747 root          1  68    0    50M  8192B accept   1   0:00   0.00% <php-cgi>
46160 root          1  68    0    50M  8192B accept   1   0:00   0.00% <php-cgi>
46492 root          1  68    0    50M  8192B accept   0   0:00   0.00% <php-cgi>
46477 root          1  68    0    50M  8192B accept   1   0:00   0.00% <php-cgi>
46536 root          1  68    0    50M  8192B accept   1   0:00   0.00% <php-cgi>

after i make a option 11 reload services i got this kind of errors, but services works again (the question is how long):

Code: [Select]

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 11

Writing firmware settings: FreeBSD OPNsense
Writing trust files...done.
Scanning /usr/share/certs/untrusted for certificates...
Scanning /usr/share/certs/blacklisted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
certctl: No changes to trust store were made.
Writing trust bundles...done.
Configuring login behaviour...done.
Configuring CRON...done.
Setting timezone: Europe/Vienna
Setting hostname: sense.biricon.eu
Generating /etc/resolv.conf...done.
Generating /etc/hosts...done.
Configuring loopback interface...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring V12_Video_MGMT_LAN interface...done.
Configuring V13_Biricon_Clients interface...done.
Configuring V14_Management_LAN interface...done.
Configuring V15_AccessPoint_Guest_LAN interface...done.
Configuring V16_VoIP interface...done.
Configuring V18_Gegensprechanlage interface...done.
Configuring V20_Bernhard_Server_LAN interface...done.
Configuring V40_Biricon_Backend_Server interface...done.
Configuring V60_Werkstatt_LAN interface...done.
Configuring V110_Biricon_Server interface...done.
Configuring V123_Chia interface...done.
Configuring V999_Funkfeuer_WAN interface...done.
Configuring V1123_WireGuardAD interface...done.
Configuring V2000_BMW_Service interface...done.
Configuring V4009_NextLayer interface...done.
Configuring V4094_Transfer_LAN interface...done.
Configuring OpenVPNCustomers interface...done.
Setting up routes...done.
Setting up gateway monitor...done.
Configuring firewall.......done.
Starting DHCPv4 service...done.
Starting NTP service...done.
Configuring OpenSSH...done.
Starting Unbound DNS...done.
Starting web GUI...done.
Configuring IPsec VPN...done.
Syncing OpenVPN settings...done.
Configuring WireGuard VPN...done.
Generating RRD graphs...done.
ntopng not running?
haproxy not running? (check /var/run/haproxy.pid).
Stopping nginx.
Waiting for PIDS: 87815.
Stopping php_fpm.
Waiting for PIDS: 84399.
redis not running? (check /var/run/redis/redis.pid).
Stopping acme_http_challenge.
Waiting for PIDS: 78678.
Stopping flowd.
Waiting for PIDS: 73903 73973.
flowd_aggregate not running? (check /var/run/flowd_aggregate.pid).
monit not running? (check /var/run/monit.pid).
setup ovpns2
ngctl: send msg: No such file or directory
error ovpns2: cannot create netflow node for ovpns2
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn2_vlanxxx [egress only]
Checking zebra.conf
2024/11/01 02:51:51 ZEBRA: [NNACN-54BDA][EC 4043309110] Disabling MPLS support (no kernel support)
OK
Starting zebra.
2024/11/01 02:51:51 ZEBRA: [NNACN-54BDA][EC 4043309110] Disabling MPLS support (no kernel support)
Checking staticd.conf
2024/11/01 02:51:51 STATIC: [PNYPZ-BCP8Y] Static Route using hn2_vlanxxx interface not installed because the interface does not exist in specified vrf
2024/11/01 02:51:51 STATIC: [RHJK1-M5FAR] static_zebra_nht_register: Failure to send nexthop 78.41.118.73/32 for 78.41.112.0/23 to zebra
2024/11/01 02:51:51 STATIC: [PNYPZ-BCP8Y] Static Route using hn2_vlanxxx interface not installed because the interface does not exist in specified vrf
2024/11/01 02:51:51 STATIC: [RHJK1-M5FAR] static_zebra_nht_register: Failure to send nexthop 78.41.118.73/32 for 78.41.118.0/23 to zebra
2024/11/01 02:51:51 STATIC: [PNYPZ-BCP8Y] Static Route using hn2_vlanxxx interface not installed because the interface does not exist in specified vrf
2024/11/01 02:51:51 STATIC: [RHJK1-M5FAR] static_zebra_nht_register: Failure to send nexthop 78.41.118.73/32 for 193.238.156.0/22 to zebra
OK
Starting staticd.
Starting CARP event handler now
Starting monit.
Starting Monit 5.34.1 daemon with http interface at /var/run/monit.sock
Starting flowd_aggregate.
Starting flowd.
rmdir: /var/etc/acme-client/home/deploy: Not a directory
rmdir: /var/etc/acme-client/home/dnsapi: Not a directory
rmdir: /var/etc/acme-client/home/notify: Not a directory
Starting acme_http_challenge.
Starting redis.
Performing sanity check on php-fpm configuration:
[01-Nov-2024 02:51:52] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Starting php_fpm.
sh: /usr/local/etc/rc.d/php-fpm: not found
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.
Starting haproxy.
Certificates generated /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
Starting ntopng.
md5sum: invalid option -- q
usage: md5sum [-bctwz] [files ...]
usage: grep [-abcDEFGHhIiLlmnOopqRSsUVvwxz] [-A num] [-B num] [-C num]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context=num] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]
01/Nov/2024 02:52:04 [Ntop.cpp:4052] WARNING: Unable to find timezone: using UTC
01/Nov/2024 02:52:04 [Redis.cpp:171] Successfully connected to redis 127.0.0.1@0
01/Nov/2024 02:52:04 [Redis.cpp:171] Successfully connected to redis 127.0.0.1@0
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn2_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded bridge0
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded lo0
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [NetworkInterface.cpp:3856] Cleanup interface dummy
01/Nov/2024 02:52:05 [Ntop.cpp:2642] Parent process is exiting (this is normal)

The log stops here and does not go back to the console menu, only a CTRL+C will return to the menu.

The Webconsole is working again, but the Dashboard does not load widgets with data. ,,Failed to load widget"

[biricon]

The Routing stuff has a interface ... but the errros from nexthop and redis-cli flushall is not clear for me and why the sense did result in crashing without big activity.

[hakuna]

This is the type of update that you wanna performe a clean install to avoid headaches, or you will spend endless time dealing with problems because of the major release.

IIRC, OPNSense team was preparing a "how-to" to make the upgrade, that alone was a clear "there are a lot of things to go wrong, fresh install it instead" message!!
No other major upgrade required that.

I was forced to fresh install it and it has been flawless but I did notice that some firewall stuff were not migrated over.
Still, zero problems.

It has been almost 2 weeks since you posted this and you are still having problems.
Backup the config and fresh install it!!

[biricon]

What kind of stuff is not migrated or part of the backup file?

[biricon]

[ironic on]i test now the config file on a new installation and i get a very helpful error [/ironic off]
 wtf