Hello,
we upgrade on friday our own firewall and since that we get random connection losts. The VLANs losts from one to another. We have around 30 VLANs on 3 Hyper-V NICs cables. The cables has the same vlans and we configure on the hyper-v the following:
NIC1: All VLANs (only tagged) used only the VLANs for internal LANs (one unused untagged vlan)
NIC2: reserved for HA (but not used now with a untagged VLAN for that)
NIC3: All VLANs (only tagged) used only for 3 WAN links (one unused untagged vlan)
It starts with the WAN links, that goes down. After that (very short behind) the internal connectios will be lost and no connectivitiy anymore.
After restore from backup the 24.4.3 the problem was gone.
Did anyone knows what was changed in the newer version that we have this kind of issue?
best regards
Today at 6am the firewall did stop working again.
The Health Check say everything fine.
***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 24.10_7 at Tue Oct 29 07:59:09 CET 2024
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 24.7.6 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 24.7.6 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-OPNBEcore 1.4_2
os-OPNcentral 1.10_1
os-acme-client 4.6
os-clamav 1.8
os-frr 1.41_1
os-haproxy 4.3_1
os-nginx 1.34_2
os-ntopng 1.3
os-redis 1.1_2
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense-business" has 70 dependencies to check.
Checking packages: ....................................................................... done
***DONE***
The VLAN Configuration is on the Hyper-V right now:
LAN all internal VLANs
WAN (extra cable not connected with the internal infrastructor) and 2 VLANs for 2 providers.
We dont use a brige on our installation.
The firewall linux was reachable via console, but all network connectivity was lost. No vlan connection, no ip connectivity anymore.
With the 2FA its not possible to login via console anymore at this state. Only a restart via STRG+ALT+DEL will be possible and than, everything is up and running again.
I suggest you read the release notes again to reaquaint with the changes. You read them first before upgrading, right?
That done, perhaps even if not in the release notes, you could consider setting only tagged VLANs on your interfaces. Not a mix of tagged and untagged, even if not currently used.
p.s. hyper-v is not a great hypervisor for freeBSD but that hasn't been a problem for you so far. I suggest to put in the to-do list to migrate when your infra is stable again.
i find out right now the behavior of the issue that does not exists on the preview versions and yes i read the changelog and search since 2 weeks .. right now i stand up several times in the night an reboot the opnsense. thats realy bad. . .
Right now i have only tagged traffic on the interfaces, but no luck.
So - i can not reproduce the issue:
- i have normaly backups, that live only in his VLANs (thats no problem.)
- i have backups from a IOT lan (for fake by tesla batterys with a PI and fake a smartmeter) for a fronius Installation. This Raspberries makes backups.
If this backups starts the problem starts too. I find out on the traffic on the interface.
After the traffic reaches 600Mbit/s (on the internal lan) the NAT stuff dies on Opensense. If i reach the 1,2Gbis/s the routing between the VLANs dies and than the Opnsense is not reachable anymore from any interface and a console login with 2FA is not possible anymore.
The installation is like describe before, but the the VM specs:
Xeon Gold 6252N (8 cores for opnsense)
8Gbit ram
On the new Dashboad vlans result in a wrong traffic statistic. Because you see the traffic of the Interface AND the traffic of the vlans and it makes not diffrent if the interface (untagged) is enabled or disabled
The script the bring the Interfaces to collaps is a simple backup script, that works in the past without issues:
#!/bin/bash
#Festplatte einbinden
mount -t cifs -o user=xxxxxxx,password=xxxxxxx,rw,file_mode=0777,dir_mode=0777 //xxx.xxx.xxx.xxx/Backup /media/nas
#Variablen
BACKUP_PFAD="/media/nas/PiBackups/TBattery03"
BACKUP_ANZAHL="14"
BACKUP_NAME="TBarrery03"
#Backup erstellen
dd if=/dev/sda of=${BACKUP_PFAD}/${BACKUP_NAME}-$(date +%Y%m%d).img bs=1MB
#Komprimieren
cd ${BACKUP_PFAD}
zip ${BACKUP_NAME}-$(date +%Y%m%d) *.img
rm ${BACKUP_NAME}-$(date +%Y%m%d).img
#Alte Sicherung löschen
pushd ${BACKUP_PFAD}; ls -tr ${BACKUP_PFAD}/${BACKUP_NAME}* | head -n -${BACKUP_ANZAHL} | xargs rm; popd
#Festplatte auswerfen
umount /media/nas
That is very interesting as it seems to suggest a hardware glitch that gets triggered with the high transfers.
I do not know hyper-v as I don't use it, so can't help but perhaps you could start focusing on interfaces (NICs)0 both virtual and physical.
Its a Gen 2 VM with its standard hardware, that used before too. Its possible that there is a BSD Hyper-V extensions, that's possible to build in or is already included. But i dont know the specs of the distro that used here.
on the console i dont see that the interface goes down.
Here is the integration article from free bsd:
https://wiki.freebsd.org/HyperV
Did you know whats included on the stock installation?
I do not know but the article you link to is talking about necessaru good bits being included since freeBSD 10 so I would be thinking that OPN will also contain them, no reason not to.
Distro base of OPN is freeBSD. We're on freeBSD 14.x-RELEASE I think as base.
Thats correct, but for FreeBSD 14 i dont find anything about Hyper-V right now. Its very interessting, that the load goes so extremly up, if the traffic is on the interface. But thats only a system load - no process uses mutch CPU at this time.
Easier said than done buy my suggestio if possible, test it on a non-hyperv host.
i dont have a none hyper-v host with the SFP+ connection for testing.
Right now .. its 2am - the firewall stop working again. Without big traffic on the interfaces anymore - backup traffic will stay in its own vlans.
Routing and Firewall Rules works.
No: Webinterface, DHCP, DNS, HAProxy and other services.
i can surf if i use a public dns, static ip and the routing between the vlans works.
a lot of jobs are missing in top and ps -aux
root@sense:~ # ps -aux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 11 799.0 0.0 0 128 - RNL 08:17 7864:06.07 [idle]
root 2 1.0 0.0 0 128 - RL 08:17 56:17.00 [clock]
root 26743 1.0 1.1 120828 93756 0 S+ 02:44 0:01.55 /usr/local/bin/php /usr/local/etc/rc.reload_all
root 0 0.0 0.0 0 2640 - DLs 08:17 261:36.23 [kernel]
root 1 0.0 0.0 11308 656 - ILs 08:17 0:00.18 /sbin/init
root 3 0.0 0.0 0 144 - DL 08:17 0:00.00 [crypto]
root 4 0.0 0.0 0 64 - DL 08:17 0:00.01 [cam]
root 5 0.0 0.0 0 1296 - DL 08:17 5:36.84 [zfskern]
root 6 0.0 0.0 0 16 - DL 08:17 0:25.14 [pf purge]
root 7 0.0 0.0 0 16 - DL 08:17 0:08.33 [rand_harvestq]
root 8 0.0 0.0 0 48 - DL 08:17 10:21.13 [pagedaemon]
root 9 0.0 0.0 0 16 - DL 08:17 4:53.15 [vmdaemon]
root 10 0.0 0.0 0 16 - DL 08:17 0:00.00 [audit]
root 12 0.0 0.0 0 96 - WL 08:17 1:58.38 [intr]
root 13 0.0 0.0 0 48 - DL 08:17 0:00.00 [geom]
root 14 0.0 0.0 0 16 - DL 08:17 0:00.00 [sequencer 00]
root 15 0.0 0.0 0 80 - DL 08:17 0:01.64 [bufdaemon]
root 16 0.0 0.0 0 16 - DL 08:17 0:01.48 [vnlru]
root 17 0.0 0.0 0 16 - DL 08:17 0:00.50 [syncer]
root 29 0.0 0.0 0 16 - DL 08:17 0:00.01 [aiod1]
root 30 0.0 0.0 0 16 - DL 08:17 0:00.01 [aiod2]
root 31 0.0 0.0 0 16 - DL 08:17 0:00.01 [aiod3]
root 32 0.0 0.0 0 16 - DL 08:17 0:00.02 [aiod4]
root 485 0.0 0.0 12640 1560 - Is 08:17 0:00.13 /usr/sbin/hv_kvp_daemon
root 487 0.0 0.0 12624 1252 - Is 08:17 0:00.00 /usr/sbin/hv_vss_daemon
root 544 0.0 0.0 14292 1920 - Ss 08:18 0:05.47 /sbin/devd
root 7508 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 8196 0.0 0.0 23444 8 - IW - 0:00.00 /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
root 8288 0.0 0.1 65940 8008 - Ss 08:18 14:29.96 /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
root 10197 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 16479 0.0 0.0 26304 3528 - S 08:18 0:08.84 /usr/local/bin/python3 /usr/local/sbin/configctl -e -t 0.5 system event config_changed (python3.11)
root 16962 0.0 0.1 27328 5096 - S 08:18 0:09.77 /usr/local/bin/python3 /usr/local/opnsense/scripts/syslog/lockout_handler (python3.11)
root 17196 0.0 0.1 19724 7120 - Is 02:45 0:00.02 sshd-session: root [priv] (sshd-session)
sshd 17294 0.0 0.1 19620 6916 - IC 02:45 0:00.01 sshd-session: root [net] (sshd-session)
root 17402 0.0 0.1 19724 7108 - I 02:45 0:00.00 sshd-session: root [pam] (sshd-session)
root 19463 0.0 0.0 0 128 - DL 08:18 0:07.61 [ng_queue]
root 20563 0.0 0.0 12828 2120 - Is 02:45 0:00.00 /usr/sbin/cron -s
root 23853 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 26286 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 29248 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 34377 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
www 34638 0.0 0.1 51684 9704 - I 01:09 0:00.00 php-fpm: pool www (php-fpm)
root 37750 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 38435 0.0 0.1 19724 7148 - Ss 02:46 0:00.02 sshd-session: root [priv] (sshd-session)
root 39482 0.0 0.1 19724 7440 - S 02:46 0:00.01 sshd-session: root@pts/1 (sshd-session)
root 40370 0.0 0.0 12736 8 - IWs - 0:00.00 daemon: /usr/local/bin/samplicate[40499] (daemon)
nobody 40499 0.0 0.0 12636 128 - I 08:18 0:12.91 /usr/local/bin/samplicate -s 127.0.0.1 -p 2055 127.0.0.1/2056
root 41241 0.0 0.1 19192 4720 - Ss 08:18 0:00.03 sshd: /usr/local/sbin/sshd [listener] 1 of 10-100 startups (sshd)
root 41487 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 42371 0.0 0.0 51572 8 - IWs - 0:00.00 /usr/local/bin/php-cgi
root 42518 0.0 0.0 51572 8 - IWs - 0:00.00 /usr/local/bin/php-cgi
root 43352 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 43445 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 43529 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 43816 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 43939 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 44294 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 44525 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 44761 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 44952 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 45192 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 45377 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 45455 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 45596 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 45747 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 45874 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 45953 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46012 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46160 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46187 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46224 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46240 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46300 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46355 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46477 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46492 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46518 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 46536 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 53194 0.0 0.1 19724 7084 - Is 02:42 0:00.02 sshd-session: root [priv] (sshd-session)
root 54037 0.0 0.1 19724 7380 - S 02:42 0:00.03 sshd-session: root@pts/0 (sshd-session)
root 60008 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 60291 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 60440 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 73267 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 73903 0.0 0.0 12724 8 - IWs - 0:00.00 flowd: monitor (flowd)
_flowd 73973 0.0 0.0 12724 1772 - Is 08:18 0:21.56 flowd: net (flowd)
root 78678 0.0 0.0 14452 2112 - S 08:18 0:01.66 /usr/local/sbin/lighttpd -f /var/etc/lighttpd-acme-challenge.conf
root 79717 0.0 0.0 13020 2004 - Ss 08:18 3:38.38 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root 84399 0.0 0.1 51692 9436 - Ss 08:18 0:04.08 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
root 84495 0.0 0.0 51684 8 - IW - 0:00.00 (php-fpm)
root 84651 0.0 0.0 51684 8 - IW - 0:00.00 (php-fpm)
www 84731 0.0 0.0 51684 8 - IW - 0:00.00 (php-fpm)
root 86186 0.0 0.1 23732 4356 - Ss 08:18 0:06.88 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
root 87815 0.0 0.0 50512 632 - Is 08:18 0:00.14 nginx: master process /usr/local/sbin/nginx
www 87993 0.0 0.0 50512 652 - I 08:18 0:00.10 nginx: worker process (nginx)
root 95557 0.0 0.0 51572 8 - IW - 0:00.00 /usr/local/bin/php-cgi
root 30939 0.0 0.0 12756 1196 v0 Is+ 08:18 0:00.00 /usr/libexec/getty Pc ttyv0
root 31334 0.0 0.0 12756 1196 v1 Is+ 08:18 0:00.00 /usr/libexec/getty Pc ttyv1
root 31666 0.0 0.0 12756 1200 v2 Is+ 08:18 0:00.00 /usr/libexec/getty Pc ttyv2
root 32068 0.0 0.0 12756 1196 v3 Is+ 08:18 0:00.00 /usr/libexec/getty Pc ttyv3
root 32335 0.0 0.0 12756 1196 v4 Is+ 08:18 0:00.00 /usr/libexec/getty Pc ttyv4
root 32739 0.0 0.0 12756 1196 v5 Is+ 08:18 0:00.00 /usr/libexec/getty Pc ttyv5
root 32826 0.0 0.0 12756 1196 v6 Is+ 08:18 0:00.00 /usr/libexec/getty Pc ttyv6
root 33023 0.0 0.0 12756 1196 v7 Is+ 08:18 0:00.00 /usr/libexec/getty Pc ttyv7
root 54414 0.0 0.0 13284 2236 0 Is+ 02:42 0:00.01 /bin/sh /usr/local/sbin/opnsense-shell
root 39613 0.0 0.0 13284 2340 1 Ss 02:46 0:00.01 /bin/sh /usr/local/sbin/opnsense-shell
root 46467 0.0 0.0 13760 3272 1 S 02:46 0:00.01 /bin/csh
root 46647 0.0 0.0 13352 2476 1 R+ 02:46 0:00.00 ps -aux
root@sense:~ #
last pid: 29210; load averages: 0.18, 0.21, 0.18 up 0+18:30:19 02:47:58
82 processes: 1 running, 81 sleeping
CPU: 0.0% user, 0.0% nice, 0.9% system, 0.0% interrupt, 99.1% idle
Mem: 118M Active, 81M Inact, 232K Laundry, 1311M Wired, 56K Buf, 6416M Free
ARC: 364M Total, 117M MFU, 69M MRU, 1681K Anon, 2415K Header, 174M Other
82M Compressed, 318M Uncompressed, 3.89:1 Ratio
Swap: 8192M Total, 58M Used, 8133M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
8288 root 4 20 0 64M 8024K kqread 6 14:30 0.74% syslog-ng
79717 root 1 20 0 13M 2004K bpf 4 3:39 0.57% filterlog
29210 root 1 20 0 14M 3052K CPU2 2 0:00 0.33% top
16479 root 1 20 0 26M 3528K select 5 0:09 0.02% python3.11
39482 root 1 20 0 19M 7456K select 6 0:00 0.02% sshd-session
544 root 1 20 0 14M 1920K select 3 0:05 0.01% devd
16962 root 1 20 0 27M 5096K select 5 0:10 0.01% python3.11
26743 root 1 21 0 118M 92M nanslp 4 0:03 0.01% php
86186 root 1 20 0 23M 4356K select 7 0:07 0.01% ntpd
84399 root 1 20 0 50M 9436K kqread 4 0:04 0.00% php-fpm
78678 root 1 20 0 14M 2112K kqread 5 0:02 0.00% lighttpd
73973 _flowd 1 20 0 12M 1772K select 0 0:22 0.00% flowd
40499 nobody 1 20 0 12M 128K sbwait 1 0:13 0.00% samplicate
87815 root 1 20 0 49M 632K pause 1 0:00 0.00% nginx
485 root 1 20 0 12M 1560K select 5 0:00 0.00% hv_kvp_daemon
87993 www 1 20 0 49M 652K kqread 6 0:00 0.00% nginx
42518 root 1 20 0 50M 8192B wait 2 0:00 0.00% <php-cgi>
42371 root 1 20 0 50M 8192B wait 5 0:00 0.00% <php-cgi>
54037 root 1 20 0 19M 7380K select 1 0:00 0.00% sshd-session
41241 root 1 20 0 19M 4788K select 3 0:00 0.00% sshd
38435 root 1 23 0 19M 7148K select 4 0:00 0.00% sshd-session
53194 root 1 23 0 19M 7084K select 0 0:00 0.00% sshd-session
46467 root 1 20 0 13M 3308K pause 3 0:00 0.00% csh
73903 root 1 20 0 12M 8192B sbwait 5 0:00 0.00% <flowd>
39613 root 1 68 0 13M 2340K wait 3 0:00 0.00% sh
54414 root 1 26 0 13M 2236K wait 3 0:00 0.00% sh
20563 root 1 68 0 13M 2120K nanslp 5 0:00 0.00% cron
34638 www 1 20 0 50M 9704K accept 2 0:00 0.00% php-fpm
487 root 1 20 0 12M 1252K select 0 0:00 0.00% hv_vss_daemon
32826 root 1 68 0 12M 1196K ttyin 0 0:00 0.00% getty
33023 root 1 68 0 12M 1196K ttyin 1 0:00 0.00% getty
30939 root 1 68 0 12M 1196K ttyin 5 0:00 0.00% getty
29248 root 1 20 0 50M 8192B accept 5 0:00 0.00% <php-cgi>
32068 root 1 68 0 12M 1196K ttyin 6 0:00 0.00% getty
31666 root 1 68 0 12M 1200K ttyin 7 0:00 0.00% getty
32739 root 1 68 0 12M 1196K ttyin 4 0:00 0.00% getty
31334 root 1 68 0 12M 1196K ttyin 7 0:00 0.00% getty
32335 root 1 68 0 12M 1196K ttyin 2 0:00 0.00% getty
7508 root 1 20 0 50M 8192B accept 1 0:00 0.00% <php-cgi>
8196 root 1 68 0 23M 8192B wait 2 0:00 0.00% <syslog-ng>
95557 root 1 20 0 50M 8192B accept 5 0:00 0.00% <php-cgi>
26286 root 1 20 0 50M 8192B accept 4 0:00 0.00% <php-cgi>
60291 root 1 20 0 50M 8192B accept 3 0:00 0.00% <php-cgi>
34377 root 1 20 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
23853 root 1 20 0 50M 8192B accept 7 0:00 0.00% <php-cgi>
60008 root 1 20 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
60440 root 1 20 0 50M 8192B accept 5 0:00 0.00% <php-cgi>
84731 www 1 68 0 50M 8192B accept 3 0:00 0.00% <php-fpm>
73267 root 1 20 0 50M 8192B accept 6 0:00 0.00% <php-cgi>
10197 root 1 20 0 50M 8192B accept 6 0:00 0.00% <php-cgi>
84495 root 1 68 0 50M 8192B accept 3 0:00 0.00% <php-fpm>
84651 root 1 68 0 50M 8192B accept 3 0:00 0.00% <php-fpm>
37750 root 1 20 0 50M 8192B accept 4 0:00 0.00% <php-cgi>
41487 root 1 20 0 50M 8192B accept 3 0:00 0.00% <php-cgi>
40370 root 1 68 0 12M 8192B kqread 2 0:00 0.00% <daemon>
45596 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
45192 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
45455 root 1 68 0 50M 8192B accept 5 0:00 0.00% <php-cgi>
46518 root 1 68 0 50M 8192B accept 3 0:00 0.00% <php-cgi>
43445 root 1 68 0 50M 8192B accept 0 0:00 0.00% <php-cgi>
44952 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
45377 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
46300 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
46187 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
43816 root 1 68 0 50M 8192B accept 0 0:00 0.00% <php-cgi>
44761 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
46240 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
43939 root 1 68 0 50M 8192B accept 0 0:00 0.00% <php-cgi>
44525 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
43529 root 1 68 0 50M 8192B accept 1 0:00 0.00% <php-cgi>
46012 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
45874 root 1 68 0 50M 8192B accept 2 0:00 0.00% <php-cgi>
43352 root 1 68 0 50M 8192B accept 1 0:00 0.00% <php-cgi>
45953 root 1 68 0 50M 8192B accept 0 0:00 0.00% <php-cgi>
44294 root 1 68 0 50M 8192B accept 0 0:00 0.00% <php-cgi>
46224 root 1 68 0 50M 8192B accept 1 0:00 0.00% <php-cgi>
46355 root 1 68 0 50M 8192B accept 0 0:00 0.00% <php-cgi>
45747 root 1 68 0 50M 8192B accept 1 0:00 0.00% <php-cgi>
46160 root 1 68 0 50M 8192B accept 1 0:00 0.00% <php-cgi>
46492 root 1 68 0 50M 8192B accept 0 0:00 0.00% <php-cgi>
46477 root 1 68 0 50M 8192B accept 1 0:00 0.00% <php-cgi>
46536 root 1 68 0 50M 8192B accept 1 0:00 0.00% <php-cgi>
after i make a option 11 reload services i got this kind of errors, but services works again (the question is how long):
0) Logout 7) Ping host
1) Assign interfaces 8) Shell
2) Set interface IP address 9) pfTop
3) Reset the root password 10) Firewall log
4) Reset to factory defaults 11) Reload all services
5) Power off system 12) Update from console
6) Reboot system 13) Restore a backup
Enter an option: 11
Writing firmware settings: FreeBSD OPNsense
Writing trust files...done.
Scanning /usr/share/certs/untrusted for certificates...
Scanning /usr/share/certs/blacklisted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
certctl: No changes to trust store were made.
Writing trust bundles...done.
Configuring login behaviour...done.
Configuring CRON...done.
Setting timezone: Europe/Vienna
Setting hostname: sense.biricon.eu
Generating /etc/resolv.conf...done.
Generating /etc/hosts...done.
Configuring loopback interface...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring V12_Video_MGMT_LAN interface...done.
Configuring V13_Biricon_Clients interface...done.
Configuring V14_Management_LAN interface...done.
Configuring V15_AccessPoint_Guest_LAN interface...done.
Configuring V16_VoIP interface...done.
Configuring V18_Gegensprechanlage interface...done.
Configuring V20_Bernhard_Server_LAN interface...done.
Configuring V40_Biricon_Backend_Server interface...done.
Configuring V60_Werkstatt_LAN interface...done.
Configuring V110_Biricon_Server interface...done.
Configuring V123_Chia interface...done.
Configuring V999_Funkfeuer_WAN interface...done.
Configuring V1123_WireGuardAD interface...done.
Configuring V2000_BMW_Service interface...done.
Configuring V4009_NextLayer interface...done.
Configuring V4094_Transfer_LAN interface...done.
Configuring OpenVPNCustomers interface...done.
Setting up routes...done.
Setting up gateway monitor...done.
Configuring firewall.......done.
Starting DHCPv4 service...done.
Starting NTP service...done.
Configuring OpenSSH...done.
Starting Unbound DNS...done.
Starting web GUI...done.
Configuring IPsec VPN...done.
Syncing OpenVPN settings...done.
Configuring WireGuard VPN...done.
Generating RRD graphs...done.
ntopng not running?
haproxy not running? (check /var/run/haproxy.pid).
Stopping nginx.
Waiting for PIDS: 87815.
Stopping php_fpm.
Waiting for PIDS: 84399.
redis not running? (check /var/run/redis/redis.pid).
Stopping acme_http_challenge.
Waiting for PIDS: 78678.
Stopping flowd.
Waiting for PIDS: 73903 73973.
flowd_aggregate not running? (check /var/run/flowd_aggregate.pid).
monit not running? (check /var/run/monit.pid).
setup ovpns2
ngctl: send msg: No such file or directory
error ovpns2: cannot create netflow node for ovpns2
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn0_vlanxxx
setup hn2_vlanxxx [egress only]
Checking zebra.conf
2024/11/01 02:51:51 ZEBRA: [NNACN-54BDA][EC 4043309110] Disabling MPLS support (no kernel support)
OK
Starting zebra.
2024/11/01 02:51:51 ZEBRA: [NNACN-54BDA][EC 4043309110] Disabling MPLS support (no kernel support)
Checking staticd.conf
2024/11/01 02:51:51 STATIC: [PNYPZ-BCP8Y] Static Route using hn2_vlanxxx interface not installed because the interface does not exist in specified vrf
2024/11/01 02:51:51 STATIC: [RHJK1-M5FAR] static_zebra_nht_register: Failure to send nexthop 78.41.118.73/32 for 78.41.112.0/23 to zebra
2024/11/01 02:51:51 STATIC: [PNYPZ-BCP8Y] Static Route using hn2_vlanxxx interface not installed because the interface does not exist in specified vrf
2024/11/01 02:51:51 STATIC: [RHJK1-M5FAR] static_zebra_nht_register: Failure to send nexthop 78.41.118.73/32 for 78.41.118.0/23 to zebra
2024/11/01 02:51:51 STATIC: [PNYPZ-BCP8Y] Static Route using hn2_vlanxxx interface not installed because the interface does not exist in specified vrf
2024/11/01 02:51:51 STATIC: [RHJK1-M5FAR] static_zebra_nht_register: Failure to send nexthop 78.41.118.73/32 for 193.238.156.0/22 to zebra
OK
Starting staticd.
Starting CARP event handler now
Starting monit.
Starting Monit 5.34.1 daemon with http interface at /var/run/monit.sock
Starting flowd_aggregate.
Starting flowd.
rmdir: /var/etc/acme-client/home/deploy: Not a directory
rmdir: /var/etc/acme-client/home/dnsapi: Not a directory
rmdir: /var/etc/acme-client/home/notify: Not a directory
Starting acme_http_challenge.
Starting redis.
Performing sanity check on php-fpm configuration:
[01-Nov-2024 02:51:52] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful
Starting php_fpm.
sh: /usr/local/etc/rc.d/php-fpm: not found
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.
Starting haproxy.
Certificates generated /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
Starting ntopng.
md5sum: invalid option -- q
usage: md5sum [-bctwz] [files ...]
usage: grep [-abcDEFGHhIiLlmnOopqRSsUVvwxz] [-A num] [-B num] [-C num]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context=num] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
01/Nov/2024 02:52:04 [Ntop.cpp:4052] WARNING: Unable to find timezone: using UTC
01/Nov/2024 02:52:04 [Redis.cpp:171] Successfully connected to redis 127.0.0.1@0
01/Nov/2024 02:52:04 [Redis.cpp:171] Successfully connected to redis 127.0.0.1@0
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn0_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded hn2_vlanxxx
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded bridge0
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [Prefs.cpp:2592] ERROR: Too many interfaces (8): discarded lo0
01/Nov/2024 02:52:05 [Prefs.cpp:2596] ERROR: Hint: reset redis (redis-cli flushall) and then start ntopng again
01/Nov/2024 02:52:05 [NetworkInterface.cpp:3856] Cleanup interface dummy
01/Nov/2024 02:52:05 [Ntop.cpp:2642] Parent process is exiting (this is normal)
The log stops here and does not go back to the console menu, only a CTRL+C will return to the menu.
The Webconsole is working again, but the Dashboard does not load widgets with data. ,,Failed to load widget"
The Routing stuff has a interface ... but the errros from nexthop and redis-cli flushall is not clear for me and why the sense did result in crashing without big activity.
This is the type of update that you wanna performe a clean install to avoid headaches, or you will spend endless time dealing with problems because of the major release.
IIRC, OPNSense team was preparing a "how-to" to make the upgrade, that alone was a clear "there are a lot of things to go wrong, fresh install it instead" message!!
No other major upgrade required that.
I was forced to fresh install it and it has been flawless but I did notice that some firewall stuff were not migrated over.
Still, zero problems.
It has been almost 2 weeks since you posted this and you are still having problems.
Backup the config and fresh install it!!
What kind of stuff is not migrated or part of the backup file?
[ironic on]i test now the config file on a new installation and i get a very helpful error [/ironic off]
wtf :(
Right now the Opnsense running on a ,,old" Sophos SG 310v2 (i3 6100, 12GB HDD, 250GB SATA SSD, LAG via 2 SFP+ for internal VLAN stuff - before LAG was done via Windows HyperV host) and we will see if this works better than a hyper-v VM. I think in a few days i know if its crashes again or not.
after ugprading to the SG310v2 with Opnsense on it - everything works without Unbound DNS Query Forwarding.
Unbound DNS does not forward any domain to the specified DNS Servers. But on the HYPER-V it works without problem. Did anyone have a idea to this problem?
Yup, I got a few of those while trying to import my backup into a fresh install.
It does mention what it doesn't understand so I had to load the file again and remove the offending settings for the restore to work.
From memory, Firewall Groups was one it refused to import.
Hopefully you can get your setup back on track now.