[solved]23.1.11 -> 23.7.4 openvpn not working and firmware crash detected

[Adures]

For now I had to rollback to 23.1.11 as 23.7.2 seems to have 2 problems (not sure if they are related in any way).

I constantly get crash reports. This opnsense instance is virutalized on xcp-ng host.

Code Select Expand


System Information:

User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0
FreeBSD 13.2-RELEASE-p2 stable/23.7-n254761-4b4f06e3731 SMP amd64
OPNsense 23.7.2 81a9dcc9c
Plugins os-ddclient-1.15 os-dyndns-1.27_3 os-sunnyvalley-1.2_3 os-theme-cicada-1.34 os-theme-rebellion-1.8.8 os-vnstat-1.3_1 os-wireguard-1.13_7 os-xen-1.2_1
Time Tue, 29 Aug 2023 23:16:51 +0200
OpenSSL 1.1.1v  1 Aug 2023
Python 3.9.17
PHP 8.2.9

PHP Errors (there are a lot of the same errors like this):

[29-Aug-2023 20:58:10 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20220829/mongodb.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so"), /usr/local/lib/php/20220829/mongodb.so.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so.so")) in Unknown on line 0
[29-Aug-2023 20:58:13 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20220829/mongodb.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so"), /usr/local/lib/php/20220829/mongodb.so.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so.so")) in Unknown on line 0
[29-Aug-2023 20:58:13 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20220829/mongodb.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so"), /usr/local/lib/php/20220829/mongodb.so.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so.so")) in Unknown on line 0
[29-Aug-2023 20:58:13 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20220829/mongodb.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so"), /usr/local/lib/php/20220829/mongodb.so.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so.so")) in Unknown on line 0
[29-Aug-2023 20:58:13 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'mongodb.so' (tried: /usr/local/lib/php/20220829/mongodb.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so"), /usr/local/lib/php/20220829/mongodb.so.so (Cannot open "/usr/local/lib/php/20220829/mongodb.so.so")) in Unknown on line 0



dmesg.boot:

Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.2-RELEASE-p2 stable/23.7-n254761-4b4f06e3731 SMP amd64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
VT(vga): text 80x25
XEN: Hypervisor version 4.13 detected.
CPU: AMD Ryzen 7 PRO 4750G with Radeon Graphics (3600.13-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x860f01  Family=0x17  Model=0x60  Stepping=1
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x40001f1<LAHF,CR8,ABM,SSE4A,MAS,Prefetch,DBE>
  Structured Extended Features=0x219c01a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA>
  Structured Extended Features2=0x400004<UMIP,RDPID>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x10cd005<CLZERO,XSaveErPtr,IBPB,IBRS,STIBP,PREFER_IBRS,SSBD>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 6438256640 (6140 MB)
avail memory = 6203113472 (5915 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 4 package(s) x 1 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0: MADT APIC ID 1 != hw id 0
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47
Launching APs: 1 3 2
wlan: mac acl policy registered
random: entropy device external interface
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
vtvga0: <VT VGA driver>
smbios0: <System Management BIOS> at iomem 0xfb610-0xfb62e
smbios0: Version: 2.4, BCD Revision: 2.4
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <Xen>
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc400-0xc40f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
uhci0: <Intel 82371SB (PIIX3) USB controller> port 0x1000-0x101f irq 23 at device 1.2 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0
vgapci0: Boot video device
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
xenpv0: <Xen PV bus>
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xen_et0: registered as a time-of-day clock, resolution 0.000001s
xenstore0: <XenStore> on xenpv0
xsd_dev0: <Xenstored user-space device> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
gntdev0: <Xen grant-table user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
fdc0: No FDOUT register!
Timecounters tick every 10.000 msec
xenballoon0: <Xen Balloon Device> on xenstore0
ugen0.1: <Intel UHCI root HUB> at usbus0
xctrl0: <Xen Control Device> on xenstore0
uhub0 on usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: Ethernet address: removed_by_author
xn1: <Virtual Network Interface> at device/vif/1 on xenbusb_front0
xn1: Ethernet address: removed_by_author
xn0: backend features:xn2: <Virtual Network Interface> at device/vif/2 feature-sg on xenbusb_front0
feature-gso-tcp4
xn2: Ethernet address: removed_by_author

xenbusb_back0: <Xen Backend Devices> on xenstore0
xbd0: 102400MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
xbd0: features: write_barrier
xbd0: synchronize cache commands enabled.
xn1: backend features: feature-sg feature-gso-tcp4
xn2: backend features: feature-sg feature-gso-tcp4
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
cd0 at ata1 bus 0 scbus1 target 1 lun 0
cd0: <QEMU QEMU DVD-ROM 0.10> Removable CD-ROM SCSI device
cd0: Serial Number QM00004
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present


Additionally openvpn is failing with errors (attached). This may be because interface is showing up as down (even though it is enabled in interfaces settings and is working fine on 23.1.11).

Does anyone have any idea why these crushes are happening and how can I pinpoint why openvpn interface is showing as down after upgrade?

[newsense]

It is unclear what the issue with OpenVPN is, not much to look at there.

23.7.3 should be out in a matter of hours and it includes OpenVPN fixes as well, so it would be best to upgrade first and then continue the thread with more verbose information if still required.


This should help for the mongodb issue

Code Select Expand

pkg remove php74-pecl-mongod

[franco]

More likely:

# pkg remove php81-pecl-mongodb


Cheers,
Franco

[Adures]

Hi, thank you for taking the time to help me with this issue.

I updated to 23.7.3. Openvpn and craching issues were still there.
Using command:
# pkg remove php81-pecl-mongodb

fixed the issue with crashes.

However openvpn server still has failed status. I suspect it has something to do with interface status and gateway (it shows up as online in 23.1.11)

[Adures]

Ok I decided to play around with it a little bit and removed old interface, gateway and legacy vpn server.
I created new OpenVPN instance. However the problem persist. New interface created automatically and gateway are offline.

Log files show this (attached) when trying to restart the server.

[Adures]

I found out what is causing the OpenVPN server to fail.

It's Topology setting.
By default it's set to subnet (which should be default according to https://community.openvpn.net/openvpn/wiki/Topology)
However, using this setting, instantly changes opnvpn status to "failed"

Setting the Topology setting to net30 or p2p changes the server status to Ok and allows connection from client.

Anyone knows why subnet setting is causing this issue?

[Adures]

Ok I updated to 23.7.4
The opnvpn server still crashes when I set topology as "subnet"
I decided to leave it as net30.

The problem is now  cannot figure out why the firewall blocks the connection. The connection from openvpn network to LAN works fine, however traffic from LAN to openvpn network is blocked (I have default allow LAN to any rule).
I attached files with this.
From what I undestand this is the autorule that is blocking me? After checking ip addr on my openvpn client I get:

Code Select Expand

inet 10.28.30.6 peer 10.28.30.5/32 scope global noprefixroute tun0 valid_lft forever prreferred_lft forever

I suspect this peer 10.28.30.5/32 is causing all the issues, but not sure how to change it.

[Adures]

Ok, going to net30 was not the solution.

I came back to the drawing board and tried to figure out why topology subnet is failing.

Unfortunately I do not understand why, but changing the IPv4 network from 10.28.30.0/24 to 10.28.30.0/26 fixed the issue.