Hello,
After the latest update, following a reboot, I need to manually restart my OpenVPN clients to get the connected.
This cannot be done without doing the following steps.
I first must execute
sudo ps auxww | grep openvpn
To get the OpenVPN PIDS (The system tries to start my clients but fails and these PIDS block manual start). Then I need to kill the PIDS in question.
sudo kill PID
After I do this i can manually start the clients from the GUI.
Does anyone have any insight on how I can fix this?
Thanks in advance,
I'm running 23.1.5_4-amd64 and just rebooted but all my clients came up straight away.
Are you positive you are getting a clean shutdown? Perhaps reboot again at the console (if you have one) and check if that's the case...
Thanks for the ideas.
FYI I looked at dmesg and didn't see anything glaringly wrong.
I gave your theory a shot.
sudo opnsense-shell reboot ─╯
The system will reboot. Do you want to proceed? [y/N]: y
>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
Stopping crowdsec_firewall.
Waiting for PIDS: 57894.
Stopping crowdsec.
Waiting for PIDS: 8413.
Stopping vnstat.
Waiting for PIDS: 29581.
Stopping netdata.
Waiting for PIDS: 80448 81906, 80448 81906, 80448 81906, 80448 81906, 80448 81906, 80448 81906, 80448 81906, 80448 81906.
Stopping suricata.
Waiting for PIDS: 18882.
Stopping flowd.
Waiting for PIDS: 34208 35086.
Stopping mdns_repeater.
Waiting for PIDS: 81454.
Stopping flowd_aggregate...done
Stopping monit.
Waiting for PIDS: 77963.
Network UPS Tools upsmon 2.8.0
nut not running? (check /var/db/nut/upsd.pid).
crowdsec not running? (check /var/run/crowdsec.pid).
crowdsec_firewall not running? (check /var/run/crowdsec_firewall.pid).
>>> Invoking stop script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
>>> Invoking backup script 'rrd'
>>> Invoking stop script 'config'
shutdown: [pid 11227]
Shutdown NOW!
*** FINAL System shutdown message from XXXX@opnsense.XXXXX.org ***
System going down IMMEDIATELY
Once I log back in the GUI same thing happened.
sudo ps auxww | grep openvpn ─╯
Password:
XXXX 87188 0.4 0.0 12828 2452 2 S+ 22:29 0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox openvpn
root 33848 0.0 0.0 17932 7200 - Ss 22:23 0:00.06 /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
root 70723 0.0 0.0 17932 7200 - Ss 22:23 0:01.36 /usr/local/sbin/openvpn --config /var/etc/openvpn/client1.conf
I have kill these PIDs to restart them.
Previously something similar would happen once a week or two (been happening since 21.7 / 22.1).
I have my clients set to reconnect every so many hours so that my VPN connections go to different servers.
Sometimes the connection fails to cycle, the client is still up but shows not connected, and holds onto the PID, which blocks the GUI from restarting / stopping / starting the client in question.
This is the first time it happens on (every) reboot (I have been running openvpn on opnsense since version 19 / 20).
The connections are coming up but show down and are unable to be affected by the GUI until killed from the shell.
Cheers,
From dmesg you want to check you get something similar to:
...
Syncing disks, vnodes remaining... 8 0 0 done
All buffers synced.
Uptime: 2d14h58m34s
...
Rebooting...
That way you know it was a clean sync to disk (the 0 0 done and All buffers synced).
The following command might reveal something:
dmesg | grep ovpnc
Link state changes and name changes should be the only lines returned.
You might need to share your OpenVPN logs to make headway... Maybe split /var/log/openvpn/latest.log into two at the reboot epoch and have a look either side. Methinks it would be a fair bit of redaction to share here and then we might miss some detail such as IP range overlap, routing table conflicts, etc.
Given that you are forcing the clients to regularly reconnect, it's probably worthwhile asking how you are doing this, e.g. are you restarting the OpenVPN service for each client, issuing a client-kill command via the management interface whilst using the explicit-exit-notify 2 configuration directive, or some other mechanism.
Also did you ever issue a hold on command via the management interface? Or do you have the --management-hold directive in your configuration? You may want to check with something like:
echo hold | socat - unix-connect:/var/etc/openvpn/client1.sock
You will likely need to install socat first. You might also want to check this following a reboot and before killing the PIDs...
If you get a hold=1 response, try issuing a hold release command to see if it starts and then a hold off command to help it persist reboots.
If present, the removal of the --management-hold directive would be important.
dmesg
ovpnc1: link state changed to DOWN
ovpnc1: link state changed to UP
ovpnc2: link state changed to DOWN
ovpnc2: link state changed to UP
ovpnc1: link state changed to DOWN
ovpnc1: link state changed to UP
ovpnc2: link state changed to DOWN
ovpnc2: link state changed to UP
ovpnc1: link state changed to DOWN
ovpnc2: link state changed to DOWN
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 0 0 done
All buffers synced.
Uptime: 11h54m41s
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p7 stable/23.1-n250411-85724e9ce22 SMP amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(efifb): resolution 800x600
CPU: Intel(R) Core(TM) i7-10810U CPU @ 1.10GHz (1600.00-MHz K8-class CPU)
Origin="GenuineIntel" Id=0xa0660 Family=0x6 Model=0xa6 Stepping=0
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
IA32_ARCH_CAPS=0x2b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics
real memory = 34358689792 (32767 MB)
avail memory = 33209929728 (31671 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 11 3 5 7 9 10 4 8 6 2
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x9b70f000-0x9b70f01e
smbios0: Version: 3.2, BCD Revision: 3.2
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 350
Event timer "HPET1" frequency 24000000 Hz quality 340
Event timer "HPET2" frequency 24000000 Hz quality 340
Event timer "HPET3" frequency 24000000 Hz quality 340
Event timer "HPET4" frequency 24000000 Hz quality 340
Event timer "HPET5" frequency 24000000 Hz quality 340
Event timer "HPET6" frequency 24000000 Hz quality 340
Event timer "HPET7" frequency 24000000 Hz quality 340
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x3000-0x303f mem 0xb0000000-0xb0ffffff,0xa0000000-0xafffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xb2100000-0xb210ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <AHCI SATA controller> port 0x3090-0x3097,0x3080-0x3083,0x3060-0x307f mem 0xb2114000-0xb2115fff,0xb211b000-0xb211b0ff,0xb211a000-0xb211a7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pci0: <serial bus> at device 25.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 28.5 on pci0
pci2: <ACPI PCI bus> on pcib2
igc0: <Intel(R) Ethernet Controller I225-V> mem 0xb1f00000-0xb1ffffff,0xb2000000-0xb2003fff irq 17 at device 0.0 on pci2
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 4 RX queues 4 TX queues
igc0: Using MSI-X interrupts with 5 vectors
igc0: Ethernet address: 20:7c:14:a2:62:f8
igc0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib3: <ACPI PCI-PCI bridge> at device 28.6 on pci0
pci3: <ACPI PCI bus> on pcib3
igc1: <Intel(R) Ethernet Controller I225-V> mem 0xb1d00000-0xb1dfffff,0xb1e00000-0xb1e03fff irq 18 at device 0.0 on pci3
igc1: Using 1024 TX descriptors and 1024 RX descriptors
igc1: Using 4 RX queues 4 TX queues
igc1: Using MSI-X interrupts with 5 vectors
igc1: Ethernet address: 20:7c:14:a2:62:f9
igc1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib4: <ACPI PCI-PCI bridge> at device 28.7 on pci0
pci4: <ACPI PCI bus> on pcib4
igc2: <Intel(R) Ethernet Controller I225-V> mem 0xb1b00000-0xb1bfffff,0xb1c00000-0xb1c03fff irq 19 at device 0.0 on pci4
igc2: Using 1024 TX descriptors and 1024 RX descriptors
igc2: Using 4 RX queues 4 TX queues
igc2: Using MSI-X interrupts with 5 vectors
igc2: Ethernet address: 20:7c:14:a2:62:fa
igc2: netmap queues/slots: TX 4/1024, RX 4/1024
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 29.0 on pci0
pci5: <ACPI PCI bus> on pcib5
igc3: <Intel(R) Ethernet Controller I225-V> mem 0xb1900000-0xb19fffff,0xb1a00000-0xb1a03fff irq 16 at device 0.0 on pci5
igc3: Using 1024 TX descriptors and 1024 RX descriptors
igc3: Using 4 RX queues 4 TX queues
igc3: Using MSI-X interrupts with 5 vectors
igc3: Ethernet address: 20:7c:14:a2:62:fb
igc3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib6: <ACPI PCI-PCI bridge> irq 16 at device 29.4 on pci0
pci6: <ACPI PCI bus> on pcib6
igc4: <Intel(R) Ethernet Controller I225-V> mem 0xb1700000-0xb17fffff,0xb1800000-0xb1803fff irq 16 at device 0.0 on pci6
igc4: Using 1024 TX descriptors and 1024 RX descriptors
igc4: Using 4 RX queues 4 TX queues
igc4: Using MSI-X interrupts with 5 vectors
igc4: Ethernet address: 20:7c:14:a2:62:fc
igc4: netmap queues/slots: TX 4/1024, RX 4/1024
pcib7: <ACPI PCI-PCI bridge> irq 17 at device 29.5 on pci0
pci7: <ACPI PCI bus> on pcib7
igc5: <Intel(R) Ethernet Controller I225-V> mem 0xb1500000-0xb15fffff,0xb1600000-0xb1603fff irq 17 at device 0.0 on pci7
igc5: Using 1024 TX descriptors and 1024 RX descriptors
igc5: Using 4 RX queues 4 TX queues
igc5: Using MSI-X interrupts with 5 vectors
igc5: Ethernet address: 20:7c:14:a2:62:fd
igc5: netmap queues/slots: TX 4/1024, RX 4/1024
pcib8: <ACPI PCI-PCI bridge> irq 18 at device 29.6 on pci0
pci8: <ACPI PCI bus> on pcib8
igc6: <Intel(R) Ethernet Controller I225-V> mem 0xb1300000-0xb13fffff,0xb1400000-0xb1403fff irq 18 at device 0.0 on pci8
igc6: Using 1024 TX descriptors and 1024 RX descriptors
igc6: Using 4 RX queues 4 TX queues
igc6: Using MSI-X interrupts with 5 vectors
igc6: Ethernet address: 20:7c:14:a2:62:fe
igc6: netmap queues/slots: TX 4/1024, RX 4/1024
pcib9: <ACPI PCI-PCI bridge> irq 19 at device 29.7 on pci0
pci9: <ACPI PCI bus> on pcib9
igc7: <Intel(R) Ethernet Controller I225-V> mem 0xb1100000-0xb11fffff,0xb1200000-0xb1203fff irq 19 at device 0.0 on pci9
igc7: Using 1024 TX descriptors and 1024 RX descriptors
igc7: Using 4 RX queues 4 TX queues
igc7: Using MSI-X interrupts with 5 vectors
igc7: Ethernet address: 20:7c:14:a2:62:ff
igc7: netmap queues/slots: TX 4/1024, RX 4/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
hdac0: <Intel Comet Lake-LP HDA Controller> mem 0xb2110000-0xb2113fff,0xb1000000-0xb10fffff irq 16 at device 31.3 on pci0
pci0: <serial bus> at device 31.5 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16950 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16950 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16950 or compatible> port 0x3e8-0x3ef irq 7 on acpi0
uart3: <16950 or compatible> port 0x2e8-0x2ef irq 7 on acpi0
uart4: <16950 or compatible> port 0x2f0-0x2f7 irq 7 on acpi0
uart5: <16950 or compatible> port 0x2e0-0x2e7 irq 7 on acpi0
acpi_syscontainer0: <System Container> on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
hwpstate_intel4: <Intel Speed Shift> on cpu4
hwpstate_intel5: <Intel Speed Shift> on cpu5
hwpstate_intel6: <Intel Speed Shift> on cpu6
hwpstate_intel7: <Intel Speed Shift> on cpu7
hwpstate_intel8: <Intel Speed Shift> on cpu8
hwpstate_intel9: <Intel Speed Shift> on cpu9
hwpstate_intel10: <Intel Speed Shift> on cpu10
hwpstate_intel11: <Intel Speed Shift> on cpu11
Timecounter "TSC" frequency 1607999430 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
hdacc0: <Intel Kaby Lake HDA CODEC> at cad 2 on hdac0
hdaa0: <Intel Kaby Lake Audio Function Group> at nid 1 on hdacc0
pcm0: <Intel Kaby Lake (HDMI/DP 8ch)> at nid 3 on hdaa0
Trying to mount root from zfs:zroot/ROOT/Current []...
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <KINGSTON SKC600MS512G S4800105> ACS-3 ATA SATA 3.x device
ada0: Serial Number 50026B77846F91A7
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 488386MB (1000215216 512 byte sectors)
uhub0: 18 ports with 18 removable, self powered
igc0: link state changed to UP
igc7: link state changed to UP
pchtherm0: <CometLake-LP Thermal Subsystem> mem 0xb211e000-0xb211efff irq 16 at device 18.0 on pci0
ig4iic0: <Intel Comet Lake-LP I2C Controller-4> at device 25.0 on pci0
ig4iic0: Using MSI
iicbus0: <Philips I2C bus (ACPI-hinted)> on ig4iic0
ichsmb0: <Intel Comet Lake SMBus controller> port 0xefa0-0xefbf mem 0xb2118000-0xb21180ff irq 16 at device 31.4 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: cannot find EC device
acpi_wmi0: Embedded MOF found
ACPI: \134_SB.WFDE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20201113/nsarguments-361)
acpi_wmi1: <ACPI-WMI mapping> on acpi0
acpi_wmi1: cannot find EC device
acpi_wmi1: Embedded MOF found
ACPI: \134_SB.WFTE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20201113/nsarguments-361)
lo0: link state changed to UP
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
pflog0: permanently promiscuous mode enabled
igc0: link state changed to DOWN
vlan0: changing name to 'igc0_vlan38'
vlan1: changing name to 'igc0_vlan51'
tun1: changing name to 'ovpnc1'
tun2: changing name to 'ovpnc2'
igc7: link state changed to DOWN
igc7: link state changed to UP
ovpnc1: link state changed to UP
ovpnc2: link state changed to UP
WARNING: attempt to domain_add(netgraph) after domainfinalize()
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded
ovpnc1: link state changed to DOWN
ovpnc2: link state changed to DOWN
ovpnc1: link state changed to UP
ovpnc2: link state changed to UP
igc0: link state changed to UP
igc0_vlan51: link state changed to UP
igc0_vlan38: link state changed to UP
So this time before I rebooted OPNsense, I manually stopped both VPN clients ...not sure why they keep cycling up and down in the above dmesg printout.
As you can see everything seems to be syncing.
OpenVPN advanced client options
persist-key
persist-tun
auth-nocache
fast-io
explicit-exit-notify 5
push-peer-info
remote-cert-tls server
server-poll-timeout 10
key-direction 1
sndbuf 393216
rcvbuf 393216
push "sndbuf 393216"
push "rcvbuf 393216"
reneg-sec 3600
replay-window 64 [15]
result of the socat command you suggested
cho hold | socat - unix-connect:/var/etc/openvpn/client1.sock ─╯
>INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info
SUCCESS: hold=0
The way I am restarting the client is through an OPNSense facility (it has worked for months, with the occasional manual kill needed)
I have a cronjob:
(https://i.imgur.com/POtHYnc.png)
After reading the following:
https://forum.opnsense.org/index.php?topic=16167.msg74095#msg74095 (https://forum.opnsense.org/index.php?topic=16167.msg74095#msg74095)
and
https://docs.opnsense.org/development/backend/configd.html (https://docs.opnsense.org/development/backend/configd.html)
I came up with this
╭─XX│ X /usr/local/opnsense/service/conf/actions.d X X ✔ │ with XXXX@opnsense │ at 11:05:28 AM XX─╮
╰─ cat actions_airvpnone.conf ─╯
[restart]
command:/usr/local/sbin/pluginctl -s openvpn restart 1
parameters:
type:script
message:reloading AirVPNOne
description: Reload AirVPNOne (opt2)
Do you know of any slick way to divide the OpenVPN log?
I am decent with *nix and terminals, but I use decent loosely (I am very good at messing things up, not so good at bash / scripting).
If not I will save it somewhere and use a notepad type program to divide it.
Cheers,
Quote from: Koldnitz on April 02, 2023, 06:14:54 PM
So this time before I rebooted OPNsense, I manually stopped both VPN clients ...not sure why they keep cycling up and down in the above dmesg printout.
Did manually stopping the clients have any impact on the problem, i.e. did they restart normally on reboot?
The cycling at the start (before rebooting) looks normal if you are restarting the clients regularly.
Startup doesn't look "normal" though. In the absence of timestamps I'm making some presumptions, but I would call that cycling atypical...
The section in question:
ovpnc1: link state changed to UP
ovpnc2: link state changed to UP
WARNING: attempt to domain_add(netgraph) after domainfinalize()
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded
ovpnc1: link state changed to DOWN
ovpnc2: link state changed to DOWN
ovpnc1: link state changed to UP
ovpnc2: link state changed to UPIf I'm not mistaken, it looks like you are loading netgraph / limiters in between the interface state changes. This could certainly be related to the problem. Could you try disabling this to see the effect on the problem?
Re client config, you may want to consider removing
perist-tun to see if rebuilding the interface helps with the problem. I would suggest
explicit-exit-notify 5 is a bit excessive, the default 1 or maybe 2 should really be sufficient. It's possible (but maybe not probable) that you are telling multiple servers that you are leaving... This setting only has an effect if you are using UDP tunnels.
It doesn't look like
management-hold is an issue and your cron jobs look fine.
Quote from: Koldnitz on April 02, 2023, 06:14:54 PM
Do you know of any slick way to divide the OpenVPN log?
I would just carve it up manually. If you have the epoch / timestamp, it should be relatively straightforward. It's really so that anyone having a look doesn't have to try and find the point of reboot as it's not always obvious. So an alternative is to write into the file at the reboot epoch and make it obvious, or post the timestamp, etc.
Also, I'd have a look yourself first before resorting to publicly posting the log.
Probably also important to try each of the suggestions one at a time to see the effect and identify potential root causes rather than doing them all at once and fixing it but not knowing why or how.
Benyamin,
I figured out my issue after the last update dropped (I have been busy).
I did what you suggested and read through my logs (there was no point in posting them, just showed a lot of connection event ... more than 5).
I had IPV6 set up with xfinity and it was sorta working throughout 22.1 / 22.7.
Once 23.1 hit something changed (other people have had issues with IPV6).
On boot this was causing the link states up and down.
My VPN provider allows 5 connections, I was hitting this with all the link up / downs and then it was saying fatal error busy.
Once I turned everything relating to IPV6 off everything works correctly / comes up at boot.
Cheers,
Glad to hear you worked it out, notwithstanding the loss of IPv6 functionality.