Messages

Benyamin,

I figured out my issue after the last update dropped (I have been busy).

I did what you suggested and read through my logs (there was no point in posting them, just showed a lot of connection event ... more than 5).

I had IPV6 set up with xfinity and it was sorta working throughout 22.1 / 22.7.

Once 23.1 hit something changed (other people have had issues with IPV6).

On boot this was causing the link states up and down.

My VPN provider allows 5 connections, I was hitting this with all the link up / downs and then it was saying fatal error busy.

Once I turned everything relating to IPV6 off everything works correctly / comes up at boot.

Cheers,

Have you seen this thread.

https://forum.opnsense.org/index.php?topic=26038.msg125602#msg125602

I just lurk here, and have a regular cable (1Gbit) setup.

There seem to be professionals who have gotten close to what you are going for.

Is mellanox supported by FreeBSD?

You might need to use a VM (again I have read a lot but never done what you are trying to do).

Cheers,

Did you follow this guide?

https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/

Cheers,

dmesg

Code Select Expand

ovpnc1: link state changed to DOWN
ovpnc1: link state changed to UP
ovpnc2: link state changed to DOWN
ovpnc2: link state changed to UP
ovpnc1: link state changed to DOWN
ovpnc1: link state changed to UP
ovpnc2: link state changed to DOWN
ovpnc2: link state changed to UP
ovpnc1: link state changed to DOWN
ovpnc2: link state changed to DOWN
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 0 0 done
All buffers synced.
Uptime: 11h54m41s
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p7 stable/23.1-n250411-85724e9ce22 SMP amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(efifb): resolution 800x600
CPU: Intel(R) Core(TM) i7-10810U CPU @ 1.10GHz (1600.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0xa0660  Family=0x6  Model=0xa6  Stepping=0
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x2b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 34358689792 (32767 MB)
avail memory = 33209929728 (31671 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 11 3 5 7 9 10 4 8 6 2
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x9b70f000-0x9b70f01e
smbios0: Version: 3.2, BCD Revision: 3.2
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 350
Event timer "HPET1" frequency 24000000 Hz quality 340
Event timer "HPET2" frequency 24000000 Hz quality 340
Event timer "HPET3" frequency 24000000 Hz quality 340
Event timer "HPET4" frequency 24000000 Hz quality 340
Event timer "HPET5" frequency 24000000 Hz quality 340
Event timer "HPET6" frequency 24000000 Hz quality 340
Event timer "HPET7" frequency 24000000 Hz quality 340
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x3000-0x303f mem 0xb0000000-0xb0ffffff,0xa0000000-0xafffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xb2100000-0xb210ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <AHCI SATA controller> port 0x3090-0x3097,0x3080-0x3083,0x3060-0x307f mem 0xb2114000-0xb2115fff,0xb211b000-0xb211b0ff,0xb211a000-0xb211a7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pci0: <serial bus> at device 25.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 28.5 on pci0
pci2: <ACPI PCI bus> on pcib2
igc0: <Intel(R) Ethernet Controller I225-V> mem 0xb1f00000-0xb1ffffff,0xb2000000-0xb2003fff irq 17 at device 0.0 on pci2
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 4 RX queues 4 TX queues
igc0: Using MSI-X interrupts with 5 vectors
igc0: Ethernet address: 20:7c:14:a2:62:f8
igc0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib3: <ACPI PCI-PCI bridge> at device 28.6 on pci0
pci3: <ACPI PCI bus> on pcib3
igc1: <Intel(R) Ethernet Controller I225-V> mem 0xb1d00000-0xb1dfffff,0xb1e00000-0xb1e03fff irq 18 at device 0.0 on pci3
igc1: Using 1024 TX descriptors and 1024 RX descriptors
igc1: Using 4 RX queues 4 TX queues
igc1: Using MSI-X interrupts with 5 vectors
igc1: Ethernet address: 20:7c:14:a2:62:f9
igc1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib4: <ACPI PCI-PCI bridge> at device 28.7 on pci0
pci4: <ACPI PCI bus> on pcib4
igc2: <Intel(R) Ethernet Controller I225-V> mem 0xb1b00000-0xb1bfffff,0xb1c00000-0xb1c03fff irq 19 at device 0.0 on pci4
igc2: Using 1024 TX descriptors and 1024 RX descriptors
igc2: Using 4 RX queues 4 TX queues
igc2: Using MSI-X interrupts with 5 vectors
igc2: Ethernet address: 20:7c:14:a2:62:fa
igc2: netmap queues/slots: TX 4/1024, RX 4/1024
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 29.0 on pci0
pci5: <ACPI PCI bus> on pcib5
igc3: <Intel(R) Ethernet Controller I225-V> mem 0xb1900000-0xb19fffff,0xb1a00000-0xb1a03fff irq 16 at device 0.0 on pci5
igc3: Using 1024 TX descriptors and 1024 RX descriptors
igc3: Using 4 RX queues 4 TX queues
igc3: Using MSI-X interrupts with 5 vectors
igc3: Ethernet address: 20:7c:14:a2:62:fb
igc3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib6: <ACPI PCI-PCI bridge> irq 16 at device 29.4 on pci0
pci6: <ACPI PCI bus> on pcib6
igc4: <Intel(R) Ethernet Controller I225-V> mem 0xb1700000-0xb17fffff,0xb1800000-0xb1803fff irq 16 at device 0.0 on pci6
igc4: Using 1024 TX descriptors and 1024 RX descriptors
igc4: Using 4 RX queues 4 TX queues
igc4: Using MSI-X interrupts with 5 vectors
igc4: Ethernet address: 20:7c:14:a2:62:fc
igc4: netmap queues/slots: TX 4/1024, RX 4/1024
pcib7: <ACPI PCI-PCI bridge> irq 17 at device 29.5 on pci0
pci7: <ACPI PCI bus> on pcib7
igc5: <Intel(R) Ethernet Controller I225-V> mem 0xb1500000-0xb15fffff,0xb1600000-0xb1603fff irq 17 at device 0.0 on pci7
igc5: Using 1024 TX descriptors and 1024 RX descriptors
igc5: Using 4 RX queues 4 TX queues
igc5: Using MSI-X interrupts with 5 vectors
igc5: Ethernet address: 20:7c:14:a2:62:fd
igc5: netmap queues/slots: TX 4/1024, RX 4/1024
pcib8: <ACPI PCI-PCI bridge> irq 18 at device 29.6 on pci0
pci8: <ACPI PCI bus> on pcib8
igc6: <Intel(R) Ethernet Controller I225-V> mem 0xb1300000-0xb13fffff,0xb1400000-0xb1403fff irq 18 at device 0.0 on pci8
igc6: Using 1024 TX descriptors and 1024 RX descriptors
igc6: Using 4 RX queues 4 TX queues
igc6: Using MSI-X interrupts with 5 vectors
igc6: Ethernet address: 20:7c:14:a2:62:fe
igc6: netmap queues/slots: TX 4/1024, RX 4/1024
pcib9: <ACPI PCI-PCI bridge> irq 19 at device 29.7 on pci0
pci9: <ACPI PCI bus> on pcib9
igc7: <Intel(R) Ethernet Controller I225-V> mem 0xb1100000-0xb11fffff,0xb1200000-0xb1203fff irq 19 at device 0.0 on pci9
igc7: Using 1024 TX descriptors and 1024 RX descriptors
igc7: Using 4 RX queues 4 TX queues
igc7: Using MSI-X interrupts with 5 vectors
igc7: Ethernet address: 20:7c:14:a2:62:ff
igc7: netmap queues/slots: TX 4/1024, RX 4/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
hdac0: <Intel Comet Lake-LP HDA Controller> mem 0xb2110000-0xb2113fff,0xb1000000-0xb10fffff irq 16 at device 31.3 on pci0
pci0: <serial bus> at device 31.5 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16950 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16950 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16950 or compatible> port 0x3e8-0x3ef irq 7 on acpi0
uart3: <16950 or compatible> port 0x2e8-0x2ef irq 7 on acpi0
uart4: <16950 or compatible> port 0x2f0-0x2f7 irq 7 on acpi0
uart5: <16950 or compatible> port 0x2e0-0x2e7 irq 7 on acpi0
acpi_syscontainer0: <System Container> on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
hwpstate_intel4: <Intel Speed Shift> on cpu4
hwpstate_intel5: <Intel Speed Shift> on cpu5
hwpstate_intel6: <Intel Speed Shift> on cpu6
hwpstate_intel7: <Intel Speed Shift> on cpu7
hwpstate_intel8: <Intel Speed Shift> on cpu8
hwpstate_intel9: <Intel Speed Shift> on cpu9
hwpstate_intel10: <Intel Speed Shift> on cpu10
hwpstate_intel11: <Intel Speed Shift> on cpu11
Timecounter "TSC" frequency 1607999430 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
hdacc0: <Intel Kaby Lake HDA CODEC> at cad 2 on hdac0
hdaa0: <Intel Kaby Lake Audio Function Group> at nid 1 on hdacc0
pcm0: <Intel Kaby Lake (HDMI/DP 8ch)> at nid 3 on hdaa0
Trying to mount root from zfs:zroot/ROOT/Current []...
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <KINGSTON SKC600MS512G S4800105> ACS-3 ATA SATA 3.x device
ada0: Serial Number 50026B77846F91A7
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 488386MB (1000215216 512 byte sectors)
uhub0: 18 ports with 18 removable, self powered
igc0: link state changed to UP
igc7: link state changed to UP
pchtherm0: <CometLake-LP Thermal Subsystem> mem 0xb211e000-0xb211efff irq 16 at device 18.0 on pci0
ig4iic0: <Intel Comet Lake-LP I2C Controller-4> at device 25.0 on pci0
ig4iic0: Using MSI
iicbus0: <Philips I2C bus (ACPI-hinted)> on ig4iic0
ichsmb0: <Intel Comet Lake SMBus controller> port 0xefa0-0xefbf mem 0xb2118000-0xb21180ff irq 16 at device 31.4 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: cannot find EC device
acpi_wmi0: Embedded MOF found
ACPI: \134_SB.WFDE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20201113/nsarguments-361)
acpi_wmi1: <ACPI-WMI mapping> on acpi0
acpi_wmi1: cannot find EC device
acpi_wmi1: Embedded MOF found
ACPI: \134_SB.WFTE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20201113/nsarguments-361)
lo0: link state changed to UP
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
pflog0: permanently promiscuous mode enabled
igc0: link state changed to DOWN
vlan0: changing name to 'igc0_vlan38'
vlan1: changing name to 'igc0_vlan51'
tun1: changing name to 'ovpnc1'
tun2: changing name to 'ovpnc2'
igc7: link state changed to DOWN
igc7: link state changed to UP
ovpnc1: link state changed to UP
ovpnc2: link state changed to UP
WARNING: attempt to domain_add(netgraph) after domainfinalize()
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded
ovpnc1: link state changed to DOWN
ovpnc2: link state changed to DOWN
ovpnc1: link state changed to UP
ovpnc2: link state changed to UP
igc0: link state changed to UP
igc0_vlan51: link state changed to UP
igc0_vlan38: link state changed to UP

So this time before I rebooted OPNsense, I manually stopped both VPN clients ...not sure why they keep cycling up and down in the above dmesg printout.

As you can see everything seems to be syncing.

OpenVPN advanced client options

Code Select Expand

persist-key
persist-tun
auth-nocache
fast-io
explicit-exit-notify 5
push-peer-info
remote-cert-tls server
server-poll-timeout 10
key-direction 1
sndbuf 393216
rcvbuf 393216
push "sndbuf 393216"
push "rcvbuf 393216"
reneg-sec 3600
replay-window 64 [15]

result of the socat command you suggested

Code Select Expand

cho hold | socat - unix-connect:/var/etc/openvpn/client1.sock                                                                                                         ─╯
>INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info
SUCCESS: hold=0

The way I am restarting the client is through an OPNSense facility (it has worked for months, with the occasional manual kill needed)

I have a cronjob:



After reading the following:
https://forum.opnsense.org/index.php?topic=16167.msg74095#msg74095
and
https://docs.opnsense.org/development/backend/configd.html

I came up with this

Code Select Expand


╭─XX│ X /usr/local/opnsense/service/conf/actions.d X                                                                      X ✔ │ with XXXX@opnsense │ at 11:05:28 AM XX─╮
╰─ cat actions_airvpnone.conf                                                                                                                                             ─╯
[restart]
command:/usr/local/sbin/pluginctl -s openvpn restart 1
parameters:
type:script
message:reloading AirVPNOne
description: Reload AirVPNOne (opt2)


Do you know of any slick way to divide the OpenVPN log?

I am decent with *nix and terminals, but I use decent loosely (I am very good at messing things up, not so good at bash / scripting).

If not I will save it somewhere and use a notepad type program to divide it.

Cheers,

Thanks for the ideas.

FYI I looked at dmesg and didn't see anything glaringly wrong.

I gave your theory a shot.

Code Select Expand

sudo opnsense-shell reboot                                                                                                                                             ─╯
The system will reboot. Do you want to proceed? [y/N]: y

>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
Stopping crowdsec_firewall.
Waiting for PIDS: 57894.
Stopping crowdsec.
Waiting for PIDS: 8413.
Stopping vnstat.
Waiting for PIDS: 29581.
Stopping netdata.
Waiting for PIDS: 80448 81906, 80448 81906, 80448 81906, 80448 81906, 80448 81906, 80448 81906, 80448 81906, 80448 81906.
Stopping suricata.
Waiting for PIDS: 18882.
Stopping flowd.
Waiting for PIDS: 34208 35086.
Stopping mdns_repeater.
Waiting for PIDS: 81454.
Stopping flowd_aggregate...done
Stopping monit.
Waiting for PIDS: 77963.
Network UPS Tools upsmon 2.8.0
nut not running? (check /var/db/nut/upsd.pid).
crowdsec not running? (check /var/run/crowdsec.pid).
crowdsec_firewall not running? (check /var/run/crowdsec_firewall.pid).
>>> Invoking stop script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
>>> Invoking backup script 'rrd'
>>> Invoking stop script 'config'
shutdown: [pid 11227]
Shutdown NOW!

*** FINAL System shutdown message from XXXX@opnsense.XXXXX.org ***

System going down IMMEDIATELY

Once I log back in the GUI same thing happened.

Code Select Expand

sudo ps auxww | grep openvpn                                                                                                                                           ─╯
Password:
XXXX   87188    0.4  0.0  12828   2452  2  S+   22:29    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox openvpn
root    33848    0.0  0.0  17932   7200  -  Ss   22:23    0:00.06 /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
root    70723    0.0  0.0  17932   7200  -  Ss   22:23    0:01.36 /usr/local/sbin/openvpn --config /var/etc/openvpn/client1.conf

I have kill these PIDs to restart them.

Previously something similar would happen once a week or two (been happening since 21.7 / 22.1).

I have my clients set to reconnect every so many hours so that my VPN connections go to different servers.

Sometimes the connection fails to cycle, the client is still up but shows not connected, and holds onto the PID, which blocks the GUI from restarting / stopping / starting the client in question.

This is the first time it happens on (every) reboot (I have been running openvpn on opnsense since version 19 / 20).

The connections are coming up but show down and are unable to be affected by the GUI until killed from the shell.

Cheers,

Hello,

After the latest update, following a reboot, I need to manually restart my OpenVPN clients to get the connected.

This cannot be done without doing the following steps.

I first must execute

Code Select Expand

sudo ps auxww | grep openvpn
To get the OpenVPN PIDS (The system tries to start my clients but fails and these PIDS block manual start).  Then I need to kill the PIDS in question.

Code Select Expand

sudo kill PID
After I do this i can manually start the clients from the GUI.

Does anyone have any insight on how I can fix this?

Thanks in advance,

Everything is denied unless explicitly allowed above the deny all rule (at bottom).

The rules are evaluated top to bottom.

In my rather painful experience those automatic rules should not be messed with ... if you do (and you are new at this) I guarantee you will be making a post asking how to get in when you are locked out of your box (assuming you are using the gui).

TLDR: If you put a deny all rule above any of the other rules ... it will be denied / never evaluated.

Cheers,

I fully defer to what pmhausen says on this.

A lot of my limited understanding on lagg (lacp) is due to reading his and some of the other professional types posts on the subject.

The overheads I spoke of were minimal I believe around or slightly less than 5 percent (I think I read this on the pfsense forums while researching the subject).

Cheers,

I ran a lagg (lacp) for a little while.

There were no real gains to be had, it added complexity and I noticed interface errors (they were not detrimental to performance).

There is often a misunderstanding on how lagg (lacp works).

It just makes the pipe bigger.  Your stream (I am calling a stream the data connection between a single computer and the switch / router ) is not split between the 2 ports if you are fully saturating one of the 2.5 GB ports.

If your network (more than one computer / device) has the bandwidth to fully saturate one 2.5G port then you might see a benefit, but if you have all 2.5 GB ports I'm assuming your modem connection is 2.5 as well so then the bottleneck will be the modem.

I do run a lagg (lacp) on my NAS and it "should" allow me to use both ports if multiple devices are accessing it at once (who knows if it actually benefits me; the NAS has multiple ports).

The consensus on these forums seems to be that it adds complexity for no real gains.

However if you are actually fully saturating the 2.5G port between your switch and your router and you are doing this with more than one device (multiple streams - this may be incorrect terminology btw) you might gain a benefit from it.

From what I understand there are overheads, so lagging two 1GBE ports does not give you 2G bandwidth.

Cheers,

Maybe try setting up your firewall rules like this:

https://digiex.net/threads/pfsense-step-by-step-guide-to-multiple-xbox-ones-open-nat-play-together-2-3-x.15094/

From a high level they seems to be doing something similar to what you are doing but using firewalls with upnp.

Sorry I can't be of more help, I copied some firewall rules from a pfsense post previously to get COD game to be open, but that's it.

I doubt that you can just patch the code for opnsense with whatever they have on pfsense (everything has changed to much I believe).

Cheers,

I used the same config (with modifications).

I made mine so it only affects a certain IP range.

I didn't use the route lines in the advanced box (never had that), and I have the don't pull routes box unchecked and the don't add remove routes box checked.

Please link the guide or give details on your set up.

I have been using the same OpenVPN set up since 20.1 and everything is working fine for the most part.

Do you have a rule making sure that all DNS queries are forwarded to the router?  You said your phones behave weirdly.  I think I have read that both apple devices and android devices will use their own preferred DNS servers at times.

I have made rules to force everything to the router, the only devices that circumvent this use DNS overs HTTPS, and according to Sensei everything is working for the most part.

From the looks of it, you are definitely using Unbound's recursive functionality.  You could try putting Cloudflare's servers in the System:Settings:General area.  If I remember correctly, that is how you set Unbound into forward mode, but I'm not sure if it works with DNS over TLS.

Your numbers are better than mine and I generally do not notice anything unless the statistics approach 0.6 to 1.0, so I am confused as well.

Do you notice any slowdown over something wired into the switch?  It could be something to do with your WIFI.  Still the fact that resetting Unbound fixes it is suspect, but I think resetting Unbound resets the whole network (watch logs it definitely resets a lot of stuff on my network).  There is a chance it is something else and resetting Unbound fixes it in a way different than what you think (this would be way beyond my understanding though).

I am by no means an expert, but I have fiddled with Unbound a lot, and lately I never have problems unless I turn on receive side scaling.

Here are some links:

This explains some of the options the gui gives
https://nlnetlabs.nl/documentation/unbound/unbound.conf/
optimization from unbound project people
https://nlnetlabs.nl/documentation/unbound/howto-optimise/
This guys explains a lot of settings
https://calomel.org/freebsd_network_tuning.html
Check tools
https://www.dnsperf.com/

Cheers,

What kind of hardware are you running Opnsense on processor / ramwise?

You have confirmed that everything is unchecked / no DNS servers are set in Networking part of System: Settings: General?

Please provide what the statistics tab is showing in Services:Unbound.

To troubleshoot this I recommend turning off the blocklist.  From reading these forums I have noticed that that functionality can add problems.

I do not use blocklist, have unbound set up to do dns over tls (using cloudflare), and resolve recursively.

These are my statistics (i7-7500u with 16gigs of ram):

Code Select Expand

Thread 0
Recursion time (average): 0.079423
Recursion time (median): 0.0789569
TCP usage: 0
IP ratelimited queries: 0
Recursive replies: 1329
Cache misses: 1329
Cache hits: 1741
Zero TTL: undefined
Prefetch: 124
Queries: 3070
Thread 1
Recursion time (average): 0.083162
Recursion time (median): 0.0833995
TCP usage: 0
IP ratelimited queries: 0
Recursive replies: 1376
Cache misses: 1376
Cache hits: 1762
Zero TTL: undefined
Prefetch: 131
Queries: 3138
Thread 2
Recursion time (average): 0.084281
Recursion time (median): 0.08
TCP usage: 0
IP ratelimited queries: 0
Recursive replies: 1246
Cache misses: 1246
Cache hits: 1697
Zero TTL: undefined
Prefetch: 134
Queries: 2943
Thread 3
Recursion time (average): 0.082296
Recursion time (median): 0.0823693
TCP usage: 0
IP ratelimited queries: 0
Recursive replies: 1301
Cache misses: 1301
Cache hits: 1770
Zero TTL: undefined
Prefetch: 109
Queries: 3071
Times
Now: 1650206772.324366
Uptime: 27955.731622
Elapsed: 27955.731622
Total
Recursion time (average): 0.082267
Recursion time (median): 0.0811814
TCP usage: 0
IP ratelimited queries: 0
Recursive replies: 5252
Cache misses: 5252
Cache hits: 6970
Zero TTL: undefined
Prefetch: 498
Queries: 12222


Cheers,

You can re-add the custom options by using mimugmail's repo https://www.routerperformance.net/opnsense-repo/

Or you can set it up the correct way... https://docs.opnsense.org/manual/unbound.html...Go to advanced configuration.

Cheers,