Topics

[suddenly all IPv6 connections get routed via the default IPv6 WAN gateway]

Hi,

recently I had to move in with a friend of mine. We both wanted to keep our LANs as close as possible to how they were prior to me moving in. So now I'm facing an admittedly overly complex and perhaps even silly network setup:

Code Select Expand

Internet-->DrayTek Vigor 167 VDSL2 Modem-->OpenWrt on NanoPi R6C-->opnSense on random AliExpress x86 mini PC(-->my LAN)
                                                               '-->AVM Fritz!Box 7490(-->roommate's LAN)
All I had to do to make this work was to:

• configure VLAN ^{(VDSL connections require VLAN tag 7 here)}, PPPoE and DHCPv6 (client) on the WAN interface and DHCPv6 (server, with static leases for predictable PD) on the LAN interface on the OpenWrt router
• disable NAT on the opnSense and Fritz!Box routers
• set up static routes for both, my roommate's and my IPv4 subnets on the OpenWrt router (to avoid double-NATting)
• do some port forwarding on the OpenWrt router

Despite the perceived weirdness of this set up, everything seems to work perfectly fine. Well, almost everything...

For sake of troubleshooting I've made a backup of my current config, reinstalled opnSense 25.1.6 and only applied the most essential settings. Most importantly, I've got two firewall rules on the LAN interface. One for IPv4 and one for IPv6. In which I've only specified the IP versions, the source addresses and the WireGuard tunnel interfaces as gateways. This policy based routing works fine, too. That is, only until the upstream router gets disconnected from the internet and re-establishes a new connection. ^{(Where I'm from it's usual for ISPs to forcefully reconnect their customers every 24 hours.)} After that it still uses the WireGuard tunnel for IPv4 connections, but suddenly all IPv6 connections get routed via the default IPv6 WAN gateway.

I have no idea why, even in theory only, this would possibly be the case and need advice how to even begin troubleshooting this.

Thanks in advance!



PS: The machine gets its IPv4 address via ISC DHCPv4 and IPv6 address via ISC DHCPv6. Static leases are set up and working. The machine gets the same IPv4 address and IPv6 suffix every time. It only ever has exactly one global scope IPv6 address. NAT is set up on only for the IPv4 and IPv6 WireGuard tunnel interfaces on the opnSense router. "Allow default gateway switching" is unchecked and "Skip rules when gateway is down" checked. All of this is complex enough as is (at least to me), so I wish to not use ULA. My roommate is using his Fritz!Box to establish a WireGuard tunnel, too, and it doesn't behave this way. This is why I assume it's an issue with (my configuration of) opnSense, hence is why I post here and not in the OpenWrt forums. 😅

[Channel]

2024-08-17 edit: A better workaround has been found by @cercle!
Set the Channel to Auto (in Interfaces -> *WIFI-Interface* -> Common wireless configuration).

2024-07-31 edit: Should anyone face the same issue, there's a workaround!
Set kern.smp.disabled to 1 (in System -> Settings -> Tunables) and reboot.
Applying changes to settings still makes the adapter go down without coming back up until rebooting.



Hi,

apart from the poor, yet for my use-case acceptable performance, I've never had issues using my trusty old Ralink adapter in hostap mode on opnSense until upgrading to 24.7.

Since then my clients can no longer find the WiFi network and applying changes to the settings cause kernel panics.

I understand FreeBSD doesn't play well with WiFi adapters, but I still think this might be some sort of regression and as such should be further investigated.  :(

System information:

Code Select Expand

User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0
FreeBSD 14.1-RELEASE-p2 stable/24.7-n267758-4ad7ad40bc77 SMP amd64
OPNsense 24.7_9 0d38c7804
Plugins os-ddclient-1.22 os-freeradius-1.9.24 os-iperf-1.0_1 os-theme-cicada-1.37
Time Wed, 31 Jul 2024 11:19:50 +0200
OpenSSL 3.0.14
Python 3.11.9
PHP 8.2.20

dmesg.boot:

Code Select Expand

---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-RELEASE-p2 stable/24.7-n267758-4ad7ad40bc77 SMP amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(vga): resolution 640x480
module run already present!
CPU microcode: updated from 0x7c to 0xf4
CPU: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz (2900.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x806e9  Family=0x6  Model=0x8e  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbb7<SSE3,PCLMULQDQ,DTES64,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x2000c04<RSBA>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8153300992 (7775 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 3 2
random: entropy device external interface
wlan: mac acl policy registered
kbd0 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.
vtvga0: <VT VGA driver>
smbios0: <System Management BIOS> at iomem 0xf05e0-0xf05fe
smbios0: Version: 2.8, BCD Revision: 2.8
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Sunrise Point-LP USB 3.0 controller> mem 0xdf810000-0xdf81ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <serial bus, USB> at device 20.1 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Sunrise Point-LP AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xdf828000-0xdf829fff,0xdf82d000-0xdf82d0ff,0xdf82c000-0xdf82c7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 3 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) I211 (Copper)> port 0xe000-0xe01f mem 0xdf700000-0xdf71ffff,0xdf720000-0xdf723fff irq 16 at device 0.0 on pci1
igb0: NVM V0.6 imgtype1
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 2 RX queues 2 TX queues
igb0: Using MSI-X interrupts with 3 vectors
igb0: Ethernet address: 40:62:31:06:c5:a7
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
igb1: <Intel(R) I211 (Copper)> port 0xd000-0xd01f mem 0xdf600000-0xdf61ffff,0xdf620000-0xdf623fff irq 17 at device 0.0 on pci2
igb1: NVM V0.6 imgtype1
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 2 RX queues 2 TX queues
igb1: Using MSI-X interrupts with 3 vectors
igb1: Ethernet address: 40:62:31:06:c5:a8
igb1: netmap queues/slots: TX 2/1024, RX 2/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
igb2: <Intel(R) I211 (Copper)> port 0xc000-0xc01f mem 0xdf500000-0xdf51ffff,0xdf520000-0xdf523fff irq 18 at device 0.0 on pci3
igb2: NVM V0.6 imgtype1
igb2: Using 1024 TX descriptors and 1024 RX descriptors
igb2: Using 2 RX queues 2 TX queues
igb2: Using MSI-X interrupts with 3 vectors
igb2: Ethernet address: 40:62:31:06:c5:a9
igb2: netmap queues/slots: TX 2/1024, RX 2/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
igb3: <Intel(R) I211 (Copper)> port 0xb000-0xb01f mem 0xdf400000-0xdf41ffff,0xdf420000-0xdf423fff irq 19 at device 0.0 on pci4
igb3: NVM V0.6 imgtype1
igb3: Using 1024 TX descriptors and 1024 RX descriptors
igb3: Using 2 RX queues 2 TX queues
igb3: Using MSI-X interrupts with 3 vectors
igb3: Ethernet address: 40:62:31:06:c5:aa
igb3: netmap queues/slots: TX 2/1024, RX 2/1024
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
igb4: <Intel(R) I211 (Copper)> port 0xa000-0xa01f mem 0xdf300000-0xdf31ffff,0xdf320000-0xdf323fff irq 16 at device 0.0 on pci5
igb4: NVM V0.6 imgtype1
igb4: Using 1024 TX descriptors and 1024 RX descriptors
igb4: Using 2 RX queues 2 TX queues
igb4: Using MSI-X interrupts with 3 vectors
igb4: Ethernet address: 40:62:31:06:c5:ab
igb4: netmap queues/slots: TX 2/1024, RX 2/1024
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
igb5: <Intel(R) I211 (Copper)> port 0x9000-0x901f mem 0xdf200000-0xdf21ffff,0xdf220000-0xdf223fff irq 17 at device 0.0 on pci6
igb5: NVM V0.6 imgtype1
igb5: Using 1024 TX descriptors and 1024 RX descriptors
igb5: Using 2 RX queues 2 TX queues
igb5: Using MSI-X interrupts with 3 vectors
igb5: Ethernet address: 40:62:31:06:c5:ac
igb5: netmap queues/slots: TX 2/1024, RX 2/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
hdac0: <Intel Kaby Lake-LP HDA Controller> mem 0xdf820000-0xdf823fff,0xdf800000-0xdf80ffff irq 16 at device 31.3 on pci0
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
ns8250: UART FCR is broken
ns8250: UART FCR is broken
uart0: <16950 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ns8250: UART FCR is broken
uart0: console (115200,n,8,1)
ns8250: UART FCR is broken
ns8250: UART FCR is broken
uart1: <16950 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
Timecounter "TSC-low" frequency 1451999960 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
hdacc0: <Intel Kaby Lake HDA CODEC> at cad 2 on hdac0
hdaa0: <Intel Kaby Lake Audio Function Group> at nid 1 on hdacc0
pcm0: <Intel Kaby Lake (HDMI/DP 8ch)> at nid 3 on hdaa0
Trying to mount root from zfs:zroot/ROOT/default []...
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <Samsung SSD 860 EVO mSATA 250GB RVT41B6Q> ACS-4 ATA SATA 3.x device
ada0: Serial Number S41MNC0KA10520Y
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 238475MB (488397168 512 byte sectors)
ada0: quirks=0x3<4K,NCQ_TRIM_BROKEN>
uhub0: 18 ports with 18 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <Ralink 802.11 n WLAN> at usbus0
run0 on uhub0
run0: <Ralink 802.11 n WLAN, class 0/0, rev 2.00/1.01, addr 1> on usbus0
run0: MAC/BBP RT3070 (rev 0x0201), RF RT3020 (MIMO 1T1R), address 00:08:ca:51:cd:e3
run0: [HT] Enabling 802.11n

info.0:

Code Select Expand

Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 76288
  Blocksize: 512
  Compression: none
  Dumptime: 2024-07-31 11:17:56 +0200
  Hostname: opnsense.home.arpa
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 14.1-RELEASE-p2 stable/24.7-n267758-4ad7ad40bc77 SMP
  Panic String: page fault
  Dump Parity: 1831400993
  Bounds: 0
  Dump Status: good

textdump attached

[In short:]

Hi,

a few weeks ago I found myself forced to move into a shared apartment. Somewhat understandably my new roommate won't allow me to replace his router, an AVM Fritz!Box 7490, with my own (previously pfSense based) router. Because I've gotten so accustomed to all its features (mainly PBR over various VPN GWs and IPBL/DNSBL), I connected my router to his. He allowed me to set my router to be an "exposed host" for both its addresses (IPv4 and IPv6) and delegated prefixes. As far as I understand, this means the Fritz!Box firewall is practically disabled for my router. Also the Fritz!Box now delegates a /63 IPv6 subnet to my router, which I split into two /64 for the LAN and WiFi interfaces.

Curiously, with the move the VPN connections to Mullvad stopped working: No matter what I did, I could no longer use them as gateways for browsing the web, ie. attempts to open websites would simply time out. Because I wanted to give opnSense a try for a long time, I took this as an opportunity to finally make the switch from pfSense.

I already was at the point where all my devices have gotten IPv4 and IPv6 addresses assigned (via DHCP/managed RA), could talk to each other (confirmed via ping and ssh) and connect to the internet, except to blacklisted hosts. From the outside I could connect to the (non-Mullvad) WireGuard VPN I've set up on opnSense to get to my home server (confirmed via 5G connection). From the inside I could connect to the (non-Mullvad) WireGuard VPN I've set up on my mail server (some cheapo OVH Kimsufi box in France) for nightly backups.

Unfortunately the issue I had with pfSense persists with opnSense no matter what I do: I still can't get traffic from within my LAN to be routed via Mullvad servers to the outside world. All attempts to connect to web servers still just time out.
At all times I've followed this guide with no significant deviations. Once I realized I still have the same problem, I removed all firewall and NAT rules except those from the guide. Still, the issue persists. So I removed all firewall and NAT rules that I've set up, also the WG interfaces, gateways, instances and peers, and started all over again, again and again. Making sure I don't miss anything, such as ticking the "allow-options" checkbox of step 9. Nothing. I even generated new keys, thus also new interfaces on Mullvad's end, via their API, with the "hijack_dns" option set to false (this used to work just fine before the move). Still nothing. So I reinstalled opnSense, updated it, only set up WAN and LAN interfaces (both DHCP) and DNS, followed the guide again.

Thanks



In short:
- connections to servers time out when enabling rules with Mullvad VPNs set as gateway
- issue is reproducible on every device in LAN
+ ping works (and changes when routing via MV GW)
+ traceroute works (and changes when routing via MV GW)
+ dig/nslookup works
+ no suspicious logs (besides ones similar to these, see screenshots)
+ Mullvad client works (but is impractical, because can't be used on some devices)
+ connecting to tunneled devices on non-MV WG VPNs works fine
* this is the guide I followed (docs.opnsense.org)
* these are screenshots of the settings (imgur.com)
* this is a video illustrating the issue (youtube.com)