Messages

Hi Community,

in the last weeks I tried everything to punch down the issue and today I fixed the issue with luck but want to share some insights from debugging.
I tried a lout about finding the right MTU, but this wasn't the issue. (you can find the best MTU with `tracepath` on Linux)

I read about how to debug wireguard on FreeBSD and set via SSH on OPNsense the debug flag for the NIC (ifconfig wg0 debug), now I saw connection aborts in the wireguard connection itself, so I recognized that I loose every 2min the VPN connection and wg client tries 5sec to reconnect - that was really strange.
I read about a post (forgot the link) where a user told about a issue with UDP itself, so today, because I was out of ideas, I tried to change the default Wireguard Port from 51820 to another high-port and suddenly, I have no more packet loss!
So maybe it was really a issue with UDP on the default wireguard port, maybe a issue with my ISP, I dont know, but now it works.

Hi Community,

I'm facing a really interesting issue with a Wireguard VPN tunnel.
Setup is "Site DZ" with virtual OPNsense and "Site O" with a hardware firewall with OPNsense, both running latest Business Firmware.

Wireguard "Site O" connects to Wireguard Server on "Site DZ", not vice versa, only one way. Every connection works fine and as expected.
On a routed LAN on "Site DZ", OPNsense is also working as reverse proxy for TLS (tcp/443). One day I thought the connection when browsing one application, which is offloaded by OPNsense, feels a bit strange, I checked ICMP Ping and saw, that I loose every than and now exact 5 Pings, so the connection aborts and I "feel" it when browsing the web applications.
I checked the ping also directly from the OPNsense firewall itself, same packet loss when pinging or MTRing.
When I'm connecting with my computer directly via a second Wireguard instance (Road Warrior), I have no issues with packet loss, so it must be an issue with the second OPNsense firewall - both Wireguard Instances have default MTU.

Today I read about the MSS clamping (https://github.com/opnsense/docs/pull/498) and set it on both firewalls, rebootet, but nothing changed the situation.

Has anyone a idea? Attached is a wireguard dump of the ICMP and Ping, maybe that helps.

Thank you all!

Hi guys,

I've figured out, that it's an issue from the BSD/HardenedBSD kernel, an erratum/bug from the AMD 10h CPU :

Code Select Expand

pve01 kernel: [  284.573818] SVM: KVM: Guest triggered AMD Erratum 383

Snipped from the pmap.c file from the kernel:

Code Select Expand

/*
* If the kernel is running on a virtual machine, then it must assume
* that MCA is enabled by the hypervisor.  Moreover, the kernel must
* be prepared for the hypervisor changing the vendor and family that
* are reported by CPUID.  Consequently, the workaround for AMD Family
* 10h Erratum 383 is enabled if the processor's feature set does not
* include at least one feature that is only supported by older Intel
* or newer AMD processors.
*/
if (vm_guest != VM_GUEST_NO && (cpu_feature & CPUID_SS) == 0 &&
    (cpu_feature2 & (CPUID2_SSSE3 | CPUID2_SSE41 | CPUID2_AESNI |
    CPUID2_AVX | CPUID2_XSAVE)) == 0 && (amd_feature2 & (AMDID2_XOP |
    AMDID2_FMA4)) == 0)
workaround_erratum383 = 1;

Has anyone some experience with an secure workaround for this? Or have I really switch CPU/Server to get an VM running for OPNsense :(

Thanks @eugenmayer for your config!
But I checked my kvm with your config and my kvm is also running with these settings.
I also tried today to install proxmox from scratch with debian, but It's also not working, same boot loop as in my video.

I don't know if qemu or proxmox is making this issues.

@OPNsense devs, please give me some hints how I can debug this issue deeper so I can help you better.

Thanks!

Try setting the Machine to q35 and the CPU to host.
I have it running with these settings on a similar CPU (although I use OVMF UEFI, not SeaBIOS)

Hi @ruffy91, thanks for the idea!
But it's also looping like in the video  :(

Is there anyway to debug this issue deeper?

Has anyone some ideas?  :(

[HP ProLiant Microserver N40L]

Hi,
I tried many configurations in the last days but nothing worked, except the 32bit version (OPNsense-19.1.4-OpenSSL-dvd-i386.iso) which is running now with the latest version 19.1.9 with kvm32 and the "hw.mca.enabled=0" hook in loader.conf.local. (according to https://pve.proxmox.com/wiki/PfSense_Guest_Notes)
But it's not the 64bit version :(
BUT, when I disable "KVM hardware virtualization", the 64bit version with kvm64 is booting & working, but extremly slow and not production ready of course...

Can anyone please give me some advice to get the 64bit version running? :)

I recorded a video from the VM which is trying to boot and I have here the full specs from my bare metal server:
HP ProLiant Microserver N40L

Video:
https://vimeo.com/343868600

CPU:

Code Select Expand

root@pve01:~# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             AuthenticAMD
CPU family:            16
Model:                 6
Model name:            AMD Turion(tm) II Neo N40L Dual-Core Processor
Stepping:              3
CPU MHz:               1500.000
CPU max MHz:           1500.0000
CPU min MHz:           800.0000
BogoMIPS:              2995.12
Virtualization:        AMD-V
L1d cache:             64K
L1i cache:             64K
L2 cache:              1024K
NUMA node0 CPU(s):     0,1
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate vmmcall npt lbrv svm_lock nrip_save

PCI:

Code Select Expand

root@pve01:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] RS880 Host Bridge
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780 PCI to PCI bridge (ext gfx port 0)
00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780/RS880 PCI to PCI bridge (PCIE port 0)
00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780 PCI to PCI bridge (PCIE port 2)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control
01:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
01:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 02)
03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10)
04:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5723 Gigabit Ethernet PCIe (rev 10)

Memory:

Code Select Expand

root@pve01:~# dmidecode -t memory
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.6 present.

Handle 0x0014, DMI type 16, 15 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: Single-bit ECC
Maximum Capacity: 8 GB
Error Information Handle: Not Provided
Number Of Devices: 2

Handle 0x0016, DMI type 17, 28 bytes
Memory Device
Array Handle: 0x0014
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 8192 MB
Form Factor: DIMM
Set: None
Locator: DIMM0
Bank Locator: BANK0
Type: Other
Type Detail: Synchronous
Speed: 1333 MHz
Manufacturer: Manufacturer00
Serial Number: SerNum00
Asset Tag: Not Specified
Part Number: ModulePartNumber00
Rank: Unknown

Handle 0x0018, DMI type 17, 28 bytes
Memory Device
Array Handle: 0x0014
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 8192 MB
Form Factor: DIMM
Set: None
Locator: DIMM1
Bank Locator: BANK1
Type: Other
Type Detail: Synchronous
Speed: 1333 MHz
Manufacturer: Manufacturer01
Serial Number: SerNum01
Asset Tag: Not Specified
Part Number: ModulePartNumber01
Rank: Unknown

Today I tried the older version 18.7 and it worked without any problems.
Nevertheless, how can I debug the issue with 19.1 so we can find the issue?
Thanks!  :D

Hi OPNsense guys!
I'm new to OPNsense but I'm trying to get it running on my old "ProLiant Microserver N40L" (AMD Turion II Neo N40L Dual-Core Processor) which is running the latest Proxmox VE (5.4-6) but it's driving me crazy because I can't get the OPNsense installer (19.1.4 DVD ISO, checksum correct) running!
I have tried many many different virtualization settings, other BIOS, different CPU types, VirtIO SCSI etc etc. but the bootloader is "looping" and I don't know why.
Attached is the screenshot with the last message I see, then the Proxmox VE BIOS image is back again and OPNsense is loading...
Can anyone please tell me, how I can debug why the installer isn't booting? Maybe my baremetal hardware isn't compatible?  :o

Thank you so much!