Messages

PS: Made this a little clearer... https://github.com/opnsense/core/commit/95ef3e91

I'm confused, the ReadTheDocs page says System-> Settings-> Tunables is for sysctl configuration parameters

How can it be for /boot/loader.conf and /etc/sysctl.conf at the same time?

I only donated $3, but it's monthly, so if you wait a year you'll have $36  ;)

I figure everyone has $3/mo (I'm unemployed, but still...) and if everyone gave $3 every month it would be a lot more sustainable than sporadic donations of larger amounts...

Just my 2 cents  :o

I found this thread wondering the same question.  Thanks for chiming in about that, Franco, it was really interesting!

Hi,

I just did a routine upgrade from 20.7.1 to 20.7.8, and after the reboot my system wouldn't boot.

I managed to boot from a kernel backup that was automatically created at `/boot/kernel.old`.  I noticed the kernel was changed during the upgrade because I put a tag in the last kernel I built, and now it's `kernel.old`:

Code Select Expand


root@gateway:/var/run # uname -a

FreeBSD gateway.webtool.space 12.2-RELEASE-p2 FreeBSD 12.2-RELEASE-p2 #0 r369071M: Wed Jan 20 13:32:25 PST 2021     avery@fabby.webtool.space:/usr/obj/usr/src/amd64.amd64/sys/CALOMEL  amd64


Here's the tail of my `pkg` log:

Code Select Expand


root@gateway:/var/log # tail pkg.log

Jan 18 18:11:52 gateway pkg[42852]: pkg upgraded: 1.12.0_1 -> 1.15.10_1
Jan 18 18:11:54 gateway pkg[21286]: py37-speedtest-cli-2.1.2 installed
Jan 18 20:43:44 gateway pkg[7559]: os-theme-cicada-1.25 installed
Jan 18 20:44:00 gateway pkg[70015]: os-theme-vicuna-1.1 installed
Jan 26 19:56:26 gateway pkg-static[78977]: os-dyndns upgraded: 1.22 -> 1.23
Jan 26 19:56:38 gateway pkg-static[78977]: opnsense upgraded: 20.7 -> 20.7.8
Jan 26 19:56:38 gateway pkg-static[72426]: py37-asn1crypto-1.3.0 de


Here's `dmesg` from the boot that couldn't complete:

Code Select Expand


root@gateway:/var/run # less dmesg.boot

---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-RELEASE-p2 #0 r369071M: Wed Jan 20 13:32:25 PST 2021
    avery@fabby.webtool.space:/usr/obj/usr/src/amd64.amd64/sys/CALOMEL amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(efifb): resolution 1024x768
CPU: Intel(R) Celeron(R) CPU  J1800  @ 2.41GHz (2416.73-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30673  Family=0x6  Model=0x37  Stepping=3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8145580032 (7768 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <LENOVO TC-03   >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: unblocking device.
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20200430/tbfadt-748)
ioapic0 <Version 2.0> irqs 0-86 on motherboard
Launching APs: 1
Timecounter "TSC-low" frequency 1208363314 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
000.000023 [4336] netmap_init               netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff81117e40, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
kernel trap 12 with interrupts disabled
kernel trap 12 with interrupts disabled
cryptosoft0: <software crypto> on motherboard
acpi0: <LENOVO TC-03> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 10b15fff vs 10b16000
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf080-0xf087 mem 0xb0000000-0xb03fffff,0xa0000000-0xafffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
ahci0: <AHCI SATA controller> port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf020-0xf03f mem 0xb0b15000-0xb0b157ff irq 19 at device 19.0 on pci0
ahci0: AHCI v1.30 with 2 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
xhci0: <Intel BayTrail USB 3.0 controller> mem 0xb0b00000-0xb0b0ffff irq 20 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pcib1: [GIANT-LOCKED]
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <PCI-PCI bridge> at device 2.0 on pci2
pci3: <PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection> port 0xd020-0xd03f mem 0xb0920000-0xb093ffff,0xb0880000-0xb08fffff at device 0.0 on pci3
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:26:55:d6:b1:15
em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: <Intel(R) PRO/1000 Network Connection> port 0xd000-0xd01f mem 0xb0900000-0xb091ffff,0xb0800000-0xb087ffff at device 0.1 on pci3
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:26:55:d6:b1:14
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <PCI-PCI bridge> at device 4.0 on pci2
pci4: <PCI bus> on pcib4
em2: <Intel(R) PRO/1000 Network Connection> port 0xc020-0xc03f mem 0xb0720000-0xb073ffff,0xb0680000-0xb06fffff at device 0.0 on pci4
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:26:55:d6:b1:17
em2: netmap queues/slots: TX 1/1024, RX 1/1024
em3: <Intel(R) PRO/1000 Network Connection> port 0xc000-0xc01f mem 0xb0700000-0xb071ffff,0xb0600000-0xb067ffff at device 0.1 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:26:55:d6:b1:16
em3: netmap queues/slots: TX 1/1024, RX 1/1024
pcib5: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pcib5: [GIANT-LOCKED]
pci5: <ACPI PCI bus> on pcib5
re0: <RealTek 810xE PCIe 10/100baseTX> port 0xe000-0xe0ff mem 0xb0a04000-0xb0a04fff,0xb0a00000-0xb0a03fff irq 18 at device 0.0 on pci5
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: Chip rev. 0x40800000
re0: MAC rev. 0x00400000
miibus0: <MII bus> on re0
rlphy0: <RTL8201E 10/100 media interface> PHY 1 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: c0:3f:d5:93:3d:a7
re0: netmap queues/slots: TX 1/256, RX 1/256
pcib6: <ACPI PCI-PCI bridge> at device 28.3 on pci0
ada0: 22902MB (46905264 512 byte sectors)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <INTEL SSDMAEXC024G3H 9CV10379> ATA8-ACS SATA 2.x device
ada1: Serial Number CVHA3433003R024D
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 22902MB (46905264 512 byte sectors)
GEOM: ada0: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-CVHA3433003W024D: the secondary GPT header is not in the last LBA.
GEOM: ada1: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-CVHA3433003R024D: the secondary GPT header is not in the last LBA.
uhub0: 7 ports with 7 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <vendor 0x05e3 USB2.0 Hub> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 1> on usbus0
uhub1: MTT enabled
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.3: <vendor 0x04d9 USB Keyboard> at usbus0
ukbd0 on uhub1
ukbd0: <vendor 0x04d9 USB Keyboard, class 0/0, rev 1.10/3.10, addr 2> on usbus0
kbd2 at ukbd0
mountroot: waiting for device /dev/gpt/rootfs...
GEOM: diskid/DISK-CVHA3433003W024D: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-CVHA3433003W024D: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-CVHA3433003W024D: the secondary GPT header is not in the last LBA.


It looks like it has something to do with disk access.  I have a ufs geom mirror running on dual mSATA SLC 24GB SSDs.

It's funny because I had *just* installed my self-built kernel right before I noticed there was an update available.  I'm glad I had it to fall back on!

Should I post a bug report on github instead of here?  Any recommendations, would love to hear it - I'll probably move `/boot/kernel` to `/boot/kernel.old` and vice-versa for now...

Thanks!

Edit:  For reference, here's my `dmesg` from when I managed to boot:

Code Select Expand


root@gateway:/var/run # dmesg -a
---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-RELEASE-p2 #0 r369071M: Wed Jan 20 13:32:25 PST 2021
    avery@fabby.webtool.space:/usr/obj/usr/src/amd64.amd64/sys/CALOMEL amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(efifb): resolution 1024x768
CPU: Intel(R) Celeron(R) CPU  J1800  @ 2.41GHz (2416.73-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30673  Family=0x6  Model=0x37  Stepping=3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8145580032 (7768 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <LENOVO TC-03   >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: unblocking device.
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20200430/tbfadt-748)
ioapic0 <Version 2.0> irqs 0-86 on motherboard
Launching APs: 1
Timecounter "TSC-low" frequency 1208363314 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
000.000023 [4336] netmap_init               netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff81117e40, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
kernel trap 12 with interrupts disabled
kernel trap 12 with interrupts disabled
cryptosoft0: <software crypto> on motherboard
acpi0: <LENOVO TC-03> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 10b15fff vs 10b16000
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf080-0xf087 mem 0xb0000000-0xb03fffff,0xa0000000-0xafffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
ahci0: <AHCI SATA controller> port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf020-0xf03f mem 0xb0b15000-0xb0b157ff irq 19 at device 19.0 on pci0
ahci0: AHCI v1.30 with 2 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
xhci0: <Intel BayTrail USB 3.0 controller> mem 0xb0b00000-0xb0b0ffff irq 20 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pcib1: [GIANT-LOCKED]
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <PCI-PCI bridge> at device 2.0 on pci2
pci3: <PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection> port 0xd020-0xd03f mem 0xb0920000-0xb093ffff,0xb0880000-0xb08fffff at device 0.0 on pci3
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:26:55:d6:b1:15
em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: <Intel(R) PRO/1000 Network Connection> port 0xd000-0xd01f mem 0xb0900000-0xb091ffff,0xb0800000-0xb087ffff at device 0.1 on pci3
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:26:55:d6:b1:14
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <PCI-PCI bridge> at device 4.0 on pci2
pci4: <PCI bus> on pcib4
em2: <Intel(R) PRO/1000 Network Connection> port 0xc020-0xc03f mem 0xb0720000-0xb073ffff,0xb0680000-0xb06fffff at device 0.0 on pci4
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:26:55:d6:b1:17
em2: netmap queues/slots: TX 1/1024, RX 1/1024
em3: <Intel(R) PRO/1000 Network Connection> port 0xc000-0xc01f mem 0xb0700000-0xb071ffff,0xb0600000-0xb067ffff at device 0.1 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:26:55:d6:b1:16
em3: netmap queues/slots: TX 1/1024, RX 1/1024
pcib5: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pcib5: [GIANT-LOCKED]
pci5: <ACPI PCI bus> on pcib5
re0: <RealTek 810xE PCIe 10/100baseTX> port 0xe000-0xe0ff mem 0xb0a04000-0xb0a04fff,0xb0a00000-0xb0a03fff irq 18 at device 0.0 on pci5
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: Chip rev. 0x40800000
re0: MAC rev. 0x00400000
miibus0: <MII bus> on re0
rlphy0: <RTL8201E 10/100 media interface> PHY 1 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: c0:3f:d5:93:3d:a7
re0: netmap queues/slots: TX 1/256, RX 1/256
pcib6: <ACPI PCI-PCI bridge> at device 28.3 on pci0
pcib6: [GIANT-LOCKED]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
orm0: <ISA Option ROM> at iomem 0xd0000-0xd0fff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
TCP Hpts created 2 swi interrupt thread and bound 0
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]...
Root mount waiting for: CAM usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <INTEL SSDMAEXC024G3H 9CV10379> ATA8-ACS SATA 2.x device
ada0: Serial Number CVHA3433003W024D
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 22902MB (46905264 512 byte sectors)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <INTEL SSDMAEXC024G3H 9CV10379> ATA8-ACS SATA 2.x device
ada1: Serial Number CVHA3433003R024D
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 22902MB (46905264 512 byte sectors)
GEOM: ada0: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-CVHA3433003W024D: the secondary GPT header is not in the last LBA.
GEOM: ada1: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-CVHA3433003R024D: the secondary GPT header is not in the last LBA.
uhub0: 7 ports with 7 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <vendor 0x05e3 USB2.0 Hub> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 1> on usbus0
uhub1: MTT enabled
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.3: <vendor 0x04d9 USB Keyboard> at usbus0
ukbd0 on uhub1
ukbd0: <vendor 0x04d9 USB Keyboard, class 0/0, rev 1.10/3.10, addr 2> on usbus0
kbd2 at ukbd0
mountroot: waiting for device /dev/gpt/rootfs...
Mounting filesystems...
tunefs: soft updates remains unchanged as enabled
GEOM: diskid/DISK-CVHA3433003W024D: the secondary GPT header is not in the last LBA.
tunefs: file system reloaded
tunefs: issue TRIM to the disk remains unchanged as enabled
GEOM: diskid/DISK-CVHA3433003W024D: the secondary GPT header is not in the last LBA.
tunefs: file system reloaded
** /dev/gpt/rootfs
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 4768551 free (4487 frags, 595508 blocks, 0.1% fragmentation)
GEOM: diskid/DISK-CVHA3433003W024D: the secondary GPT header is not in the last LBA.
Setting hostuuid: 007547df-0bdc-e311-847d-c03fd5933da7.
Setting hostid: 0xa99c24ce.
Configuring vt: blanktime.
Configuring crash dump device: /dev/null
ddb: sysctl: debug.ddb.scripting.scripts: No such file or directory
.ELF ldconfig path: /lib /usr/lib /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path:
done.
>>> Invoking early script 'update'
>>> Invoking early script 'configd'
Starting configd.
>>> Invoking early script 'templates'
Generating configuration: OK
>>> Invoking early script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
>>> Invoking backup script 'rrd'
>>> Invoking early script 'carp'
CARP event system: OK
Launching the init system...done.
Initializing...........done.
em0: link state changed to UP
em3: link state changed to UP
re0: link state changed to DOWN
Starting device manager...
uhid0 on uhub1
uhid0: <vendor 0x04d9 USB Keyboard, class 0/0, rev 1.10/3.10, addr 2> on usbus0
done.
Configuring login behaviour...done.
Configuring loopback interface...
lo0: link state changed to UP
done.
Configuring kernel modules...
kernel trap 12 with interrupts disabled
kernel trap 12 with interrupts disabled
aesni0: No AES or SHA support.
done.
Setting up extended sysctls...done.
Setting timezone...done.
Writing firmware setting...done.
Writing trust files...done.
Setting hostname: gateway.webtool.space
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAN interface...
em3: link state changed to DOWN
done.
Configuring WAN interface...
em0: link state changed to DOWN
em3: link state changed to UP
em0: link state changed to UP
done.
Creating IPsec VTI instances...done.
Creating OpenVPN instances...
tun1: changing name to 'ovpns1'
done.
Generating /etc/resolv.conf...done.
Configuring firewall........done.
Starting PFLOG...done.
pflog0: promiscuous mode enabled
Configuring OpenSSH...done.
Starting web GUI...done.
Configuring CRON...done.
Setting up routes...done.
Generating /etc/hosts...done.
Starting DHCPv6 service...done.
Starting router advertisement service...done.
Setting up gateway monitors...done.
Configuring firewall........done.
Starting PFLOG...
pflog0: promiscuous mode disabled
done.
pflog0: promiscuous mode enabled
Syncing OpenVPN settings...
ovpns1: link state changed to UP
done.
Configuring dynamic DNS clients...done.
Starting NTP service...deferred.
Generating RRD graphs...done.
Configuring system logging...done.
>>> Invoking start script 'newwanip'
Reconfiguring IPv4 on em0:
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
OK
Reconfiguring IPv4 on ovpns1: OK
Reconfiguring IPv6 on em0:
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
OK
>>> Invoking start script 'freebsd'
WARNING: attempt to domain_add(netgraph) after domainfinalize()
setup em3
setup em0 [egress only]
Starting flowd_aggregate.
Starting flowd.
>>> Invoking start script 'syslog-ng'
Stopping syslog_ng.
Waiting for PIDS: 72205, 72205.
Starting syslog_ng.
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK
>>> Invoking start script 'beep'
Root file system: /dev/gpt/rootfs
Tue Jan 26 20:06:48 PST 2021


This problem seems to be getting worse - I upgraded to 20.7.5 and my iperf3 speeds have dropped from ~2Gb/s to hovering around 1Gb/s, with VM->VM speeds at ~650Mbps  :o  :-\

CentOS 8 VM on the same machine gets around 9.4Gbps

will upload some speeds when I get a chance

Hi,

I'm trying to figure out how to set up my DCs with IPV6

I have a residential 1G/40M cable connection (Comcast ISP) with DHCPv4/v6

My LANv6 works with "track changes" which is the only way I've been able to get it working consistently, but I admit I don't know much about how IPV6 works

I just noticed if I change my WANv6 settings to "Request only an IPv6 prefix"  I get a link local address

For instance, I had "Request only IPV6 prefix" unchecked, and my WAN IPV6 address was: 2001:558:600a:c3:8590:bba3:2023:e486/128

I checked "Request only IPV6 Prefix" and rebooted, and now my address is: fe80::226:55ff:fed6:b115/64

Can any of these assist me in having a static gateway address?  I need a static gateway address in order to create static IPV6 addresses for my domain controllers.

I asked this question on pfsense forum on Reddit and some people were saying ULAs (would that double-NAT with the DCs?) Another person said EIA64 could be an option.

This is all new to me, I have no idea what they are besides looking up definitions, and I haven't heard any clear direction as to how to implement either one.

Does anyone want to demystify this for me?  How could either be used in the context of OPNsense and AD?

Thanks

Would it be possible to install a stock FreeBSD 13 kernel?  Maybe they fixed the regressions.  I'm wondering if it has something to do with HBSD compile flags for security.

Unfortunatelly this is not so easy. You cant use a precompiled kernel from an another system. It wouldn't boot.
You have to compile from source, but newer kernel means newer headers and libraries in dependency. The compilation process could failed at some point. The only solution what could work is cherry pick the fix only and implement to the original kernel source tree and compile. But this needs work too.
I was an android kernel developer many years back so i know experiencing with the kernel is always risky.

Wouldnt it be easier to do it the other way round?

Make OS work with FBSD13? To eliminate any remnance of bad plugin code?

It does work, and it's fairly easy. Just install OPNsense using opnsense-bootstrap over a FreeBSD installation.  You have to change the script if you want to install over a different version of FreeBSD (e.g. 13), but if you install 12.x you can just run the script.  Then boot from kernel.old or copy the kernel back to /boot/kernel, kldxref, etc.

I can't vouch for the helpfulness as my FreeBSD understanding is limited, I don't know much about kernel tuning.  Your identification of net.isr.maxthreads and net.isr.numthreads always returning 1 core seems more helpful than arbitrarily changing kernel.

How would you recommend tuning kernel for multi-threaded?  Is turning off hyperthreading a good idea?

Btw I didn't see much speed increase installing OPNsense 20.7 over 13-CURRENT and I'm suspect of its reliability, but there is a slight increase in speed installing OPNsense over 12.1-RELEASE and keeping FreeBSD kernel:  https://forum.opnsense.org/index.php?topic=19789.msg91356#msg91356

It would probably be more noticeable on 10G but I haven't done any benchmarking w/ it yet.

I have some iperf3 scores from a debian bullseye box on the same network:

Freebsd 12.1 Kernel on J1800 12.1 zfs installation (stock)

Code Select Expand


avery@debbox:~$ iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.39 port 39364 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  68.6 MBytes   575 Mbits/sec    0    281 KBytes       
[  5]   1.00-2.00   sec  65.6 MBytes   550 Mbits/sec    0    281 KBytes       
[  5]   2.00-3.00   sec  65.4 MBytes   549 Mbits/sec    0    281 KBytes       
[  5]   3.00-4.00   sec  65.2 MBytes   547 Mbits/sec    0    281 KBytes       
[  5]   4.00-5.00   sec  64.6 MBytes   542 Mbits/sec    0    281 KBytes       
[  5]   5.00-6.00   sec  64.9 MBytes   545 Mbits/sec    0    281 KBytes       
[  5]   6.00-7.00   sec  63.9 MBytes   536 Mbits/sec    0    281 KBytes       
[  5]   7.00-8.00   sec  65.0 MBytes   545 Mbits/sec    0    281 KBytes       
[  5]   8.00-9.00   sec  64.9 MBytes   545 Mbits/sec    0    281 KBytes       
[  5]   9.00-10.00  sec  64.9 MBytes   544 Mbits/sec    0    281 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   653 MBytes   548 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   652 MBytes   547 Mbits/sec                  receiver

iperf Done.


Freebsd 13 Kernel on J1800 12.1 zfs installation (stock - 10222020 snapshot)

Code Select Expand


avery@debbox:~$ iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.39 port 42638 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  65.9 MBytes   552 Mbits/sec    0    185 KBytes       
[  5]   1.00-2.00   sec  64.9 MBytes   545 Mbits/sec    0    185 KBytes       
[  5]   2.00-3.00   sec  64.6 MBytes   542 Mbits/sec    0    185 KBytes       
[  5]   3.00-4.00   sec  64.7 MBytes   543 Mbits/sec    0    185 KBytes       
[  5]   4.00-5.00   sec  64.9 MBytes   544 Mbits/sec    0    185 KBytes       
[  5]   5.00-6.00   sec  64.9 MBytes   544 Mbits/sec    0    185 KBytes       
[  5]   6.00-7.00   sec  64.6 MBytes   542 Mbits/sec    0    185 KBytes       
[  5]   7.00-8.00   sec  65.0 MBytes   545 Mbits/sec    0    185 KBytes       
[  5]   8.00-9.00   sec  64.5 MBytes   541 Mbits/sec    0    185 KBytes       
[  5]   9.00-10.00  sec  64.9 MBytes   544 Mbits/sec    0    185 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   649 MBytes   544 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   648 MBytes   544 Mbits/sec                  receiver

iperf Done.

OPNsense 20.7.4 Kernel on J1800 12.1 zfs installation (https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/sets/kernel-20.7.4-amd64.txz)

Code Select Expand


iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.39 port 44048 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  64.6 MBytes   542 Mbits/sec    0    170 KBytes       
[  5]   1.00-2.00   sec  63.8 MBytes   535 Mbits/sec    0    170 KBytes       
[  5]   2.00-3.00   sec  63.1 MBytes   529 Mbits/sec    0    170 KBytes       
[  5]   3.00-4.00   sec  63.4 MBytes   532 Mbits/sec    0    170 KBytes       
[  5]   4.00-5.00   sec  63.8 MBytes   535 Mbits/sec    0    170 KBytes       
[  5]   5.00-6.00   sec  63.4 MBytes   532 Mbits/sec    0    180 KBytes       
[  5]   6.00-7.00   sec  63.8 MBytes   535 Mbits/sec    0    180 KBytes       
[  5]   7.00-8.00   sec  63.4 MBytes   532 Mbits/sec    0    180 KBytes       
[  5]   8.00-9.00   sec  63.8 MBytes   535 Mbits/sec    0    180 KBytes       
[  5]   9.00-10.00  sec  63.4 MBytes   532 Mbits/sec    0    180 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   636 MBytes   534 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   636 MBytes   533 Mbits/sec                  receiver

iperf Done.


Pretty much hair splitting, but interestingly the OPNsense 20.7.4 kernel is the slowest.  I wonder if this becomes more pronounced on faster systems.  I'll test it on my 10Gbe system later and report back.

Hey,

I was reading this thread: https://forum.opnsense.org/index.php?topic=18754.15 which prompted me to do some throughput tests

I hadn't even noticed that my 10GbE iperf3 speeds were 6-8Gbps slower on my OPNsense VM than the rest of my VMs, until I did some testing - my iperf3 speeds were around 1Gbps for OPNsense 20.7.3 whereas they're consistently stuck at 9.4Gbps between everything else when there's no competing network traffic.

So I thought I'd try a different kernel.  I'd been wanting to do an installation over zfs as kind of a proof of concept, and also because I miss the zfs from my old pfsense installations (that's about all I miss), and since I was wanting to experiment with the kernel, I thought it'd be good to have snapshot capabilities so I could roll back easily if I wanted to.

I installed 20.7.3 over FreeBSD 12.1-RELEASE on a 24GB slc zmirror on this old j1800 shitbox I built from a Lenovo H500s motherboard (literally $15) using opnsense-bootstrap.  I replaced the kernel w/ the previously installed freebsd kernel after opnsense-bootstrap finished, and rebooted with it. 

This box used to get a pretty consistent 650Mbps with previous builds of OPNsense on my 1Gb comcast residential in speedtest-cli. Now it's rarely over 500Mbps, usually closer to 450.  And I'm just wondering:

What do people w/ experience with either think it is?  Does zfs tend to make systems with low processing power more sluggish? (It's got 8GB of ram, btw)  Or are there kernel optimizations for networking that the stock freebsd kernel is missing?

My arc_max is stock, which puts it up near the full 8GB, does anyone think if I tuned that, or my meta cache, I could get the speeds back up to 650Mbps again?

What about old OPNsense kernel?  I know the people in that thread said they got better speeds in ver. <19

Thanks

Would it be possible to install a stock FreeBSD 13 kernel?  Maybe they fixed the regressions.  I'm wondering if it has something to do with HBSD compile flags for security.

It is odd that so many of us seem to find an artificial ~1gbps limit when testing OPNsense 20.7 on VMware ESXi and vmxnet3 adapters. It looks like there's at least 3 of us that are able to re-produce these results now?

I've disable the hardware blacklist and did not see a difference in my test results from what I had posted here prior. The only way I can get a little bit better throughput is to add more vCPU to the OPNsense VM, however this does not scale well. For instance, if I go from 2vCPU to 4vCPU, I can start to get between 1.5gbps and 2.2gbps depending on how much parallelism I select on my iperf clients.

I don't think it's related to the "hardware" (even though in this case, it's virtual).  I think it's the upstream regression mentioned on page 1 - since I used to get better speeds than this before I upgraded.  I think I did my last LAN-side iperf3 tests around v18 or 19, and they were at least twice that.  In fact, I'm fairly certain I doubled my vCPUs and ram since because I was testing Sensei and never re-configured it for 2 vCPU/4GB after I uninstalled it.

You guys got me interested in this subject. I have tested plenty of iperf3 against my VMs in my little 3-host homelab, my 10GbE is just a couple DACs connected between the 10Gbe "backbone" IFs of my Dell Powerconnect 7048P, which is really more of a gigabit switch.

Usually the VMs will peg right up to ~9.4Gbps with little fluctuation if nothing else is happening, but I'm recording 3 720p video streams and 6 high-MP (4MP & 8MP) IP cameras right now, and have no interest in stopping any of it for testing right now.

I could have sworn I'd iperfed my OPNsense VM and gotten somewhere around 2.9Gbps vs the 9.4Gbps I got on my Linux, OmniOS or FreeBSD VMs (don't think I tested Windows, iperf3 is compiled weird in Win32 and doesn't yield predictable results).  So I expected it to be a bit slower, but not THIS much slower:

OPNsense 20.7.3 to OmniOS r151034
(on separate hosts)

This is a VM w/ 4 vCPU and 8GB ram, run on an E3-1230 v2 home-built Supermicro X9SPU-F host running ESXi 6.7U3.  The LAN vNIC is vmxnet3, running open-vm-tools.

Code Select Expand


root@gateway:/ # uname -a
FreeBSD gateway.webtool.space 12.1-RELEASE-p10-HBSD FreeBSD 12.1-RELEASE-p10-HBSD #0  517e44a00df(stable/20.7)-dirty: Mon Sep 21 16:21:17 CEST 2020     root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64

root@gateway:/ # iperf3 -c 192.168.1.56
Connecting to host 192.168.1.56, port 5201
[  5] local 192.168.1.1 port 13640 connected to 192.168.1.56 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   125 MBytes  1.05 Gbits/sec    0   2.00 MBytes       
[  5]   1.00-2.00   sec   126 MBytes  1.06 Gbits/sec    0   2.00 MBytes       
[  5]   2.00-3.00   sec   132 MBytes  1.11 Gbits/sec    0   2.00 MBytes       
[  5]   3.00-4.00   sec   131 MBytes  1.10 Gbits/sec    0   2.00 MBytes       
[  5]   4.00-5.00   sec   132 MBytes  1.11 Gbits/sec    0   2.00 MBytes       
[  5]   5.00-6.00   sec   135 MBytes  1.13 Gbits/sec    0   2.00 MBytes       
[  5]   6.00-7.00   sec   138 MBytes  1.16 Gbits/sec    0   2.00 MBytes       
[  5]   7.00-8.00   sec   137 MBytes  1.15 Gbits/sec    0   2.00 MBytes       
[  5]   8.00-9.00   sec   133 MBytes  1.12 Gbits/sec    0   2.00 MBytes       
[  5]   9.00-10.00  sec   131 MBytes  1.10 Gbits/sec    0   2.00 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.29 GBytes  1.11 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.29 GBytes  1.11 Gbits/sec                  receiver

iperf Done.


That is abysmal.  Compare that to this Bullseye VM going to same OmniOS VM (also on separate hosts)

Debian Bullseye to OmniOS r151034

Code Select Expand


avery@debbox:~$ uname -a
Linux debbox 5.4.0-4-amd64 #1 SMP Debian 5.4.19-1 (2020-02-13) x86_64 GNU/Linux

avery@debbox:~$ iperf3 -c 192.168.1.56
Connecting to host 192.168.1.56, port 5201
[  5] local 192.168.1.39 port 58064 connected to 192.168.1.56 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   688 MBytes  5.77 Gbits/sec    0   2.00 MBytes       
[  5]   1.00-2.00   sec   852 MBytes  7.15 Gbits/sec    0   2.00 MBytes       
[  5]   2.00-3.00   sec   801 MBytes  6.72 Gbits/sec  1825    730 KBytes       
[  5]   3.00-4.00   sec   779 MBytes  6.53 Gbits/sec   33   1.13 MBytes       
[  5]   4.00-5.00   sec   788 MBytes  6.61 Gbits/sec  266   1.33 MBytes       
[  5]   5.00-6.00   sec   828 MBytes  6.94 Gbits/sec  392   1.43 MBytes       
[  5]   6.00-7.00   sec   830 MBytes  6.96 Gbits/sec  477   1.49 MBytes       
[  5]   7.00-8.00   sec   826 MBytes  6.93 Gbits/sec  1286    749 KBytes       
[  5]   8.00-9.00   sec   826 MBytes  6.93 Gbits/sec    0   1.26 MBytes       
[  5]   9.00-10.00  sec   775 MBytes  6.50 Gbits/sec  278   1.38 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.81 GBytes  6.71 Gbits/sec  4557             sender
[  5]   0.00-10.00  sec  7.80 GBytes  6.70 Gbits/sec                  receiver

iperf Done.


So much better throughput. Even while that OmniOS VM is recording 8-9 streams of video over the network.

I'm going to install a FreeBSD kernel and see what happens.  Will be back with more benchmarks.

> How did you do that? [force 1Gbps NIC]

Turn off auto negotiation and set the nic's IF to 1gbps (?)

Yes, that looks weird. This happens when Sensei is configured for a WAN interface or there are connections which originate and terminate on the internal network.

LAN is the only device that appears in "protected interfaces" - WAN is not even an option...

I'm running OPNsense on ESXi 6.7U2 with a passthrough 82579LM for WAN and vmxnet3 LAN vnic.

```                        `       root@gateway.domain.example
  ` `.....---.......--.```   -/    --------------------------
  +o   .--`         /y:`      +.   OS: FreeBSD 11.2-RELEASE-p17-HBSD amd64
   yo`:.            :o      `+-    Uptime: 20 days, 16 hours, 20 mins
    y/               -/`   -o/     Packages: 176 (pkg)
   .-                  ::/sy+:.    Shell: opnsense-shell Illegal option -- Usag
   /                     `--  /    Terminal: /dev/pts/0
  `:                          :`   CPU: Intel Xeon E3-1230 V2 (4) @ 3.300GHz
  `:                          :`   GPU: SVGA II Adapter
   /                          /    Memory: 3982MiB / 8155MiB
   .-                        -.
    --                      -.                             
     `:`                  `:`                             
       .--             `--.
          .---.....----.



Thanks for your assistance -- I reached out with the UI link

 ???