Messages

oh this went very, very badly.

I set the sysctl in the webif as you suggested and instantly it killed the entire firewall. No traffic can go through, serial console spewing out debug data at max rate. I'm guessing 115200 is still a bottleneck, this clogs up the kernel from doing anything useful. Even if I wanted to SSH from one of the igb interfaces, it wouldn't work.

Not good.

Eventually manage to get a single user mode shell before those sysctls get applied on boot. I manually edit the config.xml to remove the sysctl entries. Reboot again.

Now I get a kernel panic every boot when it loads the driver. Removing the SFPs and leaving it powered off for a while doesn't help, every boot is a kernel panic.

Luckily I have a spare DEC 2700 I could install/restore my config onto and swap the hardware in place to get my network back online, but this DEC840 is pretty much a brick right now.

In the debugger after panic, I get this output:

Code Select Expand

db> bt
Tracing pid 0 tid 100000 td 0xffffffff81d81d60
kdb_enter() at kdb_enter+0x33/frame 0xffffffff82e6d6b0
panic() at panic+0x43/frame 0xffffffff82e6d710
trap_pfault() at trap_pfault+0x3da/frame 0xffffffff82e6d760
calltrap() at calltrap+0x8/frame 0xffffffff82e6d760
--- trap 0xc, rip = 0xffffffff810f9dc0, rsp = 0xffffffff82e6d830, rbp = 0xffffffff82e6d850 ---
xgbe_dump_dma_registers() at xgbe_dump_dma_registers+0x3c0/frame 0xffffffff82e6d850
axgbe_pci_init() at axgbe_pci_init+0xe0/frame 0xffffffff82e6d870
axgbe_if_attach_post() at axgbe_if_attach_post+0x395/frame 0xffffffff82e6d8b0
iflib_device_register() at iflib_device_register+0x2655/frame 0xffffffff82e6d9e0
iflib_device_attach() at iflib_device_attach+0xaa/frame 0xffffffff82e6da10
device_attach() at device_attach+0x43d/frame 0xffffffff82e6da60
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6da80
pci_attach() at pci_attach+0xc7/frame 0xffffffff82e6dab0
acpi_pci_attach() at acpi_pci_attach+0x15/frame 0xffffffff82e6daf0
device_attach() at device_attach+0x43d/frame 0xffffffff82e6db40
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6db60
acpi_pcib_pci_attach() at acpi_pcib_pci_attach+0x95/frame 0xffffffff82e6db90
device_attach() at device_attach+0x43d/frame 0xffffffff82e6dbe0
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6dc00
pci_attach() at pci_attach+0xc7/frame 0xffffffff82e6dc30
acpi_pci_attach() at acpi_pci_attach+0x15/frame 0xffffffff82e6dc70
device_attach() at device_attach+0x43d/frame 0xffffffff82e6dcc0
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6dce0
acpi_pcib_acpi_attach() at acpi_pcib_acpi_attach+0x42b/frame 0xffffffff82e6dd50
device_attach() at device_attach+0x43d/frame 0xffffffff82e6dda0
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6ddc0
acpi_probe_children() at acpi_probe_children+0x6f/frame 0xffffffff82e6de20
acpi_attach() at acpi_attach+0x9dc/frame 0xffffffff82e6deb0
device_attach() at device_attach+0x43d/frame 0xffffffff82e6df00
bus_generic_attach() at bus_generic_attach+0x2d/frame 0xffffffff82e6df20
device_attach() at device_attach+0x43d/frame 0xffffffff82e6df70
bus_generic_new_pass() at bus_generic_new_pass+0x109/frame 0xffffffff82e6dfa0
root_bus_configure() at root_bus_configure+0x26/frame 0xffffffff82e6dfc0
configure() at configure+0x9/frame 0xffffffff82e6dfd0
mi_startup() at mi_startup+0xb5/frame 0xffffffff82e6dff0


Attempting a full reinstall on this DEC840 now.

edit: the reinstall was successful

Instead of directly fetching from the serial console you can also pipe "# dmesg" into a file, but make sure it has the ---<<BOOT>>--- in there.

I can't, because it doesn't finish booting to the point where I can get a shell and look at dmesg :)

> Possibly unrelated, but is one of the ax ports configured for DHCP as a client?

that is correct, ax1 is a DHCP client to my ISP

OPNsense 26.1.8_5-amd64

I have Quantum Fiber for my internet service and my CPE equipment is a C5500XK. Unfortunately I don't have a GPON SFP that will let me bypass this equipment at this time.

I have a 10GBASE-T SFP+ 30m https://www.fs.com/products/111919.html

If I attempt to boot my DEC840 with this physical link up it gets stuck and fails to complete booting. I have tried replacing this SFP as I have many on hand, but they all produce the same behavior. Changing this out for an 1G transceiver works fine and it boots noticeably faster than the 10G.

I am attaching the boot log captured over serial console at boot via script+screen. I had to clean this up a bit as it had all the terminal control characters and color sequences littered throughout but I think it's fully intact.

What I have observed is that there are some additional i2c devices detected when the 10G module is used, and it seems to fail to configure/communicate with the SFP based on the timeouts it displays in the boot log.

If I boot the firewall without the physical link up (unplug ethernet), and then plug in after fully booted it works fine -- or at least I don't notice any issues with the link or its stability.


edit: I want to mention that this hardware configuration worked fine for a couple years, but I can't be certain when this problem started happening. I believe it started somewhere within the 25.x series.

The best way to deal with bufferbloat is to use CAKE+FQ_CoDel but OpnSense/FreeBSD only supports FQ_CoDel right now. This is a pretty good option though.

Better to turn off backups in System: Settings: Misc -- likely caused by a damaged netflow backup

I also encountered this damaged netflow backup problem. It was preventing the firewall's ability to boot. I manually deleted the /conf/netflow.tgz and cleaned out /var/netflow. I'm turning off the "Capture Local" netflow setting that saves data locally because this is just too dangerous to have such a situation where the firewall can fail like this.

I posted about this in the forum for the previous series here, but I don't think this feature is actually working correctly for UDP or there's some other subtle bug going on.

I have several outbound Wireguard VPN tunnels that go through this pfSense firewall from my servers behind it. The Wireguard tunnels will failover to my 5G backup connection if my fiber goes down, but they never switch back to the fiber connection when the fiber comes back up. I have both gateways in the group configured correctly with failover and failback. The only way I can get the tunnel to move back to the fiber is to manually restart the Wireguard services on this servers.

Has anyone else encountered this behavior?

Looked closer at this again now that I've upgraded. Still no firewall logs at all for this traffic which is super strange because all violations are supposed to be logged as I have "Default block" logging enabled.

Noticed the Wireguard rule in the Floating was only applying to the WAN IF but the OpenVPN one for all interfaces (did this change? dunno).

Made a new rule to allow Wireguard UDP 51820 on all interfaces, now it's working from the LAN.

Still doesn't explain the lack of logs, but at least there's a known fix.

Specifically this is about connecting to the VPN when you're already on the LAN. e.g., if I am away from home with VPN enabled on my phone, then come home and connect to my WiFi without disabling the VPN. A previous update about a year ago broke this functionality for both OpenVPN and Wireguard.

Current status:

OpenVPN can connect just fine when on the LAN.

Wireguard still mysteriously fails to connect when on the LAN.

Please allow renaming gateways. There has to be a way to make this possible...

I have a multi ISP configuration and when I moved recently and brought up all of my gear, got a new primary ISP installed, and discovered I cannot rename gateways with incorrect names. Creating all new gateways and gateway groups, applying them all across the firewall rules where I specify different gateways, etc is tedious and error prone. I just want to update their names 🥲

Aha! I had Failover States enabled, but not Failback States. I don't know why you wouldn't want that as the default behavior, though.

When my WAN fails over and fails back it doesn't clear firewall states so traffic still tries to use the WAN that was previously routing the traffic. This is most noticeable when my primary WAN comes back online and traffic still flows through my backup WAN because the states still exist and the network is still functional, so it's not like it's going to have any TCP RSTs or timeouts that push the traffic back to my primary WAN.

Is there a solution to this that I'm not aware of?

I am having no issues with mine, but I'm using SSL / port 636. If you can switch to that I'd recommend it as implicit TLS is more secure that explicit TLS anyway.

Something did change because it used to work just fine roaming the VPN between LTE/5G and my LAN. Even if you change it to "UDP IPv4" so those errors go away in the logs the connection gets established but traffic seems to fail to pass. It broke for me with OpenVPN and Wireguard at the same time.

It was working flawlessly in the past for several years...

With pf and ipfw it's quite easy to write a single line rule that lets you define the allowed ICMP and ICMP6 types, but with OpnSense you have to create an individual rule for each type. Can this be refactored to allow selecting multiple types just like you can select multiple interfaces?

�

Code Select Expand


feld@gw:~ $ netstat -I ax0
Name    Mtu Network        Address                             Ipkts                 Ierrs                 Idrop       Opkts                 Oerrs                  Coll
ax0    1500 <Link#5>       f4:90:ea:00:62:2d                  382472  18446744073709551610                     0  1703087022                     0                     0
ax0       - fe80::%ax0/64  fe80::f690:eaff:fe00:622d%ax0           0                     -                     -           0                     -                     -



this is impossible :)