Messages

just wanted to post a follow up. after my last changes (those few tunables), i have had very stable experience. i updated to 26.4 and migrated my rules, but i have had 100% uptime on OS and WAN since i updated and rebooted for the update.

haven't had a change to redo some of my cable management since i replaced the unifi cloud gateway fiber, but i like the pop of red :) lol

The logout logic of opnsense is a bit hit or miss... Most people never realize it because they don't keep sessions open for 4 hours.

This could have happened here.

Also changing from https to http can be catastrophic for up to 1-2 weeks. Chrome in particular if it has connected to a domain or ip using https beforehand, it refuses to acknowledge that http is a real thing.

Generally speaking, I would try incognito sessions.

i tried just accessing the with curl which also failed. guess what, it could be something wrong with OPNsense. its not bug free. its also fine. nothing is perfect. but it doesn't seem like something anyone would prioritize, which is also fine.

If it's the timeout issue, then no need to reboot.  OPNsense is fine and it's just the web session that's gone stale.

Hit the browser refresh and log in again.

yea, i think i tried refreshing the page, lol. i even closed the browser and couldn't get to the login screen. all networking seemed fine and like i said, ssh worked and i rebooted it via the console and then the web ui worked again. i didn't report anything because i had nothing in the logs, so nothing was going to done about it (which is fine)

yea, have also encountered this, i accidentally had a tab open for a long time, and when i came back nothing was working in the UI, i could ssh, rebooted and the web ui working again. normally i don't do this, but i had the tab up, got distracted, and when i came back things were weird. i looked in the logs, lighthttpd logs, and saw nothing. so i rebooted and moved on with my life, but it was strange.

i gave adguard a try and i am currently using that and working on adguard plugin, the 3rd party repo adguard home plugin doesn't really have a native interface...

That is some effort that you are putting in... AGH has a ton of options and to capture them all into an OPNsense GUI and then ensure that future updates also work.... Hats off to your effort.

I very much look forward to your plugin... Is there a GitHub link for it?

its not ready yet, plus i am still sort-of on the fence about adguard home myself, so just giving adguard home more time and if i decide to stick with it, i'll finish the work and make the PR, because yes, i probably won't want to maintain a plugin i wasn't using myself

[that hard to simply check your Pi-Hole Query Log (or the same in AdGuard) and figure out what is being blocked]

To be honest : People should really avoid needing such a feature!

Is it really that hard to simply check your Pi-Hole Query Log (or the same in AdGuard) and figure out what is being blocked and solve the issue once and for all ?!
_{Nothing personal by the way. Talking about people in general!}

And in the worst case you even have the option to use some kind of DoH/DoT feature in Mozilla Firefox for example to check if your adblocker is the issue at all :)

It is not about whether I can analyse and check what websites are being blocked and then whitelist these permanently, its about other users in the home who are not that tech-savvy... and for the times that I am not available to do the same.

Hence, my request for a API call (I know that it exists, as the OPNsense API documentation does refer to adblocking... I just haven't been able to get my head around it to make it work !) - if anyone who understands API can help; then it would be much appreciated.

i started making PR for a unbound dashboard widget that used the API to enable/disable blocking (there isn't a timer mechanism though like PiHole or Adguard). the main issue i had was that the act of enable/disable in opnsense and its custom blocklist implementation it that if you have decent sized lists, the operation can take seconds, like 5 or more on modern CPU, 10+ on smaller embedded cpus. its just not built with quick enable/disable/reloading in mind.

i gave adguard a try and i am currently using that and working on adguard plugin, the 3rd party repo adguard home plugin doesn't really have a native interface...

PR: https://github.com/opnsense/plugins/pull/5336

Issue: Lots of RX errors / missed packets when the NIC is connected to Arris ONT Calix 711GE ONT, connecting it to switch or computer gives no error. Swapping igc0/igc1 makes no difference. So many pkts are missed that it will frequently look like DNS issues, due to missed SYN-ACKs and long delays to connect to websites.

interesting, i seemed to have not had any issues with a Protectli VP2440 connected to Calix 711GE ONT, but then i replaced it with an official Deciso DEC3920 and had tons of WAN issues. my issues tho, the WAN would go completely out and only physically removing and reinserting the cable would restore the connection. i seemingly got it stable, but only after so many tweaks back and forth, i don't 100% know the root cause, only the set the tunables that seems to be stable -> https://forum.opnsense.org/index.php?msg=264927

i upgraded the protectli from 2.17 to 2.32 firmware, i left the deciso at 2.25 *shurg*

edit: oh i c, i was also running 0.9.1-rc3 on VP2440.. yea 0.9.0 coreboot was 100% broken for FreeBSD + igc because of ASPM. AMI bios completely disabled ASPM, adds 4-5w to idle, which is like 50%, but also solves the issues.

[aspm_disable]

i stumbled across this bug discussing the i226 TX hang on FreeBSD: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279245

there is shell script, aspm_disable attached at the bottom, you can easily disable ASPM on PCI device at run-time / boot. seems to work no matter the BIOS settings for ASPM control. only tested on OPNsense DEC3920.

Code Select Expand

# pciconf -l | grep igc
igc0@pci0:1:0:0:        class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc1@pci0:2:0:0:        class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc2@pci0:3:0:0:        class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc3@pci0:4:0:0:        class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000

# aspm_disable 02:00.0
  PCIe capability found at offset 0xa0
  Link Control offset: 0xb0
  Current Link Control: 0x0042
  New Link Control (ASPM disabled): 0x0040
  setpci -s 02:00.0 b0.w=0040
  ASPM disabled for 02:00.0.

i copied the script to /usr/local/bin and then added a syshook script to execute early (before network) on boot:

Code Select Expand

# chmod 755 /usr/local/etc/rc.syshook.d/early/99-aspm-disable
# cat /usr/local/etc/rc.syshook.d/early/99-aspm-disable
#!/bin/sh
/usr/local/bin/aspm_disable 02:00.0 > /dev/null


yea i saved the config.xml from first boot. it only had 3 tunables.

Code Select Expand

hw.ibrs_disable=1
vm.pmap.pti=0
ice_ddp_load=YES

yea, without bindthreads/maxthreads/rss_enabled, the IRQs are mapped to separate CPUs, but the netstat stats show all the packets being queued on 1 cpu. honestly for 1 Gbps, i find it strange that this CPU can't do it with just 1 core, but yea.

with RSS disabled, watching sysctl

Code Select Expand

# sysctl dev.igc.1.iflib.{txq0,txq1,txq2,txq3}.r_enqueues
dev.igc.1.iflib.txq0.r_enqueues: 3928608
dev.igc.1.iflib.txq1.r_enqueues: 1070
dev.igc.1.iflib.txq2.r_enqueues: 1219
dev.igc.1.iflib.txq3.r_enqueues: 235

all the packets seem to go to txq0, but maybe its because 1 Gbps isn't enough to stress even one core.

strange is that i never really root cause my connection issue. was it because i powered off the ONT for 5 minutes and it really let it reset? was it because i moved from igc0 to igc1 for WAN?

[aspm_disable]

i stumbled across this bug discussing the i226 TX hang on FreeBSD: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279245
there is shell script, aspm_disable attached at the bottom, you can easily disable ASPM on PCI device.

Code Select Expand

# pciconf -l | grep igc
igc0@pci0:1:0:0:        class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc1@pci0:2:0:0:        class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc2@pci0:3:0:0:        class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc3@pci0:4:0:0:        class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000

# aspm_disable 02:00.0
  PCIe capability found at offset 0xa0
  Link Control offset: 0xb0
  Current Link Control: 0x0042
  New Link Control (ASPM disabled): 0x0040
  setpci -s 02:00.0 b0.w=0040
  ASPM disabled for 02:00.0.

interestingly enough, disabling ASPM on igc1 had no noticeable effect on the idle power consumption.

i was able to completely eliminate the Oerrs by using traffic shaper. my bufferbloat grade was a C, so i did the traffic shaping from the OPNsense docs and got an A+, slightly lower max bandwidth, but now the WAN interface is overwhelming the ONT.

and also dialed back on the tunables, once traffic shaping seem to eliminate all interface errors.

# make sure ax0 and igc1 don't overlap cpus, leave cpu 0 for system interrupts

Code Select Expand

dev.ax.0.iflib.core_offset=1

# igc tweaks

Code Select Expand

dev.igc.1.fc=0

# enable RSS

Code Select Expand

net.inet.rss.bits=3
net.inet.rss.enabled=1
net.isr.bindthreads=1
net.isr.maxthreads=-1

# enabled zenarmor, increased buffers to prevent netmap full errors in dmesg

Code Select Expand

dev.netmap.admode=2         # added by zenarmor
dev.netmap.buf_num=1000000  # added by zenarmor
dev.netmap.generic_rings=4
dev.netmap.generic_ringsize=2048
dev.netmap.ring_num=1024    # added by zenarmor
dev.netmap.ring_size=131072

# factory defaults

Code Select Expand

hw.ibrs_disable=1
vm.pmap.pti=0
ice_ddp_load=YES


yea i don't think its an actual issue. before, my connection was dropping off completely and the only fix to physically unplug and plug the cable back in, nothing i did on CLI like if down / if up, brought the connection back, so at least i feel i am at a better spot.

ok, i couldn't wait, i swapped WAN to ax1 with UniFi 1G SFP to RJ45 adapter.

run speedtest

Code Select Expand

speedtest --server-id=8862

   Speedtest by Ookla

      Server: CenturyLink - Denver, CO (id: 8862)
         ISP: CenturyLink
Idle Latency:     2.97 ms   (jitter: 0.04ms, low: 2.92ms, high: 3.00ms)
    Download:   939.80 Mbps (data used: 655.7 MB)
                  4.18 ms   (jitter: 23.82ms, low: 2.22ms, high: 438.52ms)
      Upload:   937.74 Mbps (data used: 434.3 MB)
                 37.86 ms   (jitter: 3.14ms, low: 3.09ms, high: 50.13ms)
 Packet Loss: Not available.
and on first try, i got Oerrs on WAN. it must be related to the ONT.

Code Select Expand

vlan0.201  1500 <Link#16>         f4:90:ea:01:ef:d1             1049994     0     0   801383   238     0
also, as expected, using SFP to RJ45 adapter, the idle power went from 10.1w to 12.6w

FWIW, i ran speedtest 3x, and 3x the Oerrs increased. i just set net.link.ifqmaxlen=4096 and rebooted (its a boot-time tunable). i ran speedtest 3x and Oerrs increased only once. i just don't think this matters.

i'll try WAN on ax1 with UniFi 1G SFP to RJ45 adapter later this weekend.