Messages

Someone should start an LKML thread and CC Torvalds directly.  We have a serious problem here...

You could try enabling query logs in Unbound itself (Advanced->Logging Settings->Log Queries) which I guess would show them in the service logs (Unbound->Log File), but it comes with a performance impact warning:

You cannot view this attachment.

No idea if/how the OPNsense reports can be extended but would also be interested in that...

@stumper doesn't mention this being the case, but I've also seen this kind of issue on my previous cable modem while using the in-line Ethernet surge protection on my UPS.

---(coax)---[modem]---(cat6)---[ups]---(cat6)---[router]---

The 2.5GbE port on the modem was problematic so I used one of its adjacent 1GbE ports for a while.  It was anyway enough for my ISP plan.

My current cable modem (an ISP-issued replacement, newer model) doesn't seem to care and the 2.5GbE port negotiates fine.

Did you install any public VPN instances (e.g. Mullvad) into the WireGuard group without the "Disable routes" option?

I think I saw this too when I was adding instances recently, but I moved the VPNs off to dedicated interfaces (with gateways in my case) and rebooted to flush the routing table.

I'm only keeping my private s2s and road warrior instances on the default WG group and this seems OK.

IIRC, instances with peers having 0.0.0.0/0 or ::/0 in the Allowed IPs list might be getting added as default routes, causing a conflict.  I'd need to retest this though because my memory could be failing.

It sounds similar to, or a regression of, this older issue?

https://www.mail-archive.com/freebsd-net%40freebsd.org/msg52770.html

Does 'sysctl hw.mlxen1.stat.rx_bytes' show correctly?

(Sorry, I was looking at the packet count rather than byte count when I subtracted, but still those are crazy numbers...)

I think FreeBSD would be the best place but I've never submitted a bug there.  Maybe someone else knows.

OPNsense also tracks upstream issues: https://github.com/opnsense/src/issues

I don't know how those get tracked and resolved by upstream, though.  I was always too shy to ask but maybe this is a good time :)

Yeah, I would try a different NIC.  That's crazy.

Side note: the maximum value of a 64-bit unsigned int is 18446744073709551615 bytes (16 EiB - 1 byte) and is likely what netstat uses for its counter, at least according to ChatGPT.

Your result is only 157,015 bytes off from that.  It's probably overflowing the counter and wrapping back around to 0, so that's why you see it increase and decrease probably.

I don't know if it's correct or not but I was curious if there would be a discrepancy between the two sources.  If they agreed (and the data was obviously wrong) then I would have said the problem could be at the OS/driver level.

What you show here is nowhere near 1TB even, let alone 3.96T.  I had ChatGPT convert the numbers to human readable format:

Code Select Expand

Name     Network          RX Packets   RX Data     TX Packets   TX Data
-------  ---------------  -----------  ----------  -----------  ----------

mlxen0   <Link#6>             24,067    16.8 MiB      375,393    621.2 MiB
         Local IPv6            6,753   552.7 KiB        6,768    573.1 KiB
         0.0.0.0/22        1,810,036   597.7 MiB    3,165,999     3.22 GiB
         IPv6 Address         67,311    26.4 MiB      110,451     39.2 MiB

The netstat numbers (I think) are cumulative since the last boot or when the network interface was initialized.

I'm not sure what the reporting period of your insight data is.  In your first post you said you were looking at the interface totals on the Dashboard, which I would guess more closely align to netstat, with some rounding.

Run 'netstat -ib'.

Maybe this way we can at least rule out OPNsense and isolate the issue to FreeBSD.

At least TrueNAS remove the hard 50% limit AFAIK.

Indeed :P

You cannot view this attachment.

For your first issue, this would need some sort of sequence number when interfaces are created, same as the groups already have. It's nothing that could be reasonably auto calculated because we cannot know what the user wants all the time.

Makes sense.  I get why you might not want to introduce sequence IDs at the interface level and it would interfere with the flexibility we now have to order individual interface rules liberally within the "400000.x" range.

I was thinking that it would be trivial to auto-rebalance the sequence numbers (the part after the decimal, not before) to avoid overlaps, but I didn't consider that some users/orgs might actually have overlapping rules on purpose.  I don't know why you would do that (performance optimization?) but as long as the system allows overlaps then my idea is not safe.

For the second issue, this is also the case in the old rules view. Rules are not deleted when an interface vanishes, they stick around in the config. This is also a safety feature against accidental lockouts or deletions. Not sure if we should unravel this one.

Understood; leaving this alone.  I think it's subjective anyway depending how you look at it.  My opinion is that stale interface configs that can be silently re-activated when a device ID is re-assigned is dangerous.  I can't be sure of what's happening with a newly created interface and I feel like I'm breaking my firewall by deleting & creating interfaces.  I guess it's not really a problem though if no one's complaining.

And just one more (sorry!)

I noticed that when you delete an interface from Interfaces->Assignments, any existing rules that were present for that interface get left hanging around in the config.  Next time you assign a new interface that automatically inherits the old device identifier (e.g. opt10) then it also silently inherits the old interface's rules.

Can there be an option (or better, a prompt) to delete interface rules when an interface is removed?

I don't mind adding feature requests for either of these if you think they might be reasonable.

Since we're on the topic there's another quirk that I want to run by you.  When I set up my WG interfaces I cloned rules from WAN_VPN1 -> WAN_VPN2, and this is now reflected in the sequence of the rules.  You can clearly see where I created the first rule, cloned it, created the next two rules, cloned them, etc.

You cannot view this attachment.

I don't think this causes a problem in terms of firewalling because the traffic is anyway exclusive to each interface, but the mixed order of the interface rules is unsettling.

Since I don't manually manage sequence IDs and I typically let the system do it, I'm wondering if there's a possibility to clean up the automatic sequencing so that there are always cleanly separated ranges between interfaces?  Cloning seems problematic for overlaps.

Yep, you're right.  The groups are both set to sequence 2.  My mistake.

Thanks!

After the upgrade from 26.1.9 -> 26.1.10 I am just now realizing an overlap in rule order between two interface groups when using the "All rules" filter in the new UI.  My "IG_OUT_WAN" group is interspersed with the "IG_OUT_VPN" group.  These are the only two affected.

Curiously, both groups are using the same "300002.xxx" sort order which should not happen, right?  I think the last digit in the priority group should be unique per interface/group if I'm not mistaken.

I will roll back to the snapshot for 26.1.9 and check the rule ordering there but that's as far back as I can go.  Was there a change in 26.1.10 that might affect this, or is it likely that this happened during my rule migration several releases ago and I never noticed?

I'm curious how this can happen.  Are there issues with cloning rules between groups that might cause the priority group number to carried over, perhaps?