Messages

> Could this be a compatibility issue between FreeBSD and Broadcom network cards?

Always a possibility - did you search for the underlying FreeBSD version and your interface model(s)?

Is it just 1 interface? Or multiple of the same model?

If you have a ZFS snapshot, rolling back and testing again should be easy enough.

Did you check the suggested operating temperature for the drive in the specs?

I think there's definitely an argument for it to be disabled by default.

... I'm not sure I need to put unnecessary wear on an SSD for this.

I'd have thought that most Business Edition customers will disable it and they bring in the money!

I cannot comment on changes between 25.1 and 25.7 specifically, nor first hand experience with bge interfaces ....

... BUT if you have ASPM enabled in the BIOS, I'd try turning that off (everywhere). 

Also set a tuneable, then reboot after setting:

Code Select Expand

hw.pci.enable_aspm=0
.. has fixed all sorts of interface randomness over the years for me, although primarily on Intel.  Then if you want to narrow it down after - i.e it resolves the issue - selectively re-enable.

You can check ASPM state with:

Code Select Expand

pciconf -lcv |grep -i aspm

Installed the latest version: hostwatch-1.0.6.pkg

Writes, for me, seem to be more or less the same.  Not clear whether this is just 'how things will be' with this service enabled. 

But, for completeness, screenshot added again.  Set to 'All' interfaces which is:

Code Select Expand

Initialized 21 packet device_captures
If I filter out the various VPNs, WAN - cable modem on the WAN, so can be noisy on the front end - it's then 'Initialized 11 packet device_captures', writes roughly half - as should probably be expected.

... with only a handful of interfaces, or only a handful enabled for this service, the writes are probably negligible. 

But the writes do seem to be constant when monitoring with 'zpool iostat -v 1' - for me, whilst it is an Enterprise SSD, I think I can live without the convenience this service is designed to bring.

Not seeing any signs of logs etc growing in size, nor CPU spikes.

I updated from 25.1.x -> 25.7.11_2-amd64 and whilst I didn't see the logs/disk usage growing (due to the _2 hot fix), having automatic discovery did lead to increased writes for me.

... not massive, but unnecessary in my view.  Screenshot attached - you can see when I disabled it, just after 16:00.  Auto Neighbour Discovery is unnecessary for my usage.

Personal preference would be that this is disabled by default, but it seems like I'll just need to remember to disable it on new builds/installs now!

For anyone that is curious, you can use iostat 1 (UFS) or zpool iostat -v 1 (zfs)

Are you using unicast sync on both opnsense and pfsense?

The opnsense documentation seems to suggest specifying a unicast address, but the pfsense documentation seems to lean more towards 'not' and using multicast.

EDIT: Going back a bit, looks like someone else had an issue with Unicast:

https://forum.opnsense.org/index.php?topic=34522.0

There were Intel NIC driver changes in 24.7.8 - if you have a saved boot environment, you could try rolling back.

This is very odd, I will admit there are things that just don't make sense to me :)

- That you DO see (tcpdump) the DNS request enter the LAN interface and pass out of the PPPoE interface.  Which means it's passing through opnsense, i.e is not being blocked, dropped, etc.

- That you DONT see the issue with OpenWRT, suggests the connection itself is good

- That you DO still see the problem with a completely (completely-completely?) clean install

Have you ruled out things like specific ports (on the device, or switch ports, etc), cables, etc?

If it's easy-ish to do, I think opnsense 24.1 would be interesting to test (i.e FreeBSD 13, rather than 14.1).

I'd also make use of the Cloud Backups .... again, a life saver for when the drive decides to 'give up the ghost' at LOL o'clock :)

https://docs.opnsense.org/manual/how-tos/cloud_backup.html

> Do you mean the provider's firewall?

Your firewall, opnsense :)

From your tcpdump, you see the DNS request in the PPPoE interface dump, so it makes it through opnsense and gets 'dropped onto the wire' of the WAN interface.

Well, it seems to make it through the firewall...

Can you dig @1.1.1.1 when you see the problems with 8.8.8.8?

Are you sure your ISP doesn't rate limit UDP traffic?  If you remove opnsense completely from the equation, if you can, and use an ISP router, do you see the same then?

From your timestamps, I'm guessing you're in South East Asia (GMT+7) somewhere?  I know for a fact some of the ISPs in that region do filter/limit traffic.

EDIT: I also missed this:

Code Select Expand

21:04:17.791183 IP 100.68.87.90.50947

... you're behind CGNAT? 

I think the next step is to prove that you don't see the same problem, when you remove opnsense, i.e using the ISP supplier router if you have one.

Another question: Did you see this problem on 24.1?  Or have you only ever run 24.7?

I don't see this mentioned here ... so ...

... have you checked the firewall logs for drops? 

If you tcpdump the LAN side interface, do you see your DNS requests ingress to the interface?  For example:

Code Select Expand

tcpdump -i lan-interface host 8.8.8.8 and port 53


If you do, then if you tcpdump the WAN side interface, do you see your DNS requests egress the WAN interface?

Code Select Expand

tcpdump -i wan-interface host 8.8.8.8 and port 53


Don't disable RSS.

This is something that Crowdsec might be responsible for...is it running ?

This makes zero sense.

The default install, is with RSS disabled.  If you have problems, the absolute first thing anyone should do, is go back to stock/default.