Messages

Having the same issue here, but I do indeed have a second repository loaded - SunnyValley (For Sensei).

Scratch that... Disabling DNSSEC does not make the issue go away.  Instead, it just delays it by a few days at a time.

I've been having a similar issue with Unbound as well, but I think I have narrowed it down to DNSSEC support. Using Log Level 2, I was able to observe the following snippet of logs:

Version 18.7, but this issue was also present on 18.1.[10-13] (possibly earlier)

Aug  1 12:35:18 gateway unbound: [71731:7] info: response for ghostery-collector.ghostery.com. AAAA IN
Aug  1 12:35:18 gateway unbound: [71731:7] info: reply from <.> 2620:fe::9#853
Aug  1 12:35:18 gateway unbound: [71731:7] info: query response was nodata ANSWER
Aug  1 12:35:18 gateway unbound: [71731:7] info: resolving ghostery.com. DS IN
Aug  1 12:35:19 gateway unbound: [71731:7] info: DS response was error, thus bogus
Aug  1 12:35:19 gateway unbound: [71731:7] info: resolving ghostery.com. DS IN
Aug  1 12:35:19 gateway unbound: [71731:7] info: DS response was error, thus bogus
Aug  1 12:35:19 gateway unbound: [71731:7] info: resolving ghostery.com. DS IN
Aug  1 12:35:19 gateway unbound: [71731:7] info: DS response was error, thus bogus
Aug  1 12:35:19 gateway unbound: [71731:7] info: resolving ghostery.com. DS IN
Aug  1 12:35:19 gateway unbound: [71731:7] info: DS response was error, thus bogus
Aug  1 12:35:19 gateway unbound: [71731:7] info: resolving ghostery.com. DS IN
Aug  1 12:35:19 gateway unbound: [71731:7] info: DS response was error, thus bogus
Aug  1 12:35:19 gateway unbound: [71731:7] info: resolving ghostery.com. DS IN
Aug  1 12:35:19 gateway unbound: [71731:7] info: DS response was error, thus bogus
Aug  1 12:35:19 gateway unbound: [71731:7] info: Could not establish a chain of trust to keys for ghostery.com. DNSKEY IN
Aug  1 12:35:32 gateway unbound: [71731:b] info: resolving tmi.twitch.tv. A IN
Aug  1 12:35:32 gateway unbound: [71731:4] info: resolving tmi.twitch.tv. A IN
Aug  1 12:35:36 gateway unbound: [71731:4] info: resolving api-global.netflix.com. AAAA IN
Aug  1 12:35:36 gateway unbound: [71731:0] info: resolving secure.netflix.com. AAAA IN
Aug  1 12:35:36 gateway unbound: [71731:c] info: resolving customerevents.netflix.com. AAAA IN
Aug  1 12:35:36 gateway unbound: [71731:c] info: resolving customerevents.netflix.com. AAAA IN
Aug  1 12:35:36 gateway unbound: [71731:5] info: resolving cdn-0.nflximg.com. AAAA IN
Aug  1 12:35:36 gateway unbound: [71731:c] info: resolving customerevents.netflix.com. AAAA IN
Aug  1 12:35:36 gateway unbound: [71731:4] info: resolving secure.netflix.com. A IN

Prior to the "Could not establish a chain..." log entry, DNS resolution is working fine. "ghostery.com" just happened to be the domain that triggered it this time.  I've also witnessed it with others, including google.com, and microsoft.com.  Once the "chain of trust" error occurs, Unbound will log that it's resolving other requests, but it never actually performs the query, nor does it respond to the client's request. It requires a restart of the Unbound service to restore functionality -  at least, until it happens again.

The temporary, but undesirable solution for now is to disable DNSSEC in Unbound. Unbound doesn't get choked up with DNSSEC disabled.

A little more about my config -
I'm forwarding all my queries to Quad9 via TLS on IPv4 and IPv6.  Additionally, I run a local BIND9 server for local resolution that us configured as a Stub in Unbound.

Ah, makes sense....

dcol suggested this a bit ago and I will implement it soon so all the loader.conf stuff can reside in config.xml backup and restore as well:

https://github.com/opnsense/core/issues/2083


Cheers,
Franco

That would be awesome. I appreciate all you guys do. Opnsense is great.

Dell R610, 48gb RAM, 2x146gb SAS (RAID 1)

Did you use /boot/loader.conf for these? Or were they really gone from /boot/loader.conf.local ? It shouldn't touch .local ever...


Cheers,
Franco

I added them to loader.conf.local manually so they would be safe from being overwritten by future updates.

When I installed v18.1 originally, I did a clean install... Mainly because the v17 install used 100gb of my storage for swap by default. So, loader.conf.local was empty.

Sent from my Pixel 2 XL using Tapatalk

It should go without saying, but you must reboot after making changes to /boot/loader.conf.local ...

I have a solution for my Broadcom-NIC'ed Dell R610. It was definitely a driver issue more than a NAT or anything else.  The hw.bce.* settings only apply to Broadcom, but there's probably similar options for the others who have also replied to this thread.

In /boot/loader.conf.local:

hw.bce.verbose=0
hw.bce.tso_enable=0
hw.bce.rx_pages=8
hw.bce.tx_pages=8
hw.pci.enable_msix=1
hw.pci.enable_msi=1
net.inet.tcp.tso=0
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288

I'm now getting full bandwidth in both directions via opnsense. After getting that sorted out, I enabled dual-stack IPv4/IPv6, IDS and upstream traffic shaping. No more issues.

Gigabit.... If only.  What model card is that you're using?

I checked... I didn't have IDS enabled. Good thought, for sure.

I'm going to roll back to 17 this evening and see what happens.

Sent from my Pixel 2 XL using Tapatalk

There are no errors on any of the interfaces. All are running at 1G, full-duplex.

I'm testing via speedtest.net. Connecting my laptop to the cable modem directly, I get 367M downstream, but the same laptop through opnsense, it slams into a 100M wall. Literally 100M. It's almost as if there's a traffic shaper installed, but there isn't. The tests are all done hard-wired.

There's no gigabit internet service in this area yet. The best we can get is this oddball 360M/8M service. Maybe someday...

I may end up trying to go back to v17 and compare the sysctl output to what I'm seeing in v18.  Otherwise, I'm not sure what else to try.

Sent from my Pixel 2 XL using Tapatalk

I'd check if the Broadcom card is supported at al https://www.freebsd.org/releases/11.1R/hardware.html#ethernet

Code Select Expand

[i386,amd64] The bce(4) driver provides support for various NICs based on the QLogic NetXtreme II family of Gigabit Ethernet controllers, including the following:
QLogic NetXtreme II BCM5706 1000Base-SX
QLogic NetXtreme II BCM5706 1000Base-T
QLogic NetXtreme II BCM5708 1000Base-SX
QLogic NetXtreme II BCM5708 1000Base-T
QLogic NetXtreme II BCM5709 1000Base-SX
QLogic NetXtreme II BCM5709 1000Base-T
QLogic NetXtreme II BCM5716 1000Base-T
Dell PowerEdge 1950 integrated BCM5708 NIC
Dell PowerEdge 2950 integrated BCM5708 NIC
Dell PowerEdge R710 integrated BCM5709 NIC
HP NC370F Multifunction Gigabit Server Adapter
HP NC370T Multifunction Gigabit Server Adapter
HP NC370i Multifunction Gigabit Server Adapter
HP NC371i Multifunction Gigabit Server Adapter
HP NC373F PCIe Multifunc Giga Server Adapter
HP NC373T PCIe Multifunction Gig Server Adapter
HP NC373i Multifunction Gigabit Server Adapter
HP NC373m Multifunction Gigabit Server Adapter
HP NC374m PCIe Multifunction Adapter
HP NC380T PCIe DP Multifunc Gig Server Adapter
HP NC382T PCIe DP Multifunction Gigabit Server Adapter
HP NC382i DP Multifunction Gigabit Server Adapter
HP NC382m DP 1GbE Multifunction BL-c Adapter

Then check if the card has errors (Interfaces -> Overview or netstat -idb -I bce0).

And read the bce(4) manaual page. Maybe increasing the sysctl's hw.bce.rx_pages and hw.bce.tx_pages could help.

The ethernet device is a BCM5709, and is listed on the compatibility list for the bce driver.

I went ahead and increased the tx_pages and rx_pages to "8" (the max) and there's no difference in the throughput.

For grins, I enabled verbose debugging on the bcm driver as well. There are no notices of buffer overruns or anything else that would be of concern. In fact, I don't see much of a difference in the dmesg output between debugging off or on.

I'm at a complete loss as to what to try next. It seems very odd that the throughput is slamming into the wall it's hitting - 100Mbps.



Sent from my Pixel 2 XL using Tapatalk

The top two lines were from a FreeBSD guide for mitigating issues with Broadcom cards - specifically to disable hardware TSO offloading. The same settings can be found in the pfsense wiki.

The buffer sizes were expanded to make use of some of the 48gb of RAM the server has (from it's previous role).

I performed a clean install of 18.1 on my Dell R610 server that has quad-broadcom ports installed and restored my configuration from backup. I'm experiencing some rather significant performance issues now. I have 360Mbps downstream, but SpeedTest is only getting me ~100Mbps now.

On 17.7.11 (and 17.7.12), I added the following to /etc/loader.conf.local to mitigate the issue, but this doesn't seem to be having the same result anymore. I'm wondering if I need to return to 17?

hw.bce.tso_enable=0
net.inet.tcp.tso=0
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288

Does anyone have any thoughts?