[Solved] TCP errors for some websites

[Just]

Hello guys,

I hope this is the right thread for it. Anyway I recently switched from pfSense to opnSense and I face a kinda annoying problem.
For some websites I get alot of "TCP Dup ACK" and "Ignored Unknown Record" messages while tracing the traffic with wireshark. For some sites it makes no difference in performance, they just load fine but for some others they take like 40-60 seconds to finish loading.

For example reddit.com needs like 50 seconds until it finished loading. My wireshark trace looks most of the time like this.


My current setup is the following

ISP <-> FRITZBOX 7490 <-> OPNsense

FRITZ!Box 7490

• wan0: DHCP provided by ISP
• lan1: 10.255.0.1

OPNsense

• wan0: 10.255.0.2
• lan1: 192.168.0.1

Info about OPNsense

• Hardware is an APU2C4 (Firwamre from Dec. 2019)
• OPNsense version 19.7.9_1
• Firewall rules allow anything from the subnet (log is clear)
• using Hybrid NAT, this subnet has no extra NAT rules
• using an OpenVPN client as an alternative gateway, but not for this subnet
• using haproxy as a reverse proxy, but no other proxies

How do I know it must be an issue related to the firewall?

• no TCP errors in wireshark when I'm directly connected to the router of my ISP (loading time is like 5seconds instead 50)
• no TCP errors in wireshark when I use OpenVPN  (also just 5 seconds)

What did I already do?

• changed MTU of WAN Interface (1492, 1300...)
• disabled TCP offload engine
• tried different DNS Servers, browsers and clients (pc, mobile
• hours of googling...

I hope anybody can help me out, since I have absolutly no idea what I can do about it.

Best regards
Just

[Just]

I am not 100% sure if I solved this mystery, but I'll try to explain what I found out.

This issue seems to be an DNS problem in combination with Unbound and DNS-over-TLS using Quad 9 servers (I didn't test any other servers). I used the following guide (https://stafwag.github.io/blog/blog/2018/12/09/configure-dns-tls-on-opnsense/) for DNS over TLS and this worked fine (no DNS issues at all and there was TLS traffic on port 853).
But if I use these custom options, I have the loading problem I described in my original post. If I remove these, the problem is gone. Even when I send the queries directly to 9.9.9.9 instead to the firewall the issue is still there if I haven't removed the custom options for DNS-over-TLS .

My workaround is to use normal DNS for now, but maybe someone knows a different solution, since I would like to keep using DoT.