OPNsense Forum

English Forums => General Discussion => Topic started by: mveerman on August 18, 2026, 10:49:42 AM

Title: TCP/53 fails when DNS responses cross the TCP segmentation boundary
Post by: mveerman on August 18, 2026, 10:49:42 AM
## Problem

I am seeing a reproducible problem with **DNS over TCP through OPNsense**. The failure is strongly correlated with the size of TCP segments/packets.

The problem is not specific to Unbound or to a particular upstream DNS provider.

## Environment

* OPNsense is the gateway/firewall.
* Ziggo modem: `192.168.78.1`
* OPNsense WAN IP: `192.168.178.12`
* WAN interface: `ue0`
* Test query:
  `drill -t @192.168.178.1 rdw.nl DNSKEY`
* The DNSKEY response is approximately 724 bytes.

## Reproduction

With a TCP MSS normalization rule applied to the LAN traffic:

* MSS **700** → DNSKEY query fails
* MSS **800** → DNSKEY query succeeds
* MSS **500** → DNSKEY query fails

This is highly reproducible.

At MSS 700, the TCP handshake succeeds and the DNS query is successfully delivered to `192.168.178.1:53`. The client then waits for the DNS response and eventually closes the connection.

Example:

```
192.168.178.12:46281 > 192.168.178.1:53  SYN, MSS 700
192.168.178.1:53 > 192.168.178.12:46281  SYN-ACK, MSS 1460
192.168.178.12:46281 > 192.168.178.1:53  DNSKEY query
192.168.178.1:53 > 192.168.178.12:46281  ACK
```

But no DNS response is delivered to the client. After approximately 5 seconds the client closes the connection.

With MSS 800, the same query succeeds. The response is:

```
192.168.178.1:53 > 192.168.178.12:12536
TCP payload: 726 bytes
IP total length: 778 bytes
```

and the client immediately ACKs and receives the complete 724-byte DNS response.

## Important additional observation

This does **not appear to be an Unbound-specific problem**.

The same behaviour has been observed when testing DNS servers/providers including:

* Google DNS
* Cloudflare DNS
* Quad9
* the local Ziggo modem

The issue therefore appears to be somewhere in the OPNsense/network path handling of TCP DNS rather than the DNS resolver itself.

## Normal TCP segmentation works

I also tested downloading a large file over HTTPS through the same OPNsense installation.

A packet capture showed the remote HTTPS server sending many TCP segments with approximately 688 bytes of TCP payload:

```
78.46.170.2:443 > 192.168.178.12:17455
IP length: 740
TCP payload: 688
```

These packets are received and ACKed normally and the large HTTPS transfer works.

Therefore this does **not** appear to be a general inability of OPNsense/the NIC/network stack to handle TCP segmentation.

The interesting characteristic is that the failure appears to be specific to **TCP DNS traffic (port 53)**.

## Other troubleshooting already performed

The following have already been checked/tested:

* Hardware checksum offloading disabled.
* Hardware TSO/TCP segmentation offload disabled.
* VLAN hardware filtering disabled.
* Other relevant hardware offload settings have already been disabled.
* MSS/normalization has been explicitly tested.
* The failure is reproducible by changing only the MSS normalization value.
* The normal MSS configuration also exhibits the problem with larger DNS responses, so this is not simply a permanently configured MSS=700 issue.

OPNsense's documentation confirms that normalization/scrubbing can modify TCP MSS and perform fragment reassembly, so this seemed relevant to investigate.

## Packet capture demonstrating successful case

With MSS 800:

```
192.168.178.12:12536 > 192.168.178.1:53
DNSKEY rdw.nl

192.168.178.1:53 > 192.168.178.12:12536
IP length 778
TCP payload 726 bytes
DNS response payload 724 bytes

192.168.178.12 > 192.168.178.1
ACK
```

The query completes successfully.

With MSS 700, the handshake and DNS query complete, but the response does not reach the client.

## What I would like help identifying

At this point I suspect a problem somewhere in the OPNsense/PF networking path involving **TCP DNS and packet/segment size**, possibly normalization/scrubbing, state handling, filtering, or another PF/network-stack interaction.

The fact that ordinary HTTPS TCP segmentation works makes this particularly interesting: it seems to be **TCP segmentation/packet-size handling that is specific to TCP/53**, rather than a generic TCP segmentation failure.

I would appreciate guidance on:

1. How to determine exactly where the DNS TCP response is being lost.
2. Whether PF scrub/normalization has any known interaction with TCP DNS responses of this size.
3. Whether there are known OPNsense/pf issues involving TCP port 53 and segment sizes around this range.
4. Which additional packet captures or `pfctl` diagnostics would be most useful to determine whether PF receives, modifies, drops, or fails to forward the DNS response.