Cannot open TCP connections from DHCP-provided DNS server IPs to opnsense

[mcorddry]

Hi Folks,

Here's an interesting one. I've found that I cannot open inbound TCP connections from IPs that are configured as DNS servers in opnsense, but only when Allow DNS server list to be overridden by DHCP/PPP on WAN is checked in system_general.php. When the same DNS server IPs are specified manually and the box is not checked, TCP connections can be opened easily.

This is because the SYN ACK packets are being sent to the HWADDR of the gateway rather than the HWADDR of the DNS server. The DNS server is on the same L2 segment as the opnsense machine. I've found that this happens when system.inc:system_resolvconf_generate() injects host routes, which for some reason only happens when the system is using DHCP-provided DNS servers.

I've confirmed that this is the source of the host routes by adding a debug log statement just after the call to system_host_route() within system_resolvconf_generate() at etc/inc/system.inc:237. This is only called when Allow DNS server list to be overridden by DHCP/PPP on WAN is set.

I've also noticed that unchecking Allow DNS server list to be overridden by DHCP/PPP on WAN does not clear the host routes, so a reboot (or route del 192.168.0.32) is required to clean up the problem.

In the following examples:

• 192.168.0.254 (00:00:5e:00:01:01) is the default gateway (a different opnsense virtual IP using carp, unrelated to this machine)
• 192.168.0.32 (00:0d:b9:47:25:40) is the DNS server (linux debian)
• 192.168.0.5 (00:a0:98:6f:f4:40) is this opnsense machine
• The machine is a fresh installation of OPNsense 24.1.6-amd64 with just the LAN interface enabled
• I'm attempting to ssh from the DNS server to the opnsense machine. The TCP connection is never established.

With Allow DNS server list to be overridden by DHCP/PPP on WAN not set:

Code Select Expand

# route -v -n get 192.168.0.32
RTA_DST: inet 192.168.0.32; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
192.168.0.32 link#0
   route to: 192.168.0.32
destination: 192.168.0.0
       mask: 255.255.255.0
        fib: 0
  interface: vtnet0
      flags: <UP,DONE,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

locks:  inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
192.168.0.0 link#1 255.255.255.0 vtnet0:0.a0.98.6f.f4.40 192.168.0.5

# tcpdump -n -i vtnet0 -e src or dst host 192.168.0.32 and tcp and src or dst port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:34:21.463675 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 74: 192.168.0.32.38774 > 192.168.0.5.22: Flags [S], seq 3385995390, win 64240, options [mss 1460,sackOK,TS val 2147404981 ecr 0,nop,wscale 7], length 0
10:34:21.463721 00:a0:98:6f:f4:40 > 00:0d:b9:47:25:40, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.38774: Flags [S.], seq 704437459, ack 3385995391, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 432472754 ecr 2147404981], length 0
10:34:21.463923 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 66: 192.168.0.32.38774 > 192.168.0.5.22: Flags [.], ack 1, win 502, options [nop,nop,TS val 2147404982 ecr 432472754], length 0
10:34:21.466826 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 106: 192.168.0.32.38774 > 192.168.0.5.22: Flags [P.], seq 1:41, ack 1, win 502, options [nop,nop,TS val 2147404985 ecr 432472754], length 40
10:34:21.466862 00:a0:98:6f:f4:40 > 00:0d:b9:47:25:40, ethertype IPv4 (0x0800), length 66: 192.168.0.5.22 > 192.168.0.32.
^C

With Allow DNS server list to be overridden by DHCP/PPP on WAN set:

Code Select Expand

# route -v -n get 192.168.0.32
RTA_DST: inet 192.168.0.32; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
192.168.0.32 link#0
   route to: 192.168.0.32
destination: 192.168.0.32
    gateway: 192.168.0.254
        fib: 0
  interface: vtnet0
      flags: <UP,GATEWAY,HOST,DONE,STATIC>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

locks:  inits:
sockaddrs: <DST,GATEWAY,IFP,IFA>
192.168.0.32 192.168.0.254 vtnet0:0.a0.98.6f.f4.40 192.168.0.5

# tcpdump -n -i vtnet0 -e src or dst host 192.168.0.32 and tcp and src or dst port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:24:31.951129 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 74: 192.168.0.32.59048 > 192.168.0.5.22: Flags [S], seq 4118321306, win 64240, options [mss 1460,sackOK,TS val 2146815468 ecr 0,nop,wscale 7], length 0
10:24:31.951217 00:a0:98:6f:f4:40 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.59048: Flags [S.], seq 42904569, ack 4118321307, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2836344428 ecr 2146815468], length 0
10:24:32.947798 00:a0:98:6f:f4:40 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.59048: Flags [S.], seq 42904569, ack 4118321307, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2836345424 ecr 2146815468], length 0
10:24:32.964348 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 74: 192.168.0.32.59048 > 192.168.0.5.22: Flags [S], seq 4118321306, win 64240, options [mss 1460,sackOK,TS val 2146816482 ecr 0,nop,wscale 7], length 0
10:24:32.964389 00:a0:98:6f:f4:40 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.59048: Flags [S.], seq 42904569, ack 4118321307, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2836345438 ecr 2146816482], length 0
10:24:34.017659 00:a0:98:6f:f4:40 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.59048: Flags [S.], seq 42904569, ack 4118321307, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2836346494 ecr 2146816482], length 0
^C

I've disabled all firewalling & NAT on this machine (it's an internal test machine, no public IP) to ensure that nothing is interfering with the packets. The interfaces are only configured with ipv4.

Code Select Expand

# pfctl -sa
FILTER RULES:

INFO:
Status: Disabled                              Debug: Urgent

State Table                          Total             Rate
  current entries                        0
  searches                               0            0.0/s
  inserts                                0            0.0/s
  removals                               0            0.0/s
Counters
  match                                  0            0.0/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  map-failed                             0            0.0/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
sctp.first                  120s
sctp.opening                 30s
sctp.established          86400s
sctp.closing                900s
sctp.closed                  90s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start            60000 states
adaptive.end             120000 states
src.track                     0s

LIMITS:
states        hard limit   100000
src-nodes     hard limit    10000
frags         hard limit     5000
table-entries hard limit   200000

TABLES:
__lan_network
__lo0_network

# netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.254      UGS      vtnet0
127.0.0.1          link#2             UH          lo0
192.168.0.0/24     link#1             U        vtnet0
192.168.0.5        link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::1                               link#2                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0


This is a test machine, let me know if I can try anything else out to help isolate what seems like a small bug. Thanks much.