Show Posts
1
24.1 Legacy Series / Cannot open TCP connections from DHCP-provided DNS server IPs to opnsense
« on: May 15, 2024, 07:55:30 pm »
Hi Folks,
Here's an interesting one. I've found that I cannot open inbound TCP connections from IPs that are configured as DNS servers in opnsense, but only when Allow DNS server list to be overridden by DHCP/PPP on WAN is checked in system_general.php. When the same DNS server IPs are specified manually and the box is not checked, TCP connections can be opened easily.
This is because the SYN ACK packets are being sent to the HWADDR of the gateway rather than the HWADDR of the DNS server. The DNS server is on the same L2 segment as the opnsense machine. I've found that this happens when system.inc:system_resolvconf_generate() injects host routes, which for some reason only happens when the system is using DHCP-provided DNS servers.
I've confirmed that this is the source of the host routes by adding a debug log statement just after the call to system_host_route() within system_resolvconf_generate() at etc/inc/system.inc:237. This is only called when Allow DNS server list to be overridden by DHCP/PPP on WAN is set.
I've also noticed that unchecking Allow DNS server list to be overridden by DHCP/PPP on WAN does not clear the host routes, so a reboot (or route del 192.168.0.32) is required to clean up the problem.
In the following examples:
With Allow DNS server list to be overridden by DHCP/PPP on WAN not set:
With Allow DNS server list to be overridden by DHCP/PPP on WAN set:
I've disabled all firewalling & NAT on this machine (it's an internal test machine, no public IP) to ensure that nothing is interfering with the packets. The interfaces are only configured with ipv4.
This is a test machine, let me know if I can try anything else out to help isolate what seems like a small bug. Thanks much.
Here's an interesting one. I've found that I cannot open inbound TCP connections from IPs that are configured as DNS servers in opnsense, but only when Allow DNS server list to be overridden by DHCP/PPP on WAN is checked in system_general.php. When the same DNS server IPs are specified manually and the box is not checked, TCP connections can be opened easily.
This is because the SYN ACK packets are being sent to the HWADDR of the gateway rather than the HWADDR of the DNS server. The DNS server is on the same L2 segment as the opnsense machine. I've found that this happens when system.inc:system_resolvconf_generate() injects host routes, which for some reason only happens when the system is using DHCP-provided DNS servers.
I've confirmed that this is the source of the host routes by adding a debug log statement just after the call to system_host_route() within system_resolvconf_generate() at etc/inc/system.inc:237. This is only called when Allow DNS server list to be overridden by DHCP/PPP on WAN is set.
I've also noticed that unchecking Allow DNS server list to be overridden by DHCP/PPP on WAN does not clear the host routes, so a reboot (or route del 192.168.0.32) is required to clean up the problem.
In the following examples:
- 192.168.0.254 (00:00:5e:00:01:01) is the default gateway (a different opnsense virtual IP using carp, unrelated to this machine)
- 192.168.0.32 (00:0d:b9:47:25:40) is the DNS server (linux debian)
- 192.168.0.5 (00:a0:98:6f:f4:40) is this opnsense machine
- The machine is a fresh installation of OPNsense 24.1.6-amd64 with just the LAN interface enabled
- I'm attempting to ssh from the DNS server to the opnsense machine. The TCP connection is never established.
With Allow DNS server list to be overridden by DHCP/PPP on WAN not set:
Code: [Select]
# route -v -n get 192.168.0.32
RTA_DST: inet 192.168.0.32; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks: inits:
sockaddrs: <DST,IFP>
192.168.0.32 link#0
route to: 192.168.0.32
destination: 192.168.0.0
mask: 255.255.255.0
fib: 0
interface: vtnet0
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
locks: inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
192.168.0.0 link#1 255.255.255.0 vtnet0:0.a0.98.6f.f4.40 192.168.0.5
# tcpdump -n -i vtnet0 -e src or dst host 192.168.0.32 and tcp and src or dst port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:34:21.463675 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 74: 192.168.0.32.38774 > 192.168.0.5.22: Flags [S], seq 3385995390, win 64240, options [mss 1460,sackOK,TS val 2147404981 ecr 0,nop,wscale 7], length 0
10:34:21.463721 00:a0:98:6f:f4:40 > 00:0d:b9:47:25:40, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.38774: Flags [S.], seq 704437459, ack 3385995391, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 432472754 ecr 2147404981], length 0
10:34:21.463923 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 66: 192.168.0.32.38774 > 192.168.0.5.22: Flags [.], ack 1, win 502, options [nop,nop,TS val 2147404982 ecr 432472754], length 0
10:34:21.466826 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 106: 192.168.0.32.38774 > 192.168.0.5.22: Flags [P.], seq 1:41, ack 1, win 502, options [nop,nop,TS val 2147404985 ecr 432472754], length 40
10:34:21.466862 00:a0:98:6f:f4:40 > 00:0d:b9:47:25:40, ethertype IPv4 (0x0800), length 66: 192.168.0.5.22 > 192.168.0.32.
^CWith Allow DNS server list to be overridden by DHCP/PPP on WAN set:
Code: [Select]
# route -v -n get 192.168.0.32
RTA_DST: inet 192.168.0.32; RTA_IFP: link ; RTM_GET: Report Metrics: len 224, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks: inits:
sockaddrs: <DST,IFP>
192.168.0.32 link#0
route to: 192.168.0.32
destination: 192.168.0.32
gateway: 192.168.0.254
fib: 0
interface: vtnet0
flags: <UP,GATEWAY,HOST,DONE,STATIC>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
locks: inits:
sockaddrs: <DST,GATEWAY,IFP,IFA>
192.168.0.32 192.168.0.254 vtnet0:0.a0.98.6f.f4.40 192.168.0.5
# tcpdump -n -i vtnet0 -e src or dst host 192.168.0.32 and tcp and src or dst port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:24:31.951129 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 74: 192.168.0.32.59048 > 192.168.0.5.22: Flags [S], seq 4118321306, win 64240, options [mss 1460,sackOK,TS val 2146815468 ecr 0,nop,wscale 7], length 0
10:24:31.951217 00:a0:98:6f:f4:40 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.59048: Flags [S.], seq 42904569, ack 4118321307, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2836344428 ecr 2146815468], length 0
10:24:32.947798 00:a0:98:6f:f4:40 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.59048: Flags [S.], seq 42904569, ack 4118321307, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2836345424 ecr 2146815468], length 0
10:24:32.964348 00:0d:b9:47:25:40 > 00:a0:98:6f:f4:40, ethertype IPv4 (0x0800), length 74: 192.168.0.32.59048 > 192.168.0.5.22: Flags [S], seq 4118321306, win 64240, options [mss 1460,sackOK,TS val 2146816482 ecr 0,nop,wscale 7], length 0
10:24:32.964389 00:a0:98:6f:f4:40 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.59048: Flags [S.], seq 42904569, ack 4118321307, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2836345438 ecr 2146816482], length 0
10:24:34.017659 00:a0:98:6f:f4:40 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 74: 192.168.0.5.22 > 192.168.0.32.59048: Flags [S.], seq 42904569, ack 4118321307, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 2836346494 ecr 2146816482], length 0
^CI've disabled all firewalling & NAT on this machine (it's an internal test machine, no public IP) to ensure that nothing is interfering with the packets. The interfaces are only configured with ipv4.
Code: [Select]
# pfctl -sa
FILTER RULES:
INFO:
Status: Disabled Debug: Urgent
State Table Total Rate
current entries 0
searches 0 0.0/s
inserts 0 0.0/s
removals 0 0.0/s
Counters
match 0 0.0/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
map-failed 0 0.0/s
TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
sctp.first 120s
sctp.opening 30s
sctp.established 86400s
sctp.closing 900s
sctp.closed 90s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 60000 states
adaptive.end 120000 states
src.track 0s
LIMITS:
states hard limit 100000
src-nodes hard limit 10000
frags hard limit 5000
table-entries hard limit 200000
TABLES:
__lan_network
__lo0_network
# netstat -nr
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.0.254 UGS vtnet0
127.0.0.1 link#2 UH lo0
192.168.0.0/24 link#1 U vtnet0
192.168.0.5 link#1 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
::1 link#2 UHS lo0
fe80::%lo0/64 link#2 U lo0
fe80::1%lo0 link#2 UHS lo0
This is a test machine, let me know if I can try anything else out to help isolate what seems like a small bug. Thanks much.