Tearing hair out - packets passing fw but not being received by Unbound

[Stilez]

I am at a loss.  Getting a public IP to pass queries to Unbound should be trivially easy. It shouldn't need any customised NAT settings at all (Unbound is listening directly to the WAN), but Unbound never sees the queries. If I use NAT and port forward to localhost:53 instead (where Unbound also listens), Unbound still never sees them. In both cases, the firewall log shows they passed inbound correctly.  Only once they pass PF, they aren't showing up on the Unbound logs or being replied at all.

I've done a ton of debug, its 5 am and I'm still stumped, and honestly, I'm feeling frustrated and at a complete loss why it's not working.


Relevant background:

The router has a block of public IPs - call it 93.94.0.96/29. It's PPPoE so the WAN automatically adopts 93.94.0.97/32 as its IP, but that's okay AFAIK.  I have defined 6 virtual IPs on the WAN, one for each usable public IP address. I need the firewall to do 3 things, beyond any pass/block rules:

• If an unsolicited (new) packet comes in on one of my public IPs and isn't blocked, any outgoing responses have src=the same public IP.
  Example - incoming mail HELO from WAN on 93.94.0.102, response to PING from WAN should have src=93.94.0.102. Kinda obvious
   
  
• If a packet exits via the WAN and it's not a reply to an earlier packet, it always exits on 93.94.0.98.
  Example - outgoing http request from my desktop browser, public src IP should be 93.94.0.98.
     
• Unbound will listen and respond to queries from the WAN, on public IP 93.94.0.100 port 53 (but not other public IPs). Yes there are rules to prevent abuse

My laptop is connected to the external Internet via my Telco (phone->AP hotspot->GSM/LTE), and has been assigned an IP in the range 148.252.0.0/16 for now, which lets me test how my config  is working for external inquiries.

But none of these things is working for me, although they are basic and simple.

Outbound src IPs, trying to do 2nd bullet seems to break 1st bullet

The second bullets should just need outbound NAT with updated src =93.94.0.100. Unfortunately if I enable outbound NAT to achieve it, it *also* sets the src IP for all other egressing packets, including reply packets whose source IPs must remain as expected. I can't seem to get outbound NAT to do the one, without breaking the other.

Filter says it's passing packets to IPs where Unbound says it's listening, but Unbound never sees the packets:

This one really has me tearing out hair. I've tried from basic to esoteric, and can't find a way even to troubleshoot it.

For test purposes, Unbound is configured to accept inquiries from the WAN (both listening on WAN interface, and also specifically allowing access control set at 148.252.0.0/16 = allow.  NAT is manual only with no rules at present. I have these settings:

• In Firewall: Pass 93.94.0.100:53 with logging (log confirms packets are arriving and being passed)
• In Outbound NAT+ Unbound config: Tried unbound listening on 93.94.0.100. Also tried port forwarding 93.94.0.100:53 to 127.0.0.1:53 with associated filter rule, since Unbound is *definitewly* listening on localhost. I've tried outbound Nat (and outbound NONAT) on the public IP, as well + logging. Then tried a ton of options. Checked and logged rules by the dozen, nothing. Tried playing with reply-to, reflection settings, and a bunch of others.

Whatever I do, the firewall logs and tcpdump show the firewall is passing the queries on the IP alias with dst= 93.94.0.100:53 (or dst=127.0.0.1:53 if port forwarding to that IP/port is enabled). This is exactly what should ahppoen, but then.... the Unbound log remains stubbornly empty. It's not that the rewply isn't getting out to the client, or has the wrong IP, it;'s that it simply isn't responding at all, or even logging the query.

I don't know where to go from here.  I'm not seeing a clear reason in the logs or tcpdump, and RTFM + fiddling with settings + tools like logs, dig, tcpdump, even looking at the config directly in /var/unbound and /tmp (for pf),  are all failing to hint what's wrong. PF is plainly allowing incoming packets to either of 93.94.0.100:53 or forwarded to 127.0.0.1:53 if port forwarding is enabled for it. Unbound is listening explicitly to both of those interfaces and clearly allows both according to GUI and /var/unbound/access_control.conf, but when queried, and I have Unbound set up to log all queries and replies as received/sent, but still... Unbound never sees any of these queries even though they are shown as passed to IPs it's listening, in pf logs.
I just don't understand. Whatever I try, unbound log contains nothing and Unbound never responds. I've tried all the obvious things, and moved to esoteric ones, and still Unbound is silent.


What's needed in my config, to get my IPs working as I want? What else can I do, to troubleshoot and resolve these 2 issues?

[newsense]

Can you post the WAN rules ?

[Stilez]

Thanks. Things have moved on a bit.  I see somehow what's going on. I just don't understand why, but at least it's a testable hypothesis.

Everything I've read about a virtual IP, says that they are treated as IPs of the router. So if 1.2.3.4/32 is defined as a virtual IP on OPT6, then ping 1.2.3.4/32 from OPT6_subnet (if allowed by firewall) is in effect, pinging a router IP on OPT6.

So when I define 93.94.0.100/32 as a virtual IP, and my internetted laptop tries to DIG at 93.94.0.100:53, I'm expecting this is a router virtual IP, and also on a router-controlled interface, so provided the fw rules allow the packet to pass and provided Unbound is set to listen on WAN and allow the source IP in access-control, Unbound should be able to see the incoming packet. But strangely, it *can't*, and that's the problem.

I start to suspect that in some subtle way, packets with dst=[virtual IPs on an IF] are not "seen" by some services running on the router in the same way that the same packets with dst=[non-virtual IP on the same interface] would be seen. I don't understand how that can be, either.

(The issue with 127.0.0.1 turned out to be much simpler - NAT including port forwarding happens before fw so I needed a rule that checked and passed dst=127.0.0.1:53 not dst=93.94.0.100/32. But with the other one not working, I had overlooked that point, when I tried to port forward as a workaround.)

So the underlying issue is definitely not a WAN or floating rules issue. So far, this is where I'm at:

I have 93.94.0.100 as a direct Virtual IP with a pass rule and Unbound listening, and 93.94.0.102 as a virtual IP with :53 port forwarded to localhost:53 and an allow rule on localhost. Unbound is listening to both WAN and localhost. The queries come from the same laptop, hence same src IP. FW logs confirm both queries pass. The query to *.102 that's natted to localhost is seen, but the query that's not natted and just listened for, on the WAN itself, via its Virtual IP, isn't seen.

Bug? or something subtle?

[newsense]

The same rules would have to be defined for all IPs. Somehow it would appear you're expecting identical results for different settings

[Stilez]

I don't think so, but perhaps I misunderstand you - please explain a bit more at length if so.

Taking away the NAT aspect, which now looks like a bit of a red herring, this is what I have:

• According to docs, a virtual IP can have services attached to it by the router. (I can't find a docs reference for VIPs on OPNsense.org but it's well known, and plenty of other sources confirm it)
   
• So for example, if you create a VIP 10.10.10.10 on your LAN then the router will receive packets directed to that IP from LAN devices (and others if able), and router services can be configured to listen on 10.10.10.10 eg for web, ping, or whatever, and will respond.
   
• But when I do the same on the WAN, and put a VIP there (without any NAT), I could *not* get Unbound to pick up packets sent to that IP, whatever I tried, even though both pf logs and tcpdump confirmed the packets passed the firewall and were sent to the virtual IP on port 53, and even though Unbound conf clearly showed it was both listening on WAN and specifically had allowed access control to queries from the client's src address on the internet.
   
• The problem *wasn't* that Unbound was firewall-blocked, NATted, or not listening/denying. There weren't any NAT rules set. The packets got to the IP and interface it was listening on, from a client with access control rights to query it, and Unbound simply didn't see the packets that it should have been able to see, whatever I tried.  It didn't get as far as considering whether to accept/deny - it never saw them in the first place.
  
  But any Virtual IP on an interface should by its nature, be a router IP......... hence wha...?

[newsense]

There's no default allow in on WAN - doesn't matter how many services are listening on the FW. That's why I asked for the rules.

Also, you should be able to see in Firewall-Logs-Live what's happening with all your VIPs taking advantage of the filtering facility provided.

[Stilez]

True. WAN has a rule to allow in on dst=[93.94.0.100:53]. That's implicit in passing FW, because as you say, default deny.

So the firewall log shows clearly that the packet arrived on the WAN with dst=[93.94.0.100:53], it encountered that pass rule, and passed (bright green + "PASS" in the log). Visible in dynamic view (logs live) like you said.

Without any NAT rules active, packets with that dst should be picked up by Unbound if listening + enabled which it was. But they aren't.

[bartjsmit]

From the shell, what does this return:

sockstat -4 | grep 53

Can you do a dig +tcp from the outside?

Bart...

[Stilez]

dig +tcp domain @93.94.0.XXX A

works for 93.94.0.100 (which is the one that is redirected to 127.0.0.1); doesn't work for 93.94.0.102 (which is a VIP defined on the WAN and not redirected, which Unbound is configured to listen to and allow). The dynamic firewall log shows both the udp lookup and the tcp lookup (dig +tcp) passed the fw.

tl;dr - the query to .100 is in the Unbound log; the query to .102 is not.

sockstat -4 | grep 53

Code: [Select]

unbound  unbound    51083 3  udp4   10.0.0.1:53         *:*
unbound  unbound    51083 6  tcp4   10.0.0.1:53         *:*
unbound  unbound    51083 7  udp4   10.3.0.1:53         *:*
unbound  unbound    51083 8  tcp4   10.3.0.1:53         *:*
unbound  unbound    51083 9  udp4   10.5.0.1:53         *:*
unbound  unbound    51083 10 tcp4   10.5.0.1:53         *:*
unbound  unbound    51083 13 udp4   10.7.0.1:53         *:*
unbound  unbound    51083 14 tcp4   10.7.0.1:53         *:*
unbound  unbound    51083 15 udp4   93.94.0.97:53       *:*
unbound  unbound    51083 16 tcp4   93.94.0.97:53       *:*
unbound  unbound    51083 17 udp4   127.0.0.1:53          *:*
unbound  unbound    51083 18 tcp4   127.0.0.1:53          *:*
unbound  unbound    51083 19 tcp4   127.0.0.1:953         *:*
The other WAN virtual IPs are conspicuous by their absence. Apparently selecting the WAN is implied in unbound.conf by selecting the single /32 IP which "represents" the WAN's PPPoE link. The IP used seems to always be the lowest usable IP in the block.

This is the listening "interfaces" section from /var/unbound/unbound.conf:

Code: [Select]

# Interface IP(s) to bind to
interface: 10.0.0.1
interface: 10.3.0.1
interface: 10.5.0.1
interface: fe80::21b:21ff:fe55:2f0c%igb2
interface: 10.7.0.1
interface: fe80::21b:21ff:fe55:2f0d%igb3_vlan8
interface: 93.94.0.97
interface: fe80::222:4dff:fe6a:206%pppoe0
interface: fe80::222:4dff:fe6a:206%em0
interface: 127.0.0.1
interface: ::1
interface: fe80::1%lo0
It tells the same story. Interestingly it *is* listening on IPv6 interfaces. But for IPv4 it's only being directed to listen on the interfaces' primary IP/subnet - which for PPPoE is a single IP.

I think that PPPoE automatically picks up just a single IP to "represent" the entire WAN, even if it has a block of IPs. Then it gives just that one IP to Unbound to listen to, when you select to listen ion the interface, even if other IPs are defined on it.

So I tested that theory and it seems correct. I enabled lookups to virtual IP 93.94.0.97 as well as 93.94.0.102. I copied the fw rule, changing just the dst IP, and otherwise made no other changes - and suddenly unbound *did* start to pick up queries on VIP .97 and reply to them, but still not on VIP .102.

tl;dr - When you select listening interface = WAN, it's not actually telling Unbound to listen to the WAN. It tells Unbound to listen to just one specific IP on the WAN.

I haven't tested if manually hacking the .conf to add the missing IP or entire interface by name, would work, but it looks like it might.

But how to fix? You can add Virtual IPs to a PPPoE interface if you have a block, but you can't change the interface to be defined as 93.94.0.96/29, and this suggests that OPNsense isn't trying to truly bind Unbound to the interface (em0 or pppoe0), but only to (some representative) IPs on it. For some IF types those will be a complete list. For others it won't.

[bartjsmit]

But how to fix / where next?

Taking a step back, I would run an internal DNS server to replace unbound on the firewall, and create a 1:1 NAT from one of your public IP's with associated 53 tcp/udp pass rules.

I think you're running into trouble between OPNsense being a firewall and OPNsense being a server.

Bart...

[Stilez]

Updated previous reply. I don't think this is anything to do with Unbound. It's just being told the wrong places to listen. When you prod it on an IP it's configured to listen at (whether localhost via NAT, or the IP registered for the WAN's PPPoE link) it's fine. But if the WAN has other IPs, Unbound just isn't being told to listen to them.

Going to try one last test - adding the missing IPs to the .conf, and seeing if that fixes it.

If so then the cure is simple - add all IPs defined on listening interfaces, not just the primary ones.