Messages

I'll need to pass that from the plugin to the .sh file, or look for it in env , it's too specific for the main dnsapi. (More a note to myself not to forget and hard code it!)

[cat ${CACHE} | ${UNBOUNDCTL} load_cache]

I'm writing a short add-on for the ACME DNS-01 challenge.

The idea is that if you're using DNS alias/redirect, and a tiny "set and forget" type of domain (like one used just for your family website/email), and your provider doesn't provide an API for DNS record updates (web UI/email only), you could have an ACME challenge setup something like this, and not need to run BIND or anything else, if you're running unbound already as the main resolver.

Since many people could fall into that scenario, it seems a neat DNS-01 module to have, and its relevant to me as I'm squarely in that situation.

It also feels a neat solution from a security perspective, as it doesn't need any key or password, nor any software beyond unbound, since it's all based on local unbound.conf data that's secured by usual Unix permissions, and it doesn't risk exposing anything beyond the few text records, and then only for a few seconds (max 3-5 minutes) once every 2-3 months when DNS-01 is active. Note also, no external DNS provider API is involved or needs to propagate data, so the total start-to-end time for live data to be in the file should be the time taken for Let's Encrypt's queue to perform lookup after being notified of a DNS update, plus a fraction of a second.  The workflow is:

Done manually by the user:

   
• Create the couple of DNS records with their provider, to redirect _acme_challenge.domain.name to the IP that unbound will listen on, during the DNS-01 challenge.

In the ACME plugin:

   
• add a directive in unbound.conf, to include "/var/unbound/unbound-dns-01.conf"
• add a suitable one-line system startup script so this exists and is empty at unbound config load
  (these are probably too system specific to be in acme.sh itself and need to be in the acme plugin code or done manually. But they are trivial.)

In a new ACME DNS-01 challenge dnsapi shell module "dns_unbound_direct_access.sh"  (both add and remove parts):

   
• If Unbound isn't running, error + return (maybe an option to autostart/autostop if this isn't a problem for edge cases)
• before doing anything else, and regardless whether the call was to add or remove TXT records, set a cron job to empty the file of any records for that domain and autostop unbound if autostarted, in some amount of seconds (say 300 seconds). This is a failsafe to ensure the file is emptied and generally serves+contains nothing unless a DNS-01 query is actively in progress, which is only once every 60 - 90 days, even if something fails or exits prematurely within acme plugin or elsewhere, at any point. 
  
• Create /var/unbound/unbound-dns-01.conf which exposes a new custom view. The view is set to "deny_non_local" + "view-first:no" (ie refuses/denies anything except local-data explicitly defined within that view in unbound.conf and subfiles), and also rejects non_WAN IPs. The contents of the file are, obviously, the local-data: DOMAIN TXT 'DATA' records that Unbound should serve for DNS-01, or empty for that domain when no longer needed
• direct Unbound to reload its config (better than restarting service)
• verify that new config was loaded by unbound and exit success

I've got almost all of this prototyped, and working, with one exception, which is the dumbest thing - I can't seem to work out reliably, the best/correct way to reload unbound.conf.

From examining the codebase I've found it uses the old legacy system (.inc files) and I've played with the following commands seen in its scripts, which seem to be relevant:

   
• ...
  elif [ "${COMMAND}" = "load" -a -f "${CACHE}" ]; then
    cat ${CACHE} | ${UNBOUNDCTL} load_cache
  ...
  
• /usr/sbin/chroot -u unbound -g unbound /
  /usr/local/sbin/unbound-control -c "$unbound_main_conf_file" "$1"  2>&1
  (gets permission denied for port)
  
• /usr/sbin/chroot -u root -g wheel /
  /usr/local/sbin/unbound-control -c "$unbound_main_conf_file" "$1"  2>&1
  (no obvious permissions issue but seems to hang up a lot of the time)

I think the problem is that with unbound-control running in chroot, and potentially being called both from console (for testing) and also from a script run by ACME-plugin (presumably running as privileged user of some kind?), that there are nuances of the command, or the user/group, which I'm not getting right. Help appreciated, and once it's working here, code review time? :)

Updated previous reply. I don't think this is anything to do with Unbound. It's just being told the wrong places to listen. When you prod it on an IP it's configured to listen at (whether localhost via NAT, or the IP registered for the WAN's PPPoE link) it's fine. But if the WAN has other IPs, Unbound just isn't being told to listen to them.

Going to try one last test - adding the missing IPs to the .conf, and seeing if that fixes it.

If so then the cure is simple - add all IPs defined on listening interfaces, not just the primary ones.

dig +tcp domain @93.94.0.XXX A

works for 93.94.0.100 (which is the one that is redirected to 127.0.0.1); doesn't work for 93.94.0.102 (which is a VIP defined on the WAN and not redirected, which Unbound is configured to listen to and allow). The dynamic firewall log shows both the udp lookup and the tcp lookup (dig +tcp) passed the fw.

tl;dr - the query to .100 is in the Unbound log; the query to .102 is not.

sockstat -4 | grep 53

Code Select Expand

unbound  unbound    51083 3  udp4   10.0.0.1:53         *:*
unbound  unbound    51083 6  tcp4   10.0.0.1:53         *:*
unbound  unbound    51083 7  udp4   10.3.0.1:53         *:*
unbound  unbound    51083 8  tcp4   10.3.0.1:53         *:*
unbound  unbound    51083 9  udp4   10.5.0.1:53         *:*
unbound  unbound    51083 10 tcp4   10.5.0.1:53         *:*
unbound  unbound    51083 13 udp4   10.7.0.1:53         *:*
unbound  unbound    51083 14 tcp4   10.7.0.1:53         *:*
unbound  unbound    51083 15 udp4   93.94.0.97:53       *:*
unbound  unbound    51083 16 tcp4   93.94.0.97:53       *:*
unbound  unbound    51083 17 udp4   127.0.0.1:53          *:*
unbound  unbound    51083 18 tcp4   127.0.0.1:53          *:*
unbound  unbound    51083 19 tcp4   127.0.0.1:953         *:*

The other WAN virtual IPs are conspicuous by their absence. Apparently selecting the WAN is implied in unbound.conf by selecting the single /32 IP which "represents" the WAN's PPPoE link. The IP used seems to always be the lowest usable IP in the block.

This is the listening "interfaces" section from /var/unbound/unbound.conf:

Code Select Expand

# Interface IP(s) to bind to
interface: 10.0.0.1
interface: 10.3.0.1
interface: 10.5.0.1
interface: fe80::21b:21ff:fe55:2f0c%igb2
interface: 10.7.0.1
interface: fe80::21b:21ff:fe55:2f0d%igb3_vlan8
interface: 93.94.0.97
interface: fe80::222:4dff:fe6a:206%pppoe0
interface: fe80::222:4dff:fe6a:206%em0
interface: 127.0.0.1
interface: ::1
interface: fe80::1%lo0

It tells the same story. Interestingly it *is* listening on IPv6 interfaces. But for IPv4 it's only being directed to listen on the interfaces' primary IP/subnet - which for PPPoE is a single IP.

I think that PPPoE automatically picks up just a single IP to "represent" the entire WAN, even if it has a block of IPs. Then it gives just that one IP to Unbound to listen to, when you select to listen ion the interface, even if other IPs are defined on it.

So I tested that theory and it seems correct. I enabled lookups to virtual IP 93.94.0.97 as well as 93.94.0.102. I copied the fw rule, changing just the dst IP, and otherwise made no other changes - and suddenly unbound *did* start to pick up queries on VIP .97 and reply to them, but still not on VIP .102.

tl;dr - When you select listening interface = WAN, it's not actually telling Unbound to listen to the WAN. It tells Unbound to listen to just one specific IP on the WAN.

I haven't tested if manually hacking the .conf to add the missing IP or entire interface by name, would work, but it looks like it might.

But how to fix? You can add Virtual IPs to a PPPoE interface if you have a block, but you can't change the interface to be defined as 93.94.0.96/29, and this suggests that OPNsense isn't trying to truly bind Unbound to the interface (em0 or pppoe0), but only to (some representative) IPs on it. For some IF types those will be a complete list. For others it won't.

True. WAN has a rule to allow in on dst=[93.94.0.100:53]. That's implicit in passing FW, because as you say, default deny.

So the firewall log shows clearly that the packet arrived on the WAN with dst=[93.94.0.100:53], it encountered that pass rule, and passed (bright green + "PASS" in the log). Visible in dynamic view (logs live) like you said.

Without any NAT rules active, packets with that dst should be picked up by Unbound if listening + enabled which it was. But they aren't.

I don't think so, but perhaps I misunderstand you - please explain a bit more at length if so.

Taking away the NAT aspect, which now looks like a bit of a red herring, this is what I have:

   
• According to docs, a virtual IP can have services attached to it by the router. (I can't find a docs reference for VIPs on OPNsense.org but it's well known, and plenty of other sources confirm it)
   
• So for example, if you create a VIP 10.10.10.10 on your LAN then the router will receive packets directed to that IP from LAN devices (and others if able), and router services can be configured to listen on 10.10.10.10 eg for web, ping, or whatever, and will respond.
  
• But when I do the same on the WAN, and put a VIP there (without any NAT), I could *not* get Unbound to pick up packets sent to that IP, whatever I tried, even though both pf logs and tcpdump confirmed the packets passed the firewall and were sent to the virtual IP on port 53, and even though Unbound conf clearly showed it was both listening on WAN and specifically had allowed access control to queries from the client's src address on the internet.
  
• The problem *wasn't* that Unbound was firewall-blocked, NATted, or not listening/denying. There weren't any NAT rules set. The packets got to the IP and interface it was listening on, from a client with access control rights to query it, and Unbound simply didn't see the packets that it should have been able to see, whatever I tried.  It didn't get as far as considering whether to accept/deny - it never saw them in the first place.
  
  But any Virtual IP on an interface should by its nature, be a router IP......... hence wha...?

Thanks. Things have moved on a bit.  I see somehow what's going on. I just don't understand why, but at least it's a testable hypothesis.

Everything I've read about a virtual IP, says that they are treated as IPs of the router. So if 1.2.3.4/32 is defined as a virtual IP on OPT6, then ping 1.2.3.4/32 from OPT6_subnet (if allowed by firewall) is in effect, pinging a router IP on OPT6.

So when I define 93.94.0.100/32 as a virtual IP, and my internetted laptop tries to DIG at 93.94.0.100:53, I'm expecting this is a router virtual IP, and also on a router-controlled interface, so provided the fw rules allow the packet to pass and provided Unbound is set to listen on WAN and allow the source IP in access-control, Unbound should be able to see the incoming packet. But strangely, it *can't*, and that's the problem.

I start to suspect that in some subtle way, packets with dst=[virtual IPs on an IF] are not "seen" by some services running on the router in the same way that the same packets with dst=[non-virtual IP on the same interface] would be seen. I don't understand how that can be, either.

(The issue with 127.0.0.1 turned out to be much simpler - NAT including port forwarding happens before fw so I needed a rule that checked and passed dst=127.0.0.1:53 not dst=93.94.0.100/32. But with the other one not working, I had overlooked that point, when I tried to port forward as a workaround.)

So the underlying issue is definitely not a WAN or floating rules issue. So far, this is where I'm at:

I have 93.94.0.100 as a direct Virtual IP with a pass rule and Unbound listening, and 93.94.0.102 as a virtual IP with :53 port forwarded to localhost:53 and an allow rule on localhost. Unbound is listening to both WAN and localhost. The queries come from the same laptop, hence same src IP. FW logs confirm both queries pass. The query to *.102 that's natted to localhost is seen, but the query that's not natted and just listened for, on the WAN itself, via its Virtual IP, isn't seen.

Bug? or something subtle?

I am at a loss.  Getting a public IP to pass queries to Unbound should be trivially easy. It shouldn't need any customised NAT settings at all (Unbound is listening directly to the WAN), but Unbound never sees the queries. If I use NAT and port forward to localhost:53 instead (where Unbound also listens), Unbound still never sees them. In both cases, the firewall log shows they passed inbound correctly.  Only once they pass PF, they aren't showing up on the Unbound logs or being replied at all.

I've done a ton of debug, its 5 am and I'm still stumped, and honestly, I'm feeling frustrated and at a complete loss why it's not working.


Relevant background:

The router has a block of public IPs - call it 93.94.0.96/29. It's PPPoE so the WAN automatically adopts 93.94.0.97/32 as its IP, but that's okay AFAIK.  I have defined 6 virtual IPs on the WAN, one for each usable public IP address. I need the firewall to do 3 things, beyond any pass/block rules:

   
• If an unsolicited (new) packet comes in on one of my public IPs and isn't blocked, any outgoing responses have src=the same public IP.
  Example - incoming mail HELO from WAN on 93.94.0.102, response to PING from WAN should have src=93.94.0.102. Kinda obvious :)
  
  
• If a packet exits via the WAN and it's not a reply to an earlier packet, it always exits on 93.94.0.98.
  Example - outgoing http request from my desktop browser, public src IP should be 93.94.0.98.
     
• Unbound will listen and respond to queries from the WAN, on public IP 93.94.0.100 port 53 (but not other public IPs). Yes there are rules to prevent abuse :)

My laptop is connected to the external Internet via my Telco (phone->AP hotspot->GSM/LTE), and has been assigned an IP in the range 148.252.0.0/16 for now, which lets me test how my config  is working for external inquiries.

But none of these things is working for me, although they are basic and simple.

Outbound src IPs, trying to do 2nd bullet seems to break 1st bullet

The second bullets should just need outbound NAT with updated src =93.94.0.100. Unfortunately if I enable outbound NAT to achieve it, it *also* sets the src IP for all other egressing packets, including reply packets whose source IPs must remain as expected. I can't seem to get outbound NAT to do the one, without breaking the other.

Filter says it's passing packets to IPs where Unbound says it's listening, but Unbound never sees the packets:

This one really has me tearing out hair. I've tried from basic to esoteric, and can't find a way even to troubleshoot it.

For test purposes, Unbound is configured to accept inquiries from the WAN (both listening on WAN interface, and also specifically allowing access control set at 148.252.0.0/16 = allow.  NAT is manual only with no rules at present. I have these settings:

   
• In Firewall: Pass 93.94.0.100:53 with logging (log confirms packets are arriving and being passed)
• In Outbound NAT+ Unbound config: Tried unbound listening on 93.94.0.100. Also tried port forwarding 93.94.0.100:53 to 127.0.0.1:53 with associated filter rule, since Unbound is *definitewly* listening on localhost. I've tried outbound Nat (and outbound NONAT) on the public IP, as well + logging. Then tried a ton of options. Checked and logged rules by the dozen, nothing. Tried playing with reply-to, reflection settings, and a bunch of others.

Whatever I do, the firewall logs and tcpdump show the firewall is passing the queries on the IP alias with dst= 93.94.0.100:53 (or dst=127.0.0.1:53 if port forwarding to that IP/port is enabled). This is exactly what should ahppoen, but then.... the Unbound log remains stubbornly empty. It's not that the rewply isn't getting out to the client, or has the wrong IP, it;'s that it simply isn't responding at all, or even logging the query.

I don't know where to go from here.  I'm not seeing a clear reason in the logs or tcpdump, and RTFM + fiddling with settings + tools like logs, dig, tcpdump, even looking at the config directly in /var/unbound and /tmp (for pf),  are all failing to hint what's wrong. PF is plainly allowing incoming packets to either of 93.94.0.100:53 or forwarded to 127.0.0.1:53 if port forwarding is enabled for it. Unbound is listening explicitly to both of those interfaces and clearly allows both according to GUI and /var/unbound/access_control.conf, but when queried, and I have Unbound set up to log all queries and replies as received/sent, but still... Unbound never sees any of these queries even though they are shown as passed to IPs it's listening, in pf logs.
I just don't understand. Whatever I try, unbound log contains nothing and Unbound never responds. I've tried all the obvious things, and moved to esoteric ones, and still Unbound is silent.


What's needed in my config, to get my IPs working as I want? What else can I do, to troubleshoot and resolve these 2 issues?

I thought that I understood the way that Apple services, HP printers with Airplay implemented, and other such services worked on a segmented LAN.  The LAN here has a transparent WIFI AP that bridges wireless devices to OPT3, and filter rules on OPT3 allow access to the local resolver (Unbound: it's a small LAN with maybe <12 devices needing a static IP) and the LAN itself.

As I've understood it, devices broadcast requests/discovery announcements to their segment (bridged if needed by mdnsrepeater/avahi), and discover other devices on the LAN that way.  Irritating to have to bridge, but okay, doable.

Just now I hit a completely new thing in the zero-config realm, and I'm trying to understand how to integrate that into my LAN. I tried to use my printer from my mobile phone. The HP print service couldn't find the printer, but I could easily reach its built-in web UI from my phone, so TCP/UDP all work fine between the phone and printer. Intrigued, I ran tcpdump and found the HP Android print server plugin was sending odd DNS requests, which I now know are dns-sd packets, to the local LAN resolver, to try and find presumably statically coded or dynamic discovery records of local services.

The domain's it's looking for(initially at least) are: lb._dns-sd._udp.MY.DOMAIN and b._dns-sd._udp.MY.DOMAIN

This has thrown me on many levels. Please help me work out what to do!

I take a guess that it's piggybacking on (or similar to) Apple Airplay, and HP uses this as a kind of subnet-aware zeroconf autodiscovery, where device details are dumped into DNS TXT strings by local dns-sd aware devices, and DNS can be interrogated for them by service users. That leaves me 3 questions:

   
• I don't want to have to manually add TXT strings to Unbound any time a new service is offered by a friend's device.  But Unbound is generally read-only on DNS records, once running.  Is there a way to make this dns-sd discovery method work without ditching Unbound as the LAN resolver?  (I've never used Dyn DNS for anything but could use it if it helps)
• The HP print plugin seems to only look for dns-sd records. If it knew the device IPs, would it be able to interrogate them directly, or does it need that middleman DNS service to log them and serve them? Like, is there a way I could tell the HP thing to just add a printer at some IP or another, or is that unlikely to work?
• Should I just use different (non-HP) software? But then, friends with Apple iThings will want to print and share services on the LAN, and print software won't solve it for those. Will a fix for dns-sd combined with some kind of mdnsrepeater, finally fix all common zeroconfig discoveries, or are there others that are popular and I need to handle?

I understand the principles and have read various man pages + posts, but I'm still not geting 1:1 NAT working in this scenario.

I have two LANs - one trusted (office) and the other untrusted (domestic/family).There's a a single device that unavoidably has to be physically connected via the untrusted network, but needs to behave as if it's connected to the trusted network's subnet.
 
Getting the device to behave as if on the trusted LAN should be easy (virtual IP + 1:1 NAT) but isn't working for me.
 
To make it simple, let's give some hypothetical IPs:

   
• The "trusted" LAN is on NIC <em0> "OFFICE_NET", which has subnet 10.20.0.5/16, and the "untrusted" LAN is on NIC <em1> "HOUSE_NET", with subnet 10.50.0.5/16.
  
• Devices on HOUSE_NET are all forced onto VLAN 6 by the switches, except for the device I'm trying to fix, which is forced onto VLAN 7.   Filter rules block any traffic at VLAN 6 from reaching VLAN 7.
  
• The device needs to be accessible from OFFICE_LAN via IP 10.20.0.100. (I don't care if it has a different IP locally on VLAN 7).
  
• At the router, I've created VLAN 6+7 and their interfaces for . I've set up rules so that VLAN 6 traffic can't reach VLAN 7. I've assigned VLAN 7 the subnet 10.51.0.5/16, and enabled DHCP on that interface to issue the device with IP 10.51.0.100. I've created a virtual IP for 10.20.0.100 on OFFICE_NET.
  

In theory, all that's left is creating a 1:1 BINAT (symmetrical) entry on OFFICE_NET, so that

   
• any packets from OFFICE_NET with source in 10.20.0.5/16 and destination 10.20.0.100 that arrive at the router, are NATed to appear as destination=10.51.0.100 and arrive at VLAN 7 where they are forwarded to the device.
   
  and
  
• any packets from VLAN 7 with source 10.51.0.100 and destination in 10.20.0.5/16 that arrive at the router, are NATed to appear as source = 10.20.0.100 and arrive at OFFICE_NET where they are forwarded to their destination. (This is less common but possible, eg ping, syslog, snmp, spanning tree protocols, etc)

But I can't get this to work - meaning, I can ping the virtual IP at 10.20.0.100 from the LAN, and I can access http://10.51.0.100 from the LAN, but I can't access http://10.20.0.100 from the LAN.
 
I'm not sure what I'm doing wrong, or what else might be needed.  It shouldn't have a routing problem, as I can reach the device via its VLAN 7 IP 10.51.0.100. It's just that when I access the virtual IP, whether or not it's NATing it to dest=10.51.0.100, it's not then forwarding it to VLAN 7, as it does when I enter that IP directly.
 

What should I have in my NAT config, to make this last step work?

I've got my monitor working how I want it, with FreeNAS, and it looks like the same settings should work well on OPNSense as the OS is clearly compatible.

Since errors in loader config can be a real problem, I want to sanity check before going ahead, even though reading the forums it looks OK.
I want to load the vt driver and a couple of others, and then set some loader tunables related to them.

   
• Do I just add them to "tunables", and OPNSense middleware will figure which to apply as loader / rc / sysctl?
• Does the order matter?
• How would I recover if I find I've made a mistake and it won't boot properly, or is the system resilient against user errors in loader/rc tunable entry?

Thanks for any quick answers on this, so I can go ahead. If it works, I'll add it as a request or forum post, so others can benefit too.

In case of attempts to brute force a login on a password-secured system, it adds a layer of security for an attacker to have to guess an account name, not just a password. The "root" login name will be a "well known login" for OPNSense. It's possible to create an arbitrary named user in the admin group, that is actually used for console or webUI logins, but I'm stuck there and wondering if more is possible in two areas:

   
• The root account can still login via web, remote SSH (if enabled) or console. if the sysadmin does not believe that accidental lockout is a problem (e.g., they have a backup config and can happily reinstall), they might want to limit the root account's login to just console, or just some specific method/source IP, or block it completely other than locally as needed for OPNSense's own operations. But I can't find a way to reduce any of the root account's login rights.
  
• Similarly, I can set up other admin accounts, but even allowing login group = "wheel,admins", I can't get my alternative admin to directly login to the opnsense shell. The best I can do is login to csh and then su, which gets me there, but as a member of "admins" shouldn't it gain access to the main opnsense shell as well?
  
• Lastly, I can't find any way to specify which login routes a specific account can use - for example I might want to allow "dailyadmin" (a member of "admins") to use web + SSH from specific IP ranges, but not the system console, or allow root to use local console only as a kind of "emergency user" rarely used if locked out all other ways, that can't login from any remote system.

Can any of these be done? If not, are there good reasons why enhancement in this area would be a bad idea?