Messages

[didn't]

So that was apparently the answer--set Unbound's outgoing interface to WAN rather than all.  It seems that with that set to all, Unbound expects to be able to communicate over IPv6, which it can't do, and therefore fails.  A couple of things that didn't work, without setting the interface to WAN:

• Explicitly disabling IPv6 on LAN.  It defaulted to "track interface," whatever that means; I'd suspected that explicitly disabling IPv6 on this interface (as it already was on WAN) might result in Unbound realizing that it can't communicate over IPv6 and reverting to IPv4.  It didn't seem to change anything.
• Enabling DHCP6 on WAN.  I don't use IPv6 (I have a static IPv4 address), but my ISP will give me an IPv6 address if I ask for it.  But enabling this also didn't have Unbound working.

What I still don't understand is why pfSense worked--as far as I can tell, it has all the same buttons, switches, and knobs, and it uses most of the same underlying software.  And my pfSense box was set to "all" for the outbound interface, and it worked fine.  So I'm not sure why the difference is there, but it's working, so good enough at least for now.

I've found the control in the UI to turn up the log level for Unbound, and does it ever!  At level 4, a single query generates around 1300 lines' worth of logs.  But I think grep has found the problem:

Code Select Expand


dan@Dan-MBP-2013  ~/Downloads  grep error unbound_log.txt
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11066"] [77014:0] error: udp connect failed: No route to host for 2001:503:ba3e::2:30 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11110"] [77014:0] error: udp connect failed: No route to host for 2001:500:2d::d port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11150"] [77014:0] error: udp connect failed: No route to host for 2001:500:2d::d port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11188"] [77014:0] error: udp connect failed: No route to host for 2001:500:1::53 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11228"] [77014:0] error: udp connect failed: No route to host for 2001:500:9f::42 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11337"] [77014:0] error: udp connect failed: No route to host for 2001:7fe::53 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11377"] [77014:0] error: udp connect failed: No route to host for 2001:500:12::d0d port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11417"] [77014:0] error: udp connect failed: No route to host for 2001:500:a8::e port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11453"] [77014:0] error: udp connect failed: No route to host for 2001:503:ba3e::2:30 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11611"] [77014:0] error: udp connect failed: No route to host for 2001:500:9f::42 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11684"] [77014:0] error: udp connect failed: No route to host for 2001:500:2f::f port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11756"] [77014:0] error: udp connect failed: No route to host for 2001:7fd::1 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11795"] [77014:0] error: udp connect failed: No route to host for 2001:500:1::53 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11833"] [77014:0] error: udp connect failed: No route to host for 2001:500:a8::e port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="11872"] [77014:0] error: udp connect failed: No route to host for 2001:dc3::35 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="12111"] [77014:0] error: udp connect failed: No route to host for 2001:500:2f::f port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="12188"] [77014:0] error: udp connect failed: No route to host for 2001:500:1::53 port 53 (len 28)
<27>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="12225"] [77014:0] error: udp connect failed: No route to host for 2001:500:12::d0d port 53 (len 28)
<31>1 2022-07-09T10:44:36-04:00 opnsense.familybrown.org unbound 77014 - [meta sequenceId="12310"] [77014:0] debug: return error response SERVFAIL

Strange--I don't have an IPv6 address, and I don't think I've told OPNsense to use IPv6.  Maybe there's a control I've missed.

Edit: I do see system / settings / general / prefer to use ipv4, but that's checked already.

Edit 2: Interesting.  If I go to Services/Unbound DNS/General, click the button for Advanced, and set Outgoing Network Interfaces to WAN (rather than the default of All), it starts working.  Curious.

Whatever it might be, it has nothing to do with my multi-WAN setup.  Figuring there might be something misconfigured there or elsewhere, I decided to reinstall from scratch and try again.   Booted it up and configured the WAN and LAN interfaces from the console menu. Then logged into the web UI and stepped through the wizard. It defaults to using Unbound, so didn't change that. Entered DNS servers of 1.1.1.1 and 1.0.0.1, set an appropriate time zone, and left literally everything else at defaults. Installed the update to 22.1.10 and rebooted.

Result: my client computer can't resolve google.com. So I tried in the web UI (Interfaces/Diagnostics/DNS Lookup). And it resolves google.com just fine–but from 1.1.1.1, not from the local Unbound instance.

So I SSH'd into the OPNsense box. And after determining that neither dig nor nslookup was installed, used host:

Code Select Expand


root@opnsense:~ # host google.com localhost
Using domain server:
Name: localhost
Address: 127.0.0.1#53
Aliases:

Host google.com not found: 2(SERVFAIL)


I've installed nothing else, I've configured nothing else–it's the stock distro with the defaults across the board. And DNS resolution just isn't working.

tl;dr: Unbound doesn't appear to be responding properly to DNS queries, though DNSmasq does.  I suspect it's related to my multi-WAN setup, but I haven't been able to figure out where.

I'm running OPNsense 22.1.10; I was seeing the same behavior under 22.1.9.

Unbound won't respond to queries via dig; I get the same result using the shell on the OPNsense box itself or via a remote client:

Code Select Expand


root@opnsense:~ # dig @localhost google.com

; <<>> DiG 9.18.4 <<>> @localhost google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 2147
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;google.com. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(localhost) (UDP)
;; WHEN: Fri Jul 08 04:33:05 EDT 2022
;; MSG SIZE  rcvd: 39


Code Select Expand


✘ dan@Dan-Mac-Mini-2  ~  dig @192.168.1.1 google.com

; <<>> DiG 9.10.6 <<>> @192.168.1.1 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 3653
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;google.com. IN A

;; Query time: 1 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Fri Jul 08 04:32:37 EDT 2022
;; MSG SIZE  rcvd: 39

But when I do a DNS lookup through the web UI (Interfaces/Diagnostics/DNS Lookup), I do get a result.

The background is a little confusing, but since the problem seemed to start when I plugged in my main WAN connection, I'll try to explain as clearly as I can.

I have three Internet connections available: Cable (with a static IP) for the primary, Starlink (in bridge mode) for secondary, and cellular is third.  I'm setting up this system to replace a pfSense box, so I was trying to configure everything (or at least as much as possible) under OPNsense before moving my main home Internet connection to it.

So, initially, I put Starlink on the main WAN connection (using DHCP), and the cellular modem on WAN2 (also using DHCP), and then proceeded to set up WAN failover following https://docs.opnsense.org/manual/multiwan.html#wan-failover.  This appeared to work--I didn't actually test the failover functionality, but I had Internet access through the router, and no apparent problems with Unbound.

But realizing that my main Internet connection didn't use DHCP, I disconnected the cellular modem, moved Starlink to WAN2, and configured WAN for my static IP, leaving WAN disconnected.  This required reconfiguration of the gateway list, since there wasn't a WAN_DHCP gateway any more.  This also appeared to work; Internet access continued to be available, and Unbound continued to respond to queries as normal.

Yesterday afternoon, thinking I had everything preconfigured that I was going to be able to, I plugged my cable modem into WAN, and LAN into my switch.  And at that point, Unbound stopped working.  When I turned it off and turned on DNSmasq, it worked (and continues to work) just fine, and Internet access works well, but with Unbound enabled it no longer seems to be able to resolve DNS queries.

I've tried checking log files, but I don't see anything logged anywhere that's associated with the failing queries.  Where else should I be looking?

Did you manage to solve your issue?

No, I gave up on OPNsense and went back to pfSense.  It's working perfectly there.

Thanks for the reply.  Yes, 192.168.1.0/24 is defined as a local, trusted network on the client system.  I'll try setting topology subnet and see what that does.

tl;dr: Clients on my LAN can't connect to OpenVPN clients through OPNsense, but OpenVPN clients can reach hosts on my LAN.  I can reach OpenVPN clients (i.e., ping them) from OPNsense itself.  This started around the time I configured multi-WAN failover and upgraded to 20.7.1.

Networks:

LAN: 192.168.1.0/24
OpenVPN: 192.168.3.0/24
WAN: static IP
WAN2: 192.168.5.something (assigned by DHCP, but in that subnet)

I'm running an OpenVPN server on my OPNsense box, primarily for the sake of two remote hosts that need to be able to access services on my LAN.  At the same time, some devices on my LAN need to be able to access one of those remote hosts.

This all worked well for quite a while--on pfSense before I moved to OPNsense, then it worked under 20.1.8 and 20.1.9, and when I upgraded to 20.7 it continued to work.  Around a week ago, though, following my fourth multi-hour Internet outage in several weeks, I set up multi-WAN failover with a cellular modem (following the instructions at https://docs.opnsense.org/manual/how-tos/multiwan.html), and I also updated to 20.7.1.  And since about that time (I can't say for certain if the problem started with one or the other of these changes, but it started about the time I made them), clients on my LAN aren't able to reach the remote host via the VPN.

Specifically, the remote host is at 192.168.3.100.  If I ping that IP from my OPNsense box itself, it reaches it just fine.  But if I ping it from anywhere else on my LAN, I just get timeouts.  My Google-fu is apparently weak here; I get lots of hits about routing from VPN clients to the LAN (which already works), but nothing about routing from the LAN to those clients.  Any ideas on where to start looking?  Settings attached if they help.  I tried adding the "IPv4 remote network" as you see in those settings, but it didn't help--I'm getting the same results.

So apparently the official answer (https://twitter.com/opnsense/status/1293624624606072834 - it's on Twitter, so it must be official, right?) is "we don't know what's going on, we don't know how to track it down, and we can't be bothered to try to figure it out."  Disappointing.

For some updates--the problem is present in 20.1.9 on a Netgate RCC-VE-2440.  It's still there when I upgrade that installation to 20.7.  It's also present on a clean install of 20.7 on a Protectli FW4B, with the configuration file imported from the previous system.  So maybe it's something that's only affecting me, but it's doing it across two different releases of the software and on two different systems.

So, since this seems to be at a dead end, on to Chrony.  It installs just fine.  It starts up and appears to run without errors, which puts it ahead of ntpd.  I'm noticing what seems to be one bug, though--there's a field in the GUI for "Allowed Networks", but it only lets me enter one network--when I enter a second one, I get a validation error.

Any ideas on this?  It's somewhat disconcerting that my firewall doesn't seem to know that its VPN service is running, much less that it has two clients connected.

Thanks for the reply.  No, WAN is a simple static IP; I'm not using PPPoE at all.

Anyone?

How about new chrony plugin (devel)?

I'll consider it, but I'd still like to figure out why this one isn't working.  What does Chrony do differently that might make it a better solution?

Since this looks to me like a bug, reported here:
https://github.com/opnsense/core/issues/4245

Any other ideas?  Or since this problem is still present in 20.7, should I post this issue in that forum?

I've been having a few problems with OPNsense since I installed it, that I haven't been able to sort out.  My problem with ntpd (https://forum.opnsense.org/index.php?topic=18253.0) seems to have stumped the experts, so here's another one: OpenVPN.

I've set up an OpenVPN server on my OPNsense box  I have two remote computers connected to that server full-time, on the VPN subnet.  I know they're connected, because they're able to run backups to my FreeNAS server on my LAN every day.  But both the services widget in the dashboard, and the OpenVPN widget, say the service isn't running--see the attachments for screen shots.

A little stumped here on what to be checking--any thoughts?

Edit: The process appears to be running:

Code Select Expand


root@opnsense:~ # ps aux | grep vpn
root     6744   5.5  0.2 1066500  7996  -  Rs   Fri13    509:10.81 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
root@opnsense:~ #

Not sure if there's anything out of the ordinary in the log file:

Code Select Expand


root@opnsense:/var/log # tail openvpn.log
Aug  5 06:46:36 opnsense openvpn[14976]: OpenVPN 2.4.9 amd64-portbld-freebsd12.1 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Jul 28 2020
Aug  5 06:46:36 opnsense openvpn[14976]: library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
Aug  5 06:46:36 opnsense openvpn[91988]: MANAGEMENT: unix domain socket listening on /var/etc/openvpn/server1.sock
Aug  5 06:46:36 opnsense openvpn[91988]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Aug  5 06:46:36 opnsense openvpn[91988]: Diffie-Hellman initialized with 4096 bit key
Aug  5 06:46:36 opnsense openvpn[91988]: Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Aug  5 06:46:36 opnsense openvpn[91988]: Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Aug  5 06:46:36 opnsense openvpn[91988]: ROUTE_GATEWAY 96.68.219.30/255.255.255.252 IFACE=igb0 HWADDR=00:08:a2:0a:d5:04
Aug  5 06:46:36 opnsense openvpn[91988]: TUN/TAP device ovpns1 exists previously, keep at program end
Aug  5 06:46:36 opnsense openvpn[91988]: Cannot open TUN/TAP dev /dev/tuCLOG? ??root@opnsense:/var/log #