Previously on 23.1 I've had an issue (https://forum.opnsense.org/index.php?topic=33220) where Unbound DNS wouldn't work for WireGuard client upon reboot if Unbound wasn't set to listen on all interfaces. The fix was to restart Unbound after OPNsense boot.
I've now updated to the latest 23.7.5 and have a new problem. Now I can see Unbound resolving the DNS requests even if not set to listen to All interfaces, but WireGuard simply doesn't pass any traffic back to the endpoint (I can't ping to Internet via IP nor hostname). Now, instead of having to restart Unbound, the only way to get Internet access on the endpoint is to restart the Road Warrior WireGuard instance on the Dashboard. A second WireGuard instance for site2site communication is operating normally.
2023-09-27T22:25:03 Notice wireguard Wireguard interface WGxInternet (wg1) started
2023-09-27T22:25:03 Notice wireguard Wireguard interface WGxInternet (wg1) stopped
2023-09-27T22:17:02 Notice wireguard Wireguard interface WGSite2Site (wg2) started
2023-09-27T22:17:02 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.101.1.2/30' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) can not reconfigure without stopping it first.
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) started
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) can not reconfigure without stopping it first.
Another issue is that, unlike before, there is no Access Control list in Unbound, only the custom rules are showing. Is there a way to view all the active ACLs?
Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local. Is this dynamically linked to some other option? And the option only affects the firewall itself, interfaces continue to have DNS access even when set to Deny. Is this intended behavior?
EDIT: Allow Snoop/Deny Non-local/Refuse Non-local seem to be relevant only to manual ACL rules.
Is there a reason not to have Unbound listening on all interfaces? It's been posted a couple times in the forum why not doing so will cause problems with Unbound and dynamic interfaces.
Quote from: CJ on September 28, 2023, 09:01:44 PM
Is there a reason not to have Unbound listening on all interfaces? It's been posted a couple times in the forum why not doing so will cause problems with Unbound and dynamic interfaces.
It was easier to control access that way, though I have changed it to All in the meantime and utilized Aliases and Floating rules to control access. But it's not a factor for these issues anyway. The domains get resolved either way, but WG just doesn't get out to Internet (or back?) unless I manually restart it.
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
It was easier to control access that way, though I have changed it to All in the meantime and utilized Aliases and Floating rules to control access.
Was there specific access you were trying to prevent? OPNSense blocks all access to Unbound except LAN by default.
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
But it's not a factor for these issues anyway. The domains get resolved either way, but WG just doesn't get out to Internet (or back?) unless I manually restart it.
What do you mean by doesn't get out to the internet? Are you seeing a handshake? Can you ping? What about DNS?
Quote from: CJ on September 29, 2023, 02:20:53 PM
Was there specific access you were trying to prevent? OPNSense blocks all access to Unbound except LAN by default.
Preventing IoT, cameras and similar VLANs from resolving DNS.
Quote from: CJ on September 29, 2023, 02:20:53 PM
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
But it's not a factor for these issues anyway. The domains get resolved either way, but WG just doesn't get out to Internet (or back?) unless I manually restart it.
What do you mean by doesn't get out to the internet? Are you seeing a handshake? Can you ping? What about DNS?
Should I run packet capture to see about handshake?
I can't ping, neither by hostname nor direct IP (e.g. 1.1.1.1). Hostname gets resolved by Unbound on OPNsense and the WireGuard client receives it, but the ping to requested destination gets no reply. So, WG_client<->OPNsense works, but WG_client<->Internet doesn't unless I restart the WireGuard instance.
Could this issue (https://forum.opnsense.org/index.php?topic=36218.0) be related?
Can you confirm about Unbound ACL visibility in UI and missing options compared to documentation? Has this been changed for 23.7 or is it an issue on my end?
Ran packet capture after boot. It seems the ping request goes out to WAN but no response is visible.
nterface Timestamp SRC DST output
WGxInternet
wg1 2023-09-29
16:07:49.232765 length 88: 10.101.80.1 > 1.1.1.1: ICMP echo request, id 689, seq 1, length 64
WAN
vtnet1 2023-09-29
16:07:49.232796 xx:xx:xx:xx:xx:xx xx:xx:xx:xx:xx:xx IPv4, length 98: 10.101.80.1 > 1.1.1.1: ICMP echo request, id 689, seq 1, length 64
WGxInternet
wg1 2023-09-29
16:07:50.232109 length 88: 10.101.80.1 > 1.1.1.1: ICMP echo request, id 690, seq 1, length 64
WAN
vtnet1 2023-09-29
16:07:50.232146 xx:xx:xx:xx:xx:xx xx:xx:xx:xx:xx:xx IPv4, length 98: 10.101.80.1 > 1.1.1.1: ICMP echo request, id 690, seq 1, length 64
Quote from: Kinerg on September 29, 2023, 03:48:17 PM
Quote from: CJ on September 29, 2023, 02:20:53 PM
Was there specific access you were trying to prevent? OPNSense blocks all access to Unbound except LAN by default.
Preventing IoT, cameras and similar VLANs from resolving DNS.
That all occurs by default. Nothing will be able to reach Unbound outside of LAN unless you add a rule specifically for it regardless of what the Unbound listening interface is set to.
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
Should I run packet capture to see about handshake?
I can't ping, neither by hostname nor direct IP (e.g. 1.1.1.1). Hostname gets resolved by Unbound on OPNsense and the WireGuard client receives it, but the ping to requested destination gets no reply. So, WG_client<->OPNsense works, but WG_client<->Internet doesn't unless I restart the WireGuard instance.
Which hostname are you referring to? example.com or something local? What does the firewall log say?
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
Could this issue (https://forum.opnsense.org/index.php?topic=36218.0) be related?
I don't think so but I haven't looked at it. What rules do you have for WG?
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
Can you confirm about Unbound ACL visibility in UI and missing options compared to documentation? Has this been changed for 23.7 or is it an issue on my end?
I'm not sure what you mean. Can you elaborate? I've noticed no difference between 23.1 and 23.7 in terms of WG and made no changes for them either.
Quote from: CJ on September 30, 2023, 02:54:59 PM
Quote from: Kinerg on September 29, 2023, 03:48:17 PM
Quote from: CJ on September 29, 2023, 02:20:53 PM
Was there specific access you were trying to prevent? OPNSense blocks all access to Unbound except LAN by default.
Preventing IoT, cameras and similar VLANs from resolving DNS.
That all occurs by default. Nothing will be able to reach Unbound outside of LAN unless you add a rule specifically for it regardless of what the Unbound listening interface is set to.
It will if I have a rule to Allow IoT_net to IoT_address to enable DHCP/NTP/routing access. Or am I mistaken?
Quote from: CJ on September 30, 2023, 02:54:59 PM
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
Should I run packet capture to see about handshake?
I can't ping, neither by hostname nor direct IP (e.g. 1.1.1.1). Hostname gets resolved by Unbound on OPNsense and the WireGuard client receives it, but the ping to requested destination gets no reply. So, WG_client<->OPNsense works, but WG_client<->Internet doesn't unless I restart the WireGuard instance.
Which hostname are you referring to? example.com or something local? What does the firewall log say?
Any external hostname gets resolved, e.g. www.microsoft.com. Unbound isn't an issue regarding the WireGuard problem.
Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?
OPNsense fresh boot, before manual WG restart:
wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itselfAfter manual WG restart:
wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.
Quote from: CJ on September 30, 2023, 02:54:59 PM
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
Could this issue (https://forum.opnsense.org/index.php?topic=36218.0) be related?
I don't think so but I haven't looked at it. What rules do you have for WG?
Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges
I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.
Quote from: CJ on September 30, 2023, 02:54:59 PM
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
Can you confirm about Unbound ACL visibility in UI and missing options compared to documentation? Has this been changed for 23.7 or is it an issue on my end?
I'm not sure what you mean. Can you elaborate? I've noticed no difference between 23.1 and 23.7 in terms of WG and made no changes for them either.
I have a feeling the two issues are being conflated here. Maybe I should have made a separate post for the Unbound questions. The main issue is WireGuard not having Internet access after OPNsense boot unless the service is restarted.
The other, separate issue, is Unbound no longer showing its internal ACLs like before. Before 23.7 it listed all the automatically configured ACLs. Example in screenshot below which I found online. It used to show all internal automatically listed ACLs above
plus the custom made ones. Now, on 23.7, I can only see the custom made rules and can't find which ACLs Unbound has active.
Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local.
Quote from: Kinerg on September 30, 2023, 04:36:30 PM
Quote from: CJ on September 30, 2023, 02:54:59 PM
That all occurs by default. Nothing will be able to reach Unbound outside of LAN unless you add a rule specifically for it regardless of what the Unbound listening interface is set to.
It will if I have a rule to Allow IoT_net to IoT_address to enable DHCP/NTP/routing access. Or am I mistaken?
Well, yes, but that's because you configured it that way. You can instead just enable the specific services needed. You don't have to allow all ports.
Quote from: Kinerg on September 30, 2023, 04:36:30 PM
The other, separate issue, is Unbound no longer showing its internal ACLs like before. Before 23.7 it listed all the automatically configured ACLs. Example in screenshot below which I found online. It used to show all internal automatically listed ACLs above plus the custom made ones. Now, on 23.7, I can only see the custom made rules and can't find which ACLs Unbound has active.
Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local.
Yes, IIRC, this was done to avoid issues with dynamic interfaces such as wireguard not being picked up. So instead of a default deny with allow rules added for each interface subnet, unbound was changed to a default allow.
Quote from: Kinerg on September 30, 2023, 04:36:30 PM
Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?
OPNsense fresh boot, before manual WG restart:
wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
After manual WG restart:
wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.
Quote from: CJ on September 30, 2023, 02:54:59 PM
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
Could this issue (https://forum.opnsense.org/index.php?topic=36218.0) be related?
I don't think so but I haven't looked at it. What rules do you have for WG?
Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges
I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.
Do you have logging turned on for the WG rules? Are they showing any hits during this time?
Quote from: CJ on October 01, 2023, 03:21:05 PM
Quote from: Kinerg on September 30, 2023, 04:36:30 PM
The other, separate issue, is Unbound no longer showing its internal ACLs like before. Before 23.7 it listed all the automatically configured ACLs. Example in screenshot below which I found online. It used to show all internal automatically listed ACLs above plus the custom made ones. Now, on 23.7, I can only see the custom made rules and can't find which ACLs Unbound has active.
Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local.
Yes, IIRC, this was done to avoid issues with dynamic interfaces such as wireguard not being picked up. So instead of a default deny with allow rules added for each interface subnet, unbound was changed to a default allow.
I see, I must have missed that in the changelog. I do remember something to that effect, but not the UI changes being mentioned.
The Default Action setting doesn't work for me, though. It only affects OPNsense itself, but not the Interfaces. They continue having Unbound access regardless if set to Deny. I'm probably misunderstanding how the option should be used.
What about the missing Allow Snoop/Deny Non-local/Refuse Non-local?
Quote from: CJ on October 01, 2023, 03:23:19 PM
Quote from: Kinerg on September 30, 2023, 04:36:30 PM
Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?
OPNsense fresh boot, before manual WG restart:
wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
After manual WG restart:
wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.
Quote from: CJ on September 30, 2023, 02:54:59 PM
Quote from: Kinerg on September 28, 2023, 09:46:27 PM
Could this issue (https://forum.opnsense.org/index.php?topic=36218.0) be related?
I don't think so but I haven't looked at it. What rules do you have for WG?
Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges
I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.
Do you have logging turned on for the WG rules? Are they showing any hits during this time?
I do. They don't show anything being blocked, but they do show a different outgoing address being used. Before WG service restart, pings go out to WAN from the WG interface IP. After restart, from the WAN interface IP. NAT not working?
OPNsense fresh boot, before manual WG restart:
wan 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
After manual WG restart:
wan 2023-10-01T15:44:56 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:56 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:44:55 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
I did a fresh installation of OPNsense on a separate machine to continue my troubleshooting with the issues I've been having with the new Wireguard kernel update to 2.0 and it looks like the 'main issue' is starting to lead to here: Wireguard doesn't establish a connection at boot.
I disabled Unbound DNS to rule that out as the culprit for the issues I've been having since updating to 23.7.3 (when Wireguard was updated to 2.0) and it looks like it wasn't related.
I backed up all my configuration _before_ doing the Wireguard setup. Can reboot, use internet, etc. nothing wrong whatsoever. But then the problem started after I setup Wireguard. Just for clariy, here's what I did:
Setup an endpoint, setup the local instance, and enable it. You get a Wireguard connection from the Firewall itself and can see this with the `wg` command. You can ping and curl and verify VPN access from the firewall. Before doing anything else - Reboot the firewall. Now go back to the Wireguard plugin, and you'll see it tries to send data, but never receives. It's completely stuck, even with the default NAT rules. Verified in an SSH session too, just to make sure. Go to the plugin, uncheck it, save, check it again, save... VPN connection is back and running. So I'm starting to think the main issue/bug I've been running into the past month or whatnot is potentially a result of this issue.
All in all, can +1 the report of WG not working unless manually restarting the plugin.
Quote from: opn69a on October 02, 2023, 03:58:21 AM
Setup an endpoint, setup the local instance, and enable it. You get a Wireguard connection from the Firewall itself and can see this with the `wg` command. You can ping and curl and verify VPN access from the firewall. Before doing anything else - Reboot the firewall. Now go back to the Wireguard plugin, and you'll see it tries to send data, but never receives. It's completely stuck, even with the default NAT rules. Verified in an SSH session too, just to make sure. Go to the plugin, uncheck it, save, check it again, save... VPN connection is back and running. So I'm starting to think the main issue/bug I've been running into the past month or whatnot is potentially a result of this issue.
Yes, that sounds exactly like the issue I'm having.
Quote from: Kinerg on October 01, 2023, 03:51:06 PM
Quote from: CJ on October 01, 2023, 03:21:05 PM
Yes, IIRC, this was done to avoid issues with dynamic interfaces such as wireguard not being picked up. So instead of a default deny with allow rules added for each interface subnet, unbound was changed to a default allow.
I see, I must have missed that in the changelog. I do remember something to that effect, but not the UI changes being mentioned.
The Default Action setting doesn't work for me, though. It only affects OPNsense itself, but not the Interfaces. They continue having Unbound access regardless if set to Deny. I'm probably misunderstanding how the option should be used.
What about the missing Allow Snoop/Deny Non-local/Refuse Non-local?
No idea on the "missing" bit. Perhaps it's just a typo in the documentation. Since I leave Unbound as available to all interfaces the change didn't affect me and I didn't even realize the UI was different until you mentioned it.
The change from Allow to Deny probably requires a restart of either Unbound or OPNSense, but as I don't use it I haven't tested that.
I'm still a bit confused why you have a rule allowing access to all ports on the interface and then want to disable Unbound. Why not create an alias of only the ports you want that network to access? Or separate rules, which enables a bit more granular logging. Either way seems more secure than what you have now.
Quote from: Kinerg on October 01, 2023, 03:56:38 PM
I do. They don't show anything being blocked, but they do show a different outgoing address being used. Before WG service restart, pings go out to WAN from the WG interface IP. After restart, from the WAN interface IP. NAT not working?
OPNsense fresh boot, before manual WG restart:
wan 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
After manual WG restart:
wan 2023-10-01T15:44:56 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:56 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:44:55 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
Not sure. My WG just works so I haven't dug too deeply into it.
Quote from: opn69a on October 02, 2023, 03:58:21 AM
I did a fresh installation of OPNsense on a separate machine to continue my troubleshooting with the issues I've been having with the new Wireguard kernel update to 2.0 and it looks like the 'main issue' is starting to lead to here: Wireguard doesn't establish a connection at boot.
I disabled Unbound DNS to rule that out as the culprit for the issues I've been having since updating to 23.7.3 (when Wireguard was updated to 2.0) and it looks like it wasn't related.
I backed up all my configuration _before_ doing the Wireguard setup. Can reboot, use internet, etc. nothing wrong whatsoever. But then the problem started after I setup Wireguard. Just for clariy, here's what I did:
Setup an endpoint, setup the local instance, and enable it. You get a Wireguard connection from the Firewall itself and can see this with the `wg` command. You can ping and curl and verify VPN access from the firewall. Before doing anything else - Reboot the firewall. Now go back to the Wireguard plugin, and you'll see it tries to send data, but never receives. It's completely stuck, even with the default NAT rules. Verified in an SSH session too, just to make sure. Go to the plugin, uncheck it, save, check it again, save... VPN connection is back and running. So I'm starting to think the main issue/bug I've been running into the past month or whatnot is potentially a result of this issue.
All in all, can +1 the report of WG not working unless manually restarting the plugin.
What does your config look like? Did you change any other settings besides setting up WG?
Also, I want to clarify something. Are you attempting to initiate traffic from the OPNSense side to the client and that's not what's working? Or are you initiating traffic from the client side to OPNSense?
What do your WG configs look like? Are you using IPs, domains, etc?
I've opened bug reports for the WireGuard (https://github.com/opnsense/core/issues/6909) and Unbound (https://github.com/opnsense/core/issues/6910) issues to seek additional help. Will report back here with the findings.
Quote from: CJ on October 02, 2023, 04:47:21 PM
What does your config look like? Did you change any other settings besides setting up WG?
Also, I want to clarify something. Are you attempting to initiate traffic from the OPNSense side to the client and that's not what's working? Or are you initiating traffic from the client side to OPNSense?
What do your WG configs look like? Are you using IPs, domains, etc?
Apologies, forgot to reach back out on this!
1. Made changes to nothing else except WG because I wanted to confirm it was WG and not something else
2. Neither work prior to restarting the service, both work after
3. IPs, following the guide word-for-word from the OPNsense wiki as well. Basically kept DNS out of this as much as possible.
Hoping Kinerg's bug report and next update will resolve all this though, so guess we'll see soon-ish? :D
Quote from: opn69a on October 10, 2023, 10:04:02 PM
Apologies, forgot to reach back out on this!
1. Made changes to nothing else except WG because I wanted to confirm it was WG and not something else
2. Neither work prior to restarting the service, both work after
3. IPs, following the guide word-for-word from the OPNsense wiki as well. Basically kept DNS out of this as much as possible.
Hoping Kinerg's bug report and next update will resolve all this though, so guess we'll see soon-ish? :D
When you say it doesn't work, what specifically? Are you unable to ping, unable to resolve domains? Something else?
Lastly, are you using Keepalive 25 in your configs?
I've restarted OPNSense multiple times over the course of troubleshooting my ISP issues and never had to touch WG beyond initially setting it up.