WireGuard no Internet access unless manually restarted (and Unbound issues)

[Kinerg]

Previously on 23.1 I've had an issue where Unbound DNS wouldn't work for WireGuard client upon reboot if Unbound wasn't set to listen on all interfaces. The fix was to restart Unbound after OPNsense boot.

I've now updated to the latest 23.7.5 and have a new problem. Now I can see Unbound resolving the DNS requests even if not set to listen to All interfaces, but WireGuard simply doesn't pass any traffic back to the endpoint (I can't ping to Internet via IP nor hostname). Now, instead of having to restart Unbound, the only way to get Internet access on the endpoint is to restart the Road Warrior WireGuard instance on the Dashboard. A second WireGuard instance for site2site communication is operating normally.

Code: [Select]

2023-09-27T22:25:03 Notice wireguard Wireguard interface WGxInternet (wg1) started
2023-09-27T22:25:03 Notice wireguard Wireguard interface WGxInternet (wg1) stopped
2023-09-27T22:17:02 Notice wireguard Wireguard interface WGSite2Site (wg2) started
2023-09-27T22:17:02 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.101.1.2/30' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) can not reconfigure without stopping it first.
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) started
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) can not reconfigure without stopping it first.


Another issue is that, unlike before, there is no Access Control list in Unbound, only the custom rules are showing. Is there a way to view all the active ACLs?

Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local. Is this dynamically linked to some other option? And the option only affects the firewall itself, interfaces continue to have DNS access even when set to Deny. Is this intended behavior?

EDIT: Allow Snoop/Deny Non-local/Refuse Non-local seem to be relevant only to manual ACL rules.

[CJ]

Is there a reason not to have Unbound listening on all interfaces?  It's been posted a couple times in the forum why not doing so will cause problems with Unbound and dynamic interfaces.

[Kinerg]

Is there a reason not to have Unbound listening on all interfaces?  It's been posted a couple times in the forum why not doing so will cause problems with Unbound and dynamic interfaces.

It was easier to control access that way, though I have changed it to All in the meantime and utilized Aliases and Floating rules to control access. But it's not a factor for these issues anyway. The domains get resolved either way, but WG just doesn't get out to Internet (or back?) unless I manually restart it.

[CJ]

It was easier to control access that way, though I have changed it to All in the meantime and utilized Aliases and Floating rules to control access.

Was there specific access you were trying to prevent?  OPNSense blocks all access to Unbound except LAN by default.

But it's not a factor for these issues anyway. The domains get resolved either way, but WG just doesn't get out to Internet (or back?) unless I manually restart it.

What do you mean by doesn't get out to the internet?  Are you seeing a handshake?  Can you ping?  What about DNS?

[Kinerg]

Was there specific access you were trying to prevent?  OPNSense blocks all access to Unbound except LAN by default.

Preventing IoT, cameras and similar VLANs from resolving DNS.

But it's not a factor for these issues anyway. The domains get resolved either way, but WG just doesn't get out to Internet (or back?) unless I manually restart it.

What do you mean by doesn't get out to the internet?  Are you seeing a handshake?  Can you ping?  What about DNS?

Should I run packet capture to see about handshake?

I can't ping, neither by hostname nor direct IP (e.g. 1.1.1.1). Hostname gets resolved by Unbound on OPNsense and the WireGuard client receives it, but the ping to requested destination gets no reply. So, WG_client<->OPNsense works, but WG_client<->Internet doesn't unless I restart the WireGuard instance.

Could this issue be related?

Can you confirm about Unbound ACL visibility in UI and missing options compared to documentation? Has this been changed for 23.7 or is it an issue on my end?

[Kinerg]

Ran packet capture after boot. It seems the ping request goes out to WAN but no response is visible.

Code: [Select]

nterface Timestamp SRC DST output

WGxInternet
wg1 2023-09-29
16:07:49.232765 length 88: 10.101.80.1 > 1.1.1.1: ICMP echo request, id 689, seq 1, length 64
WAN
vtnet1 2023-09-29
16:07:49.232796 xx:xx:xx:xx:xx:xx xx:xx:xx:xx:xx:xx IPv4, length 98: 10.101.80.1 > 1.1.1.1: ICMP echo request, id 689, seq 1, length 64
WGxInternet
wg1 2023-09-29
16:07:50.232109 length 88: 10.101.80.1 > 1.1.1.1: ICMP echo request, id 690, seq 1, length 64
WAN
vtnet1 2023-09-29
16:07:50.232146 xx:xx:xx:xx:xx:xx xx:xx:xx:xx:xx:xx IPv4, length 98: 10.101.80.1 > 1.1.1.1: ICMP echo request, id 690, seq 1, length 64


[CJ]

Was there specific access you were trying to prevent?  OPNSense blocks all access to Unbound except LAN by default.

Preventing IoT, cameras and similar VLANs from resolving DNS.

That all occurs by default.  Nothing will be able to reach Unbound outside of LAN unless you add a rule specifically for it regardless of what the Unbound listening interface is set to.

Should I run packet capture to see about handshake?

I can't ping, neither by hostname nor direct IP (e.g. 1.1.1.1). Hostname gets resolved by Unbound on OPNsense and the WireGuard client receives it, but the ping to requested destination gets no reply. So, WG_client<->OPNsense works, but WG_client<->Internet doesn't unless I restart the WireGuard instance.

Which hostname are you referring to?  example.com or something local?  What does the firewall log say?

Could this issue be related?

I don't think so but I haven't looked at it.  What rules do you have for WG?

Can you confirm about Unbound ACL visibility in UI and missing options compared to documentation? Has this been changed for 23.7 or is it an issue on my end?

I'm not sure what you mean.  Can you elaborate?  I've noticed no difference between 23.1 and 23.7 in terms of WG and made no changes for them either.

[Kinerg]

Was there specific access you were trying to prevent?  OPNSense blocks all access to Unbound except LAN by default.

Preventing IoT, cameras and similar VLANs from resolving DNS.

That all occurs by default.  Nothing will be able to reach Unbound outside of LAN unless you add a rule specifically for it regardless of what the Unbound listening interface is set to.

It will if I have a rule to Allow IoT_net to IoT_address to enable DHCP/NTP/routing access. Or am I mistaken?

Should I run packet capture to see about handshake?

I can't ping, neither by hostname nor direct IP (e.g. 1.1.1.1). Hostname gets resolved by Unbound on OPNsense and the WireGuard client receives it, but the ping to requested destination gets no reply. So, WG_client<->OPNsense works, but WG_client<->Internet doesn't unless I restart the WireGuard instance.

Which hostname are you referring to?  example.com or something local?  What does the firewall log say?

Any external hostname gets resolved, e.g. www.microsoft.com. Unbound isn't an issue regarding the WireGuard problem.

Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?

OPNsense fresh boot, before manual WG restart:

Code: [Select]

wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
After manual WG restart:

Code: [Select]

wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.

Could this issue be related?

I don't think so but I haven't looked at it.  What rules do you have for WG?

Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges

I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.

Can you confirm about Unbound ACL visibility in UI and missing options compared to documentation? Has this been changed for 23.7 or is it an issue on my end?

I'm not sure what you mean.  Can you elaborate?  I've noticed no difference between 23.1 and 23.7 in terms of WG and made no changes for them either.

I have a feeling the two issues are being conflated here. Maybe I should have made a separate post for the Unbound questions. The main issue is WireGuard not having Internet access after OPNsense boot unless the service is restarted.

The other, separate issue, is Unbound no longer showing its internal ACLs like before. Before 23.7 it listed all the automatically configured ACLs. Example in screenshot below which I found online. It used to show all internal automatically listed ACLs above plus the custom made ones. Now, on 23.7, I can only see the custom made rules and can't find which ACLs Unbound has active.

Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local.

[CJ]

That all occurs by default.  Nothing will be able to reach Unbound outside of LAN unless you add a rule specifically for it regardless of what the Unbound listening interface is set to.

It will if I have a rule to Allow IoT_net to IoT_address to enable DHCP/NTP/routing access. Or am I mistaken?

Well, yes, but that's because you configured it that way.  You can instead just enable the specific services needed.  You don't have to allow all ports.

[CJ]

The other, separate issue, is Unbound no longer showing its internal ACLs like before. Before 23.7 it listed all the automatically configured ACLs. Example in screenshot below which I found online. It used to show all internal automatically listed ACLs above plus the custom made ones. Now, on 23.7, I can only see the custom made rules and can't find which ACLs Unbound has active.

Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local.

Yes, IIRC, this was done to avoid issues with dynamic interfaces such as wireguard not being picked up.  So instead of a default deny with allow rules added for each interface subnet, unbound was changed to a default allow.

[CJ]

Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?

OPNsense fresh boot, before manual WG restart:

Code: [Select]

wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
After manual WG restart:

Code: [Select]

wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.

Could this issue be related?

I don't think so but I haven't looked at it.  What rules do you have for WG?

Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges

I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.

Do you have logging turned on for the WG rules?  Are they showing any hits during this time?

[Kinerg]

The other, separate issue, is Unbound no longer showing its internal ACLs like before. Before 23.7 it listed all the automatically configured ACLs. Example in screenshot below which I found online. It used to show all internal automatically listed ACLs above plus the custom made ones. Now, on 23.7, I can only see the custom made rules and can't find which ACLs Unbound has active.

Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local.

Yes, IIRC, this was done to avoid issues with dynamic interfaces such as wireguard not being picked up.  So instead of a default deny with allow rules added for each interface subnet, unbound was changed to a default allow.

I see, I must have missed that in the changelog. I do remember something to that effect, but not the UI changes being mentioned.

The Default Action setting doesn't work for me, though. It only affects OPNsense itself, but not the Interfaces. They continue having Unbound access regardless if set to Deny. I'm probably misunderstanding how the option should be used.

What about the missing Allow Snoop/Deny Non-local/Refuse Non-local?

[Kinerg]

Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?

OPNsense fresh boot, before manual WG restart:

Code: [Select]

wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
After manual WG restart:

Code: [Select]

wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.

Could this issue be related?

I don't think so but I haven't looked at it.  What rules do you have for WG?

Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges

I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.

Do you have logging turned on for the WG rules?  Are they showing any hits during this time?

I do. They don't show anything being blocked, but they do show a different outgoing address being used. Before WG service restart, pings go out to WAN from the WG interface IP. After restart, from the WAN interface IP.  NAT not working?

OPNsense fresh boot, before manual WG restart:

Code: [Select]

wan 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet

After manual WG restart:

Code: [Select]

wan 2023-10-01T15:44:56 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:56 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:44:55 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet


[opn69a]

I did a fresh installation of OPNsense on a separate machine to continue my troubleshooting with the issues I've been having with the new Wireguard kernel update to 2.0 and it looks like the 'main issue' is starting to lead to here: Wireguard doesn't establish a connection at boot.

I disabled Unbound DNS to rule that out as the culprit for the issues I've been having since updating to 23.7.3 (when Wireguard was updated to 2.0) and it looks like it wasn't related.

I backed up all my configuration _before_ doing the Wireguard setup. Can reboot, use internet, etc. nothing wrong whatsoever. But then the problem started after I setup Wireguard. Just for clariy, here's what I did:

Setup an endpoint, setup the local instance, and enable it. You get a Wireguard connection from the Firewall itself and can see this with the `wg` command. You can ping and curl and verify VPN access from the firewall. Before doing anything else - Reboot the firewall. Now go back to the Wireguard plugin, and you'll see it tries to send data, but never receives. It's completely stuck, even with the default NAT rules. Verified in an SSH session too, just to make sure. Go to the plugin, uncheck it, save, check it again, save... VPN connection is back and running. So I'm starting to think the main issue/bug I've been running into the past month or whatnot is potentially a result of this issue.

All in all, can +1 the report of WG not working unless manually restarting the plugin.

[Kinerg]

Setup an endpoint, setup the local instance, and enable it. You get a Wireguard connection from the Firewall itself and can see this with the `wg` command. You can ping and curl and verify VPN access from the firewall. Before doing anything else - Reboot the firewall. Now go back to the Wireguard plugin, and you'll see it tries to send data, but never receives. It's completely stuck, even with the default NAT rules. Verified in an SSH session too, just to make sure. Go to the plugin, uncheck it, save, check it again, save... VPN connection is back and running. So I'm starting to think the main issue/bug I've been running into the past month or whatnot is potentially a result of this issue.

Yes, that sounds exactly like the issue I'm having.