Messages

For FQ_C bad performance or problems during slow/new start are usually caused by two reasons ECN & limit.

What is the final suggestion on ECN enable/disable? Resources, manuals, guides and forum posts are inconsistent regarding this. Some say to enable both for download and upload, some to disable it for upload. OPNsense guide isn't quite clear if it should be enabled or disabled for upload.

What does slow start refer to exactly and is there an easy way to test it?

Thank you!

Possibly related?

#7148
#6909

Hi, valid point and thanks for the advice, I can give it a try and based on the setup guide on quad9 its anyway not mentioned https://www.quad9.net/support/set-up-guides/setup-opnsense-and-dns-over-tls.

I've had issues with Quad9 DoT and DNSSEC, too. They explicitly say to disable it in their Pfsense guide:
https://docs.quad9.net/Setup_Guides/Open-Source_Routers/pfSense_%28Encrypted%29/

Not sure why it's not mentioned for Opnsense.

After this (I was pretty sure a fresh install would help, because migration could have screwed things up maybe), I decided to disable DNSSEC and DoT, but leave Unbound DNS as default DNS.

This setup is now stable for at least 24 hours!

Can you try disabling DNSSEC while using DoT? It should be disabled as your DoT/DoH server is the one ensuring DNSSEC anyway.

You need to set up NAT reflection :

https://docs.opnsense.org/manual/how-tos/nat_reflection.html

There is an issue with BSD here, either with the X520 interface, or something else within the OS. I will test the other interface cards when they arrive to see if they can improve the stability. There is still an issue with OPNsense here though where it is 20% the speed out of the box with the 10g interfaces then its FreeBSD base OS.

Hi, have you managed to test the new adapters?

Compare the content of /tmp/rules.debug before and after you hit Save and look if something similar to this is missing before saving:

Code Select Expand

nat on vtnet1 inet from (wg2:network) to any port 500 -> (vtnet1:0) static-port # Automatic outbound rule
nat on vtnet1 inet from (wg1:network) to any port 500 -> (vtnet1:0) static-port # Automatic outbound rule

nat on vtnet1 inet from (wg2:network) to any -> (vtnet1:0) port 1024:65535 # Automatic outbound rule
nat on vtnet1 inet from (wg1:network) to any -> (vtnet1:0) port 1024:65535 # Automatic outbound rule

Does running

Code Select Expand

/usr/local/etc/rc.filter_configure also fix the issue for you?

Possibly related to #6909?

You can use Unbound config to segregate responses via access-control-view

I'm getting the same error. No gateway monitor, no static routes. After 23.7.5 boot:

Code Select Expand

2023-09-27T22:17:02 Notice wireguard Wireguard interface WGSite2Site (wg2) started
2023-09-27T22:17:02 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.101.1.2/30' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) can not reconfigure without stopping it first.
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) started
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) can not reconfigure without stopping it first.


I've opened bug reports for the WireGuard and Unbound issues to seek additional help. Will report back here with the findings.

Setup an endpoint, setup the local instance, and enable it. You get a Wireguard connection from the Firewall itself and can see this with the `wg` command. You can ping and curl and verify VPN access from the firewall. Before doing anything else - Reboot the firewall. Now go back to the Wireguard plugin, and you'll see it tries to send data, but never receives. It's completely stuck, even with the default NAT rules. Verified in an SSH session too, just to make sure. Go to the plugin, uncheck it, save, check it again, save... VPN connection is back and running. So I'm starting to think the main issue/bug I've been running into the past month or whatnot is potentially a result of this issue.

Yes, that sounds exactly like the issue I'm having.

Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?

OPNsense fresh boot, before manual WG restart:

Code Select Expand

wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself

After manual WG restart:

Code Select Expand

wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)

My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.

Could this issue be related?

I don't think so but I haven't looked at it.  What rules do you have for WG?

Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges

I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.

Do you have logging turned on for the WG rules?  Are they showing any hits during this time?

I do. They don't show anything being blocked, but they do show a different outgoing address being used. Before WG service restart, pings go out to WAN from the WG interface IP. After restart, from the WAN interface IP.  NAT not working?

OPNsense fresh boot, before manual WG restart:

Code Select Expand


wan 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet


After manual WG restart:

Code Select Expand


wan 2023-10-01T15:44:56 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:56 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:44:55 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet


[plus]

The other, separate issue, is Unbound no longer showing its internal ACLs like before. Before 23.7 it listed all the automatically configured ACLs. Example in screenshot below which I found online. It used to show all internal automatically listed ACLs above plus the custom made ones. Now, on 23.7, I can only see the custom made rules and can't find which ACLs Unbound has active.

Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local.

Yes, IIRC, this was done to avoid issues with dynamic interfaces such as wireguard not being picked up.  So instead of a default deny with allow rules added for each interface subnet, unbound was changed to a default allow.

I see, I must have missed that in the changelog. I do remember something to that effect, but not the UI changes being mentioned.

The Default Action setting doesn't work for me, though. It only affects OPNsense itself, but not the Interfaces. They continue having Unbound access regardless if set to Deny. I'm probably misunderstanding how the option should be used.

What about the missing Allow Snoop/Deny Non-local/Refuse Non-local?

[plus]

Was there specific access you were trying to prevent?  OPNSense blocks all access to Unbound except LAN by default.

Preventing IoT, cameras and similar VLANs from resolving DNS.

That all occurs by default.  Nothing will be able to reach Unbound outside of LAN unless you add a rule specifically for it regardless of what the Unbound listening interface is set to.

It will if I have a rule to Allow IoT_net to IoT_address to enable DHCP/NTP/routing access. Or am I mistaken?

Should I run packet capture to see about handshake?

I can't ping, neither by hostname nor direct IP (e.g. 1.1.1.1). Hostname gets resolved by Unbound on OPNsense and the WireGuard client receives it, but the ping to requested destination gets no reply. So, WG_client<->OPNsense works, but WG_client<->Internet doesn't unless I restart the WireGuard instance.

Which hostname are you referring to?  example.com or something local?  What does the firewall log say?

Any external hostname gets resolved, e.g. www.microsoft.com. Unbound isn't an issue regarding the WireGuard problem.

Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?

OPNsense fresh boot, before manual WG restart:

Code Select Expand

wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself

After manual WG restart:

Code Select Expand

wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)

My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.

Could this issue be related?

I don't think so but I haven't looked at it.  What rules do you have for WG?

Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges

I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.

Can you confirm about Unbound ACL visibility in UI and missing options compared to documentation? Has this been changed for 23.7 or is it an issue on my end?

I'm not sure what you mean.  Can you elaborate?  I've noticed no difference between 23.1 and 23.7 in terms of WG and made no changes for them either.

I have a feeling the two issues are being conflated here. Maybe I should have made a separate post for the Unbound questions. The main issue is WireGuard not having Internet access after OPNsense boot unless the service is restarted.

The other, separate issue, is Unbound no longer showing its internal ACLs like before. Before 23.7 it listed all the automatically configured ACLs. Example in screenshot below which I found online. It used to show all internal automatically listed ACLs above plus the custom made ones. Now, on 23.7, I can only see the custom made rules and can't find which ACLs Unbound has active.

Also, the Default Action option only shows Allow/Deny/Refuse, while the manual states there should also be Allow Snoop/Deny Non-local/Refuse Non-local.