Messages

I forgot mention that on the screeen from Proxmox -> pve -> Network , there is no setup any IP and gateway on the vmbr1 (VLANs_ETH0) - I setup IP and gateway for VLANs_ETH0 on the OPNsense and on WAN the same, so only is setup on the MAGMT_ETH1 setup 192.168.1.178/24.

This is confusing. You should only have IP and gateway on vmbr2 MAGMT (and not the same as LAN 192.168.1.1/24, as others have said), leave the other two blank.

It is also not clear how many DHCP servers you have on the network.

Is the TP Link SG108PE port towards Fujitsu S920 set up as trunk?

How are your interfaces set up in OPNsense?

Possibly related to #7148 and #6909?

For FQ_C bad performance or problems during slow/new start are usually caused by two reasons ECN & limit.

What is the final suggestion on ECN enable/disable? Resources, manuals, guides and forum posts are inconsistent regarding this. Some say to enable both for download and upload, some to disable it for upload. OPNsense guide isn't quite clear if it should be enabled or disabled for upload.

What does slow start refer to exactly and is there an easy way to test it?

Thank you!

Possibly related?

#7148
#6909

Hi, valid point and thanks for the advice, I can give it a try and based on the setup guide on quad9 its anyway not mentioned https://www.quad9.net/support/set-up-guides/setup-opnsense-and-dns-over-tls.

I've had issues with Quad9 DoT and DNSSEC, too. They explicitly say to disable it in their Pfsense guide:
https://docs.quad9.net/Setup_Guides/Open-Source_Routers/pfSense_%28Encrypted%29/

Not sure why it's not mentioned for Opnsense.

After this (I was pretty sure a fresh install would help, because migration could have screwed things up maybe), I decided to disable DNSSEC and DoT, but leave Unbound DNS as default DNS.

This setup is now stable for at least 24 hours!

Can you try disabling DNSSEC while using DoT? It should be disabled as your DoT/DoH server is the one ensuring DNSSEC anyway.

You need to set up NAT reflection :

https://docs.opnsense.org/manual/how-tos/nat_reflection.html

There is an issue with BSD here, either with the X520 interface, or something else within the OS. I will test the other interface cards when they arrive to see if they can improve the stability. There is still an issue with OPNsense here though where it is 20% the speed out of the box with the 10g interfaces then its FreeBSD base OS.

Hi, have you managed to test the new adapters?

Compare the content of /tmp/rules.debug before and after you hit Save and look if something similar to this is missing before saving:

Code Select Expand

nat on vtnet1 inet from (wg2:network) to any port 500 -> (vtnet1:0) static-port # Automatic outbound rule
nat on vtnet1 inet from (wg1:network) to any port 500 -> (vtnet1:0) static-port # Automatic outbound rule

nat on vtnet1 inet from (wg2:network) to any -> (vtnet1:0) port 1024:65535 # Automatic outbound rule
nat on vtnet1 inet from (wg1:network) to any -> (vtnet1:0) port 1024:65535 # Automatic outbound rule

Does running

Code Select Expand

/usr/local/etc/rc.filter_configure also fix the issue for you?

Possibly related to #6909?

You can use Unbound config to segregate responses via access-control-view

I'm getting the same error. No gateway monitor, no static routes. After 23.7.5 boot:

Code Select Expand

2023-09-27T22:17:02 Notice wireguard Wireguard interface WGSite2Site (wg2) started
2023-09-27T22:17:02 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.101.1.2/30' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) can not reconfigure without stopping it first.
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) started
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) can not reconfigure without stopping it first.


I've opened bug reports for the WireGuard and Unbound issues to seek additional help. Will report back here with the findings.

Setup an endpoint, setup the local instance, and enable it. You get a Wireguard connection from the Firewall itself and can see this with the `wg` command. You can ping and curl and verify VPN access from the firewall. Before doing anything else - Reboot the firewall. Now go back to the Wireguard plugin, and you'll see it tries to send data, but never receives. It's completely stuck, even with the default NAT rules. Verified in an SSH session too, just to make sure. Go to the plugin, uncheck it, save, check it again, save... VPN connection is back and running. So I'm starting to think the main issue/bug I've been running into the past month or whatnot is potentially a result of this issue.

Yes, that sounds exactly like the issue I'm having.

Firewall log doesn't show anything incoming being blocked and shows ping going out, but differently before/after WG restart. Is this a gateway/routing issue?

OPNsense fresh boot, before manual WG restart:

Code Select Expand

wan 2023-09-30T15:53:46 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself

After manual WG restart:

Code Select Expand

wan 2023-09-30T15:56:41 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)

My OPNsense box has the 192.168.61.10 on the WAN interface provided by an external router and 192.168.61.1 as WAN_GW.

Could this issue be related?

I don't think so but I haven't looked at it.  What rules do you have for WG?

Block WG_net to WG_address ports 22/80/443
Allow WG_net to WG_net
Block WG_net to All_Private_IP_ranges
Allow WG_net to !All_Private_IP_ranges

I also have NAT Port Forward redirect set up so that WireGuard enabled phones work when connected to local WiFi, but that doesn't make a difference regarding this problem.

Do you have logging turned on for the WG rules?  Are they showing any hits during this time?

I do. They don't show anything being blocked, but they do show a different outgoing address being used. Before WG service restart, pings go out to WAN from the WG interface IP. After restart, from the WAN interface IP.  NAT not working?

OPNsense fresh boot, before manual WG restart:

Code Select Expand


wan 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp let out anything from firewall host itself
WGxInternet 2023-10-01T15:43:54 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet


After manual WG restart:

Code Select Expand


wan 2023-10-01T15:44:56 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:56 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet
wan 2023-10-01T15:44:55 192.168.61.10 1.1.1.1 icmp let out anything from firewall host itself (force gw)
WGxInternet 2023-10-01T15:44:55 10.101.80.1 1.1.1.1 icmp Allow WGxInternet to Internet