Messages

All traffic within the network including traffic incoming via wireguard and outgoing via WAN and VPN use pi-hole. Haven't seen any issues with that setup recently.

I will checkout the link you posted and troubleshoot further.

 I couldn't reproduce the issue so far when I am home.

Thanks for the help.

[WireGuard (Group)]

Do you only have 1 x WAN service and you run the IPv4/IPv6 VPN over it?

I have one WAN service with IP4 (no IP6). I have multiple VPN tunnels up with both IP4/6 support. Internally I use ULA IP6 to route traffic (DNS, firewall access, etc.). The same is used to redirect traffic over IP6 via the VPNs.

If you only have the one outbound WAN rule which you use for a kill-switch, how are the other connections making it out through to the Internet?

I have one outbound rule that I set and manage. There are auto generated ones which I don't touch. So far, I just assumed that they are good enough. I try to control all the traffic using inbound rules.

Your VL1 & VL2 nodes wont use the rule you have assigned to WG1. Do any other nodes use the VPN for others connections?

I have 2 devices in VL1 network that are redirected over VPN and I never had issues with them. That's why I specifically said that the issue is occurring with connections coming over Wireguard.

All VL2 devices go to the default WAN.

More information about your set up is required, including subnets and masks for VL1, VL2 & WG1.

Is the outbound VPN connecting you to another subnet or is it effectively a second gateway to the Internet?

All outbound VPN connections use Proton VPN to reach the internet.

You should also include NAT rules, be it the legacy Outbound (if you are still using them) or the Source NAT rules - logging should be enabled on these rules whilst you are troubleshooting.

Are you using any Destination NAT rules?

I left the default outbound rules as they are for the IP4 WAN NAT. Otherwise, I use the SNAT rules.

I use SNAT in conjunction with DNAT to ensure the returning traffic is routed via OpnSense.
I also use SNAT for VPNs (both IP4 & 6) to ensure that all VPN traffic appears to be originating from one interface. Proton VPN seems to have issues otherwise even though it implements its own NAT for both Ip4 & 6.

I suppose the SNAT rules are outbound rules that I manage.

All the DNAT rules are internal (none on WAN - incoming or outgoing). They are primarily used to redirect DNS, firewall GUI access and Proxmox host access.

OPNsense has a default firewall group for Wireguard. You should check this and list the rules included in the group - note, you will need to access them using Firewall -> Rules [new] and then select WireGuard (Group) from the drop-down list. Clicking on the Group name in Firewall -> Groups takes you to the old rules.

Yes. I have seen it but never touched any of the rules.

I have SNAT rules for all outbound VPNs and the filter rules I already described for the one inbound instance with multiple peers (WG1, etc.)

Are your WireGuard nodes configured with an IPv6 address on their interface?

Yes. All wireguard instances and peers are IP6 enabled.

Using tags in rules may be of assistance to you. I use tags in the majority of my rules, this includes NAT rules.

Not sure of this. I use tags for the VPN kill switch and setting DNS addresses for some interfaces via DHCP.

You should be able to easily test your WireGuard VPN when connected to your internal network - it will ease troubleshooting, including obtaining packet captures on the WireGuard interface.

You could also use traceroute with ICMP packets to see where responses get lost.

I will look into this. I already have a way to test Wireguard from the network.

I will also post some of the details you asked for once I have looked at the logs.

Thanks for the detailed analysis.

[Gateway]

What happens when you set the Gateway in these two rules to None?

First rule, gateway is None. I said default which is the same behavior as it uses the default gateway.

For second rule, setting gateway to None defeats the purpose but I will try and see what happens.

You will also have rules for your outbound traffic - what do they look like?

I have one outbound rule on the WAN interface which is a killswitch when a tag is set. The tags are not used in these two rules.

With your first rule, you could set the source to WGP1 and enable the Invert Source option. In addition, you could also enable Quick.

I can. But I use interface groups to reduce number of rules. In this case this group is to allow internet traffic over default gateway.

I will have do it as a resort if nothing else solves the issue - have separate rules for WG1 interface and remove it from the interface group. I can give it a try.

Which rules appear in the logs that are blocking these connections?

I haven't checked the logs as this issue occurs when I am connecting to home while I am away. I will simulate it and check the logs.

Firewall Interface group: IG

IG: VL1, VL2, WG1

VL1 and VL2 are VLAN interfaces
WG1 is a Wireguard interface used for incoming connections with multiple peers: WGP1, WGP2 & WGP3

Rules

Quick off
Dir: In
Action: Pass
Interface: IG
Version: IP4
Protocols: TCP/UDP
Source:any
Source port: any
Dest: not private networks
Dest port: any
Gateway: default

Quick on
Dir: In
Action: Pass
Interface: WG1
Version: IP4/IP6
Protocols: TCP/UDP
Source:WGP1
Source port: any
Dest: not private networks
Dest port: any
Gateway: VPN gateway

First rule sends all traffic to internet over the ISP connection using IP4 (IP6 is not supported by the ISP).

Second rule is supposed to override that for one of the incoming wireguard connections and send its internet traffic over VPN using both IP4 and IP6.

Problem:

When WGP1 and WGP2 are both connected behavior is unpredictable. Sometimes everything works. Sometimes both cannot get to the internet. Sometimes, one gets to the internet and the other doesn't. I can't think of any specific pattern when one happens vs another.

For internal clients not connected over wireguard, this pattern seems to work.

Hopefully, this provides enough detail. Thanks in advance for the help.

Sorry about the late reply. I somehow missed your reply asking for more details.

I am still troubleshooting the issue. I found the issue in one of the cases and fixed it. It was to do with rule tags which were used to prevent bypassing VPN by creating a kill switch rule.

In the other case, it is not consistent. It works sometimes and not at others. In this case there is no kill switch being used.

There is also a case where it constantly seem to work.

I will post the details later in the day.

After setting bridge-mssnoop to 0 and restarting the host, ip6 addresses are being assigned to devices on the bridge network.

It worked more than 3 times and is likely fixed.

Thanks for the help.

Bridge vmbr1 is already VLAN aware. I added bridge-mcsnoop 0 now to vmbr1 configuration. I have to wait until tonight to restart the host.

I will post again after I restart and check.

Thanks for the suggestions.

I apologize for the late reply.

I have one NIC for LAN and one for WAN on the mini PC I am using. So, I unfortunately cannot separate tagged and untagged networks.

On the other hand, I have been using this setup with OpnSense (directly on the hardware) without Proxmox for multiple years. It has been working. In this case, I was using Opnsense to manage VLANs but the tagged and untagged (LAN/Home) networks were on the same NIC.

I recently switched to using Proxmox with VNets to simulate the VLANs over the default bridge network (vmbr1) in Proxmox attached to the hardware NIC. The same managed switch is used in both cases (it's an Asus router/AP with VLAN support).

The router advertisement problem is occurring with the default bridge network while VLANs using the VNETs are working fine.

Strange thing is that the devices on the network acquire IP6 addresses fine once or twice after DNSMasq is restarted and stop working until the next restart.

I switched Home and Admin such that now Admin uses the untagged network and Home uses tagged network.

Now the problem occurs on the Admin interface. It looks like this problem is specific to the untagged network.

Hope that generates some interest and suggestions.

I have an interface group with a rule that sends all internet traffic over a gateway ( can be the default or VPN ).

The above rule is setup with FirstMatch disabled so that I can have rules in individual interfaces that can override the gateway.

When I have a rule in one of the member interfaces to redirect traffic from a specific ip address over a different gateway with FirstMatch enabled, I expect the traffic from that specific ip address to be sent over the 2nd gateway while the rest use the first gateway (set in the group rule).

But it doesn't work. The traffic that matches the rule in the interface is rejected or blocked. Many times, traffic from the entire interface containing the override rule is blocked.

This seems to happen quite consistently. I have two instances with the above pattern and both have the same issue.

If the override rule blocks traffic instead of redirecting it over a different gateway, it seems to work as expected.

Is this the way it is supposed to work or is it a bug?

I have OpnSense running on a Proxmox VM with multiple networks. These are not VLANs from OpnSense perspective as Proxmox vnets are used to hide them from OpnSense. In OpnSense they are all separate networks.

I use DNSMasq for DHCP and tried both native as well as separate router advertisements and both have the same problem.

I have IPV6 enabled on three networks - these are ULA addresses used for internal communication.

DMZ (VNET VLAN - one VM on a wired network)
- statically assigned IP4 and IP6 addresses using DHCP and DHCP6.  Slaac is used to generate addl. private outgoing IP6 addresses. (No issues)

Admin (VNET VLAN - fixed/known devices on wifi)
- statically assigned Ip4 and IP6 addresses using DHCP and DHCP6 (no Slaac) (no issues)

Home/LAN (all other devices using WiFi - this is not a VNET but untagged linux bridge created with a physical port on the machine)

All addresses on this network are dynamically assigned by DHCP for IPV4 which works.

In case of IP6, I tried many options (DHCP6, Slaac with Stateless DHCP for DNS address, both). In all cases, address assignment works few times after the changes are made in OpnSense and subsequently stops working.

When I check the log of DNSMasq there is RTR-Advert for the network address but no DHCP-Solicit from the clients after the first few times.

Any suggestions are appreciated.

I tried many times with all variations and I can't find a reason why it happens only on this network.

Can you paste your NAT rule for comparison?

Here are some screenshots of what works for me (on 26.1.2).  I have Unbound set to listen on All interfaces.  Not shown is the manual pass rule for the NAT rdr that was imported from my legacy ruleset into the new rules UI, but the firewall log shows it passing the traffic.  I use an alias in the NAT rule because I have this VIP referenced in several places.

You cannot view this attachment.




You cannot view this attachment.

You cannot view this attachment.

I am not using virtual IP. My assigned ip to the loopback interface is ULA.

Unbound runs on just this interface as does the Opnsense GUI. I tried the redirection for both Unbound and Opnsense. On Ip4 127.0.0.1 works but nothing works on Ip6.

I tried again. There are no errors in the log but the rule is not firing. I am using "pass" for firewall rule and not an explicit one. That works for Ip4 and not Ip6.

Rule may not be firing because the destination matches with redirect, both ip and ports. That's the case most of the time, the way it is setup. Redirect rule is for safety and pass is used to pass the traffic without explicit filter rule.

This wasn't the case until 25.x.x. Now the rules seem to be ignored when redirect is not needed and because of that pass option is not effective. Explicit rule is needed.

I will test my theory and post later.

Thanks for the screenshots. They are helpful.

I am right. Redirect rules do not fire when destination matches redirect (both port and ip) and the pass option for filter rule is not activated.

That was the issue in my case. I had to add explicit filter rules to allow traffic when there is a possibility of match between and destination and redirect.

Can you paste your NAT rule for comparison?

Here are some screenshots of what works for me (on 26.1.2).  I have Unbound set to listen on All interfaces.  Not shown is the manual pass rule for the NAT rdr that was imported from my legacy ruleset into the new rules UI, but the firewall log shows it passing the traffic.  I use an alias in the NAT rule because I have this VIP referenced in several places.

You cannot view this attachment.




You cannot view this attachment.

You cannot view this attachment.

I am not using virtual IP. My assigned ip to the loopback interface is ULA.

Unbound runs on just this interface as does the Opnsense GUI. I tried the redirection for both Unbound and Opnsense. On Ip4 127.0.0.1 works but nothing works on Ip6.

I tried again. There are no errors in the log but the rule is not firing. I am using "pass" for firewall rule and not an explicit one. That works for Ip4 and not Ip6.

Rule may not be firing because the destination matches with redirect, both ip and ports. That's the case most of the time, the way it is setup. Redirect rule is for safety and pass is used to pass the traffic without explicit filter rule.

This wasn't the case until 25.x.x. Now the rules seem to be ignored when redirect is not needed and because of that pass option is not effective. Explicit rule is needed.

I will test my theory and post later.

Thanks for the screenshots. They are helpful.

You can't use ::1 but the ULA should work.

In my setup I assigned a ULA VIP to the Loopback interface where Unbound also listens, then with a DNAT rule I forward outbound DNS on port 53 to that ULA IP.  Slightly different use case (to trap and redirect unencrypted DNS escapes) but same principle.  Seems to work OK.

My use case is the same but doesn't work for IP6 even with ULA on 26.x.x. It worked until 25.x.x.

Up to 25.x.x, Port Forward to another port on Opnsense worked.

This issue is new in 26.x.x with thd new Dest NAT section.