A firewalll rule pattern that consistently causes issues

[opnseeker]

I have an interface group with a rule that sends all internet traffic over a gateway ( can be the default or VPN ).

The above rule is setup with FirstMatch disabled so that I can have rules in individual interfaces that can override the gateway.

When I have a rule in one of the member interfaces to redirect traffic from a specific ip address over a different gateway with FirstMatch enabled, I expect the traffic from that specific ip address to be sent over the 2nd gateway while the rest use the first gateway (set in the group rule).

But it doesn't work. The traffic that matches the rule in the interface is rejected or blocked. Many times, traffic from the entire interface containing the override rule is blocked.

This seems to happen quite consistently. I have two instances with the above pattern and both have the same issue.

If the override rule blocks traffic instead of redirecting it over a different gateway, it seems to work as expected.

Is this the way it is supposed to work or is it a bug?

[Patrick M. Hausen]

Please post all details of the rules in question.

[opnseeker]

Sorry about the late reply. I somehow missed your reply asking for more details.

I am still troubleshooting the issue. I found the issue in one of the cases and fixed it. It was to do with rule tags which were used to prevent bypassing VPN by creating a kill switch rule.

In the other case, it is not consistent. It works sometimes and not at others. In this case there is no kill switch being used.

There is also a case where it constantly seem to work.

I will post the details later in the day.

[opnseeker]

Firewall Interface group: IG

IG: VL1, VL2, WG1

VL1 and VL2 are VLAN interfaces
WG1 is a Wireguard interface used for incoming connections with multiple peers: WGP1, WGP2 & WGP3

Rules

Quick off
Dir: In
Action: Pass
Interface: IG
Version: IP4
Protocols: TCP/UDP
Source:any
Source port: any
Dest: not private networks
Dest port: any
Gateway: default

Quick on
Dir: In
Action: Pass
Interface: WG1
Version: IP4/IP6
Protocols: TCP/UDP
Source:WGP1
Source port: any
Dest: not private networks
Dest port: any
Gateway: VPN gateway

First rule sends all traffic to internet over the ISP connection using IP4 (IP6 is not supported by the ISP).

Second rule is supposed to override that for one of the incoming wireguard connections and send its internet traffic over VPN using both IP4 and IP6.

Problem:

When WGP1 and WGP2 are both connected behavior is unpredictable. Sometimes everything works. Sometimes both cannot get to the internet. Sometimes, one gets to the internet and the other doesn't. I can't think of any specific pattern when one happens vs another.

For internal clients not connected over wireguard, this pattern seems to work.

Hopefully, this provides enough detail. Thanks in advance for the help.

[lmoore]

What happens when you set the Gateway in these two rules to None?

You will also have rules for your outbound traffic - what do they look like?

With your first rule, you could set the source to WGP1 and enable the Invert Source option. In addition, you could also enable Quick.

Which rules appear in the logs that are blocking these connections?

[opnseeker]

What happens when you set the Gateway in these two rules to None?

First rule, gateway is None. I said default which is the same behavior as it uses the default gateway.

For second rule, setting gateway to None defeats the purpose but I will try and see what happens.

You will also have rules for your outbound traffic - what do they look like?

I have one outbound rule on the WAN interface which is a killswitch when a tag is set. The tags are not used in these two rules.

With your first rule, you could set the source to WGP1 and enable the Invert Source option. In addition, you could also enable Quick.

I can. But I use interface groups to reduce number of rules. In this case this group is to allow internet traffic over default gateway.

I will have do it as a resort if nothing else solves the issue - have separate rules for WG1 interface and remove it from the interface group. I can give it a try.

Which rules appear in the logs that are blocking these connections?

I haven't checked the logs as this issue occurs when I am connecting to home while I am away. I will simulate it and check the logs.

[lmoore]

Do you only have 1 x WAN service and you run the IPv4/IPv6 VPN over it?

If you only have the one outbound WAN rule which you use for a kill-switch, how are the other connections making it out through to the Internet?

Your VL1 & VL2 nodes wont use the rule you have assigned to WG1. Do any other nodes use the VPN for others connections?

More information about your set up is required, including subnets and masks for VL1, VL2 & WG1.

Is the outbound VPN connecting you to another subnet or is it effectively a second gateway to the Internet?

You should also include NAT rules, be it the legacy Outbound (if you are still using them) or the Source NAT rules - logging should be enabled on these rules whilst you are troubleshooting.

Are you using any Destination NAT rules?

OPNsense has a default firewall group for Wireguard. You should check this and list the rules included in the group - note, you will need to access them using Firewall -> Rules [new] and then select WireGuard (Group) from the drop-down list. Clicking on the Group name in Firewall -> Groups takes you to the old rules.

Are your WireGuard nodes configured with an IPv6 address on their interface?

Using tags in rules may be of assistance to you. I use tags in the majority of my rules, this includes NAT rules.

You should be able to easily test your WireGuard VPN when connected to your internal network - it will ease troubleshooting, including obtaining packet captures on the WireGuard interface.

You could also use traceroute with ICMP packets to see where responses get lost.

[opnseeker]

Do you only have 1 x WAN service and you run the IPv4/IPv6 VPN over it?

I have one WAN service with IP4 (no IP6). I have multiple VPN tunnels up with both IP4/6 support. Internally I use ULA IP6 to route traffic (DNS, firewall access, etc.). The same is used to redirect traffic over IP6 via the VPNs.

If you only have the one outbound WAN rule which you use for a kill-switch, how are the other connections making it out through to the Internet?

I have one outbound rule that I set and manage. There are auto generated ones which I don't touch. So far, I just assumed that they are good enough. I try to control all the traffic using inbound rules.

Your VL1 & VL2 nodes wont use the rule you have assigned to WG1. Do any other nodes use the VPN for others connections?

I have 2 devices in VL1 network that are redirected over VPN and I never had issues with them. That's why I specifically said that the issue is occurring with connections coming over Wireguard.

All VL2 devices go to the default WAN.

More information about your set up is required, including subnets and masks for VL1, VL2 & WG1.

Is the outbound VPN connecting you to another subnet or is it effectively a second gateway to the Internet?

All outbound VPN connections use Proton VPN to reach the internet.

You should also include NAT rules, be it the legacy Outbound (if you are still using them) or the Source NAT rules - logging should be enabled on these rules whilst you are troubleshooting.

Are you using any Destination NAT rules?

I left the default outbound rules as they are for the IP4 WAN NAT. Otherwise, I use the SNAT rules.

I use SNAT in conjunction with DNAT to ensure the returning traffic is routed via OpnSense.
I also use SNAT for VPNs (both IP4 & 6) to ensure that all VPN traffic appears to be originating from one interface. Proton VPN seems to have issues otherwise even though it implements its own NAT for both Ip4 & 6.

I suppose the SNAT rules are outbound rules that I manage.

All the DNAT rules are internal (none on WAN - incoming or outgoing). They are primarily used to redirect DNS, firewall GUI access and Proxmox host access.

OPNsense has a default firewall group for Wireguard. You should check this and list the rules included in the group - note, you will need to access them using Firewall -> Rules [new] and then select WireGuard (Group) from the drop-down list. Clicking on the Group name in Firewall -> Groups takes you to the old rules.

Yes. I have seen it but never touched any of the rules.

I have SNAT rules for all outbound VPNs and the filter rules I already described for the one inbound instance with multiple peers (WG1, etc.)

Are your WireGuard nodes configured with an IPv6 address on their interface?

Yes. All wireguard instances and peers are IP6 enabled.

Using tags in rules may be of assistance to you. I use tags in the majority of my rules, this includes NAT rules.

Not sure of this. I use tags for the VPN kill switch and setting DNS addresses for some interfaces via DHCP.

You should be able to easily test your WireGuard VPN when connected to your internal network - it will ease troubleshooting, including obtaining packet captures on the WireGuard interface.

You could also use traceroute with ICMP packets to see where responses get lost.

I will look into this. I already have a way to test Wireguard from the network.

I will also post some of the details you asked for once I have looked at the logs.

Thanks for the detailed analysis.

[lmoore]

A reason for random and coincidental lack of connectivity for the WireGuard nodes may be due to DNS. Where are your WireGuard nodes pointing to for their DNS?

Perhaps you could set up an additional WireGuard instance and only have WGP1 use that instance, ensuring it goes out the VPN gateway.
 - Create a new WG instance.
 - Tag the traffic arriving on the new WG instance.
 - Create a Source NAT rule out the VPN Gateway and match the tag.
 - Create an outgoing rule setting the gateway to your VPN gateway and matching the tag.

See if you can get this working without D-NAT.

Also, OPNsense documentation has a note regarding Interface Groups and multi-wan configurations: https://docs.opnsense.org/manual/firewall_groups.html#interface-groups

Good luck.