"
Hello,
I realize this may be an edge case - I'm hoping I'm just missing a setting somewhere that can fix this. I've searched (Google, forums, Reddit/GitHub, etc.) and not found what I'm looking for - some false starts, but nothing that matches my situation closely enough.
The setup is depicted in the diagram, but I have 3 ISPs to my home, all fiber ONTs, that each get dropped to their own VLAN.
I have two fully-updated OPNsense firewalls - one for Production and one for Laboratory (HomeProd and HomeLab).
Both OPNsense firewalls have been setup using both the WireGuard Road Warrior and Multi-WAN tutorials. I have laptop clients that I may connect directly to the LAN of either firewall, and then VPN-client over to the other firewall's network.
I'm having what I believe are two separate problems, that are both manifested in this setup. The problems as I see them:
- Problem #1 is that HomeProd OPNsense is sending out HomeLab WireGuard client traffic on the wrong interface (Sending out the ISP_C instead of ISP_A, even when all 3 ISPs are "up".)
- Problem #2 is that OPNsense does not seem to accept a WireGuard VPN connection from an endpoint that's on the same subnet as the interface accepting it. So, for example, the HomeLab instance, when the ISP_C range is (for example) 70.0.0.5/24, then it cannot accept any connections from 70.0.0.6/24, because it's in the same subnet.
What this means is that connections from clients on the HomeProd LAN to the HomeLab WireGuard instance fail, because they're going out the wrong interface, and that interface is on the same subnet as the HomeLab WAN interface.
However, connections from clients on the HomeLab LAN that hit the HomeProd WAN (ISP_A) interface, work just fine, since it's a completely different ISP (and thus not sharing a subnet).
I further tested this by unplugging the ONTs for both ISP_A and ISP_B, forcing HomeProd to failover to ISP_C as its primary, and thus making both firewalls' only functioning WAN interfaces, on the same subnet. In this case, no client connections in either direction work.
However, if I disable ISP_C on HomeProd, then traffic (correctly) forced out the ISP_A interface, and connections to HomeLab WireGuard work properly (and connections in the other direction continue to work, too, of course).
Caveats:
- when attempting to connect, the corresponding "server" firewall will show the client briefly with a green checkmark, but it only ever gets the first handshake/heartbeat (set to 20 second intervals), and eventually falls off
- I do have one device that had an "established" link (an IoT camera), and it has somehow managed a still-working VPN client connection from HomeProd LAN to HomeLab WireGuard, throughout my experimentation, including a reboot of the HomeProd firewall.
- All connections to either Firewall WireGuard instance made "away" from home work perfectly fine. It's just when trying to go between the two that's problematic at this time.
I realize this may be an edge case - I'm hoping I'm just missing a setting somewhere that can fix this. I've searched (Google, forums, Reddit/GitHub, etc.) and not found what I'm looking for - some false starts, but nothing that matches my situation closely enough.
The setup is depicted in the diagram, but I have 3 ISPs to my home, all fiber ONTs, that each get dropped to their own VLAN.
I have two fully-updated OPNsense firewalls - one for Production and one for Laboratory (HomeProd and HomeLab).
Both OPNsense firewalls have been setup using both the WireGuard Road Warrior and Multi-WAN tutorials. I have laptop clients that I may connect directly to the LAN of either firewall, and then VPN-client over to the other firewall's network.
I'm having what I believe are two separate problems, that are both manifested in this setup. The problems as I see them:
- Problem #1 is that HomeProd OPNsense is sending out HomeLab WireGuard client traffic on the wrong interface (Sending out the ISP_C instead of ISP_A, even when all 3 ISPs are "up".)
- Problem #2 is that OPNsense does not seem to accept a WireGuard VPN connection from an endpoint that's on the same subnet as the interface accepting it. So, for example, the HomeLab instance, when the ISP_C range is (for example) 70.0.0.5/24, then it cannot accept any connections from 70.0.0.6/24, because it's in the same subnet.
What this means is that connections from clients on the HomeProd LAN to the HomeLab WireGuard instance fail, because they're going out the wrong interface, and that interface is on the same subnet as the HomeLab WAN interface.
However, connections from clients on the HomeLab LAN that hit the HomeProd WAN (ISP_A) interface, work just fine, since it's a completely different ISP (and thus not sharing a subnet).
I further tested this by unplugging the ONTs for both ISP_A and ISP_B, forcing HomeProd to failover to ISP_C as its primary, and thus making both firewalls' only functioning WAN interfaces, on the same subnet. In this case, no client connections in either direction work.
However, if I disable ISP_C on HomeProd, then traffic (correctly) forced out the ISP_A interface, and connections to HomeLab WireGuard work properly (and connections in the other direction continue to work, too, of course).
Caveats:
- when attempting to connect, the corresponding "server" firewall will show the client briefly with a green checkmark, but it only ever gets the first handshake/heartbeat (set to 20 second intervals), and eventually falls off
- I do have one device that had an "established" link (an IoT camera), and it has somehow managed a still-working VPN client connection from HomeProd LAN to HomeLab WireGuard, throughout my experimentation, including a reboot of the HomeProd firewall.
- All connections to either Firewall WireGuard instance made "away" from home work perfectly fine. It's just when trying to go between the two that's problematic at this time.
