Multi WAN incoming configuration

[whitwye]

The goal here is to use Multi WAN in failover configuration for outgoing connections. That is, for connections originating on the LAN, use ISP1 unless it's down, then go to ISP2.

But for incoming connections, we need to handle them on whichever ISP they come in on. We haven't fully experimented with this yet, but already we see a problem. We have ports 443 and 22 enabled for connections from a remote admin IP to "This Firewall" as a Floating Rule. We reach the admin interface fine on port 443 on the IP assigned to ISP1; but on the IP assigned to ISP2 we got the just part of the initial signin screen, and then nothing. SSH works fine to ISP1's IP; to ISP2's IP it works for long enough to log in, but then freezes up within the first minute, in repeated trials.

Is the way the gateway is handled for a failover Multi WAN config incompatible with actively using the IPs of both WANs? We have various services behind the WANs we need to NAT from each of the WANs, and have available if that WAN can be reached, regardless of the failover state for outgoing connections. This part we haven't tested yet. The docs don't seem to talk about this aspect of Multi WAN setup at all. What are the implications?

As the main admin of this, located outside the office we're setting this up for, having the admin interface accessible on both WANs is important in itself. Does this require a load-balancing rather than failover setup to work?

[whitwye]

Ah, looking at the doc I see this is partially covered in "Step 5 - Add allow rule for DNS traffic."

Does this imply that the right way to get what we need here is to assign all of the IPs on each WAN as "Destination" to use the default gateway, rather than the gateway group? I don't see any circumstance in which it even makes sense to try to respond to incoming traffic on one ISP from an IP on the other ISP. At least not for anything we handle.

Looking further into this, the Floating rules letting ports 443 and 22 in on all interfaces from the remote IP are set to use the "default" gateway, which I'm assuming means the default appropriate to each interface, not the gateway group designated for the failover setup. Do I need to put these rules into each interface individually rather than floating them, to have this work for the admin access in this situation?

[whitwye]

I removed the floating rules. In their place I put a rules on each WAN interface allowing access to ports 443 and 22 from the remote IP to the "net" of each WAN inteface, with each rule specifying the gateway appropriate to that interface. Tried this for both the destination of "This Firewall" for both interfaces.

Now neither of the WAN allows port 443 or 22 connections from the remote IP to the admin interfaces.

We've got old Linux firewalls connected to other IPs on these two WANs, which have no such problem. So it's not anything about the second ISP or their router. For that matter a differently configured pfSense installation we tried (and gave up on for other problems) handled this part of it okay. I'm sure there's a way to get there. Just not certain what the OPNsense formula for it is.

After more experimentation, setting the gateway on WAN1 to "default" allows the traffic (now with the IPs of both the remote and local end explicitly specified). But putting the gateway to the name which corresponds to the WAN does not. Due to the order in which things were initially set up (we cabled wrong and the first WAN interface got an initial DHCP assignment from the second ISP's router), that's named "GW_WAN_2", but it's set to the right IP for WAN1. Is there some intermediate level at which this is trying to assign the right IP to the wrong NIC that's tripping this up?

The second WAN is now working with this set to "default" rather than the specific gateway. I notice also that the web interface won't let gateways have their names changed, which I'd like to do for "GW_LAN_2," not least because we need to set up a second system for failover, and I'm aware of how sensitive at least pfSense is to having everything named precisely the same at every level for failover to work right.

[whitwye]

There's one key question I hope can be answered:

Why does setting the gateway for the rule to the IP (via the name associated with that IP) not work here, while setting it to "default" works? On the face of it, shouldn't these be the same, with the same result?

[whitwye]

Further question:

I noticed the LAN gateway got reset to the group-of-two WAN gateways. So I set it back to the LAN gateway. That didn't work for the LAN. But setting it to "default" rather than to the gateway name that has the IP defined works.

Now, I get the general rule that for some reason "default" works while the specific gateway names, in the firewall settings, do not. But I'd like to understand why, since it seems logically these should be equivalent.