I think I have discovered a bug with NAT and firewall (solved)

[EricPerl]

FWIW, I did all my testing in a lab environment (on my backup Proxmox/OPN within my private network).
I did NOT have to disable force gateway.
BUT the WAN side of my backup OPN (access port on the switch) is in a VLAN on my main OPN, and I ran my tests from a machine in another VLAN.
I believe that explains why I didn't need this setting.

This said, this force gateway setting is to me the most confusing of the auto generated FW rules.
I'm vaguely familiar with policy-based routing from reading the docs on multi-WAN.
The instructions modify the default LAN allow all rule (specifying the GW group) and suggest overriding it for local DNS (not specifying the GW). Makes sense.
These are first match rules for IN traffic on LAN.

But my force GW rule looks like this (WAN assigned to vtnet0):
out, last match, IPv4+6 *, source = (vtnet0), *, dest = !WAN net, *, GW = WAN-DHCP, *, *
And it comes AFTER the source = *, dest = *, GW = * FW rule you can't disable by a setting... I have no clue how the ForceGW rule ever fires (more specific?).

I see the Force GW rule firing for outbound traffic off of the FW itself (e.g. NTP requests). Expected.
Also for traffic initiated from LAN towards the !WAN (e.g. Internet or other VLANs on main OPN). Source = (vtnet0)???
In other words, I don't need Disable Force GW set to ping hosts from LAN on the same subnet as the WAN. The default FW out rule is used.
ICMP 192.168.1.100 -> 10.3.3.3 works (10.3.3.1 is the WAN GW, 10.3.3.2 is WAN IP).

On the other hand, I plain fail to get my simple 80 port forward test to work when initiated from 10.3.3.3 to 10.3.3.2 (PF to 192.168.1.100).
The backup OPN is all blue/green. I see the packet back out in a capture BUT THE MAC IS THE MAC OF THE LAN ADAPTER of my main OPN (also used by VLAN).
The packet is sent to the VLAN GW where it dies because it's a clear state violation (the request never went through the main OPN).

Disable Force GW is not helping (not totally surprising because it's reply traffic that should be sent where it came from).
I've toggled the anti-lockout rule. No change. I've removed the 80->443 redirect too so the anti-lockout rule doesn't even affect that port.

I'm done for today... I might restart from scratch tomorrow.

[chall88]

Yeah if you did it from another vlan it would have had to go through the gateway for inter-vlan routing to workso there's your extra hop. I was coming directly at it from an access port in the same vlan and ip space as the wan interface.

>On the other hand, I plain fail to get my simple 80 port forward test to work when initiated from 10.3.3.3 to 10.3.3.2 (PF to 192.168.1.100).

This is what i was seeing when not source from a network beyond the WAN subnet. I think the ACKS for the SYNs are ending up in the wrong place the gateway has no idea why it's recieving them.

>The backup OPN is all blue/green.

This was also what I was seeing in the firewall logs, which was even more confusing when I would turn the firewall completely off and it would start working

 
I know this setting is trying to be helpful and can make sense for most ISPS, but it shouldnt need to be forced either if the system routing table is sane.. It seems like a sledgehammer approach to a non-issue, unless im just ignorant of it's use case. Maybe it's this mechanism that toggles between wans for multiwan.  'automagical intransparent' features as he called them make it very difficult to try to sort things out when you're coming at the problem in a bottom up principles first fashion because things are interfering in ways you dont know. Maybe a disable unless multi-wan active would be appropriate here


The way I understand it is basically every egress on WAN is redirected at that forced gateway, which would normally just be your isp's gateway on your ip block. Which is fine, because thats the way things came in anyways, stateful firewalls will know how to deal with it.  If you treat it like any other interface and any other network, attempting to lab it in the most straightforward way possible before you put it on the real wan..... you'll run face first into what i ran into

[EricPerl]

I understand this may not be an issue at the edge of a public network (how likely are you communicate with a system in the WAN subnet, probably all peer customers).
I haven't tried yet but I might be able to convince my main gateway to "forward" that packet to the intended target (I once mistyped an interface IP with a /32 address and the effect was that all traffic was sent to the gateway) but given this is a reply, I'm not sure it will work.

Regardless, that ForceGW rule is: out, last match, IPv4+6 *, source = (vtnet0), *, dest = !WAN net, *, GW = WAN-DHCP, *, *
The destination is !WAN net. It does not apply here...

Anyway, I ended up doing something fun today so further networking experiments will wait until another day...
I might start a new topic about that rule.