strange behaviour with dynamically assigned WAN Gateway

[psuter]

I am really at a loss here:
i am setting up a new opnsense firewall (vesion 21.7.1) for someone else. To prepare everything i have attached the WAN port of the new firewall to my own private Lan network. I have set the WAN interface to request an IP by DHCP which it does.

here is a quick network overview:

my home network: 172.16.51.0/24, Gateway 172.16.51.254 (pfsnese)
new opnsense WAN interface: 172.16.51.128/24 assigned via DHCP, Gateway 172.16.51.254 assigned via DHCP
new opnsense LAN interface: 192.168.30.1/24
Qnap behind new opnsense: 192.168.30.10

clients in the 192.168.30.0 lan can access both my 172.16.51.0 network and the internet without problems.

I have set up a port forwarding for port 8443 on the WAN interface to be forwarded to 192.168.30.10:443 and i have expected to be able to then open the QNAP web-interface from my private lan via https://172.16.51.128:8443 but that does not work.

tcpdump on the opnsense router shows me, that it is getting my requests and it it sending replies to my pc in my own private lan (172.16.51.1) but on my pc tcpdump only shows my requests but no answers.

the same happens with a WAN firewall rule to allow traffic on port 443 to the opnsense web gui via the WAN Port. I simply won't get any answer back to my pc.

after hours of debugging and trying, i finally found out, that it all works perfectly fine, as soon as i hardcode the WAN Gateway IP address!
So i simply go to System -> Gateways -> single and click on the pencil to edit the WAN_DHCP gateway settings. I check the upstream Gateway checkbox and for IP Address, where it currently sais "dynamic" i enter 172.16.51.254, thats the same IP that i got from the DHCP anyway. I then click save and apply the settings.
Magically, all my port forwardings, firewall rules etc. works and i can access bot the opnsense web interface as well as the QNAP web interface from my 172.16.51 network!

i have also tried to just select the upstream gateway checkbox but leave the dynamic ip in there, which did not help, so only hardcoding the IP helps.

in the attachments you can see the non working and the working gateway list entry. as you can see, there is no difference at all in the list, only if you click on edit you can see that the non-working shows "dynamic" and the other one has the IP hardcoded.

can anybody explain to me what I'm missing here? I really don't understand this behavior and it did drive me nuts today! eager to learn why all that is so. Above all i can't understand that this setting as that gateway isn't even involved in the connection!

btw: of course i have disabled the "block private networks" and "block bogon networks" settings for my wan interface. disabling or enabling reply-to wan rules aso did not change anything, i just mention these settings as i found them mentioned in about every thread i found rearding port forwarding issues with a private IP on the wan side

cheers
Pascal

[psuter]

well it turns out that by modifying the dynamic gateway I simply killed it so that it was no longer working, so all my machines in the LAN behind the opnsense did no longer have internet access..

it also turns out that my problem can be summarized like so: port forwardings on opnsense don't work from the WAN subnet. They do work as soon as a request comes from another network on the wan-side which needs to be routed through the wan side gateway. but as soon as the clinet connecting to a NAT-ed port is on the same subnet as the WAN interface of the opnsense firewall it won't work because opnsense seems to send all outgoing traffic on the WAN interface directly to the standard gateway, even if it is in the same subnet which should not happen.

by disabling the reply-to wan rules this behavior is fixed and all works fine..
i just don't understand how reply-to and this problem is connected.. to me this looks like a bug in how reply-to is implemented.. or the description of this option in the GUI is misleading.

cheers
Pascal

[psuter]

after considering to file a bug report I found that this has been filed and discuessed with no intent on the opnsense side to fix it in the near future because there aren't enough support tickets realting to this issue.

https://github.com/opnsense/core/issues/3952

sadly this won't be fixed (any time soon)