help with NAT and port forwarding

[toxic]

Hello, I am probably just misunderstanding how network should work and I would like some help.
I have 2 separate networks living on separate vlans, opnsense is router on both vlan and gateway to internet as it's the only one connected to my ISP box. Lan1 is 10.0.10.0/24 lan1 is 10.0.20.0/24
I run gitlab with a container registry on 10.0.10.10 and have it properly exposed to internet with a fqdn, certificate, and port forwarding.
Now when I pull from this repository I use it's public name which returns my wan address, and it works fine when Ithe computer that pulls is itself on internet, works fine as well if the computer is on lan2 10.0.20.101 for instance.
But it simply times out when I use it from any host on 10.0.10.0/24, except for 10.0.10.10 itself as for him I probably put something in etc hosts to use the local loopback for all registry.gitlab.example.com

My understanding is that on 10.0.20.101 when I pull from registry.gitlab.example.com it sends out packets to my public IP, Port forwarding works fine and opnsense is probably doing NAT automatically. As you can see my understanding is probably incomplete here.
But from 10.0.10.101 when I try to pull from registry.gitlab.example.com, my understanding is that TCP packets go out to my public IP, they are properly send by opnsense to 10.0.10.10 using my Port forwarding as I see them arrive on the 10.0.10.10 server in the gitlab logs. But I guess 10.0.10.101 revives an answer directly from 10.0.10.10 for a TCP session it's trying to open with a public IP so it gets confused and just ignore the answer from 10.0.10.10 wait for an answer from a public IP that never arrives as it was just ignored.

Maybe my understanding is wrong but I welcome any help.

I do have a similar issue for other services that run on the same server on lan2.
I would really like my computers to all use DNS names and use the same names regardless of which network they are connected to, and I would rather have it done with NAT correctly and hopefully avoid having to run a DNS that manages to give different answers to the same query depending on who's asking.

I have tried to add a manual NAT rule on lan2 for my public IPs, it didn't seem to work, maybe because it's not the good solution, maybe because I have several WANs so several public IPs that I keep in a hosts alias (NAT rules should work with aliases, no?)

My gut feeling is really that NAT is the issue but I can't get my head around a solution. I like that no nat is performed between lan2 and lan1 so logs in gitlab show the real IP of the user, but I think within lan1 I need something so both the IP request and answer go through the gateway and not directly between the hosts probably staying on the switch. Or at least find something to not ignore the response;)

Any help is welcome;)
Thanks in advance !m and thanks for reading;)

[FraLem]

Hi there,
Did you check firewall rules on the WAN interface?
Rgds

[toxic]

Thanks for the idea but yes, I see nothing blocking on wan, don't believe anything on wan interface would matter anyway as long as port forwarding is working.
In the fw logs I do see the 10.0.10.101 to my public IP being allowed. I even see another line allowing between 10.0.10.101 and 10.0.10.10.
I also confirmed that http get requests successful reach from 10.0.10.101 to 10.0.10.10 when 10.0.10.101 sends them out to my public IP in http. In https I get time out for the SSL négociation, if I dig I'm pretty sure it's just the response that gets lost or ignored since it's coming from 10.0.10.10 and not from my public IP.

I haven't managed to find in a tcpdump on 10.0.10.101 if it gets the response from 10.0.10.10 or not but I'm guessing it gets through, just ignored or invalid due to origin mismatch.

I believe I need to find a way for opnsense to do NAT even within the lan1 networks if the request were destined to my public IP and are just forwarded to lan. I don't mind that the traffic goes through the router instead of staying on the switch.

[toxic]

Just made some progress, it now works for when 10.0.10.101 contacts my real wan ip, reflection works to 10.0.10.10 and https port forward is functioning.
What I did is have only one port forward that applies to wan+lan1+lan2 interface with reflection on instead of 3 forwards: 1 port forward for each interface (with or without reflection it didn't work)

Now only one thing is failing: it works fine for wan but it doesn't work for the second wan I have which is an openvpn with a dedicated public ipv4
Probably a bug

[toxic]

Noone has a hint as to why nat reflection would work for a carp wan that has a private DHCP IP but not for an OpenVPN wan that has a public static IP ? I would have expected NAT reflection to work smoothly with the static IP my VPN provider assigned to me but somehow it ain't working...

[FraLem]

I would suggest to  check on the Advance configuration of the firewall the option "reply-to on WAN rules".
Rgds
 

[toxic]

Thanks for the idea. The option is actually to disable reply to so I left it disabled as it was.

I think I had a big mess with many NAT Port forwarding disabled but for which I have seen that the associated fw pass rule still got matched... So I did some cleanup, and find I have to do 2 NAT port forward : as my real public IP is not assigned to my wan interface since my ISP box is doing NAT, I have an alias containing my real wan ip and one NAT port forward if dest is my real public IP. But I also need a second port forward where the destination match is "This firewall".

I'm wondering if I could make an alias that contains "This firewall" and a bit more, without manually typing all IPs that this firewall has...

[toxic]

well, turns out I never got it working in fact...
My ISB box is doing it properly for me : when my opnsense sends out traffic to the isp box that is destined for it's public IP, the box NAT's it to some ransom public IP and sends it back to the opnsense aitself (applying the DMZ rule).
When I try to do the port forward with reflection, I got some of it working but some not...

public ip : 82.1.2.3
opnsense : 10.0.10.1 on lan1, 10.0.20.1 on lan2 and 192.168.1.2 on wan
server : 10.0.10.2
client1: 10.0.20.2
client2: 10.0.10.3

client1 calls for 82.1.2.3:443 gets a proper answer from server
server calls for 82.1.2.3:443 gets a proper answser from himself
client2 calls for 82.1.2.3:443 and gets a timeout on trying to open the SSL tunnel

I'm stuck here... I tried portForward with "ThisFirewall", with aliases in which I put all the ips I could think of...
Is it not possible to hve an opnsense behing a NAT and still make port forward work at the opnsense level without having to rely on the ISP box ? (I was planning to change ISP and I know for a fact that the next one has a box that does not do reflection properly)

Any idea would be welcome !