Issue to reach Website hosted on internal DMZ

[Madifor]

Since some time i have a 2nd ISP Connection (Fiber) next to my current WAN connection.
The 2nd provider proviced a XGS-Pon Terminal (Fiber in -> 10gbit Coper out) and a Wifi modem/router
Created a gateway-group with ISP2 as Tier 1 and ISP1 as tier 2 , so by default all traffic goes to the internet using the 2nd Wan interface (isp)

The setup is then like the attachment "dual-wan_dual isp modem.png"

From the User i can reach the internet  and the shown web server using the public dns name which point Public-IP1.
To make this work i created the required port forwards on the ISP-1 Modem and on the Wan interface connected to  the ISP1-modem.

All working as i would like to see this

To save some energy and also because it is possible , i am now experimenting with the 2nd setup , where i removed the modem/wifi-router of ISP-2.
Adjusted the Interface configuration on the OpnSense firewall to be able to get a public IP.
So far so good..

The issue i have at this moment that it is not possible to reach the web server using the Public DNS hostname.
From the Internet i can reach the Webserver normally using the same public dns name.

It looks like OPnsense has issues returning the traffic back to PublicIP-2 when it is directly connected to the firewall, while i expect that it still has its NAT table with the port references, so shoule be able to return the traffic to the userip and initial tcp port

Anybody an idea how this can be done ?

[viragomann]

The issue i have at this moment that it is not possible to reach the web server using the Public DNS hostname.

I presume, this only is applied to a hostname pointing to IP2. However, above you only mentioned a hostname on IP1.

In OPNsense a port forwarding rule is defined on a certain interface and is by default only applied to traffic entering this interface.
To also enable the port forwarding rule on the other interface, you need to enable "NAT reflection" in the rule.
Remember that you additionally have to add a firewall rule to allow this traffic if there isn't any yet.

[Madifor]

What I am trying to figure out , why I am unable to reach the website using the public dns entry which points to IPS1-WAN IP when (default) traffic is routed to the internet using ISP2. When I enter for example webserver.mydoain as url in the browser, on the firewall I see that the request received on Wan1 interfac. The port forwarding is also happening (initial it goes to a (reverse-)web proxy and from their traffic reaches the correct webserver (based on the URL). My suspision is that it is caused by the fact that WANIPof ISP2 connection is on one of the other interfaces and then gets lost... I have trouble in this part of trouleshooting.. the packet capture feature doesn't give me a direction where to find the solution so. Hopefully some one does know which mistakes I make in my thinking / troubleshooting process.

So just to be clear when I restore the original setup where I also use the provided isp device to connect to the GXS-pon terminal(media converter)

[Madifor]

Just an update on the troubleshooting what happens , as mentioned 'Wan1' is behind the isp modem before it gets to the internet.
When i from the lan network (goes to the internet over WAN 2) tries to open the test website/ i now created on the PublicIp address of Wan 1 , on the firewall log i see the redirect rule is hit , then i see the accept rule ,  but funny enough on the pacture capture running on the test server , dont see any packet arriving.
Do i the same using a host which resides somewhere on the internet i see the same behaviour in the firewall log , so redirect rule and accept rule are hit/ used, with the difference on the test server i actually see the packet arrive and test page opens.

Doing some more trouble shooting, when i try to access the WAN1 on the interface IP directly , i only see the rdr rule kicked in, not the accept rule, but the page is loaded normally.Hopefully this helps to get to the root cause of the found issue.

[viragomann]

What I am trying to figure out , why I am unable to reach the website using the public dns entry which points to IPS1-WAN IP when (default) traffic is routed to the internet using ISP2. When I enter for example webserver.mydoain as url in the browser, on the firewall I see that the request received on Wan1 interfac.

Are you now talking from access the website from inside your network or from outside?

Before you said that you have issues accessing the server from LAN, but if so, I'm not expecting to see the packets on the WAN interface.

[Madifor]

Dispite all the advice given , and also a post regarding this issue, and possible solution (https://forum.opnsense.org/index.php?topic=42613.0) for it is not working (yet), so might am missing a small configuration failure.

Also an other difference compared to the other post is that i have asked the ISP (WAN 1) to set their Device In bridge mode , so also now for WAN-1 the Public IP is now also directly on its respective interface.
During the troubleshooting i also now testing only from a host based on the internet (a remote Pc i have access to).

Attached a packet captures what happens when i try to open the test page on Wan-IP1 (Ziggo).

Unfortunately the packet capture functionality Doesnt mix the Traffic on the selected interfaces but showed 1st the traffic on 1 interface and below that for the other interface where traffic is recorded so i have to combine them myself and placed them side by side for a better comparison what is happening.

[Madifor]

This is the latest packet capture , selecting Both WAN connections and the DMZ where the web site is hosted. As you can see in the marked area that the Reply from the server with 404 is never send back over any WAN connection it gets dropped. Under Advanced firewall settings, i adjusted selected 'use sticky connections with a 'time-out' of 5 seconds to make sure that the firewal knows which Gateway to use when returning the traffic.

The output is from the internal Packet Capture feature of OpnSense and i placed all packets in order based on the time stamps.

There you see that the message with HTTP response 404 is NOT send back to any wan interface but get stuck somewhere in the firewall , or get dropped ?

Just to be complete in with the information This WAN (ziggo) connection is configured as Secondary gateway for all traffic .
The Primary Gateway is on WAN-2.
Both Gateways are configured in a gateway group where WAN-2 has higher priority/ preference.

During the capture i wan to note that the primary wan interface was included/ selected also just to be sure that the missed packets are not send back over the wrong interface ..

All works fine when i manually shutdown the Primary Gateway (on Wan 2). Then on the remote host, i get the desired web page ..

Anybody an idea why this is happening

[viragomann]

As you can see in the marked area that the Reply from the server with 404 is never send back over any WAN connection it gets dropped.

Did you also capture the WAN2 traffic to ensure that it isn't sent out on the wrong WAN?