routing issue

[chicken]

Hi, I am working on building a new server for my personal colo use running TrueNAS Scale with OPNsense as the firewall for the various VMs.  The OPNsense instance has a PCIe Mapped NIC to it for its outside interface and then two bridge interfaces setup as VirtIO adapter types.  Everything installed fine and a VM sitting on the inside interface (bridge 100) can get to the internet fine.  The challenge I am having is getting to any of the VMs sitting behind the firewall from my desktop.

Currently the server is at my house while I am getting it built.  My house has a cable modem and then a Protectli vault running OPNsense for my home use.  Lets call the inside network behind that firewall 10.1.1.0/24.  The IP for my internet firewall is 10.1.1.1 and my pc is 10.1.1.10 and the virtual OPNsense for my colo server is temporarily on 10.1.1.20 for its outside interface.  If I use a VM sitting on the inside behind the virtual OPNsense on ip 10.10.1.10, the virtual firewall NATs it correctly to its outside interface, 10.1.1.20 and forwards it to my internet firewall at 10.1.1.1 and it NATs it again to its public IP and goes out to the internet.  Everything works great there.

The issue I have is if I try to hit a NAT translation/port forwarding on virtual OPNsense for ssh that I have forwarded to the 10.10.1.10 device from my PC at 10.1.1.10, I see in the logs it translates it fine and the server responds but instead of the virtual OPNsense forwarding the traffic directly back to my PC, it forwards it to its gateway which is my internet OPNsense firewall and it drops the traffic.  I suspect there is a setting in OPNsense where it forwards all traffic to its gateway even though the target is on the same LAN segment.  I looked around and don't see a setting forcing it to send traffic to the gateway vs directly to the PC.  I checked the arp tables on my PC and on the virtual OPNsense and they are correct so its not like the other OPNsense is proxy arping my IP.  Any pointers on trying to fix this without making changes to my cable modem firewall?  Thanks

[chicken]

Forgot to mention, both firewalls are on 24.1.2.  Thanks

[chicken]

Updated to show a drawing that hopefully explains things better.  Thanks

[zan]

I suspect there is a setting in OPNsense where it forwards all traffic to its gateway even though the target is on the same LAN segment.  I looked around and don't see a setting forcing it to send traffic to the gateway vs directly to the PC.

Try disabling "reply-to on wan" on your virtual firewall.

[newsense]

Disable the reply-to only on the individual rules on WAN for https and ssh.

And you'll need to disable the block of private networks on the WAN interface as well, else it will take precedence over your WAN rules.