NAT working only on one of two subnet

[VA2XJM]

Hello everyone,

Please keep in mind this setup is not the usual commercial networking

OPNsense 22.1.8_1-amd64
FreeBSD 13.0-STABLE
OpenSSL 1.1.1o 3 May 2022

We replaced our old SonicWall device with an OpnSense one. We operate an amateur radio mesh network with our main server needing Internet access but also allows access to services from Internet if available. We also need to be able to access the firewall from the mesh. To achieve this, we were using the interfaces EM0,1,2 as in the included schematic.

EM1 is a WAN interface that obtain a 10.0.0.0 IP from the mesh to allow management access from anywhere on the mesh.

Our main problem was that the initial setup was prone to DOS due to the limited capacity of the device, so we wish to make a "bypass" from the gateway to the server. We added EM3 so Internet traffic is forwarded from the firewall directly to the server without clogging the mesh gateway.

In the firewall when we change a NAT rule to forward to the new 192.168.124.0/24 LAN subnet allow direct access to the server instead of the 192.168.123.0/24 going toward the mesh gateway.

As soon as the rules are modified to point to the 192.168.124.0 subnet, they stop working. We took the time to compare everything from the two LAN interfaces and we cannot find WHY it is not working on one of the two subnet.

Anyone have an idea or a solution ?

Thanks

[VA2XJM]

Oh, I forgot.

From the firewall, it is possible to ping the server and make curl requests to confirm that connections can be established.

From captures, we can see requests been made on the WAN interface without returns and nothing on the LAN (EM3) interface.

[meyergru]

Two things come to mind:

1. You have not shown the outbound NAT rule(s) which could be at fault.
2. Maybe it is a routing problem, e.g. the server must have had a default gateway before it got connected to em3?

[VA2XJM]

As you can see below (attached), this is a pretty simple NAT rule that do not works, but if the NAT target is changed, it works perfectly. The filtering rules has been created as "Add associated filter rule". Even if set to "Pass", it is not working.

The two interfaces on the server are set based on the DHCP settings received, nothing is static. From the firewall it is possible to CURL the server and get returns.