Please help: Routing issues / DMZ / Double-Firewall scenario

[slowprogress_2751]

Hi all,

I'm dealing with a dual firewall setup and I have strange routing issues, especially in the DMZ (Transfer-Net) between the firewalls. May be you could help me out?

You will also find a network diagram enclosed. OPNSense (Firewall #1) is virtualized on Proxmox. Firewall #2 is a hardware Sophos XGS-Appliance.

I have a GPON Fibre-Modem in Bridge Mode from my ISP. Firewall #1 is connected to the GPON on its WAN Interface ETH1 and establishes the connection over VLAN 7 and PPPoE. Outbound-NAT Rules are in place, inverted to exclude internal traffic.

FW#1 LAN-Port (IP: 172.16.0.1) connects to FW #2 WAN-Port (IP: 172.16.0.254 / GW: 172.16.0.1). Behind Firewall #2 are several internal networks, each in its own subnet and on its own interface as Gateway. No VLANs here and NAT is disabled on Firewall #2 WAN.

To get the traffic back into the internal networks from Firewall #1, I have set static routes for each subnet (like 172.16.1.0/24 to 172.16.0.254).
In the DMZ (or the Transfer Net between the firewalls) are virtual Servers with static IP assignment in the same Subnet as the DMZ (172.16.0.0/24).

Now the weird parts:

- My internal networks can not access the internet until I set another static route 172.16.0.0/24 on FW#1 to 172.16.0.254 (FW#2 WAN). This shouldn't be necessary as these interfaces are in the same subnet.

- Additionally, with this rule set, servers in the DMZ get no Internet anymore if they have FW#1 172.16.0.1 as Gateway (asymmetric routing). It does work though, if I set FW#2 as their Gateway. But with this configuration, the complete traffic gets processed by both firewalls, as it hits both interfaces. That is very unpleasant.

In terms of firewall rules, I allowed all traffic between all subnets for now and all outgoing to the Internet.

I've read tons of topics to similar issues in the past couple of days. But I don't know what I am missing here.

The closest problem description I found is referring to a "reply-to" feature in the advanced firewall rule options. But they used two OPNSense Firewalls and I'm not sure if this is the exact same behavior.

Can you give me a hint?

[viragomann]

Outbound-NAT Rules are in place, inverted to exclude internal traffic.

Can you show, please?

Behind Firewall #2 are several internal networks, each in its own subnet and on its own interface as Gateway. No VLANs here and NAT is disabled on Firewall #2 WAN.

So you've also disabled outbound NAT /masquerading?
Run a packet capture on OPNsense to verify this.

[slowprogress_2751]

Hi, thanks for your reply! I really appreciate your help.
I have made screenshots for the Outbound NAT Config and a NAT Rule in detail:






Then I've started a ping from my workstation 172.16.1.1 to 8.8.8.8 and filtered ICMP for the packet captures. Please find the results enclosed.

Packet Capture from FW#1 LAN:


Packet Capture FW#2 WAN:


Firewall Live Log:


You can see that the traffic arrives without NAT/MASQ from Firewall #2 through the DMZ to Firewall #1.

[EricPerl]

I tried that scenario (double router with NAT disabled on the internal one) a few weeks ago.
https://forum.opnsense.org/index.php?topic=45558.msg228062#msg228062

How exactly did you route traffic back for the internal networks?
I ended up creating a gateway and a manual route to it.

One thing that was a bit different in my case is that I had nothing but the WAN port of the internal router in the edge router LAN, by design.
All hosts are in separate VLANs. It's by design for me because my internal network is a test lab (where I try things when I want to understand something I read in the forums). I believe it's actually good practice, further isolation of your DMZ hosts from the internal network.
I can use FW rules to manage what's allowed across the edge router internal networks...

Reply-to will come into play if your internal networks communicate with the DMZ hosts.
All that traffic will hit the OPN LAN gateway (won't directly target the hosts).
Apart from the extra leg and required rules, this works internal -> DMZ.
But DMZ -> Internal can't be made to work because reply traffic hits the LAN gateway and dies there (state violation due to asym routing).
You have to disable reply-to (more or less granularly) on the INTERNAL router when it's OPN. I don't know about yours.