Difficulty with DMZ traffic

[SuperModerate]

Hey everyone, I've been agonising over this for a couple of days now, hoping someone here can help me out.

I am trying to setup a couple of opnsense firewalls to create a secure internal network and DMZ environment.  The layout would be `Internet` -> `FW1` -> `DMZ` -> `FW2` -> `Internal Network`.

Eventually I would like to get physical devices for these firewalls but for now they are just running as VMs in Proxmox.  Relevant network details:
   - Proxmox host IP `192.168.50.31`
   - FW1 VM (In DMZ) LAN IP: `192.168.51.100`
   - FW2 VM (In Internal Network) LAN IP: `192.168.50.100`
   - FW2 VM WAN IP: `192.168.51.101`
   - Proxmox physical interfaces: `eno1` (1Gbps), `enp5s0` (2.5Gbps)
   - Proxmox virtual interfaces:
      - `vmbr0` (Linux bridge to enp5s0) used for the Internal Network - mapped to FW2 LAN interface
      - `vmbr1` used for the DMZ - mapped to FW2 WAN interface and FW1 LAN interface
      - `vmbr3` - (Linux bridge to eno1) - passed to FW1, unused at the moment but eventually will be connected to the WAN interface to access internet.
   - FW2 WAN interface (`192.168.51.101`) gateway set to FW1 LAN interface (`192.168.51.100`).
   - "Block private networks" unchecked on all interfaces on both FWs (will reenable where relevant once this issue is resolved)
   - NAT disabled on FW2

When I try to ping FW1 (`192.168.51.100`) from a machine on the `192.168.50.0/24` network, the ping manages to get through FW2 and to FW1 but FW1 drops it due to the "Default deny / state violaiton" rule.  I am struggling to determine why this rule is matching, my understanding is that just about everything coming in and then out of the LAN should be allowed by default.

At first I thought that this was something like asymmetric routing causing issues.  To diagnose this I tried to disable packet filtering entirely on FW1 to check packets take the same route back to host pinging the FW.  Unfortunately `traceroute` only shows the first hop (to FW2 LAN interface `192.168.50.100`), after that I just get (`* * *`).  In lieu of this I just checked the routing table and FW1 definitely sends traffic destined for `192.168.50.0/24` to `192.168.51.101` which seems correct.

I should also mention that I am having trouble capturing packets which is making diagnoses a little more difficult.  I'm not sure if I'm just using the tool wrong or what but when I set up a packet capture and send a ping that definitely travels through the interface, nothing shows up in the console.

Apologies for the kinda long winded post, I wanted to get all the details that might be relevant in.  Does anyone have any idea what the cause of this could be or how I can resolve?  Or even how I can take further steps to diagnose because I'm at a bit of a loss.

[bartjsmit]

Any reason why you wouldn't make your DMZ one-armed? I.e. with one firewall having the DMZ on a separate (VLAN) interface.

Routing tends to get a bit messy with two firewalls

[Monviech (Cedrik)]

You can look at dynamic routing protocols to make routing between multiple routers less of a hazzle since it automates it.

E.g. for simple networks using RIP could solve this with the os-frr plugin:

https://docs.opnsense.org/manual/dynamic_routing.html#rip-section
https://docs.opnsense.org/manual/how-tos/dynamic_routing_rip.html#setup-rip-between-routers

[SuperModerate]

Thanks for the replies team!

I was originally turned off VLANs because I was told they could add unnecessary complications but it looks like I may have done that by avoiding them 

I’ll have a look at both VLANs and RIP, this should be enough to get me unblocked, thanks!