Multihomed bridge

[giuliopaci]

Hi all,
  I am new to opnsense and I am trying to configure a small network with it.

The ISP CPE modem is multihomed (public network X, with X gateway, public network Y, with Y gateway, private network Z, with Z gateway).

I have a opnsense router that I phisically connected to the ISP CPE modem using an interface and to an internal switch using another interface. The ISP CPE modem is connected to internet and X and Y network are configured to be accessible from outside.

I would like to allow devices on VLAN 1000, attached to the switch, to be able to use addresses from Y network and be accessible from outside and use an address from X network for the router. I also want to use port-forwarding on X router address.

I decided to create a bridge between VLAN 1000 and ISP CPE modem, and assign WAN to it.

So:

• I created PUBLIC (VLAN 1000, vlan0.1000) and MODEM (ISP CPE modem, igc0) interfaces
• I created bridge0 bridge on top of PUBLIC and MODEM (I also disabled pfil_member and enabled pfil_bridge and set outbound NAT mode to hybrid)
• I created WLAN (bridge0), I assigned IP from X and defined X gateway
• I assigned some devices attached to VLAN 1000 IPs from Y and defined Y gateway as default
• I created LEGACY (VLAN 100, vlan0.100) and assigned private network to it, with the opnsense router acting as a gateway
• I defined some port-forwarding rules (with reflection), mapping some services from LEGACY network

Current situation is:
- I can access port-forwarded services and Y devices from internet;
- I can access port-forwarded services from LEGACY;
- I can access Y devices from the router itself;
- I cannot access Y devices from LEGACY: packets get SNATed to the X router address, but they do not reach the device;
- I cannot access port-forwarded services from Y devices: packes get DNATed to the internal address of the services, but they do not reach the service.

I spent several hours trying to understand/solve the issue, but I was not able to find any solution yet.

What can be the issue?

[pfry]

That's a rather unusual architecture. Three L3 services with no L2 partitioning? Yak!

For OPNsense -> Y, I assume that's a fixup on the modem side, but I'd have to look at the equipment to be sure.

For Legacy -> Y, it's probably a return-path issue (below).

For Y -> VLAN 100, I would assume that Y devices do not have specific routes to the VLAN 100 public NAT IPs, so packets are DNAT'd and sent to the Y gateway, which drops them (at some point). You could confirm this from packet capture (look at the destination MAC).

Ideally you could just configure a couple (additional) VRFs on the firewall and convince it that your WAN interface belongs to all three (a bit of a challenge, even on a real router). I can think of a few lousy solutions...

e.g. You could try assigning a virtual IP from Y to the firewall (bridge) and adding a static route to your Y test machine with the destination of the public NAT block and gateway of said virtual IP.

I figure I'll see if someone else here has a better solution. (Or if I'm simply dead wrong, dead.)