Two pub IPs one at ER and one at downstream router, avoiding double NAT

[live4soccer7]

I feel that I have a somewhat unique setup. 

Anyways, my ISP's modem has to RJ45 ports. Each one will serve me a different pub IP. I have two networks, one is local (same building/site) and the other is about 1/4 mile away and connected to with a bridge. The local network is easy to setup with a PUB ip and the OPNsense ER (4 ports). The second site I have another OPNsense router (4 ports). I have a router at each location as to keep the local networks independently fully functional, aside from internet access, if the bridge or internet goes down.

My issue is, how can I go about getting the "remote network" with a downstream router a public IP so that it is not double NATd and still allow the networks to communicate locally (ie, if the net goes down they can still communicate).

I have L3 switches on each side. I've messed around with those, but I lose communication between the two networks if I pass the pub IP directly to the remote network. I could create static routes to the interface pub IP on the remote router, but if the pub IP changes then that connection is lost.

I thought that I could utilize two ports on the ER for the second pub IP (allowing me to assign a local IP for local routing) and somehow allow the remote router to handle NAT for that second Pub IP and delete/bypass NAT for the second pub IP on the ER. Hopefully that makes sense, but I'm curious to hear any thoughts and/or suggestions.

[Maurice]

You can use a VLAN to bridge the modem's secondary port to the remote OPNsense's WAN interface. This should be configured on the switches, the local OPNsense doesn't have to be involved in this at all.

For connecting the two OPNsenses, use a separate VLAN.

Cheers
Maurice

[live4soccer7]

You can use a VLAN to bridge the modem's secondary port to the remote OPNsense's WAN interface. This should be configured on the switches, the local OPNsense doesn't have to be involved in this at all.

For connecting the two OPNsenses, use a separate VLAN.

Cheers
Maurice

This is something I had tried, somewhat. I was able to create VLAN3 on the L3 switch that had two untagged ports (ruckus ICX7150). One connecting the bridge and one to the modem. On the other end of the bridge it went to the WAN. The WAN is now assigned a Pub IP. How can I then get BACK to the local OPNsense using a VLAN over that same WAN connection without utilizing the internet.

Thank you very much for the response.

[Maurice]

VLAN 3 must be configured on the local and the remote switch. And it must be tagged on the ports connecting the two switches.

Then, add a dedicated VLAN for connecting the two OPNsenses. So you'll need (at least) three interfaces on each OPNsense: WAN, LAN and the OPNsense-to-OPNsense link.

Whether you configure VLANs on OPNsense itself or use multiple physical ports is up to you. If you have spare interfaces on the OPNsense routers, it's probably easier to configure VLANs on the switches only.

[live4soccer7]

VLAN 3 must be configured on the local and the remote switch. And it must be tagged on the ports connecting the two switches.

Then, add a dedicated VLAN for connecting the two OPNsenses. So you'll need (at least) three interfaces on each OPNsense: WAN, LAN and the OPNsense-to-OPNsense link.

Whether you configure VLANs on OPNsense itself or use multiple physical ports is up to you. If you have spare interfaces on the OPNsense routers, it's probably easier to configure VLANs on the switches only.

Thank you for the reply. I have been able to create everything within the switch. I can communicate locally over the vlans, however the issue a get is the modem will not assign pub IPs with this configuration for some reason. It'll want to assign an IP in the subnet to access the modem overview page (192.168.100.0/24).

Any ideas on why that could be?

[live4soccer7]

I am still trying to get the L3 switches working with this, but I'm thinking I may have to tackle this a different way. This is mostly due to my lack of knowledge on exactly what's going on and lack of experience with L3 switches. I have tried quite a few things, but it does not like something about the L3 switch. This could be the modem picking up the mac address from the switch or the bridge that goes to one of the routers etc... and I don't know how to stop that from happening.

Regardless, I'm thinking of utilizing the ER that is right next to the modem to get both pub IPs FIRST, directly from the modem.

I have 4 ports on that edge router. I would use the WAN/LAN for the local network and then use OPT2 Port for a second WAN2 and then use OPT3 to pass the traffic to the remote router. I need a full routing system at the remote location so I can utilize a failover type of internet if the bridge or net goes down at the main location. This is why I have the second router setup at the remote location.

My question here then would be, how can I have the downstream router handle NAT for that network while the local router handles NAT for the immediate network that it's attached to? I want to bypass NAT for WAN2/OPT2 on the main ER right next to the modem.