Messages

If you have separate subnets between each switch (L3) and router, remember that you also need to add static routes on the switches.

Yes, I have all of that setup. Your suggestion with the transit network from router to router worked perfectly. The issue has been resolved. Again, I really appreciate the help.

Should the transit network be created on the routers as well?

Any subnet, which is separated from the other devices, is considered as transit network.
But yes, you need to attach it to an separate interface in OPNsense. This can be a VLAN as well, of course.

Say you have
subnet1 (devices1) - Router1 -transit - Router 2 -subnet2 (devices2)
then you may have these subnets
192.168.3.0/24 - 192.168.3.0/24 - 192.168.5.0/24

I don't know, whats the sense of the L3 switches in between. But maybe you have an additional subnets between them, so you have the point the static route to the switch (next hop) instead of the router, because the routers would not be able to see each other.

Thank you. Off to the keyboard to makes some adjustments. I'm sure it'll work as it makes more sense this way. I'll post up though so that anyone else looking around may hopefully find this useful.

You need to separate the devices from the respectively other router.

Logically your network should look like this then:

subnet1 (devices1) - Router1 - Switch1 - Bridge - Switch2 - Router 2 -subnet2 (devices2)

Then you just have to add static routes for the remote subnet to the routers and point them to the respectively next hop.

Can you read my edit on the previous post and let me know if that seems correct before I go reconfiguring things. Thank you very much for taking the time to teach me.

There are L3 switches that are handling routing between the two subnets,

Then you should configure a transit network between each switch and the router, separated from the other devices. Then you can properly route the traffic over the routers.

I have done this, in a sense, but could you let me know more w regards to the transit network. I do have a transit network on the L3 switches, but the routers do not know about this network. It didn't seem pertinent, but my knowledge on this is also quite limited as this is the first time I've done this.

There is a static route from the Router2 to Switch2 to Transit network on Switch1 and then it gets distributed to either the Device1 or Router1.
Going the other way there is a static route from Router1 to Switch1 to Transit network on Switch2 and then gets distributed to either Device2 or Router2.

Should the transit network be created on the routers as well?

edit: thinking about it, I would think it would be created on the opnsense routers and the requests would then be forwarded to the OPNsense interface for that transit network on each side.

If my transit netowork is 192.168.3.0/24

Would I create a VLAN 3 and attach it to an interface and then create a gateway to the switch with that same transit network, using the just created gateway. Then do the same on the other opnsense router? All static routes would basically point to the specific hops utilizing ONLY the transit network?

A network design like this is far from ideal, you might know.
You should consider to really segment your network and put the devices of each subnet behind the respective router.

If you want to stay with this setup anyway, you can masquerade the traffic with an outbound NAT rule on the router to enable communication between the subnets:
On router1 go to Firewall > NAT > Outbound, enable the Hybrid mode.
Then add a rule:
interface: the one facing to router2
source: subent1
dest: subnet2
translation: interface address

Do the same on router2 with exchanged subnets.

I had chosen to do it this way to utilize POE on the switches to power the bridges and also "Location2" gets internet from "Location1" Each location has their own PubIP from the modem through a dedicated VLAN. This prevents double NAT at "Location2".

You cannot meaningfully bridge 2 different subnets. It needs a router between them.

There are L3 switches that are handling routing between the two subnets, but the issue lies with the opnsense autogenerated rule regarding a state violation that denies it.

Correct. Everything, for the purpose of this, is on the same subnet on switch2/router2/device2 and everything on switch1/router1/device1 is on a different subnet

I have two networks (two opnsense routers) functioning on different subnets that are remote to each other and connected via a bridge between two switches.

Traffic goes like this:

Router1 - Switch1 - Bridge - Switch2 - Router 2

This is all local/private networks.

When I try to visit a webgui of a service on the switch 2 network the traffic goes like this:
Device1 - switch1 - Bridge - switch2 - device2 with gui/service (dhcp, gets IP and gateway from router)

I can ping, traceroute etc.... just fine. I get blocked due to what I believe is asymmetric traffic routing because there are two opnsense routers and the switches that are also doing intervlan routing.

If I set the device2 gateway to the switch2 instead of the router then everything works great. This is not very practical to do on all devices.

I have posted a screenshot of the deny in the firewall rules. What kind of rule can I create here in the firewall to allow this through? I do not know how to make this traffic "symmetric", so this is where I'm at with this.

I didn't see how to upload an image, so here is a text based copy/paste found from the details in the firewall live view log.

__timestamp__    2025-02-27T10:39:17-08:00
ack    2161703274
action    [block]
anchorname   
datalen    0
dir    [in]
dst    192.168.5.230
dstport    47816
ecn   
id    0
interface    igb1
interface_name    02_LAN
ipflags    DF
ipversion    4
label    Default deny / state violation rule
length    60
offset    0
protoname    tcp
protonum    6
reason    match
rid    02f4bab031b57d1e30553ce08e0ec131
rulenr    8
seq    369640149
src    192.168.2.4
srcport    80
subrulenr   
tcpflags    SA
tcpopts   
tos    0x0
ttl    64
urp    5792

I am still trying to get the L3 switches working with this, but I'm thinking I may have to tackle this a different way. This is mostly due to my lack of knowledge on exactly what's going on and lack of experience with L3 switches. I have tried quite a few things, but it does not like something about the L3 switch. This could be the modem picking up the mac address from the switch or the bridge that goes to one of the routers etc... and I don't know how to stop that from happening.

Regardless, I'm thinking of utilizing the ER that is right next to the modem to get both pub IPs FIRST, directly from the modem.

I have 4 ports on that edge router. I would use the WAN/LAN for the local network and then use OPT2 Port for a second WAN2 and then use OPT3 to pass the traffic to the remote router. I need a full routing system at the remote location so I can utilize a failover type of internet if the bridge or net goes down at the main location. This is why I have the second router setup at the remote location.

My question here then would be, how can I have the downstream router handle NAT for that network while the local router handles NAT for the immediate network that it's attached to? I want to bypass NAT for WAN2/OPT2 on the main ER right next to the modem.

VLAN 3 must be configured on the local and the remote switch. And it must be tagged on the ports connecting the two switches.

Then, add a dedicated VLAN for connecting the two OPNsenses. So you'll need (at least) three interfaces on each OPNsense: WAN, LAN and the OPNsense-to-OPNsense link.

Whether you configure VLANs on OPNsense itself or use multiple physical ports is up to you. If you have spare interfaces on the OPNsense routers, it's probably easier to configure VLANs on the switches only.

Thank you for the reply. I have been able to create everything within the switch. I can communicate locally over the vlans, however the issue a get is the modem will not assign pub IPs with this configuration for some reason. It'll want to assign an IP in the subnet to access the modem overview page (192.168.100.0/24).

Any ideas on why that could be?

You can use a VLAN to bridge the modem's secondary port to the remote OPNsense's WAN interface. This should be configured on the switches, the local OPNsense doesn't have to be involved in this at all.

For connecting the two OPNsenses, use a separate VLAN.

Cheers
Maurice

This is something I had tried, somewhat. I was able to create VLAN3 on the L3 switch that had two untagged ports (ruckus ICX7150). One connecting the bridge and one to the modem. On the other end of the bridge it went to the WAN. The WAN is now assigned a Pub IP. How can I then get BACK to the local OPNsense using a VLAN over that same WAN connection without utilizing the internet.

Thank you very much for the response.

I feel that I have a somewhat unique setup. 

Anyways, my ISP's modem has to RJ45 ports. Each one will serve me a different pub IP. I have two networks, one is local (same building/site) and the other is about 1/4 mile away and connected to with a bridge. The local network is easy to setup with a PUB ip and the OPNsense ER (4 ports). The second site I have another OPNsense router (4 ports). I have a router at each location as to keep the local networks independently fully functional, aside from internet access, if the bridge or internet goes down.

My issue is, how can I go about getting the "remote network" with a downstream router a public IP so that it is not double NATd and still allow the networks to communicate locally (ie, if the net goes down they can still communicate).

I have L3 switches on each side. I've messed around with those, but I lose communication between the two networks if I pass the pub IP directly to the remote network. I could create static routes to the interface pub IP on the remote router, but if the pub IP changes then that connection is lost.

I thought that I could utilize two ports on the ER for the second pub IP (allowing me to assign a local IP for local routing) and somehow allow the remote router to handle NAT for that second Pub IP and delete/bypass NAT for the second pub IP on the ER. Hopefully that makes sense, but I'm curious to hear any thoughts and/or suggestions.