How to not block this traffic that's being blocked by default deny / state?

[live4soccer7]

I have two networks (two opnsense routers) functioning on different subnets that are remote to each other and connected via a bridge between two switches.

Traffic goes like this:

Router1 - Switch1 - Bridge - Switch2 - Router 2

This is all local/private networks.

When I try to visit a webgui of a service on the switch 2 network the traffic goes like this:
Device1 - switch1 - Bridge - switch2 - device2 with gui/service (dhcp, gets IP and gateway from router)

I can ping, traceroute etc.... just fine. I get blocked due to what I believe is asymmetric traffic routing because there are two opnsense routers and the switches that are also doing intervlan routing.

If I set the device2 gateway to the switch2 instead of the router then everything works great. This is not very practical to do on all devices.

I have posted a screenshot of the deny in the firewall rules. What kind of rule can I create here in the firewall to allow this through? I do not know how to make this traffic "symmetric", so this is where I'm at with this.

I didn't see how to upload an image, so here is a text based copy/paste found from the details in the firewall live view log.

__timestamp__    2025-02-27T10:39:17-08:00
ack    2161703274
action    [block]
anchorname   
datalen    0
dir    [in]
dst    192.168.5.230
dstport    47816
ecn   
id    0
interface    igb1
interface_name    02_LAN
ipflags    DF
ipversion    4
label    Default deny / state violation rule
length    60
offset    0
protoname    tcp
protonum    6
reason    match
rid    02f4bab031b57d1e30553ce08e0ec131
rulenr    8
seq    369640149
src    192.168.2.4
srcport    80
subrulenr   
tcpflags    SA
tcpopts   
tos    0x0
ttl    64
urp    5792

[viragomann]

Traffic goes like this:

Router1 - Switch1 - Bridge - Switch2 - Router 2

This is all local/private networks.

When I try to visit a webgui of a service on the switch 2 network the traffic goes like this:
Device1 - switch1 - Bridge - switch2 - device2 with gui/service (dhcp, gets IP and gateway from router)

So are the device2 and router2 directly connected to the switch2? Means, all within the same subnet.
If so all your devices would reside within the same layer 2?

[live4soccer7]

Correct. Everything, for the purpose of this, is on the same subnet on switch2/router2/device2 and everything on switch1/router1/device1 is on a different subnet

[Patrick M. Hausen]

You cannot meaningfully bridge 2 different subnets. It needs a router between them.

[viragomann]

A network design like this is far from ideal, you might know.
You should consider to really segment your network and put the devices of each subnet behind the respective router.

If you want to stay with this setup anyway, you can masquerade the traffic with an outbound NAT rule on the router to enable communication between the subnets:
On router1 go to Firewall > NAT > Outbound, enable the Hybrid mode.
Then add a rule:
interface: the one facing to router2
source: subent1
dest: subnet2
translation: interface address

Do the same on router2 with exchanged subnets.

[live4soccer7]

You cannot meaningfully bridge 2 different subnets. It needs a router between them.

There are L3 switches that are handling routing between the two subnets, but the issue lies with the opnsense autogenerated rule regarding a state violation that denies it.

[live4soccer7]

A network design like this is far from ideal, you might know.
You should consider to really segment your network and put the devices of each subnet behind the respective router.

If you want to stay with this setup anyway, you can masquerade the traffic with an outbound NAT rule on the router to enable communication between the subnets:
On router1 go to Firewall > NAT > Outbound, enable the Hybrid mode.
Then add a rule:
interface: the one facing to router2
source: subent1
dest: subnet2
translation: interface address

Do the same on router2 with exchanged subnets.

I had chosen to do it this way to utilize POE on the switches to power the bridges and also "Location2" gets internet from "Location1" Each location has their own PubIP from the modem through a dedicated VLAN. This prevents double NAT at "Location2".

[viragomann]

There are L3 switches that are handling routing between the two subnets,

Then you should configure a transit network between each switch and the router, separated from the other devices. Then you can properly route the traffic over the routers.

[live4soccer7]

There are L3 switches that are handling routing between the two subnets,

Then you should configure a transit network between each switch and the router, separated from the other devices. Then you can properly route the traffic over the routers.

I have done this, in a sense, but could you let me know more w regards to the transit network. I do have a transit network on the L3 switches, but the routers do not know about this network. It didn't seem pertinent, but my knowledge on this is also quite limited as this is the first time I've done this.

There is a static route from the Router2 to Switch2 to Transit network on Switch1 and then it gets distributed to either the Device1 or Router1.
Going the other way there is a static route from Router1 to Switch1 to Transit network on Switch2 and then gets distributed to either Device2 or Router2.

Should the transit network be created on the routers as well?

edit: thinking about it, I would think it would be created on the opnsense routers and the requests would then be forwarded to the OPNsense interface for that transit network on each side.

If my transit netowork is 192.168.3.0/24

Would I create a VLAN 3 and attach it to an interface and then create a gateway to the switch with that same transit network, using the just created gateway. Then do the same on the other opnsense router? All static routes would basically point to the specific hops utilizing ONLY the transit network?

[viragomann]

You need to separate the devices from the respectively other router.

Logically your network should look like this then:

subnet1 (devices1) - Router1 - Switch1 - Bridge - Switch2 - Router 2 -subnet2 (devices2)

Then you just have to add static routes for the remote subnet to the routers and point them to the respectively next hop.

[live4soccer7]

You need to separate the devices from the respectively other router.

Logically your network should look like this then:

subnet1 (devices1) - Router1 - Switch1 - Bridge - Switch2 - Router 2 -subnet2 (devices2)

Then you just have to add static routes for the remote subnet to the routers and point them to the respectively next hop.

Can you read my edit on the previous post and let me know if that seems correct before I go reconfiguring things. Thank you very much for taking the time to teach me.

[viragomann]

Should the transit network be created on the routers as well?

Any subnet, which is separated from the other devices, is considered as transit network.
But yes, you need to attach it to an separate interface in OPNsense. This can be a VLAN as well, of course.

Say you have
subnet1 (devices1) - Router1 -transit - Router 2 -subnet2 (devices2)
then you may have these subnets
192.168.3.0/24 - 192.168.3.0/24 - 192.168.5.0/24

I don't know, whats the sense of the L3 switches in between. But maybe you have an additional subnets between them, so you have the point the static route to the switch (next hop) instead of the router, because the routers would not be able to see each other.

[live4soccer7]

Should the transit network be created on the routers as well?

Any subnet, which is separated from the other devices, is considered as transit network.
But yes, you need to attach it to an separate interface in OPNsense. This can be a VLAN as well, of course.

Say you have
subnet1 (devices1) - Router1 -transit - Router 2 -subnet2 (devices2)
then you may have these subnets
192.168.3.0/24 - 192.168.3.0/24 - 192.168.5.0/24

I don't know, whats the sense of the L3 switches in between. But maybe you have an additional subnets between them, so you have the point the static route to the switch (next hop) instead of the router, because the routers would not be able to see each other.

Thank you. Off to the keyboard to makes some adjustments. I'm sure it'll work as it makes more sense this way. I'll post up though so that anyone else looking around may hopefully find this useful.

[viragomann]

If you have separate subnets between each switch (L3) and router, remember that you also need to add static routes on the switches.

[live4soccer7]

If you have separate subnets between each switch (L3) and router, remember that you also need to add static routes on the switches.

Yes, I have all of that setup. Your suggestion with the transit network from router to router worked perfectly. The issue has been resolved. Again, I really appreciate the help.