Local Static Routes Causing Resets

[ember1205]

I have a number of servers that I moved to a new building. Everything was originally built on IP Subnet A (class C private network) and I kept it all the same.

In the new location, everything has been built as IP Subnet B (also a Class C private network).

I built a Cisco Catalyst 4948 switch and configured VLAN 1 with an IP Address on IP Subnet A. In fact, I set the IP Address to be what the default gateway is for all of the servers (it's a Layer 3 switch).

I created VLAN 1000 with an IP Address on IP Subnet B and I allocated one specific port into that subnet so that I could connect it into the existing network that uses IP Subnet B.

The default gateway on IP Subnet B is an OpnSense firewall. I have created a gateway that points to the IP Address on VLAN 1000 on the switch for IP Subnet A, and I installed the route to use that gateway. I have a Windows workstation on IP Subnet B that I have PuTTY installed on, and I can telnet to the Cisco switch and remain connected indefinitely.

If I attempt to connect THROUGH the switch to any of the systems on IP Subnet A using only my default gateway of the OpnSense firewall, I can make a connection but it will eventually get interrupted (takes 1-2 minutes, happens with all kinds of TCP connections).

If I manually install a route on a system that points directly to the switch as the next hop for IP Subnet A, everything works fine. Not even sure where to start troubleshooting this...

[5SpeedFun]

Are you able to post a diagram?  I think it would be easier to understand.

[ember1205]

Left hand side is what existed prior to moving the systems (Subnet A). Right hand side represents the systems that were added in via the Cisco switch (Subnet B).

The OpnSense firewall has a static route for Subnet B that points to the Cisco as the gateway. All devices on Subnet A point to the OpnSense as their default gateway.

If I manually add a route to a device on Subnet A that points to the Cisco for Subnet B, I can stay connected to Subnet B systems indefinitely. Without the manually added route, traffic first flows to the OpnSense and is then forwarded to the Cisco to reach Subnet B. This works initially, but connections are reset after 1-2 minutes and I have to reconnect.

I don't know if this is because of a lack of ICMP redirect or something entirely different.