10GbE Networking, Subnets, and performance without Layer 3 switching?

[eptesicus]

I have my OPNsense firewall on an ASUS RS200 with an i3 CPU, 4 GbE NICs, and 2x 10GbE NICs that will be installed this weekend.

My two switches are Ubiquiti Unifi US-16-XG and US-24-250. I have a SAN, two NAS', and 4 ESXi hosts. Each of these is connected to the 16-XG for 10GbE and then a 1GbE on the US-24 for failover.

I'm currently working on segregating my network so that it's no longer a flat 10.0.0.0/20, but has 10 different subnets (management, storage, desktops, backup, downloads, web/dmz, remote access, guest wifi, wife's devices, and iot). My problem is that now every subnet is having it's traffic routed through the firewall's 1GbE LAN port. 10 GbE will be installed this weekend, but do I have a way without buying all new Layer 3 switches to keep traffic from completely saturating the firewall? Someone had mentioned running some VMs on my hosts to act as the gateways, but I want to see if there's a better solution before I pursue that or just leave the traffic to route through the firewall once I install 10GbE.

Thanks!

[bartjsmit]

I would agree; the traffic between VLAN's will need to go back and forth to the firewall if that is where you are routing the subnets. This limits the effective throughput to half a gigabit.

Your other alternative is to LAG as many 1Gbps ports on the firewall to cover your total traffic requirement.

Bart...

[eptesicus]

Bart - Thanks for the input. I got an Intel/Dell x540 dual-10GbE NIC, so I could LAG those two ports together and have 20GbE to and from the 10GbE core switch.

Maybe this year I'll replace the Unifi switches with the ES series to get layer 3.

[eptesicus]

So am I then setting these firewall rules up then? Should all rules go out the WAN gateway? I set the Guest, Wife's, and IOT subnets to have the gateways for the first and last rules set to WAN. Does this need to be set this way for all vlans? I had to do it this way because traffic was somehow finding its way to one of my VPN gateways, but wasn't making it out so traffic halted.