OPNsense with Layer 3 switch for routing

[the0cdguy]

Does anyone have a working config for using OPNsense with a Layer 3 switch that does all interVLAN routing?

I know similar questions have been asked before, but I didn't find a good answer to my question. I am running OPNsense on a Protectli unit connected to a D-Link 24-port switch.  The Protectli is connected to my cable modem.  OPNsense does all routing, including interVLAN, and my switch runs in layer 2 mode.  All has been well with this setup for several years. I am about to make some big changes and I think I need a dedicated Layer 3 switch.

I have 2 Gbps fiber Internet being installed and I am building a new OPNsense box (2U bare metal, Ryzen 7 7700, 64 MB RAM) and it will have a 2.5 Gbps uplink port and a 2 x 10GbE LAGG connection to my new top-level switch - a Unifi EnterpriseXG 24, which has all 10GbE ports. From there, things will fan out to other Unifi switches, including a 48-port switch.

Here's my challenge: I will have a cluster of GPU compute and storage nodes all connected at 20Gbps, along with other servers and devices connected at 2.5Gbps. The interVLAN traffic is likely to be high - so much that I am assuming that having my switches handle 100% of interVLAN traffic would be best. OPNsense would be the perimeter firewall and do NAT.  I've tried this before with my D-Link switch and ran into issues with DHCP, DNS, and asymmetric routing.  I wasn't smart enough to solve it back then, so I fell back to Layer 2 switching.

Now I want to make another go at it and I am asking for help. Is it possible to use a Layer 3 switch for interVLAN routing and use OPNsense for just perimeter services - ideally with DHCP and DNS remaining on OPNsense?

This post made me think it was possible:  https://hackmd.io/@vintage-computers/rJp5JOxw5

The idea is to still set up all the VLAN interfaces on OPNsense - just not as the .1 address on each subnet. Instead, the .1 address would be an interface on the switch and OPNsense would use that as the gateway. Does this work?

By the way, I tried setting up ISC DHCP and Bind so I wouldn't have to depend on OPNsense for those services, but that combo never worked as smoothly as Unbound and native DHCP on OPNsense.

So does anyone have a working config for using OPNsense with a Layer 3 switch that does all interVLAN routing? Alternately, does anyone use OPNsense for interVLAN routing for high speed applications - like 10, 20, or 25+ Gbps links?  I've always been told that dedicated silicon was better for full line rate switching.

[mimugmail]

The current dhcp doesnt support other pools than connected networks, you need to wait for Kea which is in development.

Then just create a transfer network between switch and FW and set your routes, thats it :)

[the0cdguy]

Thanks. That link I read suggests an approach where OPNsense is connected to the other networks, so DHCP would then work as usual. I'm skeptical.

https://hackmd.io/@vintage-computers/rJp5JOxw5

In opnSense:

- create a VLAN interface for each VLAN.
       - Give it a static address in a subnet that you want on that vlan. This will effectively be the address of the DHCP server, so don't use the .1 address so that can be the default route later (use e.g. 192.168.18.2).
       - Give the interface a gateway that's the .1 address (which will be set up later to be the address of the USW switch)
- set up a DHCP server on that VLAN interface
       - override the gateway the server sends out to be the .1 address on your VLAN subnet.

Does this sound possible?

[mimugmail]

With 2 Gateways in one network you always have trouble with asynchron routing. If you arent a pro I'd not do this

[Patrick M. Hausen]

With 2 Gateways in one network you always have trouble with asynchron routing.

Asymmetric.

[the0cdguy]

With 2 Gateways in one network you always have trouble with asynchron routing. If you arent a pro I'd not do this

Pardon me if I sound slow ???
Why would there be two gateways on one network?  I think that guy said to configure each OPNsense VLAN interface as the .2 and make the gateway the .1 on the Unifi switch.  Is this possible?

I attached a screenshot to show the Interfaces screen where I believe this setting can be made.  On my OPNsense, under Interfaces, I see:

Static IPv4 configuration
   IPv4 address    xxx.xxx.xxx.xxx
   
   IPv4 Upstream Gateway   Auto-detect  +

If I select the + sign, I get additional options to add a gateway.  If I use this, aren't I simply creating a single gateway to use for this interface?  According to that blog post, I would make that gateway the Unifi switch's interface on that VLAN - not on OPNsense.

Am I confused on this?

[mimugmail]

It will work, but OPN wants to send packets to this network via shortest path which is local interface and not layer3 switch. I didnt read the Blog, of course you can work around this, but the solution is prone to errors

[the0cdguy]

It will work, but OPN wants to send packets to this network via shortest path which is local interface and not layer3 switch. I didnt read the Blog, of course you can work around this, but the solution is prone to errors

Okay, thanks. I will go back to trying to figure out how to run a separate DHCP server - either on the switch or standalone.  What I really like about the OPNsense solution is the integration between Unbound for DNS and DHCP. Ideally, I need to find an elegant way to do something similar.

My last experience with ISC and Bind didn't work so well.  I had all kinds of permissions errors, dropped zone transfers, constant service restarts, etc.  It was a nightmare. Still, I'll dig in, educate myself, and try to get a working config up and running.

I'll also keep my eyes open for the Kea implementation when it arrives.

[Zapad]

Yes, i do that.

Switch does intervlan routing with 10Gbe line speed and OPNsense only Wan Traffic to Vlans.
Read my Posts, but my English is Bad. :)

[nope]

Just in case someone is as stupid as me and ends up on this page frustrated for not finding the kea configuration setting for default route or default gateway in the web gui for their layer3 switch or secondary router.

Uncheck the "Auto collect option data" to find all the settings you where frustrated wasn't there:

[bimbar]

You might want to use the switch as DHCP relay.