Ping-Pong between L3 switch and OPNsense

[HarryDasBrot]

Hello Everyone,

I am still new to OPNsense and advanced Routers/Firewalls. I can get OPNsense working and have internet access but once I integrated into my Network, I can't get internet to work. I am missing some config on the OPNsense.

My set-up:

Topology:
ISP - OPNSense - L3 Switch - LAN devices (multiple VLANs)

L3 Switch IP: 172.16.10.1
OPNsense IP: 172.16.10.6

Problem: I have internet access in OPNsense but not in L3 switch and LAN devices.

Config:
- L3 acts as DHCP server and default gateway for all LAN devices is 172.16.10.1
- OPNsense LAN receives IP and default gateway from OPNsense

Troubleshooting:

Traceroute from LAN device:


Traceroute from L3 Switch:


L3 Switch Routes:


OPNsense Routes:

(no static route for 172.16.10.0/24 network because the comment at the bottom says that "Do not enter static routes for networks assigned on any interface of this firewall")

OPNsense Gateways:


OPNsense Interfaces:


What can I do?

[Monviech (Cedrik)]

Well think about it, if both routers have each other as their default route they will create a routing loop.

Dont let the Opnsense get DHCP on the LAN interface, configure it static and dont set a gateway on LAN.

If you want to keep your L3 router in place, use static routes. Would be best to just use it as a normal switch though.

[HarryDasBrot]

Well think about it, if both routers have each other as their default route they will create a routing loop.

Dont let the Opnsense get DHCP on the LAN interface, configure it static and dont set a gateway on LAN.

If you want to keep your L3 router in place, use static routes. Would be best to just use it as a normal switch though.

I get the suggestion but my challenge is that I am using Wireguard and need to access other VLANs. My PC is connected to OPNsense via Wireguard and without the gateway configured, I cannot reach other VLANs in the LAN network behind the L3 switch. If there is a solution for this, I would appreciate your input.

[Monviech (Cedrik)]

Using two routers complicates things. There is no reason for the switch to do the routing in your simple network.

Configure the VLANs on the OPNsense and create a trunk to your switch.

https://docs.opnsense.org/manual/how-tos/vlan_and_lagg.html

[Patrick M. Hausen]

I get the suggestion but my challenge is that I am using Wireguard and need to access other VLANs. My PC is connected to OPNsense via Wireguard and without the gateway configured, I cannot reach other VLANs in the LAN network behind the L3 switch. If there is a solution for this, I would appreciate your input.

Instead of configuring the gateway on the LAN interface add the gateway at System > Gateways > Configuration and then at System > Routes > Configuration add routes only for the networks that are "behind" your L3 switch.

The default route of that L3 switch should point to OPNsense.

[HarryDasBrot]

I get the suggestion but my challenge is that I am using Wireguard and need to access other VLANs. My PC is connected to OPNsense via Wireguard and without the gateway configured, I cannot reach other VLANs in the LAN network behind the L3 switch. If there is a solution for this, I would appreciate your input.

Instead of configuring the gateway on the LAN interface add the gateway at System > Gateways > Configuration and then at System > Routes > Configuration add routes only for the networks that are "behind" your L3 switch.

The default route of that L3 switch should point to OPNsense.

I have not configured manually any gateways for my LAN interface. OPNsense received the gateway from my L3 switch like all other devices. Does it mean that I cannot use DHCP for the OPNsense LAN interface?

In the screenshot i posted above with the OPNsense gateways, I have WAN gateway set with a higher priority. Should that not take care of traffic to be routed first to WAN instead of LAN?

The routes I added in OPNsense are only for the other VLANs that are only accessabile through the L3 switch.

In the L3 switch I have already configured the default route to point at OPNsense LAN address.

[Patrick M. Hausen]

I have not configured manually any gateways for my LAN interface. OPNsense received the gateway from my L3 switch like all other devices. Does it mean that I cannot use DHCP for the OPNsense LAN interface?

No, you cannot use DHCP on a system that is itself a router except on a single WAN link.

[EricPerl]

Wouldn't all this "just work" if the VLANs were handled by OPN and the switch only dealt with tagging/untagging?

[Patrick M. Hausen]

Wouldn't all this "just work" if the VLANs were handled by OPN and the switch only dealt with tagging/untagging?

It also "just works" if you "just" add static routes. Never use DHCP on internal interfaces - execept as a server, of course.