OPNsense Forum

English Forums => General Discussion => Topic started by: mtlynch on March 26, 2023, 09:48:43 PM

Title: Host on VLAN A can't ping VLAN B within same router
Post by: mtlynch on March 26, 2023, 09:48:43 PM
I'm having some trouble with inter-VLAN routing within the same OPNsense system.

I've set up VLANs for the first time on my OPNsense box (running opnsense-business, 22.10.2, amd64).

I have two VLANs on the OPNsense system, both with the same parent (igb1):

VLAN A: 10.0.10.1/24
VLAN B: 10.0.20.1/24

I have two hosts:

Host A1: 10.0.10.101 (on VLAN A)
Host B1: 10.0.20.103 (on VLAN B)

I want to allow a host on VLAN A to send traffic to hosts on VLAN B. More specifically, I want hosts on VLAN A to be able to make HTTP requests to port 1313 on a specific host on VLAN B, but I'm having trouble getting even a simple ping to work.

I tried adding a firewall rule to VLAN A:

Action: Pass
Interface: VLAN A
Direction: In
TCP/IP Version: IPv4
Protocol: any
Source: VLAN A net
Destination: VLAN B net

But pings from Host A1 to Host B1 fail. I tried setting up the inverse rule on VLAN B's firewall, but I get the same result. From my direct LAN, I can ping both A1 and B1 successfully.

Interestingly, if I check for ICMP messages in Firewall > Log Files > Live View, I see this:

wan 2023-03-26T15:42:02-04:00 [my public IP, redacted] 10.0.20.103 icmp let out anything from firewall host itself (force gw)

And if I traceroute from A1 to B1, I see that the packets seem to go to the gateway on VLAN A out to the WAN interface:

$ sudo traceroute 10.0.20.103
traceroute to 10.0.20.103 (10.0.20.103), 30 hops max, 60 byte packets
1  10.0.10.1 (10.0.10.1)  0.213 ms  0.166 ms  0.130 ms
2  [redacted] ([my ISP gateway].1)  14.033 ms  14.010 ms  13.952 ms
3  [redacted] ([my public IP])  2.603 ms  2.470 ms  2.665 ms


Somehow, packets from VLAN A to VLAN B are being routed from VLAN A's gateway to OPNsense's WAN interface instead of to VLAN B on the same physical interface. I haven't added any routes to OPNsense beyond the defaults.

Does anyone have suggestions about what I'm configuring incorrectly?
Title: Re: Host on VLAN A can't ping VLAN B within same router
Post by: Greelan on March 26, 2023, 10:13:49 PM
What upstream gateway is set on the VLAN interface assignments?
Title: Re: Host on VLAN A can't ping VLAN B within same router
Post by: mtlynch on March 26, 2023, 10:26:42 PM
QuoteWhat upstream gateway is set on the VLAN interface assignments?

They are both set to "Auto-detect".
Title: Re: Host on VLAN A can't ping VLAN B within same router
Post by: JeGr on March 28, 2023, 12:16:13 PM
Could you please show both interface configurations? They are using different VLAN IDs, correct?
Could you add screens of your ruleset, too?

As per the ICMP: do you have another rule on Floating or the interface they are originating that has the WAN interface enforced as Gateway? I don't suppose so but let's just check.

Cheers
Title: Re: Host on VLAN A can't ping VLAN B within same router
Post by: mtlynch on March 29, 2023, 02:27:32 AM
Quote from: JeGr on March 28, 2023, 12:16:13 PM
As per the ICMP: do you have another rule on Floating or the interface they are originating that has the WAN interface enforced as Gateway? I don't suppose so but let's just check.

Ah, that was it! Thanks so much!

I had a rule on VLAN A that specified WAN_DHCP as the Gateway to allow VLAN A hosts to access the Internet.

If I move the WAN_DHCP rule to the end, pings from VLAN A can reach VLAN B.

I didn't realize that the firewall rules could influence packet routing.

Instead, I've created an alias for internal_networks with Content 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16. Then, I created a rule on VLAN A that's:

Protocol: IPv4
Source: any
Destination: internal_networks (invert)
Gateway: any

That seems like it's a less messy way of allowing Internet access on VLAN A.
Title: Re: Host on VLAN A can't ping VLAN B within same router
Post by: JeGr on March 29, 2023, 12:25:44 PM
Indeed they do :)
That's what's called policy (as in rules) based routing. You force the traffic that matches a specific rule through the attached gateway - even if it wouldn't go there in the first place. That's why PBRs should be handled with care ;) as they can leak or expose information that was to go into another VLAN through a WAN per accident. Normally that should be blocked upstream but a few ISPs or cloud providers take that as a kinda serious offense if they catch RFC1918 packets travelling upstream as they should not be routed :)

Cheers