Host on VLAN A can't ping VLAN B within same router

[mtlynch]

I'm having some trouble with inter-VLAN routing within the same OPNsense system.

I've set up VLANs for the first time on my OPNsense box (running opnsense-business, 22.10.2, amd64).

I have two VLANs on the OPNsense system, both with the same parent (igb1):

VLAN A: 10.0.10.1/24
VLAN B: 10.0.20.1/24

I have two hosts:

Host A1: 10.0.10.101 (on VLAN A)
Host B1: 10.0.20.103 (on VLAN B)

I want to allow a host on VLAN A to send traffic to hosts on VLAN B. More specifically, I want hosts on VLAN A to be able to make HTTP requests to port 1313 on a specific host on VLAN B, but I'm having trouble getting even a simple ping to work.

I tried adding a firewall rule to VLAN A:

Action: Pass
Interface: VLAN A
Direction: In
TCP/IP Version: IPv4
Protocol: any
Source: VLAN A net
Destination: VLAN B net

But pings from Host A1 to Host B1 fail. I tried setting up the inverse rule on VLAN B's firewall, but I get the same result. From my direct LAN, I can ping both A1 and B1 successfully.

Interestingly, if I check for ICMP messages in Firewall > Log Files > Live View, I see this:

Code: [Select]

wan 2023-03-26T15:42:02-04:00 [my public IP, redacted] 10.0.20.103 icmp let out anything from firewall host itself (force gw)
And if I traceroute from A1 to B1, I see that the packets seem to go to the gateway on VLAN A out to the WAN interface:

Code: [Select]

$ sudo traceroute 10.0.20.103
traceroute to 10.0.20.103 (10.0.20.103), 30 hops max, 60 byte packets
 1  10.0.10.1 (10.0.10.1)  0.213 ms  0.166 ms  0.130 ms
 2  [redacted] ([my ISP gateway].1)  14.033 ms  14.010 ms  13.952 ms
 3  [redacted] ([my public IP])  2.603 ms  2.470 ms  2.665 ms
Somehow, packets from VLAN A to VLAN B are being routed from VLAN A's gateway to OPNsense's WAN interface instead of to VLAN B on the same physical interface. I haven't added any routes to OPNsense beyond the defaults.

Does anyone have suggestions about what I'm configuring incorrectly?

[Greelan]

What upstream gateway is set on the VLAN interface assignments?

[mtlynch]

What upstream gateway is set on the VLAN interface assignments?

They are both set to "Auto-detect".

[JeGr]

Could you please show both interface configurations? They are using different VLAN IDs, correct?
Could you add screens of your ruleset, too?

As per the ICMP: do you have another rule on Floating or the interface they are originating that has the WAN interface enforced as Gateway? I don't suppose so but let's just check.

Cheers

[mtlynch]

As per the ICMP: do you have another rule on Floating or the interface they are originating that has the WAN interface enforced as Gateway? I don't suppose so but let's just check.

Ah, that was it! Thanks so much!

I had a rule on VLAN A that specified WAN_DHCP as the Gateway to allow VLAN A hosts to access the Internet.

If I move the WAN_DHCP rule to the end, pings from VLAN A can reach VLAN B.

I didn't realize that the firewall rules could influence packet routing.

Instead, I've created an alias for internal_networks with Content 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16. Then, I created a rule on VLAN A that's:

Protocol: IPv4
Source: any
Destination: internal_networks (invert)
Gateway: any

That seems like it's a less messy way of allowing Internet access on VLAN A.

[JeGr]

Indeed they do
That's what's called policy (as in rules) based routing. You force the traffic that matches a specific rule through the attached gateway - even if it wouldn't go there in the first place. That's why PBRs should be handled with care as they can leak or expose information that was to go into another VLAN through a WAN per accident. Normally that should be blocked upstream but a few ISPs or cloud providers take that as a kinda serious offense if they catch RFC1918 packets travelling upstream as they should not be routed

Cheers