[SOLVED] VLAN Traffic Issues

[RDLsysadmin01]

Hi everyone,

I don't think this is the right place to post this issue but after 30 minutes or so of searching the various forum topics, using the search bar and looking for a "problems" forum, I can't seem to find any solution to my problem. Admins, please feel free to move this post if you want or point me in the direction of where I can post this so it's in the accurate spot.

First off: Thank you in advance to anyone who can help me with my issue!
Secondly: Please bare with me as I'm brand new to OPNSense and have not had much formal education on systems like it. I've mainly only ever used residential systems and made them work for my purpose.

Setup: I have a firewall with OPNSense installed that has 6 RJ45 ports (igc0, igc1...ect) operating at 1gb each and 2 SFP+ ports operating at 10gb each but those 2 SFP ports do not register in OPNSense. (Or maybe they do? It shows 4 ix ports under the drop down in the "Assignments" tab but when I plug a RJ45 cable into the SFP+ to RJ45 adapters and the light comes on, no matter what, nothing changes in OPNSense as to any ports activating but that's a separate issue)
My WAN connection is on igc0 and my LAN connection on igc1.
I have 2 VLAN networks setup with both of them having igc1 (IE: the LAN interface) set as the parent under "Interfaces ---> Other Types ---> VLAN"
The first VLAN has an ID of 10 and is the Access Control VLAN for my Access control RFID door readers and has a subnet of 192.168.10.1/24
The second VLAN has an ID of 100 and is the Guest WIFI VLAN for my Aruba Instant On Access Points with a subnet of 192.168.11.1/24
I also have several Ubiquiti Unifi POE+ switches that all my equipment is plugged in to. These are setup to use VLAN network 10 and VLAN network 100 tags as well as default network which is my standard 192.168.1.0/24 network that everything not assigned to a VLAN operates on. I can change individual ports to run on the 2 VLAN networks using these switches and when I change anything from the default network to one of the VLAN's using the management portal of the switches, that device connects to the appropriate VLAN and obtains a DHCP lease from the firewall since DHCPO is turned on for each VLAN.

It's worth mentioning that that default network (IE: 192.168.1.0/24 network) is setup without DHCP in the firewall as I have my own DHCP/DNS server which is a windows 2019 server.


Here's my main problem: When I am connected to the default network (IE: 192.168.1.0/24) and try to ping any of the devices on the VLAN's or on my default network, it works just fine. I can also access my file shares on my default network because I'm connected to my default network (duh right?). When I'm on either VLAN network however, I can ping anything on the default network but I can not access any of my file shares on the default network presumably because I'm on a different subnet?

Now obviously, windows firewall would prevent me accessing the file shares from a different subnet but the firewall for that server (Let's call that server App1) is disabled so that's not the problem and once more, I can ping App1 from either VLAN, I just can't access the file shares. I do have 2 firewall rules in the firewall rules section under the VLAN's that states:

Firewall Rule #1 in the top position:
Action: Pass
Quick: Enabled
Interface: AccesscontrolVLAN (The name of the VLAN Interface as well)
Direction: in
TCP/IP Version: IPv4
Protocol: Any
Source: Any
Destination: Any

Firewall Rule #2 in the bottom position:
Action: Pass
Quick: Enabled
Interface: AccesscontrolVLAN (The name of the VLAN Interface as well)
Direction: out
TCP/IP Version: IPv4
Protocol: Any
Source: Any
Destination: Any

Now from what I understand about OPNSense, it uses the firewall rules of the VLAN (in this case the AccesscontrolVLAN) to route traffic in/out of that VLAN to the WAN and LAN interfaces and that the WAN and LAN interfaces are basically "Open" and do not block anything to VLAN's so you have to control all the VLAN traffic using the rules under the specific VLAN you want to apply those rules to? So in this particular case, there should be nothing stopping me from accessing the default network servers, shares, ect... from the VLAN's since I basically have a rule that says all traffic in and another rule that says all traffic out of the VLAN's is supposed to be accepted... Right??? or am I crazy/missing something?

Again, I can ping anything on the VLAN's from my default network and visa versa I can ping anything on my default network from my VLAN's but I can not access/open a file share in windows file explorer if the file share is located on my default network and I'm trying to access it from the VLAN.

Sorry for the lengthy post, I just wanted to make sure I provided as much info as needed to make sure I'm not missing something important.

Thanks again to anyone who can help me out with this issue!

[Seimus]

To keep it simple for now >
Rules by default dont do any routing, they just block or permit per the 5-tuple. Routing is done based on the routing table.

You also do not want to Open "WAN" this is exposing you to the internet. When you have a state created when traffic for comes back, the initialization state is applied for the switches sIP/dIP. You dont want to do IN permit on WAN, if there is no particular reason.

You need to have rules, per VLAN/Interface.

If you have 3 Networks/3 VLANs you need to set in the FW on each of these Interfaces rules Ingress to allow traffic. Egress (out) rules are not needed as OPNsense has an implicit Egress rule permitting everything in OUT direction.

Can you provide Pictures of those rules for the Interface you try to reach from but its not working?
File shares are SMB?

Always keep in mind, and to simplify things as you are new to this. You only care for IN rules.

Regards,
S.

[RDLsysadmin01]

Seimus,

Thank you for the clarification. I do not currently have any WAN rules set to open other than a couple for my servers that host websites. Those work fine and I do not believe those would conflict with my VLAN as the rules apply to my default network (192.168.1.0/24) not the VLAN networks.

As far as the rules for my AccesscontrolVLAN, see the attached photo for details on those rules. I did not know that OPNSense blanketed outgoing connections but it makes sense. In the attached picture there is an out rule, could that be causing issues or is it basically null/void because of OPNSense having the default out allow configuration you spoke of?

[Seimus]

As mentioned, the Out rule is not needed. Cause OPNsense has an Implicit OUT permit for all interfaces by default.

You care about the IN rule. As you expect to come traffic IN on that Interface to take action either block or permit. That OUT rule you have there doesn't need to be there, additionally if you keep it it does what the default does that is if traffic is about to go OUT of that interface you apply that rule action.

Please enable login on that IN RULE. Click that i icon so its blue and go to live view, there create a filter Source IP at least from which you try to access the file server and check if its hitting the permit RULE.

And if possible share a screenshot.

Regards,
S.

[cookiemonster]

I also have several Ubiquiti Unifi POE+ switches that all my equipment is plugged in to. These are setup to use VLAN network 10 and VLAN network 100 tags as well as default network which is my standard 192.168.1.0/24 network that everything not assigned to a VLAN operates on. I can change individual ports to run on the 2 VLAN networks using these switches and when I change anything from the default network to one of the VLAN's using the management portal of the switches, that device connects to the appropriate VLAN and obtains a DHCP lease from the firewall since DHCPO is turned on for each VLAN.

I hope not getting the wrong end of the stick but if I read this correctly, then you have mixed tagged and untagged traffic coming into the same interface into OPN. If so, bear in mind for weird behaviour. Not what you seem to be dealing with at present but something to be reminded of.

[RDLsysadmin01]

Seimus,

Thanks again for your speedy reply. You asking me to do some of this stuff led me to a discovery.

I have attached the requested screenshots but I believe I have figured out the issue. As you can see in the screenshot for the Live view, the traffic is passing correctly from the AccesscontrolVLAN to the LAN. That's why I'm able to ping everything. (IP's are blocked out for security)

I discovered through the file explorer under my pc where network stores shows up that one of the stores was working and one was not. I had not previously looked at that particular page as I was using file explorer to connect to the stores I wanted directly in the file explorer address bar. This led me to looking at the properties of the 2 stores which as you can see, they are both the same NTFS file system type but again one is working and one is not.

I further investigated and figured out that the one that is working was created based on its IP address and the one that is not working was created based on PC name not the IP of that server which is 192.168.1.x as you can see in the filter.

This should mean that  it's a DNS problem correct? I believe it's due to the fact that my VLAN has a DHCP and DNS server of the firewall while my default network DHCP and DNS server is not the firewall but rather my internal Domain Controller that is also on the default network.
Does that mean I can make the VLAN's use my default network DNS server of 192.168.1.x and it'll work?
Is it even possible to route DNS traffic on the VLAN's to my default network DNS server since the subnets are different? BTW: The default network DHCP/DNS server is also the domain controller for my network.



Cookiemonster,

I'm still fairly new to all this. Can you describe in more detail what you mean by tagged and untagged? The unifi switch routes all traffic to "Third-Party-Router" for all networks. One network is the default network which routes traffic to the OPNsense firewall 192.168.1.0/24 network. The VLAN's in OPNSense are configured with Tags 10 and 100 depending on the VLAN. The unifi switches are also configured with those tags. Traffic seems to be routing correctly but maybe I have missed something or am not aware of best practices for how it should be setup?

[Seimus]

So as you can see on Live view,

ICMP
1st is hit the rule on IN for that Interface of the particular VLAN
2nd is hit the rule on OUT Interface of the network where your server resides

If you do the same but trying to reach the File Server you should see exactly the same behavior but different protocol and ports.


Now in regards of your question and findings:

I further investigated and figured out that the one that is working was created based on its IP address and the one that is not working was created based on PC name not the IP of that server which is 192.168.1.x as you can see in the filter.

So basically the other one that is using a domain name. If you try to connect using the domain/host name, the source from which you are reaching will create a DNS query to its DNS server and ask for the IP. If your DNS server for that particular VLAN doesn't know it, you cant connect.

This should mean that  it's a DNS problem correct? I believe it's due to the fact that my VLAN has a DHCP and DNS server of the firewall while my default network DHCP and DNS server is not the firewall but rather my internal Domain Controller that is also on the default network.

Could be. Try to do from that machine that its not working in CLI or Konsole. If it cant translate e.g. no IP is provided you know for sure.

Code Select Expand

nslookup <yourdestinationhostname>

Does that mean I can make the VLAN's use my default network DNS server of 192.168.1.x and it'll work?
Is it even possible to route DNS traffic on the VLAN's to my default network DNS server since the subnets are different? BTW: The default network DHCP/DNS server is also the domain controller for my network.

Yes, you can tell your DHCP server (OPNsense in this example) what DNS server to advertise within the DHCP offer.
Yes, you can route between two or more different networks, thats why we call such devices routers.
If you keep your any any rule as is, traffic for DNS will be permitted to the DNS server you choose. If you will remove that rule and want to to explicit rules than you need to as well create a rule IN permitting DNS traffic.

P.S. you can even use other DHCP server than is OPNsense, just by creating relay on OPN, the OPN will forward DHCP discovery to the dedicated DHCP server. If your DHCP server can offer addresses for multiple networks its not a problem (I have this setup, as I have DNS/DHCP/NTP server standalone from OPN)

[RDLsysadmin01]

If you do the same but trying to reach the File Server you should see exactly the same behavior but different protocol and ports.

Unfortunately I do not see the same behavior. On windows 10 with CMD I can run ipconfig and it will show me my VLAN address of 192.168.10.x and then I can type ping 192.168.1.x being the file share server and I get a reply. Same thing if I go into file explorer and type the ip address \\192.168.1.x\ in the file explorer search bar.

It does not work though when I use the server name. if i do ping ServerName i get no response and same if I try accessing in file explorer using \\ServerName\

Could be. Try to do from that machine that its not working in CLI or Konsole. If it cant translate e.g. no IP is provided you know for sure.

Like stated above, when I try to ping/access the server I can get ping and access to work using IP but not ServerName.

Yes, you can tell your DHCP server (OPNsense in this example) what DNS server to advertise within the DHCP offer.
Yes, you can route between two or more different networks, thats why we call such devices routers.
If you keep your any any rule as is, traffic for DNS will be permitted to the DNS server you choose. If you will remove that rule and want to to explicit rules than you need to as well create a rule IN permitting DNS traffic.

So I have not changed my any rule and I do have my DNS server listed in "System ---> Settings ---> General ---> DNS Servers" and it is a address on my default network of 192.168.1.xxx with gateway set to none

Is that an appropriate setup so the VLAN's use that DNS server? Also, unbound DNS & OpenDNS are both disabled so I believe it should be using the DNS server in "System ---> Settings ---> General ---> DNS Servers"

P.S. you can even use other DHCP server than is OPNsense, just by creating relay on OPN, the OPN will forward DHCP discovery to the dedicated DHCP server. If your DHCP server can offer addresses for multiple networks its not a problem (I have this setup, as I have DNS/DHCP/NTP server standalone from OPN)

Can you explain the "Relay" function you mention here? I have a DHCP/DNS/NTP server on my default network of 192.168.1.0/24 which is what I have programmed into the DNS server section at "System ---> Settings ---> General ---> DNS Servers" so in theory it should be using that server for DNS but your comment implies that the VLAN's may also be able to use it? Can you help me understand how?

Do I just need to turn DHCP off on the VLAN's and create new scopes on my DHCP server that match those subnets and then it will work or am I over simplifying that/completely wrong?

I've attached images of the DNS Server section of OPNSense as well as a screen grab of my DHCP Server Scope Settings with some info blocked for security reasons but you should be able to understand it. As you can see I only have one scope on my DHCP server and it's 192.168.1.1-192.168.1.250 with a few addresses exluded from that range. Do I simply need to make 2 other scopes that are encompassing of the VLAN's subnets and then turn off DHCP on the VLANs????


Thanks again!

[cookiemonster]

I'm still fairly new to all this. Can you describe in more detail what you mean by tagged and untagged? The unifi switch routes all traffic to "Third-Party-Router" for all networks. One network is the default network which routes traffic to the OPNsense firewall 192.168.1.0/24 network. The VLAN's in OPNSense are configured with Tags 10 and 100 depending on the VLAN. The unifi switches are also configured with those tags. Traffic seems to be routing correctly but maybe I have missed something or am not aware of best practices for how it should be setup?

i might have only made unnecessary noise. If you have the VLANs correctly setup with only tagged frames i.e. a trunk, then all is good. Seems like you and Seimus are on top of it, so apologies for the noise.

[Seimus]

I'm still fairly new to all this. Can you describe in more detail what you mean by tagged and untagged? The unifi switch routes all traffic to "Third-Party-Router" for all networks. One network is the default network which routes traffic to the OPNsense firewall 192.168.1.0/24 network. The VLAN's in OPNSense are configured with Tags 10 and 100 depending on the VLAN. The unifi switches are also configured with those tags. Traffic seems to be routing correctly but maybe I have missed something or am not aware of best practices for how it should be setup?

i might have only made unnecessary noise. If you have the VLANs correctly setup with only tagged frames i.e. a trunk, then all is good. Seems like you and Seimus are on top of it, so apologies for the noise.

No, you didn't create any noise. This is a valid point, and should be repeated as many times needed, and specifically to newcommers in the world of Network. If you wouldn't said it I would ;)

We need to drill into people some common best practices.

Regards,
S.

[Seimus]

Unfortunately I do not see the same behavior. On windows 10 with CMD I can run ipconfig and it will show me my VLAN address of 192.168.10.x and then I can type ping 192.168.1.x being the file share server and I get a reply. Same thing if I go into file explorer and type the ip address \\192.168.1.x\ in the file explorer search bar.

It does not work though when I use the server name. if i do ping ServerName i get no response and same if I try accessing in file explorer using \\ServerName\

I mean if it worked you would see the same.

Like stated above, when I try to ping/access the server I can get ping and access to work using IP but not ServerName.

Sorry I expressed it wrongly. What I ment is to do nslookup on the device that the server that is not working. If you cant translate the domain/hostname than you know its really the DNS.

So I have not changed my any rule and I do have my DNS server listed in "System ---> Settings ---> General ---> DNS Servers" and it is a address on my default network of 192.168.1.xxx with gateway set to none

It does not work though when I use the server name. if i do ping ServerName i get no response and same if I try accessing in file explorer using \\ServerName\

This specifies the DNS server OPN itself should use, not the Hosts that get IP from DHCP from OPNsense. If you want to tell the HOST what is the DNS server to be used, you need to configure it in the DHCP there.

Can you explain the "Relay" function you mention here? I have a DHCP/DNS/NTP server on my default network of 192.168.1.0/24 which is what I have programmed into the DNS server section at "System ---> Settings ---> General ---> DNS Servers" so in theory it should be using that server for DNS but your comment implies that the VLAN's may also be able to use it? Can you help me understand how?

No, as mentioned above this is applicable only for OPNsense itself not for hosts that get IP from OPNsense DHCP server. You need to explicitly in the OPNsense DHCP configuration state what is the DNS server to be used.

Do I just need to turn DHCP off on the VLAN's and create new scopes on my DHCP server that match those subnets and then it will work or am I over simplifying that/completely wrong?

I've attached images of the DNS Server section of OPNSense as well as a screen grab of my DHCP Server Scope Settings with some info blocked for security reasons but you should be able to understand it. As you can see I only have one scope on my DHCP server and it's 192.168.1.1-192.168.1.250 with a few addresses exluded from that range. Do I simply need to make 2 other scopes that are encompassing of the VLAN's subnets and then turn off DHCP on the VLANs?

Create proper pools and ranges on your DHCP server, in OPNsense disable the DHCP server and configure relay server as well interfaces that should be using this relay.
https://docs.opnsense.org/manual/dhcp.html#dhcrelay

Btw, try to connect to that Fileserver using the IP not the hostname.

Regards,
S.

[Seimus]

You have multiple ways how to fix this if its DNS server related:

1. [Easy] Use IP of the server instead his hostname
2. [Easy] On OPNsense DHCP set the DNS server to be your standalone DNS server
3. [Depends on your skills] Create on your standalone DHCP server proper pools & on OPNsense Create Relay to forward DHCP to the Server

[RDLsysadmin01]

You have multiple ways how to fix this if its DNS server related:

1. [Easy] Use IP of the server instead his hostname
2. [Easy] On OPNsense DHCP set the DNS server to be your standalone DNS server
3. [Depends on you skills] Create on your standalone DHCP server proper pools & on OPNsense Create Relay to forward DHCP to the Server

Okay so I was able to fix it. I chose your option 2. I feel kind of stupid for not seeing it before but there is a option for the DNS server in the AccesscontrolVLAN under "Services ---> ISC DHCPv4 ---> AccesscontrolVLAN" so I set the correct DNS server (It had the gateway of the vlan in it instead) and now it works. I don't know why I didn't notice that before when setting it up but now it all works as it should. Thanks guys for all the help! And especially thanks for dealing with my newbie-ness.

[Seimus]

No worries about that, beginners mistakes. Usually People dont have standalone DNS servers so its easy missable.

As well if all is solved for you in regards of this topic adjust the Topic subject with [SOLVED] front of it.

Regards,
S.

[RDLsysadmin01]

Thanks guys!