Can't get firewall to pass some subnet traffic

[thinkentropy]

I had the bright idea of moving my Opnsense from bare metal to virtualized in Proxmox. Everything seems to be working except the firewall seems to be blocking some subnet traffic that I think should be passing. I've been struggling with this for days so if I can't figure this out I will go back to bare metal. Any help would be so greatly appreciated!! Here is my current setup--

I have Opnsense (192.168.1.1) running in a VM in Proxmox (192.168.1.101). They're both sharing a physical network interface over a virtual bridge within Proxmox. I setup this network and called it "LAN," 192.168.1.0/24.

I created a 2nd virtual bridge within Proxmox for another physical network interface on my machine. In Opnsense, I assigned this network as 192.168.3.0/24 and called it "VLANs". I created a Truenas VM in Proxmox and assigned it to use the 2nd virtual bridge I setup. In Opnsense, I assigned IP 192.168.3.248 to this Truenas VM.

In my firewall rules, I have added "pass" rules to allow all traffic into the "VLANs" interface, and "pass" rules to allow all traffic into the "LAN" interface. I can successfully ping computers to and from both subnets. Pinging 192.168.3.248 from 192.168.1.222 works.



The problem is when I try and access the Truenas webgui. The firewall live view shows "Default Deny" blocking the TCP request on port 80 going into the VLANs interface.



Shouldn't the rules I created allow this TCP traffic to pass?

[viragomann]

Shouldn't the rules I created allow this TCP traffic to pass?

Yes, but the blocks seems to be non-SYN packets (state violation).
Tick the "i" at the right to display the details and check the TCP flag.
Filter rules are only applied to SYN packets. If this is permitted, OPNsense adds the connection to its state table and let subsequent packets pass.

Possibly you did something wrong with the VLAN configuration, so that the SYN packet can go directly from the client to the server, but the response packet has to pass OPNsense.

[meyergru]

This could be a case of asymmetric routing or maybe, problems because of mixing tagged and untagged VLANs on the same interface.

Since it is Proxmox, you may have also configured both subnets on that...

[thinkentropy]

Shouldn't the rules I created allow this TCP traffic to pass?

Yes, but the blocks seems to be non-SYN packets (state violation).
Tick the "i" at the right to display the details and check the TCP flag.
Filter rules are only applied to SYN packets. If this is permitted, OPNsense adds the connection to its state table and let subsequent packets pass.

Possibly you did something wrong with the VLAN configuration, so that the SYN packet can go directly from the client to the server, but the response packet has to pass OPNsense.

Hello! Thank you so much for sharing your insight into this! I am super grateful.



You are correct--if I tick the information button to view the details of the block, it does show the TCP flag is SA. I will start looking into this today to see if I can figure out how to fix this as I'm not familiar with SYN packets at the moment. I will have to do some reading today.

This could be a case of asymmetric routing or maybe, problems because of mixing tagged and untagged VLANs on the same interface.

Since it is Proxmox, you may have also configured both subnets on that...

Thank you!!

I don't have any Vlan tagging (at least not intentionally). But, it I agree it sounds like I have indeed screwed something up in Proxmox. I will poke through it today and see if I can figure out what I did wrong. Really appreciate your help thank you thank you!

[viragomann]

Since your OPNsense is virtualized you can consider to do the whole VLAN stuff in Proxmox and just provide the respective virtual interfaces to the VM.

[thinkentropy]

<EDIT>

I took a look at my wiring today and I realized I had my computer (192.168.1.222) accidentally connected to both interfaces (LAN and VLANs) via a switch. That probably explains the asymmetric routing(!). But now that I removed my computer from the VLANs interface, my computer can't reach the truenas VM (192.168.3.248) at all.

My computer 192.168.1.222 can successfully ping 192.168.3.1.

The truenas VM 192.168.3.248 can successfully ping 192.168.1.1.

But 192.168.1.222 cannot talk to 192.168.3.248.

And whereas I was seeing block activity in the firewall live view before, now I'm not getting any, as though I broke the connection entirely. So, I don't even know where to go from here if the Opnsense firewall rules aren't to blame. I've also adjusted all of the networking-related knobs and dials within Proxmox that I am aware of to try and correct this.

At this point I believe this is a Proxmox VM configuration problem, and not anything with Opnsense. So I don't expect anyone here to help troubleshoot Proxmox. Unless anyone has any suggestions to try, I am very reluctantly moving everything back to bare metal this weekend.

Thank you again to everyone that has offered suggestions up to this point!

<EDIT 2>

For grins I just tried connecting both of the network ports into a switch and now things behave the way I would expect them to. My computer on 192.168.1.222 can reach the truenas gui at 192.168.3.248. I can also create firewall rules in Opnsense that pass or block traffic the way I would expect them to. I am obviously new to networking, so possibly I fundamentally misunderstand how routers help facilitate the movement of traffic between subnets.

Each of the physical network interfaces on my machine have their own separate multiport switch. I did that so I could segregate all of my devices--e.g. webcams would all connect to one switch (and hence one network interface), all MAIN LAN devices connect to another switch (and hence a second network interface), and so on. In that configuration I assumed all of the subnet traffic would have to pass THROUGH the router. That's how I previously had everything setup when I had Opnsense running on bare metal and it seemed to work just fine.

Now, it appears as though I need the two network interfaces plugged into the same switch to pass subnet traffic. I guess I'll have to just more reading, and I'll keep playing with it for the next few days and see if I can better understand what is happening.