[Solved] Block rule matching, but pass rule is not matching

[whiteoak]

I'm very new to using OPNsense. I followed https://www.youtube.com/watch?v=VcTGKBHcqmk to set up Proxmox + OPNsense, but after getting everything set up, I lost access to the Proxmox web UI and SSH.

I'm trying to connect via:

Laptop → Orbi router in AP mode → Mini PC running Proxmox/OPNsense ETH1 bridged → Proxmox static 10.0.0.7

I can ping 10.0.0.7, and in the OPNsense logs I see calls out to NTP from 10.0.0.7. I also have an internet connection through OPNsense/Proxmox. But when I try to open the web UI or ssh, I hit the base rule "Default deny / state violation rule". I tried adding both an instant first match pass and a first match block rule specific to 10.0.0.7. The block rule hits, the pass does not as far as I can tell.

[meyergru]

If you followed that video, you would have 10.0.0.7/24 as the OpnSense address and 10.0.0.x/24 as Proxmox address - they must be different (!), but on the same subnet and both are attached to the interface that serves as the vmbr0 LAN bridge.

Thus, you need no rules for proxmox, because it is on the same subnet as your laptop and opnsense (LAN interface) unless you have a topology that differs from that video in which case this would not work at all like so.

[whiteoak]

As far as I know, 10.0.0.7 is Proxmox and 10.0.0.1 is OPNsense. Before setting up OPNsense, https://10.0.0.7:8006 is what I used to connect to the Proxmox web UI and https://10.0.0.1 is what I connect to to get to the OPNsense UI. It's very possible I messed something up and right now I can't get to Proxmox, so I can't check how that's set up.

I didn't think I need any rules and everything else is working as expected:

  1. 10.0.0.7 (proxmox) is passing to NTP through OPNsense (via Default allow LAN to any rule)
  2. Ping from my laptop is reach 10.0.0.7 and getting a response.

What I don't understand is:

  1. Without any additional rules, laptop to 10.0.0.7:8006 is blocked via Default deny / state violation rule
  2. When I add an explicit instant first Pass for 10.0.0.7:8006 it doesn't seem to hit and it still gets blocked by the default deny
  3. When I add an explicit instant first Block for 10.0.0.7:8006 does hit

I feel like I'm at the end of what I can figure out and if nothing else might wipe everything and restart.

[Patrick M. Hausen]

1.0.0.7 or 10.0.0.7?

[whiteoak]

Sorry typo! 10.0.0.7. Fixed above.

[whiteoak]

In the rules.jpg screenshot I posted, I have the Pass for Proxmox before the Block. In this case it blocks on the default deny. If I flip the order so the Proxmox block is first, then that block hits rather than the default deny. They are copies of each other so I expected them to match in the same way. The only difference is the Action.

[meyergru]

I still do not get what your topology is. If both your proxmox and opnsense LAN are connected to the same bridge / physical interface, then in order to access your proxmox, there is absolutely no need for any firewall rule - UNLESS your topology is not the one from the video.

If the start of the opnsense VM under proxmox kills your connection, there must be a clash of addresses, maybe opnsense and proxmox have the same one or you did not detach your normal router and that one has the same IPv4 as your opnsense VM.

[whiteoak]

Keep in mind I'm pretty new to all this and I don't have access to Proxmox to verify some of what I'm about to say.

I set up Proxmox with my mini PC connected to my Orbi router. It got 10.0.0.7 via DHCP during setup from that router and set that as the static IP. The ONPSense ARP table has a Changwang Technology inc. MAC address for 10.0.0.7. What else can I look at?

Also, I'm not sure how how OPNSense rules work, but it doesn't make sense that I would see a Block for my 10.0.0.7 rule, but then when I change the action to Pass it hits the default deny instead.

[meyergru]

1. You first said your Orbi router is in "AP mode" - in contrast to you now saying that you got an address via DHCP from that same router, which now seems to be your internet gateway? Choose one.

2. You say you have set your Proxmox to 10.0.0.7 - which of the OpnSense VM logical ports (LAN or WAN) is connected to the vmbr0 bridge? What is the other logical port connected to and what are you trying to achieve? Mind you, if only one port is connected, OpnSense cannot do anything. Also, whatever port is connected to vmbr0, it should have another IP in 10.0.0.x/24 but you cannot create effective rules within the same subnet. Thus, even if a rule to 10.0.0.7 fires, it is probably an artefact, but has no effect.

If you lose connectivity once you start the OpnSense VM, it is highly likely that you have an IP collision or you have created a network loop, causing broadcast storms. AFAICT, any rules are most probably not your root problem.

I suggest you draw a topology map with the interfaces and their connections, including subnets.

[whiteoak]

1. Yes, the Orbi was my internet gateway during initial setup: ONT -> Orbi -> MiniPC on ETH1. I created two more bridges, one to be used for WAN and another for VLANs. Then I installed ONPSense, disconnected the Orbi and connected by laptop to ETH1 to configure OPNSense. Then I changed the Orbi to AP mode and set it up: ONT -> MiniPC -> Orbi.

2. Since I don't have access to Proxmox I can't double check this, but I believe OPNSense LAN is connected to vmbr0 and WAN is connected to vmbr1. Proxmox is also connected to vmbr0.

How would I be able to diagnose if there is a IP collision or broadcast storm? Pings are reaching 10.0.0.7. 10.0.0.7 is seemingly reaching out to NTP. I have connectivity through the AP to the internet.

[meyergru]

I can only repeat that if your topology matches what you have depicted, OpnSense does not effectively regulate traffic to your Proxmox. It is (or better: should be) directly connected to vmbr0, so nothing keeps it from being accessed. And it seems so, since you can ping it.

At this point, I only have those ideas:

If you have access to your proxmox console, you can check its network settings, you can also start and stop VMs to see if OpnSense causes the problem. You can also check if the Proxmox GUI is running (if not, it would explain why you can ping but not get at the GUI). BTW: There was someone who said that switching the browser suddenly made it work.

You also seem to have OpnSense access, so what method does it use to access internet via WAN? DHCP? PPPoE? Does it get a routeable IP or something in RFC1918?

You have switched physical eth1/eth0 and vmbr0/vmbr1, are you sure that this is correct - also in the VM?

[whiteoak]

Brilliant! I forgot I can get to console by plugging in to HDMI. The proxmox address was 10.0.0.7/24(!) and my laptop was getting a 10.0.128.x address via DHCP. I set a 1.0.0.x static IP on my laptop and can reach the UI now. Thank you getting me unstuck with this, meyergru.

I'm interested to learn a bit more what's going on here. So was it Proxmox dropping the connection or OPNSense? And why would OPNSense show matches to the default deny, but it the case where I added an explicit block to Proxmox, it would show that one as matching?

[meyergru]

Proxmox dropped it because it thinks the IP is not within its subnet (which is correct). Since it did not react to the ARP requests, I think that OpnSense just saw the request and logged it - probably, because the request was not answered via ARP, it got directed to the default gateway (OpnSense in this case).

[whiteoak]

Thanks again for your help and time.