Proxmox+OPNSense+VM Routing Issues?

[AryasDad]

I have been testing a setup of OPNSense running on Proxmox as a potential replacement for my old pfsense system that never seemed to be able to achieve 1 GB throughput (or near enough) for my fiber internet.  I believe it is hardware choking it as I can not seem to get it over the hump.

I am currently testing the new Proxmox system behind my pfsense system as I have some complicated setups with VLANS and HAPRoxy that I want to get setup on OPNsense before I transition to ensure limited downtime.

In my existing setup, I have a separate Ubuntu server running some tasks on separate hardware.  My hope is to roll that into Proxmox as a VM to reduce the amount of boxes.  So I have generated the Ubuntu VM on Proxmox.  This is where I am running into issues.  I cannot seem to get my Ubuntu VM (UVM) to connect to the internet.

My test setup is this:

WAN -> PFSense -> LAN (17216.1.1/16) -> HomeLab VLAN (10.0.1.1/24) -> UniFi switch w/ VLAN Tagged -> New Proxmox box w/ 6 ports

Proxmox box setup
Port 1 bridge setup with 10.0.1.254 for managing Proxmox from my LAN network
Port 6 bridged (vmbr6) to act as WAN port for OPNSense (vtnet0)
Port 5 bridged (vmbr5) to act as  LAN port for OPNSense (vtnet1)

I have tried the three following ways with varied results:

Option 1
- UVM w/ virtual port based on vmbr5 (port 5 above), tagged with VLAN 105
- Setup VLAN 105 on OPNSense
- Create OPNSense interface for VLAN w/ LAN Parent interface with IP 10.105.0.253
- DHCP 10.105.0.1/29 for VLAN Interface

In this setup, I get DHCP address assigned to UVM as 10.105.0.249, DNS seems to function as it revolves google.com to IP with ping.  However, I do not have internet connection, as ping fails.  I cannot curl or anything else.  I can ping OPNSense gateway at both 10.105.0.253 (setup as interface IP in OPNSense for VLAN 105) and 10.0.1.253 (how I access OPNSense from my main LAN).  But beyond that, no route is established.

Option 2
- Create new vmbr7 bridge in Proxmox, not tied to port with CDIR 10.105.0.1/29
- UVM w/ virtual port based on vmbr7, no VLAN tag as vtnet03
- Create OPNSense interface for  vtnet03 with IP 10.105.0.253
- DHCP 10.105.0.1/29 for vtnet03 Interface

In this setup, I get DHCP to assign IP to UVM, but nothing else works.  No DNS, no internet, cannot ping anything.

Option 3
- UVM w/ virtual port based on LAN (vmbr5 above), no VLAN tag
- Gets IP assigned from pfsense VLAN for HomeLab

In this setup, I get IP assigned from my pfsense box and I can DNS, reach internet, etc.  But of course, can ping OPNSense on 10.0.1.253.  In this scenario, I am essentially bypassing OPNSense.  So , this will not ultimately work in my scenario when I remove pfsense.

So, why can I not reach internet.  It seems like either 1 or 2 above would work.  I am not sure why the internet route is breaking down.  Based on my searches on this, I see posts saying to ensure there is a route from the VLAN interface to the gateway (WAN in my case?).  However, auto outbound NAT sows a connection between Ubuntu interfaces in both option 1 or 2 above.  Does that not establish that route? 

Note, in these scenarios, OPNSense can reach the internet, resolve DNS, etc.  So the breakdown apparently seems to be within OPNSense itself, but I cannot figure out where.  I do not think I am doing anything exotic that I haven't already done in my original pfsense setup.  Th only difference is that the Ubuntu setup is now virtual inside Proxmox instead of physical hardware.  I am just duplicating that setup (option 1) in this test setup.

I have turned off Firewall in Proxmox on the virtual ports to ensure it wasn't causing the issues.  I have also made sure I have Pass all traffic on the the interfaces in OPNSense in both option 1 and 2 above. 

Any help on getting this working would be greatly appreciated. 

[AryasDad]

I just setup one of my other ports on my Proxmox+OPNsense test as a separate interface OPT5.  I then plugged my laptop into it.  I get DHCP assignment and can resolve DNS, but nothing beyond that.  Just like my VLAN virtual port test above.

So it seems like it is something in my OPNSesne setup that is preventing connection form my downstream ports/VLANS to the WAN port so it can resolve internet connections.

I am not sure what it is as I never had issues like this with pfsense.  I would just setup the interface/VLAN and connections worked.

Any insight on how to get this working, would be greatly appreciated.

[AryasDad]

Not sure what I mucked up, but I decided to reset OPNSense back to defaults to help another issue and now it is working.  So I had some rule or other configuration messed up somewhere that apparently was causing the problem.