OPNsense VM behind USG router (Isolated Homelab Networks)

[ishmael]

I'm looking for some help with troubleshooting a nested OPNsense VM running on an XCP-ng cluster. My end goal is to setup several isolated networks behind this VM and learn enough about OPNsense to feel comfortable replacing the Ubiquiti USG with a dedicated OPNsense box. The isolated networks will consist of sandbox environments that may or may not have internet access when "live", but will be cut off from my home network.

The OPNsense VM install went fine, and I have internet access from the VM/web UI to complete updates, etc. The trouble I'm having is that none of the endpoints on the internal networks can reach the internet. For troubleshooting purposes, I added FW rules to open up the VM WAN, LAN1, and LAN2. The endpoints cannot reach any IPs on the main network, either.

I've attempted to simplify my approach, since my config is definitely a "boil the ocean" setup, especially for my basic skill level. None of my attempts have proven successful though, and I feel like I'm stuck and missing fundamental steps (again, likely due to a lack of understanding). Below is an overview of the configuration where I've landed, and attached is a crude network diagram I worked up. It feels like I'm close? I've spent many hours flailing about to get to this point. Any suggestions or guidance would be greatly appreciated!

A few troubleshooting details:

• Endpoints on the LAN1 and LAN2 networks can reach the WAN IP (10.0.0.2), but not the gateway (10.0.0.1).
• Using the diagnostic tools (ping) from the OPNsense web UI, I can reach any endpoint on the LAN1, LAN2, the gateway (10.0.0.1), *and* anything on the home network (192.168.1.1/25).
• Moving one of the Nix endpoints to the home network results in expected network connectivity, so there's not a configuration issue there (that I can think of).

Code: [Select]

# USG: Settings: Networks
[labnet]
Router: Security Gateway
IPv4
Host Address:         10.0.0.1/30
VLAN ID: 10
DHCP Mode: DHCP Server
DHCP Range: 10.0.0.2 - 10.0.0.2


# XCP-ng: Pool: Network
[labnet WAN]
VLAN: 10

[labnet LAN1]
VLAN: 100

[labnet LAN1]
VLAN: 200


# XCP-ng: OPNsense VM: Network
[VIF #0]
labnet LAN1
10.1.1.1

[VIF #1]
labnet WAN
10.0.0.2

[VIF #2]
labnet LAN2
10.2.2.1


# Interfaces
[WAN]
Block private networks (Disabled)
Block bogon networks (Disabled)
Static IPv4
10.0.0.2/30
Auto-detect

[LAN1]
Block private networks (Disabled)
Block bogon networks (Disabled)
Static IPv4
10.1.1.1/24
Auto-detect

[LAN2]
Block private networks (Disabled)
Block bogon networks (Disabled)
Static IPv4
10.2.2.1/24
Auto-detect

[Settings]
Disable hardware checksum offload
Disable hardware TCP segmentation offload
Disable hardware large receive offload
Disable VLAN Hardware Filtering

# System: Gateways: Single
[WAN_VLAN10]
Interface: WAN
Address Family: IPv4
IP Address: 10.0.0.1
Priority: 1

[LAN1_GW]
Interface: LAN1
Address Family: IPv4
IP Address: 10.1.1.1
Priority: 255

[LAN2_GW]
Interface: LAN2
Address Family: IPv4
IP Address: 10.2.2.1
Priority: 255

# System: Settings:     General
Hostname:         OPNsense
Domain:         internal.lan
DNS server opt: Allow DNS server list to be overridden by DHCP/PPP on WAN

# Firewall: NAT: Outbound
Automatic outbound NAT rule generation (no manual rules can be used)

# Firewall: Rules
[WAN]
IPv4 * * * * * * *
IPv4 * * * * * * *

[LAN1]
IPv4 * * * * * * *
IPv4 * * * * * * *

[LAN2]
IPv4 * * * * * * *
IPv4 * * * * * * *

# Services: DHCPv4
[WAN]
Disabled

[LAN1]
Enabled
From: 10.1.1.100
To: 10.1.1.199
Domain name: lan100.internal.lan

[LAN2]
Enabled
From: 10.2.2.100
To: 10.2.2.199
Domain name: lan200.internal.lan

# Services: Unbound DNS: General
Enable Unbound