OPNsense on KVM on Ubuntu host, host can't connect to WAN

[philkonick]

Hi there

Hopefully someone can point me in the right direction, I'm pretty new to custom home networking and Linux VMs.

I have a mini PC running Ubuntu headless, with Cockpit as a web interface and the Virtual Machines plugin which uses KVM. OPNsense is installed on a VM, with both physical interfaces assigned to the VM via direct attachment.

Out the box everything worked great, OPNsense connected fine to WAN via DHCP as my ISP requires. Its also the DHCP server for my LAN, and all my LAN devices can connect to the web no problem.

The only problem now is that the Ubuntu host can't connect to the web, and I want to use it for other things like a containerised Plex server, and maybe more stuff in the future. The host can't ping the VM's IP, and vice versa. But all the LAN devices can connect to both perfectly. I can also access the console of the OPNsense VM via the Cockpit web interface on the host machine.

Now I'm pretty sure this problem is because the LAN interface is assigned to the VM via direct attachment, and from what I've read I need to do it via bridge/VLAN instead? What would the best solution here be? I'm nervous to just go ahead and try stuff in case I lock myself out of the web interfaces and need to sit on the floor and revert configs on the physical host all night.

Thanks in advance for any advice!

[dseven]

TL;DR https://github.com/cockpit-project/cockpit-machines/issues/285

[philkonick]

Amazing, I didn't even think it could be a Cockpit specific issue.

I'll try creating and switching to a bridge for the LAN interface when I get home today. Hopefully this goes smoothly.

[philkonick]

I tried using a bridge in Cockpit instead but couldn't seem to get it right. Would someone be able to assist me with more detailed steps? I'm not married to using Cockpit to get this to work, but it would be nice to be able to at least monitor and manage everything after setting up.

My setup:
-   Mini PC with 2 physical ethernet ports and a WiFi card running Ubuntu 24.02.1 LTS headless
-   VM on Ubuntu host setup with Cockpit/KVM running OPNsense (OPNsense is DHCP server)
-   Old 4 port router/AP running in AP mode, no DHCP etc (before I get something better)
-   My PC

Mini PC network interfaces:
-   enp3s0
  o     Intel gigabit ethernet interface
  o     Used for WAN, plugs into fibre ONT
  o     Auto IP
-   enp4s0
  o     Intel gigabit ethernet interface
  o     Used for LAN, plugs into AP, and cabled to PC
  o     Static IP 192.168.1.2
-   virbr0
  o     Bridge
  o     Currently inactive
  o     Static IP 192.168.1.9
-   wlp2s0
  o     Realtek PCIe WiFi adapter, 2.4G
  o     Currently disabled

Current VM setup:
-   2 out of 4 CPU cores
-   4 out of 8GB RAM
-   30GB storage allocated
-   enp3s0 connected via “Direct attachment” to macvtab2 and virtio model
  o     Called vtnet0 in OPNsense
  o     Used for WAN
  o     DHCP enabled (as per ISP requirements)
-   enp4s0 connected via “Direct attachment” to macvtab3 and virtio model
  o     Called vtnet1 in OPNsense
  o     Used for LAN
  o     Static IP, 192.168.1.1

What I tried (in Cockpit on the host):
-   Changed VM interface using enp4s0 to “Bridge to LAN” using virbr0
-   Assigned virbr0 the enp4s0 interface in host networking tab
-   Gave virbr0 a static IP 192.168.1.9
-   Did NOT change the WAN interface
-   Did NOT change anything in OPNsense
-   Restarted VM

OPNsense booted successfully as I could see in the console in Cockpit, but the network didn’t connect. I could see OPNsense still had the right IPs set for the same vtnet interfaces, but no devices on the physical network could get DHCP or ping the VM IP.

From what I’ve read I’m thinking it could be because I assigned the interface an IP? Or maybe I also need to configure something in OPNsense to use a different interface than the current one? The problem with that is that I can’t connect to the OPNsense web interface to change anything unless I have direct attachment setup.

Any help would be appreciated.

[philkonick]

Solved my own problem, it was embarrassingly easy in the end.

What I did, for anyone finding this in future:
- Create a new bridge interface in Cockpit called bridge0 (it seems that virbr0 that I thought was a bridge was actually something else?)
- Assign the new bridge interface the enp4s0 physical interface I'm using for LAN
- New bridge interface keeps the same IP
- Stop VM, change VM network interface type from direct to bridge, change assignment to bridge0, start VM

And everything just worked.

I didn't have to change anything in OPNsense, and I didn't need to do anything through the host console. Just make sure the PC you're connecting to the Cockpit host with has a static IP and you shouldn't even lose web interface access while making the interface changes.