Searching for advice - Virtualization and OPNsense

[ezra55]

Hey guys,

I had OPNsense running behind my modem/router combination for a while now (Double NAT). This gave some errors from time to time.
Now I found out that I could set the router in bridge mode. I have OPNsense running on my Ubuntu 16.04 virtualbox host. It has 2 NICs.

Everything works as it should although I have a weird feeling about the Host directly connected to the modem. I have set the secondary NIC on the host (WAN) to a LAN ip (address line in /etc/network/interfaces) 192.168.111.111 which is bogus ofcourse.
Problem is that the NIC has to be up on the host in order to get connectivity in OPNsense (Virtualbox sees the NIC when its down).

UFW rules on the Host are: default deny incoming and some ports on tcp opened or limited.

Like I said I don't really feel secure right now, does anyone have some advice on what to do or to test?

Below are the schematics.

Kind regards,

Ezra

Edit: I might add ( don't think its really necessary but hey...) that I have all traffic tunneled through a OpenVPN client via OPNsense (except my Chromecast for netflix -> default GW)

[bartjsmit]

Hi Ezra,

Any mileage in replacing Ubuntu with VMware? This will let you run a virtual switch on the WAN side without any need to assign it an IP address.

ESXi is admittedly only free as in beer, not as in speech and the HCL is a lot more restrictive.

Bart...

[ezra55]

I've looked at it but some drivers where missing on installing esxi... to much hassle. All my images are also in .vdi so that means converting them all. Also I can't run software raid on esxi and I've had big trouble setting up hardware raid on asrock motherboard.

So I have a good working solution right now... I might change over to VMware in the future...

Thanks for your time.

[ezra55]

Alright, done some searching and I came up with NIC specific rules. Though... I still think I can do more...

Code Select Expand

root@virtualhost:~# ufw status
Status: active

To                         Action      From
--                         ------      ----
8822/tcp on enp2s0         ALLOW       Anywhere                 
443/tcp on enp2s0          ALLOW       Anywhere                 
2049/tcp on enp2s0         ALLOW       Anywhere                 
3390:3393/tcp on enp2s0    ALLOW       Anywhere                 
18083/tcp on enp2s0        ALLOW       Anywhere                 
Anywhere on enp3s5         DENY        Anywhere                 
8822/tcp (v6) on enp2s0    ALLOW       Anywhere (v6)             
443/tcp (v6) on enp2s0     ALLOW       Anywhere (v6)             
2049/tcp (v6) on enp2s0    ALLOW       Anywhere (v6)             
3390:3393/tcp (v6) on enp2s0 ALLOW       Anywhere (v6)             
18083/tcp (v6) on enp2s0   ALLOW       Anywhere (v6)             
Anywhere (v6) on enp3s5    DENY        Anywhere (v6)             

Anywhere                   DENY OUT    Anywhere on enp3s5       
Anywhere (v6)              DENY OUT    Anywhere (v6) on enp3s5 

[s4rs]

Why not use KVM instead of Virtual Box? No need to assign an IP address of the hypervisor nic, just use it for layer 2. I'm using a 128G USB 3.0 key for the hypervisor and guest.

I'm using Fedora server 25. Cockpit makes the network setup a breeze. No editing of network config files. Just set the adapter up for link local on the hypervisor, create a bridge and your good to go. Basically the same thing as a vmware vswitch.

[ezra55]

Thanks for your reply. That would have been a good solution from the start.
My host now handles much external rsyncs and stores all my media etc which some of the vms use. Like stated above I don't feel like reinstalling everything from the get go...

Could you point me to some documentation how to manage the vms with kvm?