OPNsense as a VMware VM

[spetrillo]

Hello all,

I have seen some posts on running OPNsense as a vm, as well as some older guides on the Internet. Does anyone have an up to date guide on how to do this? I am trying to consolidate all my servers as vms, for ease of use and management. I have gen7 i5 with 24 gigs of RAM to build on.

Thanks,
Steve

[bartjsmit]

Hi Steve,

There's not much to it; boot a FreeBSD 12 64-bit VM from the ISO and assign the vNIC's for the external networks or VLAN's. I use 2 vCPU and 4 GB of RAM which comfortably exceeds the system requirements.

Don't forget the os-vmware plugin.

Bart...

[spetrillo]

Bart,

Thanks for the info...maybe I am making more of it than I really have to. One question regarding the specs you have provisioned for the vm. Are you running any services, other than those that come with the default install?

Thanks,
Steve

[bartjsmit]

Light use (e.g. OpenVPN server) VM uses a few 100 MB RAM and very little CPU.

More detail in PM I sent you.

Bart...

[spetrillo]

For those of you that have viirtualized OPNsense do you use the vnics or do you passthrough to the physical NICs? If you passthrough how does that affect the use of vnics that sit on those physical NICs for other vms? Do I need to have other NICs for the other vms?

[Patrick M. Hausen]

If you passthrough how does that affect the use of vnics that sit on those physical NICs for other vms? Do I need to have other NICs for the other vms?

If you pass through a NIC (or any other PCIe device) to a VM in ESXi, that NIC is not available for ESXi. It's exclusively inside that VM. I do that for NVME drives with TrueNAS SCALE in ESXi.

[phoenix]

For those of you that have viirtualized OPNsense do you use the vnics or do you passthrough to the physical NICs? If you passthrough how does that affect the use of vnics that sit on those physical NICs for other vms? Do I need to have other NICs for the other vms?

I've used OPNsense for years as a VM (on an ESXi server) and had no problems maxing out 1GB download speed on my last ISP where I had an FTTP connection, I used vNICs without any problems.

[spetrillo]

Should I use the VMXNET3 or E1000 adapter for any of my network connections? I see the SRV IO passthrough option but not going to use that just yet. I actually could use the SRV passthrough for the WAN interface, since the fw is the only device that will ever interface to the WAN.

[phoenix]

I'd suggest you use SR-IOV as it's the most performant but if you don't want to then VMXNET3 and I'd ignore the E1000 unless you really can't use anything else.

[spetrillo]

Ok some more questions...

I have several vlans that I want to attach to my OPNsense vm. Not knowing if I am doing it right or wrong I created a VSS for each vlan, so I have a total of 5 virtual switches:

1) WAN
2) LAN/Mgmt
3) Wifi
4) Streaming
5) Server

So my OPNsense vm has 5 vnics attached, one for each vlan coming off of a separate virtual switch. When I boot the OPNsense vm in this config the LAN port takes on the IP of the streaming vlan, even without defining the streaming vlan to the vm. Not sure how this could happen, but I am assuming there is some form of cross contamination with the 5 vnics/vswitches?

[the-mk]

there are three areas to configure: physical networks (the nics you have in your ESXi I assume), those you assign in the virtual switch config area to a nic, and the vlan configuration is happening in the port group area.
from your description I assume you missed or messed up the vlan config in the port group area, since you assign port groups to the nics in your OPNsense virtual machine.

[bartjsmit]

I created a VSS for each vlan

On a stand-alone ESXi host you generally create a virtual switch per physical switch. You define VLAN's across your entire site and trunk the VLAN's that connect to virtual machines to your ESXi.

To maximise aggregate throughput and availability, you can have more than one uplink between your vSwitch and your physical switches. A single link is fine for a home system where you'll likely have more single points of failure and won't generate enough traffic to saturate the link.

In your case, create one vSwitch and configure your external switch to tag the four VLAN's to the ESXi port (five if your WAN is a VLAN in the trunk). Configure port groups with the same VLAN on the vSwitch and test.

Bart...

[spetrillo]

I created a VSS for each vlan

On a stand-alone ESXi host you generally create a virtual switch per physical switch. You define VLAN's across your entire site and trunk the VLAN's that connect to virtual machines to your ESXi.

To maximise aggregate throughput and availability, you can have more than one uplink between your vSwitch and your physical switches. A single link is fine for a home system where you'll likely have more single points of failure and won't generate enough traffic to saturate the link.

In your case, create one vSwitch and configure your external switch to tag the four VLAN's to the ESXi port (five if your WAN is a VLAN in the trunk). Configure port groups with the same VLAN on the vSwitch and test.

Bart...

Since I have 4 physical NICs for the LAN traffic, split by traffic type(Mgmt, WiFi, Streaming, Server) would I create one vswitch and add all the NICs to that vSwitch? Then I would tag all the vms as 0 and let the switch ports determine the vlan?

[bartjsmit]

Since I have 4 physical NICs for the LAN traffic, split by traffic type(Mgmt, WiFi, Streaming, Server) would I create one vswitch and add all the NICs to that vSwitch? Then I would tag all the vms as 0 and let the switch ports determine the vlan?

There are no VLAN tags on the VM's. Their VLAN membership is determined by the port group number of their vNIC. The VM OS has no configuration for VLAN at all. You can set one of its interfaces to another VLAN simply by picking another port group in the vNIC dropdown in VM settings.

The physical switch ports or the host physical NIC's do not determine the network. If you have VLAN's, the network is determined by the VLAN number. You can use multiple connections between the vSwitch and the physical switch like described here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-D34B1ADD-B8A7-43CD-AA7E-2832A0F7EE76.html

Bart...

[spetrillo]

Since I have 4 physical NICs for the LAN traffic, split by traffic type(Mgmt, WiFi, Streaming, Server) would I create one vswitch and add all the NICs to that vSwitch? Then I would tag all the vms as 0 and let the switch ports determine the vlan?

There are no VLAN tags on the VM's. Their VLAN membership is determined by the port group number of their vNIC. The VM OS has no configuration for VLAN at all. You can set one of its interfaces to another VLAN simply by picking another port group in the vNIC dropdown in VM settings.

The physical switch ports or the host physical NIC's do not determine the network. If you have VLAN's, the network is determined by the VLAN number. You can use multiple connections between the vSwitch and the physical switch like described here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-D34B1ADD-B8A7-43CD-AA7E-2832A0F7EE76.html

Bart...

That document really talks to teaming of NICs, which is not what I am doing. As mentioned in my last post each vNIC/NIC is a separate vlan, which then equates to separate physical ports on my physical switch. I was creating a virtual switch to correspond to each vNIC/NIC/Switch port.