How can I get OPNSense to run in a Hyper-V VM?

[maharris]

Hi,

I want to set up OPNsense in a Hyper-V VM for purposes of experimentation and learning.  However, I am completely blocked by an issue and hoping others have seen this and know how to overcome it.  The problem is that after I boot the OPNsense dvd from ISO, and as it begins to work through the installer, I see alternating messages of "hv_sock0 on vmbus0" followed by "hv_sock0: detached".  Each message displays on a new line causing the entire terminal to scroll and there is a new message about once per second.  This causes the VM to be unusable because it destroys anything printed to the screen so I can't even work through the installation.  And, this never stops for as long as I allow the VM to run.

I see this on the latest version of Windows 11 and Hyper-V where I have a "work VM" that is also running the latest Windows 11 and Hyper-V and it is in this nested VM where I am trying to deploy a lab with OPNsense.   It's not a nested virtualization problem though because I also see this on the host's Hyper-V.  It's also not a new OPNsense 23.7 issue because I've tried OPNsense 23.1, 22.1.2, and 21.1 and experienced the same issue.  It's also not specific to my Intel 13900KS based desktop hardware because I also see this issue in a VM I created in Azure and on an old surface laptop 2.

I partly think it's a FreeBSD (and or + Hyper-V) issue because I also see this on the latest version of FreeBSD.  Is there any way to install OPNsense on anything else?

Has anyone experienced this?  Can anyone confirm they've just downloaded the OPNsense dvd on their Windows 11 device and installed OPNSense into a Hyper-V VM by booting from the OPNSense DVD?  Does anyone have any other ideas?

I just want to get OPNsense working in a Hyper-V VM.

Thanks!

[Maurice]

Have been using OPNsense on Hyper-V for years, both in production as well as for my test setups. Works fine. I'm typically using pre-built vhdx images though, not the DVD installer.

I've just tried installing from the 23.7 iso real quick and don't see these messages. You might want to share your VM settings. Do you use any special / custom integration services?

Cheers
Maurice

[maharris]

I'm not using anything out of the ordinary as far as I know. 

I use the following powershell script to create my VM.  Once this completes, I use Hyper-V manager's vmconnect window to start and connect the VM.  (Its a mess but I think anyone could run this with no more than an update to the path to the DVD and change of 2 switch names.)

In my case, I literally run this script, and start the VM to and see this issue within a minute or so.

EDIT:
Also - where can I find pre-prepared VHDX images?  I'm willing to try that route too but I have not found any already made available.

Code: [Select]

# MUST RUN AS ADMINISTRATOR

$StartTime = $(get-date);

$HOSTNAME = "ROUTER"

# $ISO = "C:\Users\maharris\Downloads\OPNsense-21.1-OpenSSL-dvd-amd64.iso"
#$ISO = "C:\Users\maharris\Downloads\OPNsense-22.1.2-OpenSSL-dvd-amd64.iso"
# $ISO = "C:\Users\maharris\Downloads\OPNsense-23.1-OpenSSL-dvd-amd64.iso"
$ISO = "C:\Users\maharris\Downloads\OPNsense-23.7-dvd-amd64.iso"

$VMName = $HOSTNAME;
$Notes = ""
$vhdSize = 512GB;
$ProcessorCount = 8;
$Gen = 1;
$StartupRAM = 4GB;
$MinRAM = 2GB;
$MaxRAM = 8GB;
$EnableDynamicMemory = $true;
$EnableSecureBoot = $false;
$SwitchName1 = "Default Switch";
$SwitchName2 = "Internal-LAN1";
$ROOT = Get-VMHost | Select-Object -ExpandProperty VirtualMachinePath;


# Set up VM directory structure to put all VM resources in the ROOT path
# VM ROOT ($LABROOT)
# ├── Smart Paging
# │   └── Virtual Machines
# │       └── <GUID>
# ├── Snapshots
# ├── Virtual Hard Disks
# │   └── <$VMName>_Disk.vhdx
# └── Virtual Machines
#     ├── <GUID>
#     ├── <GUID>.vmcx
#     ├── <GUID>.vmgs
#     └── <GUID>.VMRS
if (!(Test-Path $ROOT))
{   
    mkdir $ROOT;
}

$LABROOT = "$ROOT\$VMName";
if (!(Test-Path $LABROOT))
{   
    mkdir $LABROOT;
}

$LABDISKROOT = $LABROOT + '\Virtual Hard Disks\';
if (!(Test-Path $LABDISKROOT))
{   
    mkdir $LABDISKROOT;
}

$LABVMROOT = $LABROOT + '\Virtual Machines\';
if (!(Test-Path $LABVMROOT))
{   
    mkdir $LABVMROOT;
}

$PAGINGROOT = $LABROOT + '\Smart Paging\';
if (!(Test-Path $PAGINGROOT))
{   
    mkdir $PAGINGROOT;
}

$SNAPSHOTROOT = $LABROOT + '\Snapshots\';
if (!(Test-Path $SNAPSHOTROOT))
{   
    mkdir $SNAPSHOTROOT;
}



# https://docs.microsoft.com/en-us/powershell/module/hyper-v/get-vmswitch?view=windowsserver2022-ps
$switch1 = Get-VMSwitch -Name $SwitchName1;
$switch2 = Get-VMSwitch -Name $SwitchName2;


# https://docs.microsoft.com/en-us/powershell/module/hyper-v/new-vhd?view=windowsserver2022-ps
$DISK01PATH = $LABDISKROOT + "\$VMName"+"_DISK01.vhdx"
if (!(Test-Path $DISK01PATH))
{   
    # https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/best-practices-for-running-linux-on-hyper-v
    # When creating the VHDX, use 1MB BlockSizeBytes (from the default 32MB)
    $vhdx = New-VHD -Path $DISK01PATH -SizeBytes $vhdSize -Dynamic -BlockSizeBytes 1MB;
}
else
{
    $vhdx = Get-VHD -Path $DISK01PATH;
}



# https://docs.microsoft.com/en-us/powershell/module/hyper-v/new-vm?view=windowsserver2022-ps
$vm = New-VM -Name $VMName -MemoryStartupBytes $StartupRAM -SwitchName $switch1.Name -Generation $Gen -Path $ROOT -VHDPath $vhdx.Path;

if ($switch2 -ne $null)
{
    Add-VMNetworkAdapter -VMName $VMName -SwitchName $switch2.Name
}


# https://docs.microsoft.com/en-us/powershell/module/hyper-v/set-vmprocessor?view=windowsserver2022-ps
$vm | Set-VMProcessor -Count $ProcessorCount;

# https://docs.microsoft.com/en-us/powershell/module/hyper-v/enable-vmintegrationservice?view=windowsserver2022-ps
$vm | Get-VMIntegrationService -Name "Guest Service Interface" | Enable-VMIntegrationService;

# https://docs.microsoft.com/en-us/powershell/module/hyper-v/set-vmmemory?view=windowsserver2022-ps
$vm | Set-VMMemory -DynamicMemoryEnabled $EnableDynamicMemory -MinimumBytes $MinRAM -MaximumBytes $MaxRAM;

# https://docs.microsoft.com/en-us/powershell/module/hyper-v/set-vmcomport?view=windowsserver2022-ps
$vm | Set-VMComPort -Number 2 -Path "\\.\pipe\dbg1";

# https://docs.microsoft.com/en-us/powershell/module/hyper-v/set-vm?view=windowsserver2022-ps
$vm | Set-VM -AutomaticCheckpointsEnabled $false -AutomaticStartAction StartIfRunning -DynamicMemory -SmartPagingFilePath $PAGINGROOT -SnapshotFileLocation $SNAPSHOTROOT -EnhancedSessionTransportType HvSocket -Notes $Notes;

# https://docs.microsoft.com/en-us/powershell/module/hyper-v/add-vmdvddrive?view=windowsserver2022-ps
$dvd = $vm | Add-VMDvdDrive -Path $ISO -Passthru;

# https://docs.microsoft.com/en-us/powershell/module/hyper-v/set-vmfirmware?view=windowsserver2022-ps
if ($Gen -eq 2)
{
    $vm | Set-VMFirmware -FirstBootDevice $dvd;

    if ($EnableSecureBoot) {
        # Sets Secure Boot Template.
        $vm | Set-VMFirmware -SecureBootTemplate 'MicrosoftUEFICertificateAuthority';
    } else
    {
        # Disables Secure Boot.
        $vm | Set-VMFirmware -EnableSecureBoot:Off;
    }
}


if ($Gen -eq 2)
{
    $vmNIC = Get-VMNetworkAdapter -VM $vm | Where SwitchName -eq $SwitchName1;
    $vmDVD = Get-VMDvdDrive -VM $vm;
    $vmDisk1 = Get-VMHardDiskDrive -VM $vm -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0;
    Set-VMFirmware -VM $vm -BootOrder $vmDVD, $vmDisk1, $vmNIC;
}

# Disable-VMIntegrationService -Name "Time Synchronization" -VM $vm
# Disable-VMIntegrationService -Name "Guest Service Interface" -VM $vm

$StopTime = $(get-date);
$ElapsedTime = $StopTime - $StartTime;
$TotalTime = "{0:HH:mm:ss}" -f ([datetime]$ElapsedTime.Ticks);
Write-Host
Write-Host $TotalTime;


[Bob.Dig]

I'm not using anything out of the ordinary as far as I know. 
...
I use the following powershell script to create my VM. 

How about not using that script and use the Windows-GUI?

[Maurice]

Why Gen1? Haven't used that for years. And FreeBSD doesn't support Dynamic Memory.

I'd start by creating a Gen2 VM with the wizard. Default settings, other than Dynamic Memory and Secure Boot (both disabled). Then go from there.

If you want to try a pre-built vhdx, see my signature.

[maharris]

Thanks @Bob.Dig.  I'm stumped on what my script is configuring to cause FreeBSD to do this but maybe this is the issue.  Attempt #1 using the manual approach did not lead the same issue and seems to be working as expected.

@Maurice - sorry, I didn't see the signature but will certainly explore what you have.  As for why Gen1, this was actually the first time I've tried to use Gen1 in many years too.  I did so because in the course of troubleshooting I read something somewhere bout FreeBSD possibly not supporting Gen2 so tried it (along with dozens of other small changes) to see if anything helped.  I didn't see dynamic memory not being supported either but the 1 manually configured VM is running with it enabled. 

So, it's looking like my script enables/disables something that causes FreeBSD to enable/disable hv_sock0 continuously.  I'll have to dig into that once I catch up to where I should be after being blocked on this issue for embarrassingly long. 

Thanks for your help!

[Bob.Dig]

If you want to try a pre-built vhdx, see my signature.

Wow, great service you do there, thank you.

[dracocephalum]

The "hv_sock0" messages could be related to this:
https://reviews.freebsd.org/D24061

Did you enable the "Integration Services -> Guest Services" for your OPNSense VM? If so try to disable it and see if it can fix this issue.

[AseKarlsson]

Hi All,
Is there any good guide or Youtube video for the Hyper-V setup?

1. I tried to setup the Virtual Switches (both switches as external because I want them to communicate to the network and not only internally in the Hyper-V environment), but cannot get OPNsense installation to detect IPs correctly.

In Windows the LAN network adaptor has a set IP and the WAN network adaptor has DHCP.
In ipconfig on the host they show as they sould with the correct IPs.

2. I mount the .ISO in Hyper-V but it seems to run on the ISO and not offering an installation option?

Ideas?

//Ase

[Patrick M. Hausen]

2. I mount the .ISO in Hyper-V but it seems to run on the ISO and not offering an installation option?

Did you login at the console with user "installer" and password "opnsense" as documented?

[AseKarlsson]

Thanks Patrick,

Question 2 solved there, been running on a dedicatet Intel NUC for a while so the root came automatically

//Ase

[AseKarlsson]

I can try to open up my case a bit more:

1 physical server (Windows Server with the Hyper-V role installed) with 2 NICs.

The Host using 1 NIC static IP of 192.168.0.10 and have created a Virtual Switch (external) for this that many different Hyper-V computers use on this server (working great).

Would like to use this Virtual Switch for LAN in OPNsense.
OPNsense LAN side should have IP 192.168.0.1 as the gateway address for the network.

The second NIC I would like to be at DHCP so I get the IP from the ISP, modem in bridged mode.

How to setup the Virtual Switches to achieve this?

Thanks,
Ase

[AseKarlsson]

LAN part solved!
WAN side now also solved!