[HOWTO] install on Odroid H3+ with qemu-kvm

[0n3man]

I upgraded to a 1Gig WAN connection and my old PC Engines APU2 wasn’t capable of supporting the data rate.  Looking around I saw Odroid had the H2 platform, but unfortunate for me they stopped building it.   Luckily shortly after I started my search they came out with the Odroid H3+.  I’ve been very happy with the arm based Odroid platorms so I figured I’d give it a try.  It runs about $250US for a complete system with the 4 port netboard.  My home network has four LAN segments and this seemed to be the best option when considering cost, performance and interfaces. 

The one thing I wasn’t aware of wast the fact that the H3+ comes with 2.5G realtek NICs and realtek is problematic for the freeBSD operating system.  It’s a driver issue.  Looking at post multiple people indicated they had things working with the older H2+.  I initial configured the H3+ with a direct hardware install and prior to putting it inline as a Firewall the box appeared to be working.  As soon as the H3+ was connected to my WAN side router freeBSD would crash immediately.   I looked around for possible setting that might fix this and tied the few I found, but nothing worked.  I saw multiple post that talked about running freeBSD on top of a hypervisor when realtek NICs were involved and so I gave it a try.

I start my hypervisor install using proxmox because it had been mentioned multiple times.  With the proxmox GUI it was a pretty simple install process.  Opnsense came up and work fine.  Two thing about this approach bothered me.  First was the fact that the supported version of proxmox cost over $100 per year.  This was way too expensive considering I wouldn’t use this approach if the direct install approach worked. They have a cost free non-supported option, but the packages are not considered to be stable.   I didn’t like the  thought of having less than stable packages on my firewall.  I could hear my user base (wife and kids) the first time it died.  The second thing was the fact, that while I didn’t notice any delay in network traffic, when I viewed the Opnsense gui dashboard the CPU performance seemed to lag and glitch.  This gave me some concern about the overhead of running on proxmox.  I’d read that it was only had a 1.5% overhead, but based the glitching I don’t think that number was right.

I set out looking for an alternative to proxmox.  In the past I’ve used VM esxi, but once again that’s really intended to be a paid product.  I came across this article  https://www.linkedin.com/pulse/installing-opnsense-firewall-kvm-sergio-maeso-jim%C3%A9nez/?trk=public_post-content_share-article which covered most of what I needed to know to get qemu kvm running.  The article set up a number of things that were not required if you were just going to run Opnsense on the platform.  Below is what I put in place to make things work. 

The first thing you need to do on the H3+ if you plan to use the 4 port netboard is to make sure you’re using the netboard version of the bios.   Instruction for that can be found here: https://wiki.odroid.com/odroid-h3/hardware/h3_bios_update.

You then need to install Debian.  I used the used the amd64 network install image available here: https://www.debian.org/CD/netinst/.   To make this work you need to use a usb NIC, as the realtek NICs require updated firmware.  On the install I just assigned the full hard drive to a single partition, as this is easist.  Once the sytem is up and running with Debian I then modified /etc/apt/sources.list adding “contrib non-free” to the end of all the repository lines.  With the comment lines remove  the file looked like this after the modifications:

Code: [Select]

deb http://deb.debian.org/debian/ bullseye main  contrib non-free
deb-src http://deb.debian.org/debian/ bullseye main  contrib non-free

deb http://security.debian.org/debian-security bullseye-security main  contrib non-free
deb-src http://security.debian.org/debian-security bullseye-security main  contrib non-free

deb http://deb.debian.org/debian/ bullseye-updates main  contrib non-free
deb-src http://deb.debian.org/debian/ bullseye-updates main  contrib non-free

This updates the firmware:

Code: [Select]

apt update
apt install firmware-realtek

You need to install the qemu packages:

Code: [Select]

apt install -y tmux htop libvirt-daemon-system  qemu-kvm libvirt-clients bridge-utils libguestfs-tools  genisoimage virtinst libosinfo-bin

Sudo isn’t installed on Debian by default so you might want to install it:

Code: [Select]

apt install sudo
If you install sudo you need to add your user to the sudo group to make it work.

You can now set the interfaces up by modifying the file /etc/network/interfaces.  Below are the contents of my interface file.  The commented out allow-hotplug lines were the lines associated with the USB NIC that were setup as part of the Debian install process.  The usb NIC is not required after the firmware update.  I put a static IP on the land bridge (br_lan) so I can directly access this box on the LAN segment of my network. The the IP address, gateway and DNS server need to be updated for your network. 

Code: [Select]

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#allow-hotplug enx000ec6b47101
#auto enx000ec6b47101
#iface enx000ec6b47101 inet dhcp

auto enp2s0
iface enp2s0 inet manual

auto br_lan
iface br_lan inet static
  bridge_ports enp2s0
#  bridge_waitport 0 # no delay before a port becomes available
#  bridge_fd 0 # no forwarding delay
#  bridge_maxwait 0
  address 192.168.10.1
  netmask 255.255.255.0
  gateway 192.168.10.2
  dns-nameservers 192.168.10.2

auto enp1s0
iface enp1s0 inet manual

auto br_wan
iface br_wan inet manual
  bridge_ports enp1s0

auto enp3s0
iface enp3s0 inet manual

auto br_dmz1
iface br_dmz1 inet manual
  bridge_ports enp3s0

auto enp4s0
iface enp4s0 inet manual

auto br_dmz2
iface br_dmz2 inet manual
  bridge_ports enp4s0

At this point you can reboot the box and move your network cable to the LAN interface.  With the setup above, the LAN interface is the NIC port on the H3+ mainboard that is directly above the blue USB ports.  The WAN port will be the second NIC port that comes on the H3+ mainboard.   With the mainboard on top and the netboard on the bottom, DMZ1 is the port to the left on the netboard and DMZ2 is the port to the right of DMZ1. 

Myself I still like to use the old networking tools like ifconfig and netstat so I install net-tools:

Code: [Select]

sudo apt install net-tools
At this point we can build our virtualized Opnsense image.   You need to grab the nano amd64 image from the Opnsense download page and place it on your H3+.  You’ll need to decompress the file via something like:

Code: [Select]

bzip2 -d OPNsense-22.7-OpenSSL-nano-amd64.img.bz2
This then gets coverted to the proper file system for qemu via:

Code: [Select]

qemu-img convert -f raw -O qcow2 OPNsense-22.7-OpenSSL-nano-amd64.img opnsense.qcow2
You want to resize the image, here we add 25G to the disk:

Code: [Select]

qemu-img resize opnsense.qcow2 +25G
Move it to the appropriate place:

Code: [Select]

mv  opnsense.qcow2 /var/lib/libvirt/images/
You need to discribe the VM enviroment to qemu.   Create the file opnsense-kvm-config.xml with the following contents.  This exposes the 4 bridges created in the interface file above to Opnsense.  It allocates 6G of RAM to the VM.  Adjust as needed:

Code: [Select]

<domain type='kvm'>
  <name>opnsense</name>
  <memory unit='MiB'>6144</memory>
  <currentMemory unit='MiB'>6144</currentMemory>
  <vcpu>4</vcpu>
  <os>
    <type arch='x86_64'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features><acpi/><apic/><pae/></features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type='pci' index='0' model='pci-root'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/opnsense201.qcow2'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <interface type='bridge'>
      <source bridge='br_wan'/>
      <model type='virtio'/>
      <target dev='vnet0'/>
      <alias name='net0'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br_lan'/>
      <model type='virtio'/>
      <target dev='vnet1'/>
      <alias name='net1'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br_dmz1'/>
      <model type='virtio'/>
      <target dev='vnet2'/>
      <alias name='net2'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br_dmz2'/>
      <model type='virtio'/>
      <target dev='vnet3'/>
      <alias name='net3'/>
    </interface>
    <serial type='pty'><target port='0'/></serial>
    <console type='pty'><target port='0'/></console>
    <memballoon model='none'></memballoon>
  </devices>
</domain>

Use the above file to set up the VM under qemu via:

Code: [Select]

virsh define opnsense-kvm-config.xml
See that qemu knows about the vm via:
[/code]
virsh list --all   

Code: [Select]

Start the vm and connect to the vm’s console
[code]
virsh start opnsense
virsh console opnsense
Run through the Opnsense install process with the following interface setup adjusted for your needs:

Code: [Select]

WAN  - vtnet0
LAN   - vtnet1
DMZ1 - vtnet2
DMZ2 - vtnet3

At this point you should have a functional system.  You can break from the virsh via <ctrl>]. I’m guessing you’ll want Opnsense to start automatically when the box reboots.  You configure this via:

Code: [Select]

virsh autostart opnsense
Just to make sure qeme is running on startup

Code: [Select]

systemctl enable libvirtd
If you need to change the amount of RAM at some point you can use:

Code: [Select]

virsh setmaxmem opnsense 7G --config
virsh setmem opnsense 7G --config
My box sits in a basement closet without a monitor and keyboard.  I like to use a serial port to access the box.  Here’s a picture of the usb ttl connected to the H3+ pins.  I believe they had the transmit and receive data ports backward on the Hard Kernal site:
   
I couldn't get the picture to post here.   I have it listed as an attachment now, we'll see if the attachment get's posted. The pins used are 4,6,8 and 10 on the H3+ board.  The wires are connected in the exact same order on the USB to TTL board as they are on the H3+.  The wire to the left is light brown and then the next wire over is orange.

To make this work you need to modifiy some lines in /etc/default/grub on the Debian host image:

Code: [Select]

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200n8 console=tty0 systemd.show_status=1"

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
In the above  GRUB_CMDLINE_LINUX_DEFAULT line I removed the quiet parameter because I want to get some indication the system is booting.  With quiet you see a few initial lines but then nothing until you get the login prompt.  Removing quiet gives you all the kernal lines as the system is booting.

Once you modify the file you need to put the changes in place via:

Code: [Select]

sudo update-grub
You should probably not allow ssh login to the Debian image.  That can be fixed via:

Code: [Select]

sudo echo  PermitRootLogin no >> /etc/ssh/sshd_config
systemctrl restart sshd

I believe that’s all I needed to do to get a working Opnsense system running on the qemu kvm hypervisor.   My belief is the hypervisor uses about 3-5% of CPU, but that’s just an estimate. 

[Abbott]

is it not suported a native install on Odroid H3+?

[0n3man]

I did a native install today.  You have to do a few things to get the network interfaces to work.  I mention what I did here https://forum.opnsense.org/index.php?topic=32591.msg161414#msg161414