Messages

Turns out there were additional ASPM bios options that I missed.  Once I turn them all off the networking seems to be good.  Its only been running for a little over 24 hours, but it's clear things are better than they were.  As a side note I had set hw.pci.enable_aspm to 0 under tunables, but this did not have a positive effect like disabling ASPM in the bios. 

I have an odroid H4+ with the net card v2 giving a total of 6 intel i226-v interfaces.  This is a clean install as I was looking to replace my current FW.  I initially had issues with the hard drive, which I got past. I then noticed the WAN interface seemed to be dropping out every couple of days.  I found the post that provided steps to update the nic firmware.  So I'm running the latest version of the intel nic firmware (FXVL_125C_V_2MB_2.32.bin).  This did not fix the issue.  I found a post where it was recommended to disable Energy-Efficient-Ethernet by setting the sysctl value for each nic to 1.  Turns out 1 is the default value after the install.  There was another post that indicated fc (flow-control) on the NIC could be a problem.  The default value here was 3, which means flow control is on.  I set it to 0 for each NIC.  Still no help.  I also turned off ASPM with sysctrl and also via the bios.

Once I noticed the WAN interface was dropping I decided to see how things faired while streaming video.  This causes the communications to drop out usually under an hour.  It's strange how things happen.  First normal traffic like DNS will stop being sent out over the WAN port.  However the video will keep streaming for a while. Anyway I'm looking for any other suggestions on how I might fix this.  I'm considering loading some form of VM host as I did that initially when I put my previous hardware in place.  Any thoughts on how to get pass this issue are appreciated.

Install the os-cpu-microcode-intel plugin and reboot.

I'm in the process of upgrading my OPNsense box to and odroid H4+.  I was having all kinds of SATA errors.  It turns out the latest H4+ bios version of the microcode is 0x10, while this plugin injects version 0x1d.  It seems this microcode update fixes my SATA issue.  I also set the tunables:

hw.pci.enable_aspm=0
vm.pmap.pcid_enabled=0

When installing the plugin, it indicated it was going be removed from a later OPNsense release after June 2025. I did ask odroid support for an upgrade bios with the latest intel microcode.  We'll see how they respond. 

So my question is, are there plans to remove the os-cpu-microcode-intel?  I'm hoping the response is that removal is on hold, with no new date in sight.

A while back I tried to get opnsense running on and odroid H3+ to replace an older system I had in place.  I build the system, jumped through the hoops to get the RTL driver going.  I then installed the configuration from my old system and reassigned the NIC interfaces to my 4 LAN segments.  Sadly, the system just crashed immediately when I connected all my LAN segments.  I spent a little time troubleshooting but didn't have any luck with the native install.  So, I went another route to get things working, running opnsense in a virtual machine.  Today I decided to give the native install another shot.  It still failed in the same manner.  This time I decided to disable any service that wasn't essential.  To my delight the system worked.  I then went about re-enabling the services I'd been using.  Everything went fine until I enable intrustion detection in IPS mode.   At which point the system immediately crashed.  Intrustion detection without IPS and promiscuous mode worked.  As such there must be a bug with IPS.  Just wanted to post in case anyone else was using and odroid H3 and hit this immediate crash issue. 

I did a native install today.  You have to do a few things to get the network interfaces to work.  I mention what I did here https://forum.opnsense.org/index.php?topic=32591.msg161414#msg161414

I just did an install today.  I used two usb nics to set up WAN and LAN.  On the WAN I had to configure a static IP.  This was fine because I was just connecting on my existing network, behind my existing router/fw, so I didn't need to get DHCP information directly from my ISP.  I did initially try to use DHCP on the WAN interface, but for some reason when doing this the interface was constantly going down and up.  Not an issue with static IP.  Then i put a laptop connected to the LAN interface so I could access the web interface.  I logged in and updated the firmware.  I had to run updates twice to get to latest versions.  Then I went to the plugins page under firmware and installed os-realtek-re.  Rebooted, removed USB NICs, logged in via console port and reassigned the new re0 and re1 interfaces to WAN and LAN.   After that I installed configuration from my old opnsense FW and put the device in line.  Works fine now.

You need to make sure you also install the network version of the bios mentioned here https://wiki.odroid.com/odroid-h3/hardware/h3_bios_update

A lot of times when you get the cert error it's because the time isn't set correctly on your system. 

I'd go with opensource qemu-kvm instead of proxmox.  I wrote up the steps I took to get opnsense running on gemu-kvm here:  https://forum.opnsense.org/index.php?topic=31650.msg152895#msg152895.  It isn't that hard.  Doesn't give you the nice GUI like proxmox.  However, if you're only installing a couple of VMS that you plan to run all the time using something with stable packages is a better choice. 

Recommendations for hardware really depend on your throughput goal.  I've used a pe engines APU2 platform for years for home. It only good for under 100M.  I've also use one of the Qotom boxes https://www.amazon.com/gp/product/B0B7JLWCML/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 when I upgraded to 1G.  I actually like the Odroid H3+ with the optional netboard, which gives you a total of 6 nics.  With out IPS the H3+ can handle the 1G speed.  If you enable IPS it passes about 600M of traffic.  The other items you mentioned aren't all that processing intensive.  I like the H3+ because in the space that it competes it is best when considering cost, performance and capbilities.  The H3+ is what I'm currently using.

My experience on the Odroid H3+ was the system crashing when the system was put in line. The realtek driver seems to have a bug.  The NICs worked after I enabled the driver from plugins but once I put it line the system was useless. As a result I ended up going the Opnsense VM running on top of qemu kvm.  I wrote it up the install process here https://forum.opnsense.org/index.php?topic=31650.0.

My experience on the Odroid H3+ was the system crashing when the system was put in line. The realtek driver seems to have a bug.  As a result I ended up going the Opnsense VM running on top of qemu kvm.  I wrote it up the install process here https://forum.opnsense.org/index.php?topic=31650.0.  I posted on the wrong page, so it wouldn't apply unless you're using realtek for the 2.5G.

I upgraded to a 1Gig WAN connection and my old PC Engines APU2 wasn't capable of supporting the data rate.  Looking around I saw Odroid had the H2 platform, but unfortunate for me they stopped building it.   Luckily shortly after I started my search they came out with the Odroid H3+.  I've been very happy with the arm based Odroid platorms so I figured I'd give it a try.  It runs about $250US for a complete system with the 4 port netboard.  My home network has four LAN segments and this seemed to be the best option when considering cost, performance and interfaces. 

The one thing I wasn't aware of wast the fact that the H3+ comes with 2.5G realtek NICs and realtek is problematic for the freeBSD operating system.  It's a driver issue.  Looking at post multiple people indicated they had things working with the older H2+.  I initial configured the H3+ with a direct hardware install and prior to putting it inline as a Firewall the box appeared to be working.  As soon as the H3+ was connected to my WAN side router freeBSD would crash immediately.   I looked around for possible setting that might fix this and tied the few I found, but nothing worked.  I saw multiple post that talked about running freeBSD on top of a hypervisor when realtek NICs were involved and so I gave it a try.

I start my hypervisor install using proxmox because it had been mentioned multiple times.  With the proxmox GUI it was a pretty simple install process.  Opnsense came up and work fine.  Two thing about this approach bothered me.  First was the fact that the supported version of proxmox cost over $100 per year.  This was way too expensive considering I wouldn't use this approach if the direct install approach worked. They have a cost free non-supported option, but the packages are not considered to be stable.   I didn't like the  thought of having less than stable packages on my firewall.  I could hear my user base (wife and kids) the first time it died.  The second thing was the fact, that while I didn't notice any delay in network traffic, when I viewed the Opnsense gui dashboard the CPU performance seemed to lag and glitch.  This gave me some concern about the overhead of running on proxmox.  I'd read that it was only had a 1.5% overhead, but based the glitching I don't think that number was right.

I set out looking for an alternative to proxmox.  In the past I've used VM esxi, but once again that's really intended to be a paid product.  I came across this article  https://www.linkedin.com/pulse/installing-opnsense-firewall-kvm-sergio-maeso-jim%C3%A9nez/?trk=public_post-content_share-article which covered most of what I needed to know to get qemu kvm running.  The article set up a number of things that were not required if you were just going to run Opnsense on the platform.  Below is what I put in place to make things work. 

The first thing you need to do on the H3+ if you plan to use the 4 port netboard is to make sure you're using the netboard version of the bios.   Instruction for that can be found here: https://wiki.odroid.com/odroid-h3/hardware/h3_bios_update.

You then need to install Debian.  I used the used the amd64 network install image available here: https://www.debian.org/CD/netinst/.   To make this work you need to use a usb NIC, as the realtek NICs require updated firmware.  On the install I just assigned the full hard drive to a single partition, as this is easist.  Once the sytem is up and running with Debian I then modified /etc/apt/sources.list adding "contrib non-free" to the end of all the repository lines.  With the comment lines remove  the file looked like this after the modifications:

Code Select Expand


deb http://deb.debian.org/debian/ bullseye main  contrib non-free
deb-src http://deb.debian.org/debian/ bullseye main  contrib non-free

deb http://security.debian.org/debian-security bullseye-security main  contrib non-free
deb-src http://security.debian.org/debian-security bullseye-security main  contrib non-free

deb http://deb.debian.org/debian/ bullseye-updates main  contrib non-free
deb-src http://deb.debian.org/debian/ bullseye-updates main  contrib non-free


This updates the firmware:

Code Select Expand


apt update
apt install firmware-realtek


You need to install the qemu packages:

Code Select Expand


apt install -y tmux htop libvirt-daemon-system  qemu-kvm libvirt-clients bridge-utils libguestfs-tools  genisoimage virtinst libosinfo-bin


Sudo isn't installed on Debian by default so you might want to install it:

Code Select Expand


apt install sudo

If you install sudo you need to add your user to the sudo group to make it work.

You can now set the interfaces up by modifying the file /etc/network/interfaces.  Below are the contents of my interface file.  The commented out allow-hotplug lines were the lines associated with the USB NIC that were setup as part of the Debian install process.  The usb NIC is not required after the firmware update.  I put a static IP on the land bridge (br_lan) so I can directly access this box on the LAN segment of my network. The the IP address, gateway and DNS server need to be updated for your network. 

Code Select Expand


# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#allow-hotplug enx000ec6b47101
#auto enx000ec6b47101
#iface enx000ec6b47101 inet dhcp

auto enp2s0
iface enp2s0 inet manual

auto br_lan
iface br_lan inet static
  bridge_ports enp2s0
#  bridge_waitport 0 # no delay before a port becomes available
#  bridge_fd 0 # no forwarding delay
#  bridge_maxwait 0
  address 192.168.10.1
  netmask 255.255.255.0
  gateway 192.168.10.2
  dns-nameservers 192.168.10.2

auto enp1s0
iface enp1s0 inet manual

auto br_wan
iface br_wan inet manual
  bridge_ports enp1s0

auto enp3s0
iface enp3s0 inet manual

auto br_dmz1
iface br_dmz1 inet manual
  bridge_ports enp3s0

auto enp4s0
iface enp4s0 inet manual

auto br_dmz2
iface br_dmz2 inet manual
  bridge_ports enp4s0


At this point you can reboot the box and move your network cable to the LAN interface.  With the setup above, the LAN interface is the NIC port on the H3+ mainboard that is directly above the blue USB ports.  The WAN port will be the second NIC port that comes on the H3+ mainboard.   With the mainboard on top and the netboard on the bottom, DMZ1 is the port to the left on the netboard and DMZ2 is the port to the right of DMZ1. 

Myself I still like to use the old networking tools like ifconfig and netstat so I install net-tools:

Code Select Expand


sudo apt install net-tools

At this point we can build our virtualized Opnsense image.   You need to grab the nano amd64 image from the Opnsense download page and place it on your H3+.  You'll need to decompress the file via something like:

Code Select Expand


bzip2 -d OPNsense-22.7-OpenSSL-nano-amd64.img.bz2

This then gets coverted to the proper file system for qemu via:

Code Select Expand


qemu-img convert -f raw -O qcow2 OPNsense-22.7-OpenSSL-nano-amd64.img opnsense.qcow2

You want to resize the image, here we add 25G to the disk:

Code Select Expand


qemu-img resize opnsense.qcow2 +25G

Move it to the appropriate place:

Code Select Expand


mv  opnsense.qcow2 /var/lib/libvirt/images/

You need to discribe the VM enviroment to qemu.   Create the file opnsense-kvm-config.xml with the following contents.  This exposes the 4 bridges created in the interface file above to Opnsense.  It allocates 6G of RAM to the VM.  Adjust as needed:

Code Select Expand


<domain type='kvm'>
  <name>opnsense</name>
  <memory unit='MiB'>6144</memory>
  <currentMemory unit='MiB'>6144</currentMemory>
  <vcpu>4</vcpu>
  <os>
    <type arch='x86_64'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features><acpi/><apic/><pae/></features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type='pci' index='0' model='pci-root'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/opnsense201.qcow2'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <interface type='bridge'>
      <source bridge='br_wan'/>
      <model type='virtio'/>
      <target dev='vnet0'/>
      <alias name='net0'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br_lan'/>
      <model type='virtio'/>
      <target dev='vnet1'/>
      <alias name='net1'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br_dmz1'/>
      <model type='virtio'/>
      <target dev='vnet2'/>
      <alias name='net2'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br_dmz2'/>
      <model type='virtio'/>
      <target dev='vnet3'/>
      <alias name='net3'/>
    </interface>
    <serial type='pty'><target port='0'/></serial>
    <console type='pty'><target port='0'/></console>
    <memballoon model='none'></memballoon>
  </devices>
</domain>


Use the above file to set up the VM under qemu via:

Code Select Expand


virsh define opnsense-kvm-config.xml

See that qemu knows about the vm via:
[/code]
virsh list --all   

Code Select Expand


Start the vm and connect to the vm's console
[code]
virsh start opnsense
virsh console opnsense

Run through the Opnsense install process with the following interface setup adjusted for your needs:

Code Select Expand


WAN  - vtnet0
LAN   - vtnet1
DMZ1 - vtnet2
DMZ2 - vtnet3


At this point you should have a functional system.  You can break from the virsh via <ctrl>]. I'm guessing you'll want Opnsense to start automatically when the box reboots.  You configure this via:

Code Select Expand


virsh autostart opnsense

Just to make sure qeme is running on startup

Code Select Expand


systemctl enable libvirtd

If you need to change the amount of RAM at some point you can use:

Code Select Expand


virsh setmaxmem opnsense 7G --config
virsh setmem opnsense 7G --config

My box sits in a basement closet without a monitor and keyboard.  I like to use a serial port to access the box.  Here's a picture of the usb ttl connected to the H3+ pins.  I believe they had the transmit and receive data ports backward on the Hard Kernal site:
   
I couldn't get the picture to post here.   I have it listed as an attachment now, we'll see if the attachment get's posted. The pins used are 4,6,8 and 10 on the H3+ board.  The wires are connected in the exact same order on the USB to TTL board as they are on the H3+.  The wire to the left is light brown and then the next wire over is orange.

To make this work you need to modifiy some lines in /etc/default/grub on the Debian host image:

Code Select Expand


GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200n8 console=tty0 systemd.show_status=1"

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"

In the above  GRUB_CMDLINE_LINUX_DEFAULT line I removed the quiet parameter because I want to get some indication the system is booting.  With quiet you see a few initial lines but then nothing until you get the login prompt.  Removing quiet gives you all the kernal lines as the system is booting.

Once you modify the file you need to put the changes in place via:

Code Select Expand


sudo update-grub

You should probably not allow ssh login to the Debian image.  That can be fixed via:

Code Select Expand


sudo echo  PermitRootLogin no >> /etc/ssh/sshd_config
systemctrl restart sshd


I believe that's all I needed to do to get a working Opnsense system running on the qemu kvm hypervisor.   My belief is the hypervisor uses about 3-5% of CPU, but that's just an estimate.