Messages

The board sucking is mostly about obnoxious IOMMU grouping and not getting past POST if a flash drive is connected to the front USB ports. It does have AMDv enabled and the CPU type is host.

MTUs were plain boring 1500 so I tried another comparison.

Booting into bare metal openWRT, I reached 880/40 with all offloads disabled and plain old defaults set.  Still slower than macbook direct to the cable modem but faster than anything I've done with opnsense. 

While there I looked at the NIC firmware.  I could only get it from ethtool and it isnt in a versioning that matches what everyone else is talking about.

Code Select Expand

root@OpenWrt:~# ethtool -i eth1
driver: igc
version: 6.6.110
firmware-version: 2023:889d
expansion-rom-version:
bus-info: 0000:07:00.0
Well I have no idea what version firmware the NIC is running at this point, but if I'm reading this right, its from 2023?

Another thing I noticed is the CPU was hardly tickled by any of these tests in bare metal openWRT.  In the OPNsense VM those fans spun up with every test, and I hadn't turned on an IDS/IPS or anything yet.  So this suggests two issues.  One, there's overhead on my VM setup that is way beyond what it should be.  Two bare metal openWRT got a lot closer to line level than anything else but still couldn't reach it either. I threw in an i350 NIC and tested again and with bare metal openWRT with no vlans or offloads got 872/40.

Finally I set up the KVM to use openWRT but otherwise with the same configuration as OPNsense's VM. I ran the speedtest.net page again and this time the fans didnt spin up so I thought I might have stumbled onto something....nope.  CPU cores assigned to that VM were maxed out on the download, 30% on the upload. and the speeds? 647/40.

So our experiments now look like this.
speedtest comparisons:
direct macbook to cable modem: 916/40
opnsense virtualized (with vlans and rules): 699/41
opnsense bare metal (with vlans and rules): 816/39
opnsense bare metal (with vlans and rules and hardware offload fully enabled): 824/40
openwrt bare metal (no vlans, default rules, no offloads): 880/40
openwrt bare metal (no vlans, default rules, HW offloads): 842/40
openwrt virtualized (no vlans, default rules, no offloads): 647/40

YAY! A new low score!

There is obviously something very wrong with my virtualization setup.  OPNsense still had weird losses bare metal, but the virtualization is hammering the CPU while the bare metal isnt.  neither performs well, but openWRT is best bare metal, OPNsense is best virtualized.

This is bizarre.  VirtuIO is not this heavy.

Drunk and tired from Thanksgiving, I went and tried one more thing before stumbling off to bed. I turned off ASPM. To my surprise it made absolutely no difference.

So tomorrow after work I will try the MTU link, different NICs, and using a similarly configured VM running openWRT instead of opnsense.

Maybe this hardware is just cursed. There is no reason my firewalla purple should be running laps around it.

Well macvtap is used on the wan for the vm to ensure a misconfigured host can never claim it. The host does whatever virtuoso tells it. So if MTU is screwy, that could explain why it was messed up in bare metal as well.

I have a spectrum cable internet connection. I'll look at that thread tomorrow after work. See you then.

I can't do passthru on this board because I'm cheap and it sucks. But I did try booting into a live instance of opnsense. It was better but still nowhere near line level.

I did see that.  On the host, I see 4 queues are already enabled on each interface.  I made sure to disable the ipv6 multicast snooping as directed in the link by setting nmcli connection modify i226vLAN ipv6.method "ignore" for the interface and bridge on the host and set the XML config for the VM in virsh edit to add "trustGuestRxFilters='yes'" on the device line for the virtual NICs.  The link you posted said "You can also enable multiqueue on the VM NIC interfaces, especially, if you have multiple threads active. There is no need for enabling this in OpnSense." so I left that alone.

As per this suggestion, I turned it on in OPNsense with the tunables in the RSS guide and set the bits value to 1 as I have only given this VM 2 cores so far.  running the test again, the performance dropped to 636/38.  I had to run it twice because the upload on the first test topped out at 11.

I've set up a fedora server and threw OPNsense into a VM on it.  The system has a ryzen 5800xt CPU and a dual port intel i226v NIC installed.  I have pinned 2 CPU cores to the OPNsense VM and fed 1 thread from each core to the VM.  The WAN port is passed to the VM via direct attachment (bridge right now, though I will probably change that to private) and the LAN port uses linux bridges for the untagged traffic as well as the vlan ports on it.  The plan is to pass them all as separate interfaces into OPNsense, then tell the host to grab an IP on whatever bridge I need it to talk to.

OPNsense is configured right now to only use the WAN and LAN ports.  I havent set up the vlan interfaces yet.  Firewall rules are still defaults.

My interent connection speed is 1000 down/40 up, but in this configuration I can only hit 670 down/40 up on speedtest.net from a client device on the network.  I tried doing this bare metal for comparison and it hit 824/40.  connecting a macbook to the modem directly got me to 916/40. I'm using virtuio since passthrough is just not going to happen with this crappy motherboard but supposedly that should be fine.

I know there's going to be some overhead with this, and I haven't done any performance tuning yet because I'm still testing out basic functionality first, but this seems a bit severe.  I have clearly overlooked something, but I'm stumped as to what I'm missing.

I checked the read me first posts on here.  I saw there were some mitigations I could disable in OPNsense and did try that but no change.  There is a firmware update possibly out there for this NIC but the linux host has custody of the NIC and it's not telling me what version firmware it has on it.  That update seems to be more about stability than performance anyway and I'm getting stable connections, just not performant ones.

What am I overlooking?