Upgraded a protectli VP2410 to a VP2420 and cross interface performance hit.

[dmax68]

Hey all, first time on the forum but not a first time user. been on opnsense since it forked from pfsense.
got an issue that has me stumped. I've been running a Protectli VP2410 for the last 3 years and it has been rock solid stable. Not running Suricata or IDP/IPS or anything special. Just a dual wan, a LAN, VLAN and a DMZ (Nextcloud) with some rules and DHCP on the LAN/VLAN. Both of my ISP's announced that 2g fiber was coming soon so I splurged and picked up a VP2420 to get the 2.5g interfaces. I did not restore my config from the old unit. I just documented my settings and entered them into a fresh opnsense install on the new unit. Everything works great except traffic to/from the DMZ. Haven't tried the VLAN as that one is wireless so I expect a perfomance hit
1. from the public internet, uploads/downloads to the nextcloud instance in the DMZ (192.168.11.23) runs between 1 and 2 mbps (on the old device it easily hit 700+mbps)(I know this as this is how I moved ISO's from home to work)
2. from my workstation (192.168.10.167) to the DMZ machine (192.168.11.23) iperf is showing 1.47 Mbits/sec. the same test to a file server (192.168.10.20) in the same lan segment as my workstation is 826 Mbits/sec.
3. From a test machine I spun up in the DMZ (192.168.11.20) to the target machine (192.168.11.23) I am seeing 9.55Gbit/sec (same hypervisor host)
I just today stood up the iperf tests but last week I put the old VP2410 back in play for a day and all the performance issues went away.
What am I missing?

[OPNenthu]

All of your cases where the speeds are slow seem to involve the hypervisor (your DMZ host).  I saw something similar on a Linux machine that is acting as a VM host and which connects to a switch port that is a trunk.  I had to try different combinations of software bridge + VLAN interface setups on the linux host to resolve the abysmally slow transfers. I was only getting several hundred kbps at one point.  We have the i226-v NIC in common; could be an issue with that.

I would eliminate the DMZ host from testing to try and isolate that as the source of the problem.  Since you already confirmed the transfer between physical hosts on the same VLAN is OK, the next thing to try is two physical hosts on different VLANs.  If inter-VLAN routing is OK then I think you can at least be sure the router is functioning OK.  Then focus back on the hypervisor setup.

[Patrick M. Hausen]

Is the OPNSense itself virtualised? If yes are you using VirtIO (vtnet) interfaces?

[meyergru]

You said you did not carry over the configuration. I would try to do that and just reassign the interfaces. That way, you would be sure that there is no setting that you once had and now forgot.