Traffic through VIP is roughly half that of traffic to real IP, any ideas why?

[dispo2]

I have two opnsense (26.7.3_8) running on different tin as a HA pair (active / standby)

I have a 500/50 Mbit internet connection and I have noticed (via speedtest.net) that my test results when using the real IP address as gateway are full 500Mbit but when I use the VIP as gateway the test speed max'es out at roughly half that.

I am trying to understand why as I did not think running a VIP / running traffic through a VIP added any major layer of additional processing.

tl;dr

These are proxmox hosted VM's with all of the recommendations (that I can find) enabled, cpu=host, hardware offloads disabled, hw.ibrs_disable=1, vm.pmap.pti=0, 8Gb RAM.

Hardware is quadcore Intel(R) Celeron(R) J6412 @ 2.00GHz and quadcore Intel(R) Pentium(R) N6415 @ 1.20GHz
Network is 4xIntel i226-V 2.5GbE, interfaces are configured as virtIO

LAN and WAN interfaces are their own NIC, not bridges

I have a virtual IP configured on the LAN via CARP, this is used for the gateway address in DHCP for seemless failover for LAN devices.
I have a virtual IP configured on the WAN for outbound NAT

These were originally 2 vCPU running the default v86-64-v2-aes profile with the default cpuunits. I have changed these to 3 vCPU type host with cpuunits=2048. This has helped a bit but has not significantly increased throughput through the VIP.

Using the real IP as the gateway address for a LAN device gets full internet speed so the hardware is obviously capable of it.
I cannot understand why, when using the VIP as gateway address, I get roughly half that.

When running the speed tests with the real ip of the active opnsense as gateway I can see traffic on the active firewall (as expected) and no traffic on the standby firewall (as expected).
Changing the gateway to the standby firewall reverses which firewall shows traffic (as expected).
Changing the gateway to the VIP shows the traffic via the active (CARP master) firewall (as expected).

In the above changes the only difference is the target gateway IP on the LAN side, all required NAT (at the WAN) is the same, all required firewall rule matching is the same, all hardware abilities and settings are the same.

So I cannot understand why the VIP is a bottleneck. Increasing cpu helps which implies an additional overhead for the VIP path but never solves it.

Any suggestions / ideas / tuneables I have missed ?