I know this is crazy, but I can replicate it. I have a rule that forces certain IP's out WAN2 and it works perfectly (just setting the gateway to WAN2 in the rule).
When i run ANY speedtest-cli from these clients (all ethernet) on a laptop/desktop everything works as expected when the rule is enabled. But from a raspberry pi running raspberry pi os aarch64, I get super slow speeds. When i disable the rule, speeds on the pi revert back to normal on WAN1 or WAN2.
So I am only getting these super slow speeds when I force WAN2 on a raspberry pi. I have tried both a pi3b and pi4. It's not the pi computing power, memory, or sd card since it works fine if I force failover by unplugging WAN1 and disabling the rule.
Any ideas on how I can further troubleshoot?
Just working from the troubleshooting you've already done;
If only WAN1 is physically available, with the "rule disabled" and the speeds are "fine" then it stands to reason that the rule is forcing it into a scenario that either affects the routing or behavior the NIC/nic drivers processing.
1. On the rasberry-pi, with rules enabled, is there any meaningful change in total CPU utilization on the PI?
2. Can you share specifically what rules you've implemented to force clients to WAN2?
Rule is Below... Glances shows similar utilization of CPU etc iin both scenarios, with plenty of headroom.
Firewall: Rules: LAN
Action: Pass
Quick: Checked
Interface: LAN
Direction: IN
Protocol: Any
Source: Starlink Only (an alias for the static IP)
Destination: Any
Gateway: WAN_Starlink (i.e. WAN2)
Ok from what I can tell, that's just a firewall rule, not a routing rule. It's simply allowing traffic. That doesn't "force" traffic.
Will need to understand more about your LAN configuration. Are you using VLans? What does the network/gateway situation look like? What method are you using to redirect traffic?
I believe using the firewall rule is the reccomended way of forcing devices out a specific WAN port.
Take a look at the following guidance:
https://docs.opnsense.org/manual/multiwan.html
Or more simply; just a multi-gateway setup:
https://docs.opnsense.org/manual/gateways.html
@axsdenied That is all setup correctly... I understand Multi-Wan and use it a LOT at many different locations. I am purposely forcing a raspberry pi to utilize WAN2 for monitoring reasons (and this works) however it greatly reduces the speed. If I use the exact same setup with a linux vm, the speeds are not reduced. The raspberry speed is fine on the default gateway when you remove the rule, its only slow when you are forcing WAN2 (while the linux vm is not).
Understood, but you have to understand, when attempting to help someone you can't take anything as fact, even if it 100% is fact. I can't assume anything :)
Part of the reason I was asking how the change happens, again a firewall rule is simply an open door, not a traffic router, is potentially something silly happening like IP conflicts.
i.e some IP or secondary IP is static, when it switches over it conflicts with the Pi, or it with somethign else, when it's routing through WAN2. Which can show up like "slowness" and not impact anything else.
Interesting theory, just checked and pi gets its IP from a static opnsense DHCP, only one entry in DHCP leases, and ARP table for that IP.