When my WAN fails over and fails back it doesn't clear firewall states so traffic still tries to use the WAN that was previously routing the traffic. This is most noticeable when my primary WAN comes back online and traffic still flows through my backup WAN because the states still exist and the network is still functional, so it's not like it's going to have any TCP RSTs or timeouts that push the traffic back to my primary WAN.
Is there a solution to this that I'm not aware of?
There is this:
https://docs.opnsense.org/manual/how-tos/multiwan.html#failover-and-failback-states
Aha! I had Failover States enabled, but not Failback States. I don't know why you wouldn't want that as the default behavior, though.