Firewall rule for SOCKs not working the way i would expect

[WonderFrank]

Hi all
I have a host in my network, we'll call it 192.168.0.5. On my OpnSense firewall i have 2 gateways, one is my modem (default) and the 2nd is a OpenVPN connection elsewhere.

What i want is for all traffic from 192.168.0.5 on port 1080 to go over my default gateway, and all other traffic to go over the VPN. This would mean that traffic going to a SOCKS proxy (1080) would go straight out of the network whereas traffic going elsewhere would go over the VPN.

I have 3 rules to try and achieve that:
Rule 1:
Direction: In
Protocol: IPv4 TCP/UDP
Source: 192.168.0.5
Source port: *
Destination: *
Destination Port: 1080
Gateway: WAN_DHCP

Rule 2:
Direction: In
Protocol: IPv4 *
Source: 192.168.0.5
Source Port: *
Destination: *
Destination Port *
Gateway: VPN

I have a 3rd rule that blocks any traffic from that host that doesnt go over the VPN (this works fine).

The trouble is that this only works for a few moments. Initially SOCKS traffic is routed straight out the network but after 5 min of connection all traffic goes back over the VPN. The only way to fix that is to reset the states and then it works again for 5 min or so and then back to normal.

Any ideas?

[WonderFrank]

I tried switching to sloppy state and it made no difference on the assumption that the data was asymmetric.
Ive checked ntopng and i can see all the traffic coming from the host is going on point 1080

[WonderFrank]

I trust this is something that no one has seen before?