Sometimes my firewall does async routing and it breaks wireguard on LAN

[lerra]

Hi, quite often I have a problem with my wireguard clients not being able to connect when they are on the LAN (igb0), especally switching to LAN or from the WAN (igb1).

In the android client I see the logs are complaining on the handshake, I either turn off wifi and use mobile internet or turn off wireguard and then it works.

I identified that there is no issue with wireguard, the issue is with opnsens and it does some wierd async routing and I think its connected with an old state or something but it holds on to it forever.

When I can reproduce the error I see the following,

WAN (igb1) & LAN (igb0)

I can see the client sending wireguard traffic to the firewall

root@fw1:~ # tcpdump -n -i igb0 port 31343
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 192.168.1.98.31343 > 88.41.98.3.31343: UDP, length 96
IP 192.168.1.98.31343 > 88.41.98.3.31343: UDP, length 96

The firewall response back the traffic to WAN interface
root@fw1:~ # tcpdump -n -i igb1 port 31343
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 88.41.98.3.31343 > 192.168.1.98.31343: UDP, length 160
IP 88.41.98.3.31343 > 192.168.1.98.31343: UDP, length 160

Sometime it just works without any problem and then all the traffic can be seen only on the LAN interface
root@fw1:~ # tcpdump -n -i igb0 port 31343
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP 88.41.98.3.31343 > 192.168.1.98.31343: UDP, length 160
IP 88.41.98.3.31343 > 192.168.1.98.31343: UDP, length 160
IP 192.168.1.98.31343 > 88.41.98.3.31343: UDP, length 96
IP 192.168.1.98.31343 > 88.41.98.3.31343: UDP, length 96
IP 192.168.1.98.31343 > 88.41.98.3.31343: UDP, length 128


Here is my ruleset matching the wireguard port
root@fw1:~ # pfctl -sr|grep 31343
pass in on igb1 reply-to (igb1 88.41.98.1) inet proto udp from any to any port 31343:45845 keep state label "3f93bfdaf0522bab14245550032cb1b4"
pass in on igb1 inet6 proto udp from any to any port 31343:45845 keep state label "3f93bfdaf0522bab14245550032cb1b4"
pass in quick on igb0 inet proto udp from <rfc1918> to (self) port = 31343 keep state label "3d7b6eba4b03445e9f08da21278a9e39"
pass in quick on igb0 inet6 proto udp from <rfc1918> to (self) port = 31343 keep state label "3d7b6eba4b03445e9f08da21278a9e39"


Do you have any ideas? I spent so much time on this issue and I'm stuck now. Thanks