[solved] Broken OpenVPN policy routing

[szty0pa]

Updating to 22.1 i noticed that my firewall rules stopped working as they should (they were fine up to the 22.1/FreeBSD 13 upgrade).

If i have only a single firewall rule like:

• allow in quick ipv4+6 * [local addresses alias:*] to [local addresses alias:*] through [default] gw

i can access my local machines just fine, but if i add another rule below this like:

• allow in quick ipv4+6 * [local addresses alias:*] to [local addresses alias:*] through [default] gw
• allow in quick ipv4 tcp/udp [interface net:*] to [any:*] through [vpn] gw

then the connection breaks as according to the firewall logs the router tries to route its own [interface address] through the [vpn gw]. I can see blocked outgoing packets on the vpn interface with a destination of the router's own originating interface address. (Say [interface net] is 192.168.1.0/24, [interface address] is 192.168.1.1, then i see blocked outgoing traffic on the vpn interface with destination of 192.168.1.1.)
The system routing table looks fine (though the whole 'Use' column has 'NaN' values), and all connections work from the router itself (which is not firewalled).

Has anyone also experienced this? How should it be fixed without having to have an [allow any to any through default gw] rule, which obviously makes routing and firewalling pointless?

[Fright]

Updating to 22.1 i noticed that my firewall rules stopped working as they should

what version did you upgrade from?

to [any:*] through [vpn] gw
then the connection breaks as according to the firewall logs the router tries to route its own [interface address] through the [vpn gw]

by what rule did the firewall allow this traffic before adding a new rule?

[szty0pa]

what version did you upgrade from?

At first from 21.7.7 to 21.7.8 then to 22.1 back to back.

by what rule did the firewall allow this traffic before adding a new rule?

This traffic i see is mostly my pc (and other devices on the same network segment) trying to connect to the firewall for DNS and NTP (ports 53 and 123). The first example rule did allow this as the [local addresses alias] contains all unicast, multicast and local(host) addresses used on my networks. The strange thing is that the firewall tries to use the vpn gateway in the second example rule to route traffic to itself.

[szty0pa]

It might actually not be the firewall's fault (my rules are in place and are working well for about 5 years now), but some trouble with the automatic gateway selection and/or openvpn.
I have a static route to my cable modem in the routing table through the physical interface the modem is connected to. If i disable the openvpn gateway i can ping the cable modem all right, but if i enable the openvpn gateway (the static route is not set up through it nor am i pulling routes from the openvpn server!), i cannot ping the cable modem, as the router would send the packets to it through the openvpn interface despite the static route, gateway priority and default route setting anyway!

[Fright]

but some trouble with the automatic gateway selection and/or openvpn

looks like it, yes
sorry, It's hard for me to guess without seeing the rules. and even better rules for the "working" and "current" configurations.
it may be related to https://github.com/opnsense/core/issues/5329 but imho it should have become noticeable earlier

[szty0pa]

Sorry for the long delay, work happened... :(
I have made a secret-redacted config xml to show my config, and in the same time i figured out it was actually an OpenVPN bug, it was just a coincidence that it's rule came after the ones it broke in the firewall.
v22.1.1 fixed it. :)