[Solved] Routing firewall traffic over VPN interfaces

[KeyHand]

I'm in the process of setting up a new OPNsense installation and have so far configured

• several subnets on multiple VLANs
• a number of OpenVPN and WireGuard interfaces in client mode
• firewall and NAT rules to enable selective traffic routing for hosts in the VLAN subnets over the VPN links

For hosts on the various LAN segments, everything is working as it should.  The problem is getting OPNsense itself to use these interfaces for outbound traffic, with the specific use case of having Unbound use the WireGuard and OpenVPN interfaces for all outbound requests; something I have working in other pfSense installations.

As the traffic is originating from the firewall itself, no outbound NAT should be required.  However, when troubleshooting this issue, I tried creating NAT rules and it did not made a difference.  The firewall logs do not show any traffic explicitly being blocked, not even a 'default deny'.  I even turned on logging for every single firewall rule in the system, which only resulted in a wall of 'let out anything from firewall host itself' messages.

`pftop` shows the following in these scenarios:

1. Configuring Unbound to explicitly use the WAN interface, then running `drill google.com` on the firewall

Code: [Select]

udp      Out 127.0.0.1:58049          127.0.0.1:53               MULTIPLE:SINGLE       00:00:13  00:00:17        2      116
udp      Out <WAN_IP>:24639        198.97.190.53:53           MULTIPLE:SINGLE       00:00:08  00:00:22        2      802
udp      Out <WAN_IP>:62446        202.12.27.33:53            MULTIPLE:SINGLE       00:00:05  00:00:25        2     1181
udp      Out 127.0.0.1:36195          127.0.0.1:53               MULTIPLE:SINGLE       00:00:04  00:00:26        2      164
udp      Out 127.0.0.1:54392          127.0.0.1:53               MULTIPLE:SINGLE       00:00:37  00:00:00        2      136
udp      Out <WAN_IP>:29434        193.0.14.129:53            MULTIPLE:SINGLE       00:00:05  00:00:25        2     1181
udp      Out <WAN_IP>:4269         198.97.190.53:53           MULTIPLE:SINGLE       00:00:05  00:00:25        2      701
udp      Out <WAN_IP>:24497        192.112.36.4:53            MULTIPLE:SINGLE       00:00:05  00:00:25        2      701
udp      Out <WAN_IP>:50107        65.22.160.17:53            MULTIPLE:SINGLE       00:00:05  00:00:25        2      415

2. Configuring Unbound to explicitly use the the OpenVPN and WireGuard interfaces, then running `drill google.com` on the firewall

Code: [Select]

PR       DIR SRC                      DEST                             STATE                AGE       EXP     PKTS    BYTES
udp      Out <OVPN1_IP>:60017           193.0.14.129:53              SINGLE:NO_TRAFFIC   00:00:15  00:00:45        1       56
udp      Out <OVPN2_IP>:42628           192.5.5.241:53               SINGLE:NO_TRAFFIC   00:00:10  00:00:50        1       56
udp      Out <WG1_IP>:40007           192.33.4.12:53               SINGLE:NO_TRAFFIC   00:00:09  00:00:51        1       56
udp      Out <OVPN1_IP>:53387           199.7.91.13:53               SINGLE:NO_TRAFFIC   00:00:09  00:00:51        1       56
udp      Out <OVPN1_IP>:57877           199.7.83.42:53               SINGLE:NO_TRAFFIC   00:00:07  00:00:53        1       56
udp      Out <OVPN2_IP>:8751            199.7.83.42:53               SINGLE:NO_TRAFFIC   00:00:06  00:00:54        1       56
udp      Out <OVPN1_IP>:18903           199.9.14.201:53              SINGLE:NO_TRAFFIC   00:00:06  00:00:54        1       56
udp      Out <OVPN2_IP>:10430           192.5.5.241:53               SINGLE:NO_TRAFFIC   00:00:05  00:00:55        1       56
udp      Out <OVPN2_IP>:63408           192.5.5.241:53               SINGLE:NO_TRAFFIC   00:00:04  00:00:56        1       56
udp      Out <OVPN2_IP>:44057           202.12.27.33:53              SINGLE:NO_TRAFFIC   00:00:04  00:00:56        1       56
udp      Out <WG1_IP>:52863           198.97.190.53:53             SINGLE:NO_TRAFFIC   00:00:03  00:00:57        1       56
udp      Out <OVPN1_IP>:54187           202.12.27.33:53              SINGLE:NO_TRAFFIC   00:00:03  00:00:57        1       56
udp      Out <WG1_IP>:36030           192.58.128.30:53             SINGLE:NO_TRAFFIC   00:00:01  00:00:59        1       56
udp      Out <WG1_IP>:43964           192.203.230.10:53            SINGLE:NO_TRAFFIC   00:00:00  00:01:00        1       56
udp      Out <WG1_IP>:16896           198.41.0.4:53                SINGLE:NO_TRAFFIC   00:00:14  00:00:46        1       56
udp      Out <WG1_IP>:44421           198.41.0.4:53                SINGLE:NO_TRAFFIC   00:00:14  00:00:46        1       56
udp      Out <OVPN1_IP>:51771           192.58.128.30:53             SINGLE:NO_TRAFFIC   00:00:13  00:00:47        1       56
3. Running `curl -L http://1.1.1.1` on the firewall

Code: [Select]

tcp      Out <WAN_IP>:5919         1.1.1.1:80               FIN_WAIT_2:FIN_WAIT_2   00:00:04  00:01:26       10      938

4. Running `curl -L --interface ovpnc1 http://1.1.1.1` on the firewall

Code: [Select]

tcp      Out <OVPN_IP>:24788           1.1.1.1:80                 SYN_SENT:CLOSED       00:00:07  00:00:26        3      180

Have I overlooked a simple configuration option?  If so, I would really appreciate if someone could point me in the right direction.

[KeyHand]

After some more digging, the culprit seems to be (the confusingly named) Disable force gateway.  The description text ("Disable automatic rules which force local services to use the assigned interface gateway.") gives a better idea of what it does.

By default it's checked, meaning the firewall is restricted to only sending traffic through the default gateway.  Unchecking the option results in several rules being created which allow traffic egress through all interfaces with an upstream gateway.

Perhaps the name of the option could be changed.  Or, at the very least, make the firewall logging more obvious when this option is enabled.  Currently there doesn't seem to be any indication whatsoever that traffic is being blocked.

Now the next problem: a race condition between Unbound and OpenVPN/WireGuard where unbound.conf gets generated before the VPN interfaces come up...