I'm in the process of setting up a new OPNsense installation and have so far configured
- several subnets on multiple VLANs
- a number of OpenVPN and WireGuard interfaces in client mode
- firewall and NAT rules to enable selective traffic routing for hosts in the VLAN subnets over the VPN links
For hosts on the various LAN segments, everything is working as it should. The problem is getting OPNsense itself to use these interfaces for outbound traffic, with the specific use case of having Unbound use the WireGuard and OpenVPN interfaces for all outbound requests; something I have working in other pfSense installations.
As the traffic is originating from the firewall itself, no outbound NAT should be required. However, when troubleshooting this issue, I tried creating NAT rules and it did not made a difference. The firewall logs do not show any traffic explicitly being blocked, not even a 'default deny'. I even turned on logging for every single firewall rule in the system, which only resulted in a wall of 'let out anything from firewall host itself' messages.
`
pftop` shows the following in these scenarios:
1. Configuring Unbound to explicitly use the WAN interface, then running `
drill google.com` on the firewall
udp Out 127.0.0.1:58049 127.0.0.1:53 MULTIPLE:SINGLE 00:00:13 00:00:17 2 116
udp Out <WAN_IP>:24639 198.97.190.53:53 MULTIPLE:SINGLE 00:00:08 00:00:22 2 802
udp Out <WAN_IP>:62446 202.12.27.33:53 MULTIPLE:SINGLE 00:00:05 00:00:25 2 1181
udp Out 127.0.0.1:36195 127.0.0.1:53 MULTIPLE:SINGLE 00:00:04 00:00:26 2 164
udp Out 127.0.0.1:54392 127.0.0.1:53 MULTIPLE:SINGLE 00:00:37 00:00:00 2 136
udp Out <WAN_IP>:29434 193.0.14.129:53 MULTIPLE:SINGLE 00:00:05 00:00:25 2 1181
udp Out <WAN_IP>:4269 198.97.190.53:53 MULTIPLE:SINGLE 00:00:05 00:00:25 2 701
udp Out <WAN_IP>:24497 192.112.36.4:53 MULTIPLE:SINGLE 00:00:05 00:00:25 2 701
udp Out <WAN_IP>:50107 65.22.160.17:53 MULTIPLE:SINGLE 00:00:05 00:00:25 2 415
2. Configuring Unbound to explicitly use the the OpenVPN and WireGuard interfaces, then running `
drill google.com` on the firewall
PR DIR SRC DEST STATE AGE EXP PKTS BYTES
udp Out <OVPN1_IP>:60017 193.0.14.129:53 SINGLE:NO_TRAFFIC 00:00:15 00:00:45 1 56
udp Out <OVPN2_IP>:42628 192.5.5.241:53 SINGLE:NO_TRAFFIC 00:00:10 00:00:50 1 56
udp Out <WG1_IP>:40007 192.33.4.12:53 SINGLE:NO_TRAFFIC 00:00:09 00:00:51 1 56
udp Out <OVPN1_IP>:53387 199.7.91.13:53 SINGLE:NO_TRAFFIC 00:00:09 00:00:51 1 56
udp Out <OVPN1_IP>:57877 199.7.83.42:53 SINGLE:NO_TRAFFIC 00:00:07 00:00:53 1 56
udp Out <OVPN2_IP>:8751 199.7.83.42:53 SINGLE:NO_TRAFFIC 00:00:06 00:00:54 1 56
udp Out <OVPN1_IP>:18903 199.9.14.201:53 SINGLE:NO_TRAFFIC 00:00:06 00:00:54 1 56
udp Out <OVPN2_IP>:10430 192.5.5.241:53 SINGLE:NO_TRAFFIC 00:00:05 00:00:55 1 56
udp Out <OVPN2_IP>:63408 192.5.5.241:53 SINGLE:NO_TRAFFIC 00:00:04 00:00:56 1 56
udp Out <OVPN2_IP>:44057 202.12.27.33:53 SINGLE:NO_TRAFFIC 00:00:04 00:00:56 1 56
udp Out <WG1_IP>:52863 198.97.190.53:53 SINGLE:NO_TRAFFIC 00:00:03 00:00:57 1 56
udp Out <OVPN1_IP>:54187 202.12.27.33:53 SINGLE:NO_TRAFFIC 00:00:03 00:00:57 1 56
udp Out <WG1_IP>:36030 192.58.128.30:53 SINGLE:NO_TRAFFIC 00:00:01 00:00:59 1 56
udp Out <WG1_IP>:43964 192.203.230.10:53 SINGLE:NO_TRAFFIC 00:00:00 00:01:00 1 56
udp Out <WG1_IP>:16896 198.41.0.4:53 SINGLE:NO_TRAFFIC 00:00:14 00:00:46 1 56
udp Out <WG1_IP>:44421 198.41.0.4:53 SINGLE:NO_TRAFFIC 00:00:14 00:00:46 1 56
udp Out <OVPN1_IP>:51771 192.58.128.30:53 SINGLE:NO_TRAFFIC 00:00:13 00:00:47 1 563. Running `
curl -L http://1.1.1.1` on the firewall
tcp Out <WAN_IP>:5919 1.1.1.1:80 FIN_WAIT_2:FIN_WAIT_2 00:00:04 00:01:26 10 938
4. Running `
curl -L --interface ovpnc1 http://1.1.1.1` on the firewall
tcp Out <OVPN_IP>:24788 1.1.1.1:80 SYN_SENT:CLOSED 00:00:07 00:00:26 3 180
Have I overlooked a simple configuration option? If so, I would really appreciate if someone could point me in the right direction.
After some more digging, the culprit seems to be (the confusingly named) Disable force gateway (https://docs.opnsense.org/manual/firewall_settings.html#disable-force-gateway). The description text ("Disable automatic rules which force local services to use the assigned interface gateway.") gives a better idea of what it does.
By default it's checked, meaning the firewall is restricted to only sending traffic through the default gateway. Unchecking the option results in several rules being created (https://github.com/opnsense/core/blob/stable/21.1/src/etc/inc/filter.lib.inc#L594) which allow traffic egress through all interfaces with an upstream gateway.
Perhaps the name of the option could be changed. Or, at the very least, make the firewall logging more obvious when this option is enabled. Currently there doesn't seem to be any indication whatsoever that traffic is being blocked.
Now the next problem: a race condition between Unbound and OpenVPN/WireGuard where unbound.conf gets generated before the VPN interfaces come up...