Hi,
I am trying to allow a Wireguard peer ("Mobile phone") access to one of our servers ("HPC-1") which is located behind the OPNsense. Below is a scheme of my current setup. I have combined the interfaces ax0, ax1, igc0 and igc1 into a transparent filtering bridge ("bridge0") and created the Wireguard interface "WG_HLB" to allow externals access to server HPC-1.
When I try to ping from "Mobile phone" (10.200.0.4) to "HPC-1" (10.93.151.5), I can see that everything works fine until the ICMP response enters bridge0 because then it gets forwarded to the "WAN" interface instead of "WG_HLB".
The same happens when I try to ping from "HPC-1" to "Mobile phone", so it happens regardless of being an ICMP request or a reply.
During my research I often stumbled across the "reply-to" option in the advanced firewall rules and tried to create a gateway for the Wireguard interface (which did not feel right because in that VPN network there is no gateway which I could use), so that I can then set reply-to to the newly created gateway. But it didn't help :-( It seems to be impossible to forward a packet to the Wireguard interface. When the packet originates from the firewall itself, everything works fine, but in case of forwarded packets, it behaves strangely.
+------------------+
| bridge0 |
|10.93.151.2/24 |
|192.168.42.254/24 |
+----------+ | |
| Switch 2 |>-----<|ax0 (SWITCH2) |
+----------+ <|ax1 (SWITCH1) |
| <|igc0 (LAN) |
| | | +-------------+
+-------------+ | igc1 (WAN)|>--------------<| Gateway |
| HPC-1 | | | | 10.93.151.1 |
| 10.93.151.5 | +------------------+ +-------------+
+-------------+ | |
| | +--------------+
| wg0 (WG_HLB)|>--------------<| Mobile phone |
| 10.200.0.1/24| | 10.200.0.4 |
+------------------+ +--------------+
I hope you can help me and point me in the right direction. I am used to and do understand how Linux iptables works but pf is new for me and I don't know how routing decisions are made. Thanks in advance!
- Vincent
Here are some additional infos:
OPNsense version:
24.1.10_8
Output of route get 10.200.0.4:
route to: 10.200.0.4
destination: 10.200.0.4
fib: 0
interface: wg0
flags: <UP,HOST,DONE,STATIC>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1420 1 0
Relevant firewall rules (bridge0):
IPv4 * hpc_1 * WireGuard (Group) net * * * Test
Relevant firewall rules (WG_HLB) (wg_ems_1 = "Mobile phone"):
IPv4 * wg_ems_1 * * * * * Test
Routes:
ipv4 default 10.93.151.1 UGS NaN 1500 bridge0 WAN_LAN_BRIDGE
ipv4 9.9.9.9 10.93.151.1 UGHS NaN 1500 bridge0 WAN_LAN_BRIDGE
ipv4 10.93.151.0/24 link#11 U NaN 1500 bridge0 WAN_LAN_BRIDGE
ipv4 10.93.151.2 link#11 UHS NaN 16384 lo0 Loopback
ipv4 10.200.0.0/24 link#10 U NaN 1420 wg0 WG_HLB
ipv4 10.200.0.1 link#10 UHS NaN 16384 lo0 Loopback
ipv4 10.200.0.2 link#10 UHS NaN 1420 wg0 WG_HLB
ipv4 10.200.0.3 link#10 UHS NaN 1420 wg0 WG_HLB
ipv4 10.200.0.4 link#10 UHS NaN 1420 wg0 WG_HLB
ipv4 127.0.0.1 link#6 UH NaN 16384 lo0 Loopback
ipv4 192.168.0.0/24 link#3 U NaN 1500 igc2 MGMT
ipv4 192.168.0.1 link#3 UHS NaN 16384 lo0 Loopback
ipv4 192.168.42.0/24 link#11 U NaN 1500 bridge0 WAN_LAN_BRIDGE
ipv4 192.168.42.254 link#11 UHS NaN 16384 lo0 Loopback
ipv6 ::1 link#6 UHS NaN 16384 lo0 Loopback
ipv6 fe80::%lo0/64 link#6 U NaN 16384 lo0 Loopback
ipv6 fe80::1%lo0 link#6 UHS NaN 16384 lo0 Loopback
Output of pfctl -s all (only section "FILTER RULES")
The output below contains shows rules which are also configured in the firewall but should not relate to this problem. I did not remove them from the output because, who knows, maybe the problem is indeed caused by one of those rules.
FILTER RULES:
scrub in all fragment reassemble
block drop in log on ! bridge0 inet from 10.93.151.0/24 to any
block drop in log on ! bridge0 inet from 192.168.42.0/24 to any
block drop in log on ! igc2 inet from 192.168.0.0/24 to any
block drop in log inet from 192.168.0.1 to any
block drop in log inet from 10.93.151.2 to any
block drop in log inet from 192.168.42.254 to any
block drop in log on ! wg0 inet from 10.200.0.0/24 to any
block drop in log inet from 10.200.0.1 to any
pass in log quick on lo0 inet6 all flags S/SA keep state label "a5d4bbc7020fdea51eaec95d0484424f"
block drop in log quick inet6 all label "5d75d96ba523ccd456ab15a327c7fed5"
block drop in log inet all label "02f4bab031b57d1e30553ce08e0ec131"
block drop in log inet6 all label "02f4bab031b57d1e30553ce08e0ec131"
pass in log quick inet6 proto ipv6-icmp all icmp6-type unreach keep state label "1d245529367b2e34eeaff16086aeafe9"
pass in log quick inet6 proto ipv6-icmp all icmp6-type toobig keep state label "1d245529367b2e34eeaff16086aeafe9"
pass in log quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state label "1d245529367b2e34eeaff16086aeafe9"
pass in log quick inet6 proto ipv6-icmp all icmp6-type neighbradv keep state label "1d245529367b2e34eeaff16086aeafe9"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type echoreq keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type echoreq keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type echorep keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type echorep keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type routersol keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type routersol keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type routeradv keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type routeradv keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type neighbrsol keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type neighbrsol keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type neighbradv keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type neighbradv keep state label "acdbb900b50d8fb4ae21ddfdc609ecf8"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echoreq keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echoreq keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbrsol keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbrsol keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbradv keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbradv keep state label "42e9d787749713a849d8e92432efdfaa"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type echoreq keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routersol keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routeradv keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbrsol keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbradv keep state label "8752fca75c6be992847ea984161bd3f1"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type echoreq keep state label "71dd196398b3f1da265dbd9dcad00e70"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type routersol keep state label "71dd196398b3f1da265dbd9dcad00e70"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type routeradv keep state label "71dd196398b3f1da265dbd9dcad00e70"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type neighbrsol keep state label "71dd196398b3f1da265dbd9dcad00e70"
pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type neighbradv keep state label "71dd196398b3f1da265dbd9dcad00e70"
block drop in log quick inet proto tcp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5"
block drop in log quick inet proto udp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5"
block drop in log quick inet6 proto tcp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5"
block drop in log quick inet6 proto udp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5"
block drop in log quick inet proto tcp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5"
block drop in log quick inet proto udp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5"
block drop in log quick inet6 proto tcp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5"
block drop in log quick inet6 proto udp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5"
pass log quick inet6 proto carp from any to ff02::12 keep state label "cf439d72ef4d245e8ad4a1405df1f665"
pass log quick inet proto carp from any to 224.0.0.18 keep state label "2ffa978d51f7b3fbc9000c2895106ee7"
block drop in log quick proto tcp from <sshlockout> to (self) port = ssh label "669143f420c3ab4118bcb0bf4b5fd823"
block drop in log quick proto tcp from <sshlockout> to (self) port = https label "6baefc2a9cf2536834c092a51134a45c"
block drop in log quick from <virusprot> to any label "8e367e2f9944d93137ae56d788c5d5e1"
pass in log quick on bridge0 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "6535985f9b71fbf131e835573f0fbd39"
pass in log quick on bridge0 proto udp from any port = bootpc to (self) port = bootps keep state label "18d2197f62c02e2805760125772e8084"
pass out log quick on bridge0 proto udp from (self) port = bootps to any port = bootpc keep state label "0c8a2f76a6bf525b6db290a7b5d9452b"
block drop in log quick on bridge0 inet from <bogons> to any label "b7742298e96ac942013f7c8069844c9e"
pass in quick on lo0 all no state label "7535c94082e72e2207679aadb26afd92"
pass out log all flags S/SA keep state allow-opts label "fae559338f65e11c53669fc3642c93c2"
pass in quick on igc2 inet all flags S/SA keep state label "26bc48a00c910a34e1f07d315761ff9b"
pass in quick on bridge0 inet from <vm_nexus> to any flags S/SA keep state label "fc2b181061cf2940897ea395ccf69adb"
pass in quick on bridge0 inet proto udp from any to (self) port = domain keep state label "f067b1faa6b53e5042a3b66bb05888c6"
pass in quick on bridge0 inet proto udp from any to (self) port = ntp keep state label "2d3f3e85f0fc7a76934d9e9878d29611"
pass in quick on bridge0 inet proto icmp from any to (self) keep state label "672b6fdc198e24bb0406173c57dfe700"
pass in quick on bridge0 inet from ! <vlan_151_and_wenger> to any flags S/SA keep state label "c0f040cdb9321d2e436ec789e30f99a8"
pass in quick on bridge0 inet from <vlan_151> to ! <private_networks> flags S/SA keep state label "18050b9753eadebc8a0387465aa28f5d"
pass in quick on bridge0 inet proto tcp from <hpc_1> to <msa> port = microsoft-ds flags S/SA keep state label "c4c49506400e994fc4cf8110cea5f1d8"
pass in quick on bridge0 inet from <hpc_1> to <wenger_sps> flags S/SA keep state label "08c2ea68466fd79b6040d0afcc24c136"
pass in quick on bridge0 inet proto tcp from ! <vlan_151_and_wenger> to <wenger_sps> port = 1020 flags S/SA keep state label "a5237445322cd82ac5e4cbe9231e44f6"
pass in quick on bridge0 inet from <hpc_1> to (wireguard:network) flags S/SA keep state label "4bde604028d750dbd4e908cfdc15293b"
pass in quick on wg0 inet from <wg_ems_1> to any flags S/SA keep state label "4ad65e1831461df4f3db0c2880a6a38f"
Do you think this could be a bug and I should file an issue on GitHub?
What is the default gateway on HPC-1? I'm assuming it's 10.93.151.1. The bridge is not going to intercept and route the way you seem to expect, because the ethernet frames will be directed to your gateway's MAC address...
Thank you!! That was the problem. I misconfigured the firewall to distribute 10.93.151.1 as gateway address via DHCP instead of its own IP (10.93.151.2) and so the firewall's IP stack was never invoked I guess. It works now with updated DHCP settings :-)