Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - maciekish

#1
As per FreeBSD docs this is not supported. Only pass/block is supporter on an L2 bridge. Other OSes like RouterOS claim to be able to do it but FreeBSD cannot. Case closed.
#2
TL;DR
OPNsense (25.7.3_7) is a transparent bridge between a Fiber ONT and my router. I'm selectively policy-routing some destinations (e.g., 4.ident.me) out a WireGuard client (wg0). States show route-to (wg0 10.49.0.1) and outbound NAT on wg0, gateways are online, WG handshakes OK... but flows to those sites time out. From a MacBook that connects to the same WG server (bypassing OPNsense), both IPv4+IPv6 work instantly. On OPNsense I even see a NAT'd SYN with source 10.49.0.2 hit the WAN interface (!), while wg UDP on WAN only shows keepalives. Where is this breaking? I do not want to use OPNsense as my router due to all the nice UniFi integrations with my other gear, it's not an option. OPNsense needs to stay a transparent bridge + PBR VPN. Config file attached.



Topology / role
    •    Path: Fiber ONT ↔ OPNsense (transparent bridge) ↔ UniFi Dream Machine (main LAN router)
    •    OPNsense bridges igc0 (WAN) and igc1 (LAN) as bridge0 and holds a management IP on the bridge.
    •    Goal: keep the box completely transparent L2, but policy-route certain destinations out WireGuard (split-VPN egress).

Interfaces & addressing (current)
    •    bridge0 (OPT1 / "Bridge"): 192.168.100.254/24 (management)
    •    igc1 (LAN/UDM): member of bridge0
    •    igc0 (WAN/ONT): member of bridge0
         •    DHCPv4: 192.168.100.101/24
         •    DHCPv6: 2001:1a10:192e:ea00:2d0:b4ff:fe05:5ab1/64
    •    wg0 (RBX_VPN):
         •    v4: 10.49.0.2/16 (gateway 10.49.0.1)
         •    v6: 2001:db8:a160::2/48 (gateway 2001:db8:a160::1)

Bridge filtering tunables (set):
net.link.bridge.pfil_bridge=1, net.link.bridge.pfil_member=0

WireGuard (client) status

ifconfig wg0
# wg0 up, 1420 MTU, v4 10.49.0.2/16, v6 2001:db8:a160::2/48

wg show
interface: wg0
  listening port: 13186
peer: M3iqRxgbMgKSpr8KHX2rPvS8UqjV5aSDzm510gdYrBE=
  endpoint: [2001:41d0:203:24b3::200]:51820
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: ~1 minute ago
  transfer: 3.66 MiB recv, 4.16 MiB sent
  persistent keepalive: every 5s

Gateways: RBX_VPN_V4=10.49.0.1, RBX_VPN_V6=2001:db8:a160::1 → both Online, RTT ~134 ms, RTTd ~2 ms. I can ping both tunnel gateways from OPNsense.

Policy-routing rules
    •    I tested Floating → Match (Quick, In) on WAN + LAN (member interfaces) and also Bridge tab rules (disabled now).
    •    Destination is an alias rbx_split_vpn containing:

4.ident.me
whatismyipaddress.com


    •    Gateway on the (active) Floating rule: RBX_VPN_V4 for IPv4. (I also have an IPv6 twin rule, but testing v4 first.)

What pf actually loaded (snippets):

pfctl -sr -vv | egrep -n 'rbx_split_vpn|route-to|wg0'
...
269:@66 pass in log quick on bridge0 route-to (wg0 10.49.0.1) inet from any to <rbx_split_vpn:5> ...
273:@67 pass in log quick on igc1    route-to (wg0 10.49.0.1) inet from any to <rbx_split_vpn:5> ...
277:@68 pass in log quick on igc0    route-to (wg0 10.49.0.1) inet from any to <rbx_split_vpn:5> ...

So the route-to to wg0 is definitely there.

Outbound NAT

Manual/advanced mode; NATting on wg0 for both families:

pfctl -sn | grep -n wg0
2:nat log on wg0 inet  all -> (wg0:0) port 1024:65535
3:nat log on wg0 inet6 all -> (wg0:0) port 1024:65535

The symptom

From a LAN client behind the bridge:

curl -4 -m 5 https://65.108.151.63/ -v  # (4.ident.me's v4)
# -> Connection timed out after 5 seconds

On OPNsense, the state shows PBR and NAT to wg0:

pfctl -vvss | egrep -A4 '65\.108\.151\.63|route-to'
all tcp 65.108.151.63:443 <- 192.168.100.2:62353      CLOSED:SYN_SENT
  id: 2ee9d16800000000 ... route-to: 10.49.0.1@wg0
  origif: bridge0
all tcp 10.49.0.2:16029 (192.168.100.2:62353) -> 65.108.151.63:443 SYN_SENT:CLOSED
  origif: wg0

But packet capture tells a different story:
    •    On LAN member (igc1):

tcpdump -ni igc1 host 65.108.151.63
... TCP SYN 192.168.100.2:52258 > 65.108.151.63:443 (retries)

    •    On WAN (igc0) during the same test:

tcpdump -ni igc0 host 65.108.151.63
... TCP SYN 10.49.0.2:15350 > 65.108.151.63:443 (retries)

    •    And wg UDP on WAN during the test shows only keepalives, no burst that would indicate the flow is being encapsulated:

tcpdump -ni igc0 udp port 51820
... steady 64/96-byte keepalives both directions every 5s

So, I'm seeing a NAT'd SYN with src 10.49.0.2 on WAN, but I don't see that traffic encapsulated as UDP/51820 by WG. No SYN-ACK ever returns. Meanwhile, connecting my MacBook directly to the same WG server (bypassing OPNsense) gives me working v4+v6 exit immediately (whatismyipaddress.com shows the WG server's IPs).

Things I tried
    •    Floating Match (Quick, In) on WAN+LAN members (active).
    •    Also tried Bridge tab PBR rules (now disabled).
    •    Disabled "reply-to on WAN rules" globally to rule out return-path pinning → no change.
    •    Reset states multiple times after changes.
    •    Verified alias resolves to 65.108.151.63.
    •    Verified WG gateways are Online and pingable.

The questions
    1.    With pfctl -vvss showing route-to (wg0 10.49.0.1) and NAT on wg0, should I ever see a raw TCP SYN with src 10.49.0.2 on the physical WAN (igc0)? I expected only UDP/51820 there. Is this a normal FreeBSD/WG quirk in captures, or an indication the packet is escaping un-encapsulated?
    2.    Is there any gotcha with PBR on a transparent bridge that would cause the state to carry route-to wg0 but the hand-off to WG not actually happen?
    3.    Should I remove IPs from the bridge members (set WAN/LAN to "None" and leave only the management IP on bridge0) for this to work reliably? My concern was that WG still needs upstream to reach its endpoint; with only bridge0 holding an IP, will WG still reach the internet cleanly?
    4.    Any WireGuard AllowedIPs/route install interactions to watch on OPNsense/FreeBSD? (In my config, the peer's AllowedIPs is 0.0.0.0/0, ::/0, and in the OPNsense WG "server" section I have disableroutes=1.)
    5.    For IPv6: I'm temporarily using a doc-prefix inside the tunnel and doing NAT66 on the server side for testing. That works fine from my Mac, but on OPNsense the v6 steering similarly times out unless I disable the v6 match rule. Any v6-specific PBR/WG/bridge caveats I should account for?

Extra snippets (in case they help)

pfctl -sr -vv (selected)
13:@2  block drop in log on ! wg0 inet  from 10.49.0.0/16 to any
21:@4  block drop in log on ! wg0 inet6 from 2001:db8:a160::/48 to any
...
277:@68 pass in log quick on igc0 route-to (wg0 10.49.0.1) inet from any to <rbx_split_vpn:5> ...

wg show (peer)
endpoint: [2001:41d0:203:24b3::200]:51820
allowed ips: 0.0.0.0/0, ::/0
persistent keepalive: every 5s

Outbound NAT
nat on wg0 (inet + inet6) -> (wg0:0) port 1024:65535  # logging enabled

Alias (rbx_split_vpn):

4.ident.me
whatismyipaddress.com




If anyone has a working recipe for WireGuard PBR on a pure bridge (no L3 on members) or can explain why I'd see 10.49.0.2 → 65.108.151.63:443 on igc0 instead of a UDP-encapsulated flow, I'd love pointers. Happy to test anything and post more captures. Thanks!