Topics

I have two servers connected with a slow, but secure tinc bridge and a fast, but unencrypted VXLAN link.
Both servers have a OPNsense VM running.
I want to send specific traffic over the VXLAN and everything else over the tinc link. Because of stupid software design I cannot use separate IP addresses for this (that would be easy), I have to change routing depending on the packet.

Code Select Expand

[SRV1] 10.8.0.1 --- 10.8.0.241 [OPN1] 172.16.4.1 --- VXLAN --- 172.16.4.2 [OPN2] 10.8.0.242 --- 10.8.0.2 [SRV2]
    \                                                                                                       /
     ------------------------------------------- tinc bridged to LAN ---------------------------------------

I have created a firewall rule on the LAN telling OPNsense to use the 172.16.4.2 as gateway for packets with a destination port 444.
This works. Packets appear on the OPN2 VXLAN interface with correct source and port. But the connection does not work.

What's weird is that OPN2 shows this in the firewall log:
nterface   Time   Source   Destination   Proto   Label   
LAN   Jul 9 12:39:05   10.8.0.1:64796   10.8.0.2:444   tcp   let out anything from firewall host itself   
VLAN   Jul 9 12:39:05   10.8.0.1:64796   10.8.0.2:444   tcp   USER_RULE

Why from the firewall host itself? It's clearly from another machine. Does OPNsense / pf get confused because the packets arrive at the "wrong" interface?
There are no drop log entries anywhere...

I have an OPNsense VM with several WAN interfaces, because my provider's routing only works correctly if you also have a MAC address for every additional public IP you use.
I have created a 1:1 NAT on WAN2 to forward everything to a single internal IP. This works for connections from the internet to WAN2.
When the internal IP tries to connect to the internet, that fails. I see the packets as PASS in the firewall log, but they don't get anywhere. Traceroutes stop at the OPNsense box.

The final solution was to create a separate firewall rule telling OPNsense to use the WAN2 gateway for all packets originating from the internal IP and going to the internet.

Shouldn't that be more intuitive? I already told OPNsense I want to translate all packets from that IP on the WAN2 interface to the WAN2 address. Can't it pick the correct gateway by itself? Or is there a better way to do this?