Weird routing / gateway problem

[ChrisH]

I have two servers connected with a slow, but secure tinc bridge and a fast, but unencrypted VXLAN link.
Both servers have a OPNsense VM running.
I want to send specific traffic over the VXLAN and everything else over the tinc link. Because of stupid software design I cannot use separate IP addresses for this (that would be easy), I have to change routing depending on the packet.

Code: [Select]

[SRV1] 10.8.0.1 --- 10.8.0.241 [OPN1] 172.16.4.1 --- VXLAN --- 172.16.4.2 [OPN2] 10.8.0.242 --- 10.8.0.2 [SRV2]
    \                                                                                                       /
     ------------------------------------------- tinc bridged to LAN ---------------------------------------
I have created a firewall rule on the LAN telling OPNsense to use the 172.16.4.2 as gateway for packets with a destination port 444.
This works. Packets appear on the OPN2 VXLAN interface with correct source and port. But the connection does not work.

What's weird is that OPN2 shows this in the firewall log:
nterface   Time   Source   Destination   Proto   Label   
LAN   Jul 9 12:39:05   10.8.0.1:64796   10.8.0.2:444   tcp   let out anything from firewall host itself   
VLAN   Jul 9 12:39:05   10.8.0.1:64796   10.8.0.2:444   tcp   USER_RULE

Why from the firewall host itself? It's clearly from another machine. Does OPNsense / pf get confused because the packets arrive at the "wrong" interface?
There are no drop log entries anywhere...

[ChrisH]

Okay, the connection seems to work now - stupid local firewall got activated somehow on SRV2.

But still, why the weird log message?

[ChrisH]

I solved this with separate subnets for each server. Seems like OPNsense indeed got confused because the packets had the "wrong" IP address for the interface.

[mimugmail]

Did you setup VXLAN via CLI on the OPNsense boxes? Brave!

[ChrisH]

No, the Hoster does the VXLAN stuff. For OPNsense it's just plain VLAN (and not even that, because I tag the VLAN on the hypervisor side )

[mimugmail]

Phew I played around with VXLAN on OPNsense via CLI, was working. But implementing via GUI is pain.

Nice hoster with nice switches it seems ..