How can we nat a remote network using vxlan wireguard ?

[Damien66]

Hello,
  I have configured two remote sites to share the same vlans, the same addresses.
The sites are connected via VXLAN through a wireguard tunnel. For this, I used and added the two docs :
- https://docs.opnsense.org/manual/how-tos/wireguard-s2s.html
- https://docs.opnsense.org/manual/how-tos/vxlan_bridge.html
The configuration works. I get DHCP when I connect a pc to the different vlans on the remote site.
I can ping between a pc on site A and a pc on site B (and vice versa).

It's almost perfect BUT...the remote client cannot access the Internet (no NAT)
However, the local client can go internet (same DHCP server, same vlan, same gateway). For me, the configuration of the local and remote client is similar.

In the test below, we see a ping from the local client (192.168.50.44 VLAN50) to 8.8.8.8.
You can see that it's pinged to my public output 82.XX.XX.64 (WAN)... and so the “echo reply” is perfect.

Code: [Select]

------------------------------------------------------
NAT works on vlan50's for local host
--------------------------------------------------------
VLAN50Free
vlan0.50 2024-11-22
09:55:03.071020 0c:b2:b7:cf:da:8e a8:b8:e0:01:ef:b3 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 50629, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.50.44 > 8.8.8.8: ICMP echo request, id 27503, seq 1, length 64

InternetWAN
igc0 2024-11-22
09:55:03.071099 a8:b8:e0:01:ef:b1 20:66:cf:60:32:31 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 50629, offset 0, flags [DF], proto ICMP (1), length 84)
    [b]82.XX.XX.64 > 8.8.8.8[/b]: ICMP echo request, id 11344, seq 1, length 64

InternetWAN
igc0 2024-11-22
09:55:03.087745 20:66:cf:60:32:31 a8:b8:e0:01:ef:b1 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 118, id 0, offset 0, flags [none], proto ICMP (1), length 84)
    8.8.8.8 > 82.XX.XX.64: ICMP echo reply, id 11344, seq 1, length 64

VLAN50Free
vlan0.50 2024-11-22
09:55:03.087763 a8:b8:e0:01:ef:b3 0c:b2:b7:cf:da:8e ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 117, id 0, offset 0, flags [none], proto ICMP (1), length 84)
    8.8.8.8 > 192.168.50.44: ICMP echo reply, id 27503, seq 1, length 64
But the remote PC (passing through wireguard, arriving on bridge between vxlan and opt2) can't go online. No NAT.
Yet this remote pc 192.168.50.45 can ping 192.168.50.44 (local pc) and 192.168.50.1 (vlanFree interface ip)

Code: [Select]

----------------------------------------------------------
But a remote host doesn't go internet. No NAT
----------------------------------------------------------
VLAN50Free
vlan0.50 2024-11-22
09:53:46.895976 ec:b1:d7:99:25:9c a8:b8:e0:01:ef:b3 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 53111, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.50.45 > 8.8.8.8: ICMP echo request, id 1, seq 236, length 40

InternetWAN
igc0 2024-11-22
09:53:46.896003 a8:b8:e0:01:ef:b1 20:66:cf:60:32:31 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 53111, offset 0, flags [none], proto ICMP (1), length 60)
    [b]192.168.50.45 > 8.8.8.8[/b]: ICMP echo request, id 1, seq 236, length 40
We can see that the ip on the InternetWAN interface should be my public ip 82.XX.XX.64... but it tries with the local ip 192.168.50.45. Of course it doesn't work.
Any ideas? Why does opnsense manage differently a local pc and a remote pc having the same network in the same vlan?
Thank you very much

[Monviech (Cedrik)]

Try creating an Alias with RFC1918 networks.

In your current NAT rule, replace Source with that Alias.

That way any RFC1918 address will be NATed before they go out of the WAN interface.

[Damien66]

I've tried but it doesn't make any difference. The local pc in 192.168.50.44 goes to the Internet and the remote pc in 192.168.50.45 doesn't go to the Internet.
In fact, instead of to nat 192.168.50.0/24 (VLAN50Free net ), it is all the local networks with this modification.

The very strange thing is why opnsense handles two PCs on the same LAN differently.

[Monviech (Cedrik)]

I still have my vxlan test setup from writing the docs. I can test how NAT behaves when packets pass from one bridge to the other over the vxlan tunnel.

There might be some limitations I have not seen.

[Damien66]

If you can test it, I'd like to.
I've probably activated or deactivated a parameter, or made a configuration error.
But I can't find the trick

[Damien66]

Hello,
I've noticed a few things. In firewall/ rules/ vlan50, I have a rule to open ICMP from 192.168.50.0/24 to gateway 192.168.50.1.

For your information, my opnsense is connected to a switch on 2 ports:
- one port only with vlan50
- one trunk port (corresponding to the vxlan bridge)

When I have a local PC connected to the switch, if I disable my ICMP rule, the client can no longer ping the gateway. It's perfect.
192.168.50.44 --> 192.168.50.1 BLOCK

But a remote pc in the same vlan (arriving on the bridge) manages to ping the gateway
192.168.50.45 --> 192.168.50.1 PASS

I would have liked the flow arriving on the bridge to enter the switch on the trunk port and exit on the vlan50 port.
However, there seems to be a magic path from the bridge to vlan50 inside the opnsense (without using the rules in vlan50).
Do you know what could explain this behavior?

[Monviech (Cedrik)]

Hello, I didn't look at the initial issue yet. Is the NAT still an issue?

I'm working on getting my test setup back up.

Can you provide a complete network diagram of your setup?

[Damien66]

Yes, NAT is always a problem for the remote PC.
I'll try to make you a network diagram (with pen and paper)

[Damien66]

Here's an (ugly) diagram.
I want vlan100 pc's (DHCP site A on the firewall) to exit via the firewall's WAN.
I want vlan50 pc's (DHCP site B on opnsense B) to exit via the WAN of opnsense B.
Right now it's almost perfect:
- pc vlan 100 on site A or site B goes to the Internet via the firewall WAN
- pc vlan 50 on site B goes to the Internet via the WAN of opnsense B
- but pc vlan 50 on site A is not connected to the Internet on opnsense B.

On my diagram, at the bottom, I've circled in red the client that isn't working: 192.168.50.45.
It gets DHCP (provided by opnsense B on its port n°3 vlan50), and it can ping gateway 192.168.50.1 or pc 192.168.50.44.
But it can't go to the Internet.

I'd like the route for client 192.168.50.45 to be :
- bridge (vxlan / port n°1) --> ok, I can see the packets arriving.
- port w on switch B (trunk)
- switch B port x
- port n°3 of opnsense B (to arrive on vlan50 interface, and pass through filter/NAT rules)

But I can see that this remote client doesn't use the filter rules, nor the NAT rules, even though it's in the right vlan.
I think there's some kind of bypass (without going through switch B) and the path is :
- bridge (vxlan / port n°1)
- port n°3 of opnsense B

In this way, the remote client doesn't behave like the local client (it doesn't match the NAT rules linked to vlan 192.168.50.0/24, nor the filtering rules).

For your information, in the other direction (example with vlan 100), this works very well. My client 192.168.100.30 on site B gets DHCP managed by site A's firewall, and it goes on the Internet via the WAN of site A's firewall.
The difference is that opnsense A doesn't manage DHCP or inter-vlan routing. Opnsense A only acts as a bridge.

In conclusion, perhaps I should be able to “tell” opnsense B not to make an internal link between its port n°1 (bridge) and n°3 (vlan50), to skew the flow to go through switch B.

I hope I've given enough information
Thanks again for your help.

[Monviech (Cedrik)]

I have just tested vxlan again and for me it works (NAT and all), but I found something interesting on the way.

Can you check the bridges on both firewalls to see if they accidentally have the same MAC address?

[Damien66]

I haven't defined a MAC address in the bridge interface config (see image).
Looking at the command line, I see two different MAC addresses:

Opnsense site A

Code: [Select]

[code]bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1370
        description: PONT (opt5)
        options=0
        ether 58:9c:fc:10:81:3d
        inet 10.28.79.1 netmask 0xfffffffc broadcast 10.28.79.3
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vxlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 55
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 20000
        groups: bridge
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
opnsense site B

Code: [Select]

bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1370
        description: PONT (opt5)
        options=0
        ether 58:9c:fc:10:ff:8a
        inet 10.28.79.2 netmask 0xfffffffc broadcast 10.28.79.3
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vxlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 23 priority 128 path cost 55
        member: igc3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 8000
        groups: bridge
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
I have two different addresses

[Monviech (Cedrik)]

Yeah thats good. In my tests I had an auto generated MAC address that was the same, so it caused ARP to fail and thats why my client could not reach the internet through the vxlan tunnel.

Your setup looks a bit more complex. Outr issues we had are not related sadly. So my documentation is fine the way it is.

Your issues might stem from complexity, or a network design issue.

The "bypass" you might see, is the router doing its job, routing. The bridge has an IP address and knows the local route to the other vlan or WAN directly.

I would try to reduce the complexity of this setup.

[Damien66]

Ok thanks I'll see how to do that.

One last question: in the documentation, “2 VPN setup”, it says :
Create Firewall rules that allow VXLAN (UDP/4789) and ICMP traffic for:
- Firewall ‣ Rules ‣ Loopback
- Firewall ‣ Rules ‣ IPsec (or Wireguard)

For Wireguard, I understand...
But you also have to open vxlan and ICMP on lo1 previously created? If you don't put any rules on lo1 in your test configuration, does it still work?

Thanks for your documentation. I wouldn't have got this far without it.

[Monviech (Cedrik)]

I think the wireguard rules on loopback are not needed. I was unsure so I put them but it also works without them.

The rules on the actual tunnel are needed though as they transport the vxlan protocol.

NP, VXLAN can be quite fun. But whenever possible I would go with standard routing since it has less random issues.

[Damien66]

What do you have in mind when you say “standard routing”?

I'll see if I can “block” default routing in opnsense to force it to go to swicth... but probably impossible since that's its job