[SOLVED] VXLAN between OPNsense and Ubuntu - can't get it to work.

[Monviech (Cedrik)]

I really need insight of somebody who has managed to get VXLAN on the OPNsense working.

As baseline, I did a basic VXLAN setup between two Ubuntu 22.04 LTS Servers on the same Network as sanity check.

Code Select Expand

VM1 (vxlan0 id 42 10.30.40.3/24) eth0 172.16.0.186/24 <-----> 172.16.0.187/24 eth0 (10.30.40.4/24 vxlan0 id 42) VM2

VM1:

Code Select Expand


eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.186  netmask 255.255.255.0  broadcast 172.16.0.255

vxlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1355
        inet 10.20.40.3  netmask 255.255.255.0  broadcast 10.20.40.255
        vxlan id 42 remote 172.16.0.187 local 172.16.0.186 dev eth0 srcport 0 0 dstport 4789


VM2:

Code Select Expand


eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.187  netmask 255.255.255.0  broadcast 172.16.0.255

vxlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1355
        inet 10.20.40.4  netmask 255.255.255.0  broadcast 10.20.40.255
        vxlan id 42 remote 172.16.0.186 local 172.16.0.187 dev eth0 srcport 0 0 dstport 4789


This works, sanity check complete.
- eth0 vm2: 172.16.0.187 can ping eth0 vm1: 172.16.0.186 and the other way around.
- VXLAN vm2: 10.20.40.4 can ping VXLAN vm1: 10.20.40.3 and the other way around.

-----------------------------------------------------------------------------------------------------------------------------------

Now I thought, I could do it between OPNsense 23.7.4 (FreeBSD) and Ubuntu 22.04.3 LTS. I was wrong it seems. But I don't understand why. Please help me :)

Code Select Expand

VM1 (vxlan0 id 42 10.30.40.3/24) eth0 172.16.0.186/24 <-----> 172.16.0.254/24 eth0 (10.30.40.1/24 vxlan0 id 42) OPNsense

VM1:

Code Select Expand


eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.186  netmask 255.255.255.0  broadcast 172.16.0.255

vxlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1355
        inet 10.20.40.3  netmask 255.255.255.0  broadcast 10.20.40.3
        vxlan id 42 remote 172.16.0.254 local 172.16.0.186 dev eth0 srcport 0 0 dstport 4789


OPNsense:
(Configured through GUI)

Code Select Expand


hn6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: hn6_LAN (lan)
options=180018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE,NETMAP>
ether 00:15:5d:00:c9:52
inet 172.16.0.254 netmask 0xffffff00 broadcast 172.16.0.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

vxlan2: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1355
description: vxlan2 (opt6)
options=80020<JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:00:62:52
inet6 fe80::5a9c:fcff:fe00:6252%vxlan2 prefixlen 64 tentative scopeid 0x13
inet 10.20.40.1 netmask 0xffffff00 broadcast 10.20.40.255
groups: vxlan
vxlan vni 42 local 172.16.0.254:4789 remote 172.16.0.186:4789
media: Ethernet autoselect (autoselect <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


OPNsense Firewall rules:

Code Select Expand


root@opn01:~ # pfctl -s rules | grep -i vxlan
scrub on vxlan2 all fragment reassemble
block drop in log on vxlan2 inet6 from fe80::5a9c:fcff:fe00:6252 to any
block drop in log on ! vxlan2 inet from 10.20.40.0/24 to any
pass in log quick on hn6 inet proto udp from any to any port = vxlan keep state label "6937e3837a1e6539757d540ce21cbb7f"
pass in quick on vxlan2 inet all flags S/SA keep state label "33f3a4eb4db2eb24493d44990290c399"


This doesn't work, sanity gone:
- hn6 opnsense: 172.16.0.254 can ping eth0 vm1: 172.16.0.186 and the other way around.
- VXLAN opnsense: 10.20.40.1 CAN'T ping VXLAN vm1: 10.20.40.3 and the other way around.

Packet Capture:
- Ping from 10.20.40.3 (VM1) to 10.20.40.1 (OPNsense) doesn't work.

VM1:

Code Select Expand


root@vm1: tcpdump -i any proto ICMP -n
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
19:53:51.654332 lo    In  IP 10.20.40.3 > 10.20.40.3: ICMP host 10.20.40.1 unreachable, length 92


OPNsense:

Code Select Expand


root@opn01:~ # tcpdump -i hn6 port 4789 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on hn6, link-type EN10MB (Ethernet), capture size 262144 bytes
21:51:58.795397 IP 172.16.0.186.53062 > 172.16.0.254.4789: VXLAN, flags [I] (0x08), vni 42
ARP, Request who-has 10.20.40.1 tell 10.20.40.3, length 28


Code Select Expand


root@opn01:~ # tcpdump -i vxlan2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vxlan2, link-type EN10MB (Ethernet), capture size 262144 bytes

... NOTHING


Packet Capture:
- Ping from 10.20.40.1 (OPNsense) to 10.20.40.3 (VM1) doesn't work

Code Select Expand


root@opn01:~ # ping -S 10.20.40.1 10.20.40.3
PING 10.20.40.3 (10.20.40.3) from 10.20.40.1: 56 data bytes
ping: sendto: No route to host


-----------------------------------------------------------------------------------------------------------------

Since there was a "No route to host", I have verified the routing tables on OPNsense. They seem to look fine.

Code Select Expand


root@opn01:~ # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
10.20.40.0/24      link#19            U        vxlan2
10.20.40.1         link#19            UHS         lo0


-------------------------------------------------------------------------------------------------------------------------

I don't know what else to do or where else to look. Either it's broken and really doesn't work, or I don't understand it (more likely). I appreciate any hints.

[lilsense]

disable the firewall completely and test. I believe this is firewall related.

[Monviech (Cedrik)]

Thanks for the suggestion, that ruled out that the firewall is the culprit. So Layer 3 should be working right.

I think that arp learning is the culprit, so the troubleshooting continues in Layer 2.

Opnsense ARP Table:

Code Select Expand


root@opn01:~ # arp -a
? (10.20.40.1) at 58:9c:fc:00:62:52 on vxlan2 permanent [ethernet]
opn01 (172.16.0.254) at 00:15:5d:00:c9:52 on hn6 permanent [ethernet]
? (172.16.0.186) at 00:15:5d:00:c9:76 on hn6 expires in 962 seconds [ethernet]


VM1 ARP Table:

Code Select Expand


root@vm1:# arp -a
? (10.20.40.1) at <incomplete> on vxlan0
opn01 (172.16.0.254) at 00:15:5d:00:c9:52 [ether] on eth0


This explains the ARP Requests the OPNsense receives, but doesn't reply to.  "? (10.20.40.1) at <incomplete> on vxlan0" shows that VM1 doesn't know where 10.20.40.1 is, because it never got an ARP reply from the OPNsense.

I forced VM1 to register the MAC of vxlan2 of the OPNsense:

Code Select Expand


root@vm1:# sudo arp -s 10.20.40.1 58:9c:fc:00:62:52

? (10.20.40.1) at 58:9c:fc:00:62:52 [ether] PERM on vxlan0


Then I tried to force the OPNsense to register the MAC of vxlan0 of VM1:

Code Select Expand


root@opn01:~ # sudo arp -s 10.20.40.3 d6:c6:72:fa:cb:5b
arp: writing to routing socket: Cannot allocate memory


How curious, that error message ( https://forum.netgate.com/topic/162645/arp-writing-to-routing-socket-cannot-allocate-memory/2 ) means that "The Firewall cannot allocate memory in the ARP table for an IP in a subnet the firewall doesn't have an interface in"

What this means is, the OPNsense and the FreeBSD Operating System below, don't recognize "vxlan2" and the IP network "10.20.40.0/24" on it as an interface (Even though ifconfig shows the interface with IP 10.20.40.1/24 AND its even in the routing Table as shown above AND it even has a permanent ARP entry in the ARP Table of the OPNsense)

---------------------------------------------------------------------------

Pinging now from VM1 to OPNsense shows this in Packet Capture:

Code Select Expand


root@opn01:~ # tcpdump -i hn6 -n port 4789
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on hn6, link-type EN10MB (Ethernet), capture size 262144 bytes
08:16:42.161162 IP 172.16.0.186.43134 > 172.16.0.254.4789: VXLAN, flags [I] (0x08), vni 42
IP 10.20.40.3 > 10.20.40.1: ICMP echo request, id 51, seq 1, length 64


The echo request is received, but it goes nowhere because there is no interface that responds to it.

---------------------------------------------------------------------------

I came to the conclusion that VXLAN will not work regardless of what I do because the above tests clearly show it failing at the system level. If somebody still has an idea, I'd be happy to dig deeper into this.

[Monviech (Cedrik)]

Troubleshooting continues on system level.

Since I found out that the vxlan2 interface is not recognized somehow, I checked dmesg:

Code Select Expand


vxlan2: cannot initialize interface: can only specify interface with a group address
vxlan2: promiscuous mode enabled


In the OPNsense GUI, the "Multicast group" in "Edit VxLan" is not an required field. So I left it empty. Which resulted in the above error. This error happens ONLY if you also have "interface" other than "none"

I added a Multicast Group Address 239.0.0.1 and rebooted.

Code Select Expand


vxlan0: Ethernet address: 58:9c:fc:10:b1:43
vxlan0: changing name to 'vxlan2'
vxlan2: link state changed to UP


That seems to make the Operating System recognize vxlan2 as valid interface.

Now I can add a static ARP on the opnsense:

Code Select Expand


root@opn01:~ # sudo arp -s 10.20.40.3 d6:c6:72:fa:cb:5b

root@opn01:~ # arp -a
? (10.20.40.3) at d6:c6:72:fa:cb:5b on vxlan2 permanent [ethernet]
? (10.20.40.1) at 58:9c:fc:10:b1:43 on vxlan2 permanent [ethernet]


The Problem now is, that the vxlan2 interface seems to be in Multicast mode, and not in Unicast mode anymore:

Code Select Expand


vxlan2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1355
description: vxlan2 (opt6)
options=80020<JUMBO_MTU,LINKSTATE>
ether 58:9c:fc:10:b1:43
inet6 fe80::5a9c:fcff:fe10:b143%vxlan2 prefixlen 64 scopeid 0x12
inet 10.20.40.1 netmask 0xffffff00 broadcast 10.20.40.255
groups: vxlan
vxlan vni 42 local 172.16.0.254:4789 group 239.0.0.1:4789
media: Ethernet autoselect (autoselect <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


Now its this

Code Select Expand


vxlan vni 42 local 172.16.0.254:4789 group 239.0.0.1:4789


as opposed to this:

Code Select Expand


vxlan vni 42 local 172.16.0.254:4789 remote 172.16.0.186:4789


That means the populated "remote" field in the GUI is ignored now.

---------------------------------------------------------------------------------

Packet Capture, PING Test:

Now the ICMP Echo Request gets through from VM1 to the OPNsense vxlan2, but there is still no Echo Reply. It's an improvement nontheless.

Code Select Expand


root@vm1: ping 10.20.40.1
PING 10.20.40.1 (10.20.40.1) 56(84) bytes of data.
^C
--- 10.20.40.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

root@vm1:~$ tcpdump -i any proto ICMP -n
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:30:27.806017 vxlan0 Out IP 10.20.40.3 > 10.20.40.1: ICMP echo request, id 61, seq 1, length 64
08:30:28.806352 vxlan0 Out IP 10.20.40.3 > 10.20.40.1: ICMP echo request, id 61, seq 2, length 64

root@opn01:~ # tcpdump -i hn6 proto UDP and port 4789
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on hn6, link-type EN10MB (Ethernet), capture size 262144 bytes
10:30:27.808498 IP 172.16.0.186.43134 > opn01.vxlan: VXLAN, flags [I] (0x08), vni 42
IP 10.20.40.3 > 10.20.40.1: ICMP echo request, id 61, seq 1, length 64
10:30:28.808904 IP 172.16.0.186.43134 > opn01.vxlan: VXLAN, flags [I] (0x08), vni 42
IP 10.20.40.3 > 10.20.40.1: ICMP echo request, id 61, seq 2, length 64

root@opn01:~ # tcpdump -i vxlan2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vxlan2, link-type EN10MB (Ethernet), capture size 262144 bytes
10:30:27.808532 IP 10.20.40.3 > 10.20.40.1: ICMP echo request, id 61, seq 1, length 64
10:30:28.808943 IP 10.20.40.3 > 10.20.40.1: ICMP echo request, id 61, seq 2, length 64


[Monviech (Cedrik)]

I have solved it myself. It was a user error:

- When VXLAN should be in UNICAST mode, the multicast group parameter has to be empty, and the interface has to be set to none.
- When VXLAN should be in MULTICAST mode, the group parameter has to be populated, and the interface has to be set to an interface that should listen to and send those broadcasts.

So I removed the interface definitions of the vxlan interfaces from the OPNsense and Ubuntu VM, and removed the multicast groups.

The interfaces got up, and I could ping back and forth and ARP worked. Nice job me :)

https://github.com/opnsense/core/issues/6893