OpenVPN TAP + CARP + External AP = Problems

[mark-p]

Hello posters, it's been a while I've used one of these boards!

I would like to ask you about a setup I'm currently trying to wrap my head around. I have already "solved" it, but the solution makes me scratch my head as to why it had to be done that way.

Firstly let me outline the topology:
- 2 OPNSense machines, both have the same interfaces configured, same order, names.
- Physical interfaces: 1 WAN interface, 1 LAN interface, 1 PFSYNC interface.
- Two CARP IPs are configured, 1 for WAN, 1 for LAN.
- External mesh devices set in bridge mode (they get their configurations through DHCP (hosted on OPNSense) alright) that act as APs, only one these units is connected to a switch that is connected to LAN, the other ones use the master node for all their requests.

Into the problem!

The other day I wanted to implement a OpenVPN TAP VPN server, sitting on OPNSense. I have roughly followed these guides (and a few others):

• https://homenetworkguy.com/how-to/configure-openvpn-opnsense/
• https://forum.opnsense.org/index.php?topic=5716.0
• https://docs.opnsense.org/manual/how-tos/lan_bridge.html

Fast forward: I have a functioning setup with a bridge that bridges LAN and the TAP interface. The bridge DOES NOT have an IP address, everything sits on the original LAN interface.

And here comes the strange thing: while obtaining this new possibility, some of the access points stopped working. More precisely, the clients connected to the "master node" (i.e. the one connected to the LAN switch with an Ethernet cable) could access Internet correctly and could ping the router, its backup and the CARP VIP.

The other two access points' clients could: connect to the access point, but would get no Internet connection. Upon further inspection through an app called Fing and a smartphone connected to one of the "faulty" nodes, I noticed I could perform a network scan, I could ping the router and its backup, I could ping ANY other client on the LAN, but I could not ping the LAN VIP (which is the set as the default gateway distributed by DHCP). ALL of the faulty access point actually had a valid IP configuration, no collisions, nothing.

Realizing it had something to do with the bridge and the OpenVPN TAP server, I decided to undo everything, and... indeed the access point started working correctly again.

Before applying the solution I decided to perform a small packet sniffing experiment in order to isolate what the access points were doing in order to be able to communicate with EVERYTHING BUT THE GATEWAY (VIP).

And here's another interesting part, both APs would constantly send ARP requests asking for the address associated with the VIP... and they would also get an answer from the master router... But when I looked at the interface associated with said traffic... it was "OpenVPN TAP" instead of LAN. I also observed that there was quite a bit of traffic (many small packets, no actual communication) coming in "OpenVPN TAP" while no client was connected.

Hence, here's my question: "how the heck did those packet get into the OpenVPN TAP interface instead of the LAN one?". To the best of my understanding the flow should be something like:

client -> AP "secondary" -> AP "master" -> ETH Switch -> LAN interface on Router

How do they get into "OpenVPN TAP" which has NO address, NO tunnel address, NO DHCP (the one in LAN is actually responding).

How I resolved the issue

By observing the behavior described above, I assigned the LAN IP to the bridge, and moved the DHCP of LAN to the bridge. The bridge is now responsible for the IP address of the router in LAN and bridges the physical interface and the "OpenVPN TAP". This produced a working setup both OpenVPN-wise and Wifi-wise. How would you explain this phenomenon?

Thank you for the time taken reading this mess!

[Patrick M. Hausen]

Explain this phenomenon: that's exactly how it's supposed to work. In FreeBSD a member interface of a bridge cannot have an IP address. The IP address must be on the bridge interface. The LAN bridge section of the OPNsense documentation as well as the FreeBSD handbook state this. You must assign your logical interface to the bridge.

If the first installation worked despite the unsupported setup, it was by accident.

[mark-p]

Thank you for your answer. I apologize for my ignorance, I come from the Linux world, and there I would never have dreamed of using a bridge the way I did here. However I am still perplexed, the chance of the aforementioned setup to "just work" without a rationale is pretty low in my humble opinion. Yet what actually defies my understanding is why those packets STILL flow through the OpenVPN TAP albeit coming FROM a physical interface. Why would there be a reason for the router to take all the packets FROM the other APs (but not the ones from the "master node") and handle them differently? And, most importantly, why would they flow in that interface?

I believe I have seen this kind of bridge called "transparent bridge", and that it would "act as a switch" (let's forget the term bridge for a second).

If that were to be the case the bridge would only "bridge" the two collision domains spanned by the LAN and the OpenVPN TAP, which is exactly what I was seeking.

EDIT:

On top of what I said, the aforementioned behavior SHOULD affect the WHOLE network, rather than just a piece of it, both APs and wired interface reside in the same collision domain and also have the same IP configuration, same net and broadcast. Why would a bridge impair clients that connect to Wifi, while it would leave alone wired clients?

Moreover, in this guide no IP address configuration on the bridge is mentioned (the purpose is the same): https://docs.netgate.com/pfsense/en/latest/recipes/openvpn-bridged.html .

[Patrick M. Hausen]

This guide is wrong. A member of a bridge must not have a layer 3 address in FreeBSD. It breaks multicast among other things. You should always assign the bridge as LAN or whatever interface is appropriate.