mDNS Repeater and Access Point challenges and network degradation

[datniha]

Hi,

I have identified a strange problem using the mDNS Repeater plugin. (I see the same behavior using UDP Broadcast Relay). I am running OPNsense version 24.7.8, and all is fully updated. Let me explain my setup:

My OPNsense installation is running on dedicated hardware. Only connections are to the WAN and my level 2 bridge (Aruba).  All network devices are connected to the bridge or wifi using two Aruba access points (also connected to the bridge). Beside a LAN I have three VLANs divided into the following subnets:

x.x.1.0/24: LAN (full access to all LAN, VLAN and WAN, exclusively for the OPNsense box, the bridge and access points - sort of administrative only)
x.x.20.0/24: HOME (full access to all LAN, VLAN and WAN)
x.x.30.0/24: IOT (access restricted to WAN only)
x.x.40.0/24: Guest (access restricted to WAN only)

Now, to make my devices on the HOME VLAN aware of the devices and services on the IOT VLAN, I have enabled mdns-repeater between the two VLAN's (IOT and HOME) only. When enabled the network becomes congested and degrades exponentially in time with **lots** of mDNS packages (224.0.0.251:5353) being transmitted and re-transmitted (identified with wireshark). After a few minutes my IOT devices fails to respond.

An increasing amount of the mDNS packages originates from my two access points (x.x.1.4 and x.x.1.5). It appears as if they relay the multicast from the devices connected to them.

As a workaround, I have created a firewall rule blocking all mDNS communication on the LAN interface regardless of origin. This reduces the number of mDNS packages on the other VLANS significantly and my network is almost back to normal. However, this made me a but curious.

I then disabled mDNS-Repeater, in the GUI.

Started the mdns-repeater on the command line as "mdns-repeater -f vlan02 vlan03" (vlan02 and vlan03 represents respectively HOME and IOT). The "-f" clause causes the program to output to the terminal, and you can follow what is going on.

Much to my surprise I discovered that a lot of traffic comes from my two access point, eg as

data from=x.x.1.5 size=1045
repeating data to vlan02
repeating data to vlan03
data from=x.x.1.4 size=1045
repeating data to vlan02
repeating data to vlan03

This is in contrast of what I'd expected due to the firewall rule mentioned above and due to the LAN being excluded in the call. I am not sure if this is the correct behavior, or a bug!

Now, running the mDNS Repeater as: "mdns-repeater -f -b x.x.1.0/24 vlan02 vlan03" blocking all multicast requests from my LAN subnet, I can only identify "data from="... my two VLANs HOME and IOT. All good.

The moment I disable my firewall rule blocking mDNS traffic the following is in the output:

skipping packet from=x.x.1.5
data from=x.x.20.y size=646
repeating data to vlan03
data from=x.x.30.z size=64
repeating data to vlan02

This corresponds to the behavior, I'd expected when LAN interface is unselected in the GUI (normal operation).

After having spent numerous hours debugging my network I am arriving at the conclusion that I will, seen from a network perspective, be better off by running mdns-repeater as a demon enabled via "/etc/rc.conf" with blocking of the LAN set as parameter. However, I am uncertain which position this will bring me into taking future upgrades og OPNsense into consideration. I'd preferred to be able to achieve the same using the web-interface.

Any recomendation, help or pointers are very much appreciated.

All the best
Datniha

PS: This is my first post only being running OPNsense for almost a year. I have searched the archives for similar issues with Multicast DNS but haven't found anything helpful. My apologies if this problem has been discussed earlier.

[datniha]

I made a few further investigations.

TLDR: The only viable solution is to set the blocklist in the configuration file of mdns-repeater: "/usr/local/etc/rc.d/mdns-repeater". In this example as 

mdns_repeater_blocklist="-b x.x.1.0/24"

It appears that multicast traffic is snatched by mdns-repeater before the packet goes into the firewall. It means that even though you set firewall rules up to block UDP multicast packets to 224.0.0.251:5353 from problematic interfaces (LAN in my case, subnet x.x.1.0/24), they will still be retransmitted to the selected interfaces!

The only uncertainty with this solution is if the file "/usr/local/etc/rc.d/mdns-repeater" is being overwritten in a future update, and the modification hence needs to be reapplied with potential network degradation in the meantime.

This might be a bug
.
If not a bug, I suggest that giving access to setting up a blocklist in mdns-repeater using the web-interface would be a nice to have feature.
/Datniha

PS: How can I mark this as "Solved"?

[datniha]

Pull request #4362 to the plugins repository created.

https://github.com/opnsense/plugins/pull/4362

[OPNenthu]

I'm using mDNS repeater plugin to share an AirPrint/Bonjour printer across a couple VLANs. One difference is that I only have one AP, but so far I'm not seeing any mDNS storms.  To be fair I only have a couple Chromecasts and a handful of iOS/Android devices on the IoT subnet together with the printer, but the Chromecasts are known to be chatty.

I am seeing mDNS traffic in 'pf' logs (screenshot attached). 

.30.105 and .30.106 are the Chromecasts and they are noisy, but I don't consider this a flood.  The printer, .30.2, is currently asleep.  I don't know who has that IPv6 link-local address in IOT, as I don't use DHCPv6 and don't see it listed in the NDP Table in OPNsense.  (Identifying IPv6 SLAAC clients is a pain point that I'm looking for a solution to).

I wonder if there's something about your setup that's causing what you're seeing?  Couple random thoughts:

- Leaks due to untagged frames on OPNsense trunk?
- Similar issue to what many UniFi customers are reporting with the new U7 Pro WiFi-7 APs?
https://www.youtube.com/watch?v=P7MBZ80HzmI

Ubiquti have implemented some work-arounds:
https://help.ui.com/hc/en-us/articles/27053048844055-Ensure-Your-IoT-Devices-Work-with-WiFi-7-on-U7-Access-Points

[OPNenthu]

One more thought-

I'm not sure about the Aruba APs, but I'm using an Asus router in AP mode with FreshTomato firmware.  It has options pertaining to multicast proxying and I have them all disabled. 

Ditto with my UniFi L3 switch (also has mDNS forwarding and IGMP options).

I may be wrong for doing this, but my reasoning is that I want OPNsense to handle all that and not have anything routed at the switch / AP level.

[datniha]

@OPNenthu, thanks a lot for your time and effort in helping with this problem.

First of all I do agree that it is definitely also my preferred way to solve this at the root of the problem - likely my Aruba APs. I have disabled IGMP Snooping on my APs and my Aruba L2 switch. No specific option to handle mDNS frames. It did not have the desired effect; my APs located on my LAN Net continues to relay mDNS frames to my IOT and HOME VLANs. I have raised the issue in the Aruba community - until now no responses. :-|

The problem may be related to untagged frames from the OPNSense Trunk. Unfortunately, the link to the video is broken for me. I will dig a bit deeper into this.

Any more pointers or help solving the rootcause is very much appreciated. In the meantime...

Since mDNS happens at the network layer (physical part of the network), I am not sure if any firewall rule will help here since in my understanding it happens at the transport layer... 🤔
I do not have any specific firewall rules for mDNS. All networks do have WAN access though and it appears as 224.0.0.251 falls in that category. (see Screenshot 2024-11-21 at 10.09.30).

What the screenshot also reveals is that mDNS frames originating from a AP (10.83.1.4) is by the default rule blocked for transmitting on other VLANs, while my devices 10.83.20.40 and 10.83.20.41 are allowed access (my rule for for HOME Net). The blocking will however not help, since the mdns-repeater daemon catches the frame before the firewall rule is applied. I have verified this by running mdns-repeater in interactive mode. See output below (note line in bold text).

For now I have "solved" the problem by adding this line to the file located at "/usr/local/etc/rc.d/mdns-repeater":

# NB: This line is added:
mdns_repeater_blocklist="-b 10.83.1.0/24"

Which seems to be persistent probably until an upgrade. I'll like to gain access to this setting in the mDNS Repeater GUI plugin in OPNSense - which is why I raised the pull request. I am aware that this is a workaround, though. But I am also aware that others have the same problem, apparently in newer APs from at least Ubiquity and Aruba.

----

/usr/local/bin/mdns-repeater -f -p /var/run/mdns-repeater.pid -b 10.83.1.0/24 vlan02 vlan03
mdns-repeater: blacklist addr 10.83.1.0 mask 255.255.255.0 net 10.83.1.0
mdns-repeater: dev vlan02 addr 10.83.20.1 mask 255.255.255.0 net 10.83.20.0
mdns-repeater: dev vlan03 addr 10.83.30.1 mask 255.255.255.0 net 10.83.30.0
data from=10.83.20.43 size=85
repeating data to vlan03
skipping packet from=10.83.1.5
data from=10.83.30.9 size=426
repeating data to vlan02
data from=10.83.20.43 size=153
repeating data to vlan03

[OPNenthu]

I need to correct some earlier misstatements I made.

Since mDNS happens at the network layer (physical part of the network), I am not sure if any firewall rule will help here since in my understanding it happens at the transport layer... 🤔
I do not have any specific firewall rules for mDNS. All networks do have WAN access though and it appears as 224.0.0.251 falls in that category. (see Screenshot 2024-11-21 at 10.09.30).

I think you're right that mDNS is both a link-layer and transport-layer protocol, but it doesn't cross network segments without either a repeater or an IGMP proxy.  'pf' sees the UDP traffic, but now that we're discussing this I do realize that my 'pass' rules are pointless in this case.  I will disable them and see how it goes.

My initial thoughts were to make sure that nothing else besides the router is reflecting the mDNS traffic so that it doesn't flood, but this is wrong.  I think repeaters are needed wherever you have a 'hop' in your network.  I'm not sure how it's working for me (2 hops from PC to wireless printer according to traceroute) so I need to go back and review my setup.  I know that I disabled the mDNS proxies, at least at the AP.

Will watch with interest to see what develops regarding your fix to cut down the noise.  8)

[tohil]

Hi

i'm using Aruba Instant ON gear as well. the untagged traffic of the Access Points (the APs self ip, management) is in connected in a dedicated management-subnet/vlan 6. AP switch ports are set untagged for this vlan.

I use not the UDP Broadcast Relay service on my opnsense box, where you can configure the listening interfaces.I have no issues with Aruba Instant ON 3.0.0.x with this config.

But as you mentioned on other posts and in the AIO community, the "Shared Services" Feature is not working well. I tried to setup Spotify and Sonos Sharing on a customer with AIO and its not realy working.

regards tohil

[datniha]

Thanks for your input @tohil.

I do not follow you entirely when it comes to your network setup... sorry 🤔

Anyway, this week an update came out from HPE allowing to disable "Detect and Control Service Sharing" under the networks tile. I can confirm this works perfectly fine. So the problem is at least solved for me and others using HPE ION.

Others may still be impacted. Hence I think my suggested change to the mDNS Repeater in Opnsense to allow you to blacklist networks and host (using net mask 32). It could come in handy if you want to e.g. mute a host.

@OPNenthu:
Yes,- some sort of repeater is needed to cross networks. However, at least in my setup, for each of my VLAN's I am using the Opnsense router as gateway (x.x.x.1)...and the behavior I observe indicates that the mDNS packets are being routed to the gateway and therefore being repeated...

All the best
/Datniha