UDP Broadcast Relay

[marjohn56]

Opening a discussion here to move this daemon development forward. Although my work on the plugin is complete and working well, testing of the daemon itself, while not having any issues with it performing in the way it's designed, has brought up some options that may improve it further. We've been doing some of this by PMs, but it will be easier to throw it open to all.


@bertofurth has been doing some work on the daemon code and we want to reach out for further testing. I'll let him kick off with what he has discovered.

[bertofurth]

Hi everyone,

Thanks for welcoming me to this forum. I'm not very experienced with OPNsense or FreeBSD but marjohn56 has been extremely helpful and generous with his time trying to bringing me up to speed.

I wanted to get your feedback in regards to the "udp-broadcast-relay" tool. It's a similar concept to the "mdns-repeater" plugin but for other UDP based discovery protocols as well.

I'm going to dedicate this particular message to how the tool, in it's original form, currently works (Cutting and pasting from the docs I'm trying to write). If you're already familiar with the tool then please go to my next post where I'm going to point out some small problems I've found and some questions I have and ask for some feedback about what (if anything) should be done.


udp-broadcast-relay is a tool that allows automatic network discovery and configuration protocols that normally only work within a single LAN to successfully operate over multiple subnets / VLANs / broadcast domains.

The way udp-broadcast-relay works is that it listens for broadcast or multicast packets on a specified UDP port and then relays the packets to other specified interfaces.

Some examples of UDP based Network Discovery Protocols include...
     - mDNS (Chromecast, Apple Bonjour Networking, others)
     - SSDP (UPnP, DLNA Media Servers, others)
     - Microsoft Windows Network Discovery without preconfigured AD or WINS server
     - Multi Player Game Server Discovery (Minecraft, Warcraft, etc)
     - IoT Device Discovery (Sonos Speakers, Lifx Bulbs, etc)
                 
A separate instance of udp-broadcast-relay is run for each UDP port being relayed and each instance is assigned a unique numerical "ID" by the user. By default udp-broadcast-relay listens for broadcast packets but can be configured to listen for protocol specific IP multicast addresses instead.

Packets received by the tool are relayed as they have come from the original broadcaster except...
     - The source IP may be optionally set as either the original packet source IP (default), another specific IP address, or to the IP address of the outgoing interface.
     - For broadcasts the destination subnet wide broadcast address is modified to suit the output interface. So an incoming UDP broadcast with destination (e.g.) 192.168.1.255, might be relayed to an output interface with a corresponding modified address of (e.g.) 192.168.99.255, or 10.2.3.255, depending on the IP address of the outgoing interface.
     - The TTL is NOT decremented by 1 like a normal router would do. Instead it is set to a value corresponding to the ID of the udp-broadcast-relay instance that is being run (ID + 64).

On the behavior with TTL, this is so that the tool can use the TTL as a kind of "tag" to identify already relayed packets in order to avoid floods. If it sees that an incoming packet has the same TTL as the "tag" value then it is discarded. This is useful in networks where broadcasts/multicasts are incorrectly transmitted back to the source by faulty/poorly configured switches. It also means that if packets are being relayed by one router, but then need to be relayed by another, then the two instances of udp-broadcast-relay running on the separate routers need to use different IDs otherwise the second router in the path will discard the packets.

This tool is geared towards home networks, or to small business networks where administrators don't have the time or resources to set up such things a Windows Servers with Active Directory, or DNS servers that cater for an Apple Networking environment. Where the tool only operates over a small number of networks the extra overhead of relayed broadcasts is not at all detrimental.

It is not easily suited to a sophisticated corporate or service provider network with a large number of subnets. In these cases there's the potential for this tool to generate a very large number of broadcasts across networks. In networks where there are multiple redundant paths between subnets there is the potential for loops to occur unless very careful configuration is applied. Since this tool does not decrement IP header TTL, if loops occur then packet storms can result.

I have attached a screen shot of the beta plugin written by marjohn56 as running on my network. I have tested Chromecast, Apple Networking, Windows Network Discovery, Media Service Discovery, and some gaming apps across subnets and they appear to work nicely.

One thing to note is that Chromecast mDNS Discovery works *WELL* with this tool. What tends to happen with some other similar tools (e.g. avahi reflector. (N.B. Correct me if this is now wrong)) is that Chromecast devices on other subnets will pop in and out of the list of available devices.

This is because most Chromecast devices are programmed to ignore discovery packets from sources not in the local subnet. This means that relayed client discovery packets are discarded and clients are only seeing remote Chromecast devices briefly when the Chromecast devices periodically announce themselves every 2 or 3 minutes.

The udp-broadcast-relay tool has the ability to set the source IP address of relayed discovery packets to the local IP address of the outbound router interface. This means that remote Chromecast devices do not ignore discovery packets forwarded from remote clients and they answer the requests straight away.

When using this tool firewalls, NAT and other such facilities operating on the host system need to be taken into account. This tool does not automatically create firewall rules that allow discovery traffic to flow between clients and servers, or for servers to respond to clients. In addition this tool does not disable any configured NAT rules for relayed packets.

If another service on the host system is using a UDP port that is also specified as a port for a udp-broadcast-relay instance then there may be a conflict causing either the service or udp-broadcast-relay to fail. Use a command such as "sockstat -l -P udp" or "netstat -p udp -n" to view UDP ports that are currently in use on a system.

Here are some examples follow of how to configure this tool in the proposed OPNsense plugin for common Broadcast/Multicast based discovery systems. (Please help add to this list if you know of other UDP based protocols that work in a similar way). Note that in all cases you also need to specify at least 2 interfaces, and specify a unique ID. Also note that the special value "Source Address = 1.1.1.1" means to relay packets using the outbound interface IP address.

mDNS (Chromecast/Apple Bonjour) : Port = 5353, Multicast Address = 224.0.0.251, Source = 1.1.1.1
SSDP (UPnp/DLNA Media) : Port = 1900, Multicast Address = 239.255.255.250
Windows Networking Netbios Name Service : Port = 137
Windows Networking SMB : Port = 138
Syncthing Discovery : Port = 21027
Lifx Bulb Discovery : Port = 56700
Broadlink IR Emitter Discovery : Port = 80
Warcraft 3 Server Discovery : Port = 6112


Thanks everyone

[bertofurth]

My previous message in this thread details how udp-broadcast-relay works.

Here I'd like to note some of the problems, observations and questions I've had while testing the tool using the latest beta plugin written by marjohn56. Please comment on one, some or all. I've summarized them here and I've put the details of each further below....

1) Should we change the tool to preserve TTL and use something else as the "ID" marker (maybe DSCP)?
2) Should we allow multiple multicast addresses to be specified per instance (e.g. mDNS on 224.0.0.251 and non standard 224.0.0.250)?
3) How important / useful would be adding IPv6 functionality?
4) When creating a plugin like this one based on an existing external tool that hasn't really been actively maintained, how much modification is it normally acceptable to do?
5) Very rare corner case involving "IPv4 Upstream Gateway" and Multicasted packets.


Details:

1) TTL : The tool currently marks the TTL of outbound relayed packets with a value of (instance ID + 64). The problem I see here is that I suspect there are some devices out there that are picky about the TTL because they want to ensure that they are only dealing with local traffic. For example from https://tools.ietf.org/html/rfc6762#section-11

11.  Source Address Check

   All Multicast DNS responses (including responses sent via unicast)
   SHOULD be sent with IP TTL set to 255.  This is recommended to
   provide backwards-compatibility with older Multicast DNS queriers
   (implementing a draft version of this document, posted in February
   2004) that check the IP TTL on reception to determine whether the
   packet originated on the local link.  These older queriers discard
   all packets with TTLs other than 255.

To be frank, I haven't seen any direct examples of this kind of behavior. Does anyone know of examples where devices are picky about TTL?

I have tested a modified version of the tool that uses IP DSCP as the field to mark and preserves the TTL of the original packet. Since DSCP is only 6 bits it means that the range of the instance ID shrinks from (1 - 99) down to (1 - 63). I didn't touch the ECN bits (least significant 2 bits in ToS) because I've seen some evidence that a small number of hosts might not like it when ECN is modified inappropriately.

Another alternative might be to use the IP Packet Header Identification field to store a "tag" but it may be that some devices might react poorly to seeing a lot of packets with the same IP ID.



2) Multiple Multicast Addresses per instance.
I've been informed that in some cases mDNS can not only use the standard 224.0.0.251 address but might also use a non standard 224.0.0.250 address as well. Is this common? Does this happen in any other discovery protocols that you know of? If so then we might need to consider adding support for Multiple Multicast Addresses per entry. I *think* the original tool supports this now, but the beta plugin for OPNsense doesn't. Thoughts?


3) IPv6 : The original tool only supports IPv4. Do you think there's any significant call for IPv6 support? In my own network which is dual stacked I noticed that a fair few devices are sending discovery messages with both protocols. That being said everything still worked fine with IPv4 only. If an IPv6 version were developed should it be integrated into the original or should it become a completely separate tool and plugin?


4) How much to modify the original : What's the consensus on how much a tool being converted to an OPNsense package can be modified or improved? I have some suggestions seen above here in regards to TTL and IPv6 which would significantly alter the original tool. Given that the tool doesn't seem to be really under significant active development is that appropriate to do or do we want to stick as closely as possible to the original?


5) Corner case with "IPv4 Upstream Gateway" and Multicast packets (rare)
I've left this one till last because it is the least important issue, and I don't expect many others to encounter it but it's the one I spent the most time and effort trying to troubleshoot and therefore I have some emotional investment in it!! I couldn't get Chromecast working properly while this problem was present.

To summarize, if an interface is configured with an explicit "IPv4 Upstream Gateway" pointing at host "w.x.y.z", then an automatic rule is generated that for some reason causes pf to set the destination MAC address to "01:00:5e:x:y:z" for all multicast traffic _originating_ from the interface (that is, using the source IP address of the interface)

This means that multicast traffic generated from the interface has an invalid multicast mac address and it is therefore dropped by receiving hosts and other network devices (like my OpenWRT based access point.)

A bug was raised about this behavior, but for some reason it wasn't fixed and it looks like it's not going to be.

https://github.com/opnsense/core/issues/3629

The issue affects other multicast related plugins like "mdns-repeater" and it also occurs when you do a simple "ping" to a multicast address with a source IP of an affected interface.

The workaround/solution for me was simply to remove the explicit "IPv4 Upstream Gateway" on my affected interface and set it back to the default of "Auto-detect". (I tested this and everything worked fine with Chromecast after I did this) The bug implies that another workaround is to construct an extra pf rule on the affected interface that excludes multicast traffic from the auto-generated "IPv4 Upstream Gateway" rule.

The problem with all of this is that I misinterpreted the purpose of this "IPv4 Upstream Gateway" field. Based on the name of the field I thought it was to specify a host to use as a default gateway if the interface was up but it's something else entirely. I didn't need to specify it but as I was configuring the affected interface for the first time I thought it was something I had to set up.



Thanks everyone, I look forward to your feedback.

[marjohn56]

I'll throw some responses in here, as I started the whole shebang. In answer to your questions

• If this does not affect anything that uses DSCP then I think it's a good idea, remembering that changing the TTL was why we use it in some cases.
• I have gone back to the original Linux version and can see no reference to multiple multicast instances, where did you see that?
• Not much I think, as IPv4 is never going to disappear or only if certain devices are IPv6 specific, which I doubt we'll see in our lifetime!
• I  think as long as you give credit to the original and state where the fork came from you can do what you wish, a bit like Opnsense.
• That one is for the devs. Franco if you read this it's your baby, or we could add a rule creation to the plugin itself.

[bertofurth]

Hi marjohn56,

Thanks again for the excellent work you did to produce this plugin. I'm already using it at home and now my kids can play minecraft together without me having to make sure their iPads are on the same SSID!! :-)



With the multiple mcast addresses, if you run the binary with multiple "--multicast <addr>" specified then it will listen for multicasts using all the specified addresses. e.g if you wanted to relay udp port 7777 being sent using either 224.1.1.1 or 224.2.2.2 you could use.....

root@OPNsense:~ # /usr/local/sbin/udpbroadcastrelay  --id 22 --dev re0 --dev re0_vlan7 --port 7777 --multicast 224.1.1.1 --multicast 224.2.2.2 -f -d
ID set to 22
Port set to 22
Forking Mode enabled
ID: 22 (ttl: 86), Port 7777
re0: 1 / 10.0.2.10 / 10.0.2.255
re0_vlan7: 6 / 10.0.7.1 / 10.0.7.255
found 2 interfaces total
IP_ADD_MEMBERSHIP:              10.0.2.10 224.1.1.1
IP_ADD_MEMBERSHIP:              10.0.7.1 224.1.1.1
IP_ADD_MEMBERSHIP:              10.0.2.10 224.2.2.2
IP_ADD_MEMBERSHIP:              10.0.7.1 224.2.2.2
Done Initializing

I thought this might be useful because I was lead to believe that mDNS might sometimes work on multiple multicast addresses at the same time (both 224.0.0.251 and 224.0.0.250) but it seems that this probably isn't the case.

Does anyone know of any multicasted discovery protocols that work on different mcast addresses but the same UDP port?



Completely agree on the IPv4/IPv6. I was just wondering if anyone had an example of a case where they needed IPv6 urgently.


Thanks again marjohn56!

[marjohn56]

OK, I'll add the option to add multiple multicast addresses. Hopefully I can do it this weekend.

[nivek1612]

Great thread

Marjohn has done some great work getting the plugin sorted. One day he will forgive me for all the work I caused him by saying “eh take a look at this pimd thing I’ve found” as always he did his normal “that’s really good but here’s something better”

Back to the questions

I’ve not needed  to specify more than one mcast address but I agree allowing it will future proof the plugin

I like the idea of using the DSCP as the id and allowing the user to manually set the TTL value maybe via advanced options. Again future proofing

[marjohn56]

Multiple multicast is done, I'll post it later. I cannot do the Instance ID/TTL change until the daemon is updated. I'll send out the plugin shortly.

[bertofurth]

Hi marjohn56,

I couldn't see a way to add an attachment to a PM so here are my proposed changes attached to this public post!!

I've attached 2 files that I've modified and I think will help to move udp-broadcast-relay to using DSCP for ID marking rather than TTL.

I've put a switch "-t" in to revert back to the original behavior of using TTL in case anyone finds that modifying DSCP causes problems.

Because DSCP is only 6 bits, I've had to reduce the range of the ID from 1 -> 99 down to 1 -> 63. Is that ok? If not then we can just make it DSCP == (id & 63) or something.

I've also fixed a few small bugs and spelling errors and I've added a few things to the "help" dialog of the cli.....you might not like that stuff so feel free to remove it.

I tested the compiled binary and it seems to work well with your most recent plugin. Everything in my network still seems to see each other.

Thanks marjohn56!

berto.

[bertofurth]

Oh one more thing...I was going to suggest that if you approve of the "-t" option to revert to the ttl to being the tag then perhaps it could be added as an "Advanced" option in the OPNsense plugin.

Thanks again!

berto.

[marjohn56]

OK,  I'll take a gander. I'll PM you.

[mimugmail]

Hi guys,

I dont need/use any functions of this tool, but if you have a stable version, a stable repo and a stable build process, I can add it as a port to FreeBSD to make it some kind of official so the plugin get's merged.

[marjohn56]

Thanks Michael, I'll checkout Berto's changes and then I'll update the repo that's waiting on ports now.

[marjohn56]

Hi guys,

I dont need/use any functions of this tool, but if you have a stable version, a stable repo and a stable build process, I can add it as a port to FreeBSD to make it some kind of official so the plugin get's merged.

Updated the commit on ports. It's there waiting for you to do your thing. This has 'quite considerable' changes, tidy ups and additions that Berto has done too.

[mimugmail]

Which commit on ports? I only saw a force push to the plugin branch ...