Traffic shaping backup/streaming traffic

[mosfet]

Hi there OPNsense community 

I am trying to set up traffic shaping to limit Dropbox (or other backup solution) bandwidth usage and prioritize Netflix (or other service) streaming. Both these services run on port 80/443, so using the port as a method for filtering the traffic into each queue is not working.

Is there another way of doing this? Surely, this must be a common problem and perhaps the single biggest reason to use traffic shaping?

I am now using rules to send VoIP traffic as highest priority (weight 100) and sending the bulk 80/443 traffic in a queue with weight 50. This ensures my skype calls are lag-free, but does not solve the lag free streaming when backing up data.

[PimB]

It's a hassle but you could lookup the ASN('s) of Netflix

[Ciprian]

I like "Weighted Fair Queuing" + "FQ CoDel" for anything as such: since I'm not able to differentiate traffic at Layer 7, at least I force the traffic to be weighted and fairly shared among internal clients, as well as different services for the same client.

The most you can do is to give a higher priority in shaper to clients you use for streaming (even if for cloud bkp too), than to those clients that don't stream. Meaning, that weight 50 bulk 80/ 443 you mentioned should be further split to weight 66 based on streaming clients' IP addresses, and weight 33 for non-streaming. Maybe you will still have cloud bkp & streaming "fights", but only on those devices that do both streaming and cloud backups, and only when they do it in the same time. This cases, the weighted fair queuing should really help.

[mosfet]

It's a hassle but you could lookup the ASN('s) of Netflix

Thanks for the input. A quick google search returned this:
https://www.reddit.com/r/PFSENSE/comments/6wd2ay/netflix_asnips_alias_for_routing_enjoy/

Haven´t tried it yet, so I´m not sure how updated it is.

Special mention of this post in the thread above:
https://www.reddit.com/r/PFSENSE/comments/6wd2ay/netflix_asnips_alias_for_routing_enjoy/dm8j7ux

[mosfet]

I like "Weighted Fair Queuing" + "FQ CoDel" for anything as such: since I'm not able to differentiate traffic at Layer 7, at least I force the traffic to be weighted and fairly shared among internal clients, as well as different services for the same client.

The most you can do is to give a higher priority in shaper to clients you use for streaming (even if for cloud bkp too), than to those clients that don't stream. Meaning, that weight 50 bulk 80/ 443 you mentioned should be further split to weight 66 based on streaming clients' IP addresses, and weight 33 for non-streaming. Maybe you will still have cloud bkp & streaming "fights", but only on those devices that do both streaming and cloud backups, and only when they do it in the same time. This cases, the weighted fair queuing should really help.

Thanks for the great input! I just realized I may have misunderstood the "Weight" value. I thought this was a priority of the traffic (i.e. higher weighted traffic has higher priority than lower weighted queues). Is it actually a percentage of the allocated bandwidth within a pipe?

[Ciprian]

Thanks for the great input! I just realized I may have misunderstood the "Weight" value. I thought this was a priority of the traffic (i.e. higher weighted traffic has higher priority than lower weighted queues). Is it actually a percentage of the allocated bandwidth within a pipe?

Yes, only not a percentage, but a proportion: say you have (like upon) a weight of 100, one of 66, and one of 33, this case every concurrent traffic made by 3 clients corresponding one to each weight you will have a shape of 3 datagrams (packets) routed for client in weight 100, followed by 2 datagrams routed for client in weight 66 and followed by 1 datagram routed for client in weight 33. Further, say that client in weight 100 finished its traffic activity, weight 66 will still get 2 datagrams routed for every 1 packet routed for client in 33. Or else, if 66 finished its transfer, 100 will get 3 datagrams passed for every 1 datagram of 33.

Theoretically, lets say you have 100 clients, and each has a corresponding individual weight: first 100 datagrams will be passed and routed for client in 100, followed by 99 datagrams passed and routed for client in 99, and so on and so forth... making client in weight 1 having only 1 datagram passed and routed for it only after 4999 datagrams have been passed and routed for everybody else... But this is a case quite not likely to be encountered in real setups.

Speaking about such theoretical setups, this is the point of all net neutrality debate: no ISP should be allowed to implement such a setup, neither towards services, neither between own clients.

[mosfet]

Thank you for your great explanation!