Messages

[while a speed test (or multiple bulk downloads for that matter) is running]

You do not implement shaping to achieve higher results in a throughput oriented speed test. That's not possible. If the provider's network or some interchange is not grossly oversubscribed you will max out your local link, anyway.

The intention of shaping is to be able to do a video conference while a speed test (or multiple bulk downloads for that matter) is running without getting flaky video or audio.

And you do sacrifice a little bit of your peak bandwidth for that, yes.

Kind regards,
Patrick

I agree.  That's an example for one of the cases where traffic shaping may have benefits; only I never had such a case.  Should I come across one, traffic shaping would probably the first thing I'd try.

Sorry but I have to correct this, as this things plague as well the enterprise world.

After all experimentation I've done, I still can only say that when you have issues that would seem to make traffic shaping useful, the issues don't go away with traffic shaping, and the only solution is to get more bandwidth.

No. To get more BW is not the solution, this is a bandage fix. Because there are and there will be cases where no matter what BW you have you still will be affected. The proper solution is to have correct and proper Queue management disciplines with proper BW sizing.

By saying that you need 'proper BW sizing' you're implying that you may need more bandwidth because traffic shaping doesn't help when you don't have enough bandwidth.  So we don't disagree other than that I have never actually seen any effects of traffic shaping that would make it worthwhile to use it.

IIUC you're now saying that traffic shaping may randomly have benefits (because some ppl claim that there are benefits, but obviously noone is able to show the benefits in practise) --- or not.  All the while its disadvantages remain excluded from the consideration.

This means that everyone needs to do their own testing and decide if they think it has benefits or not, and they can claim whatever they want.  That's a kind of mysticism, and it doesn't provide valid answers.

These are no random benefits, there are benefits which are tied to the specific implementation. Disadvantages are not excluded, they are being considered within that specific implementation.

There is nothing mystic, magic or madeup about using QoS/Shaper with Queue management. All of the algorithms were tested, there are papers and RFCs with test results. You can go and read the RFCs or lookup the white papers.


Regards,
S.

I don't doubt that some implementation of traffic shaping does what the implementation is supposed to do.  That can be tested in a lab environment.

What I'm not seeing is a benefit in practise, outside some lab environment, with internet connections you can get from some ISP.  I'm also not saying that there can't be benefits in cases in which there are benefits, only I've never come across such a case.

And as long as I don't see any benefit from traffic shaping I don't see why I should bother to use it.

You are obviously wrong. See this for an example: https://forum.opnsense.org/index.php?topic=49509.msg251434#msg251434

I'm not wrong.  The graphics you point to seem inconclusive.  They look pretty similar, and the differences can be due to lots of other factors we don't know about.  I'm getting differences between speedtests depending on wich server is being used and differences between tests when the same server is being used, with or without traffic shaping.  The only effect of traffic shaping I can observe here is that it lowers the usable bandwidth, and that doesn't seem to be an advantage.

I do not mean: Do your own testing, like you seem to imply.

Perhaps I misunderstood you.

Do you suggest that ppl just follow someones advice blindly without testing?

Why would this survivorship bias only consider the overlooked failures and not overlooked successes and everything else?  That doesn't make any sense.

IIUC you're now saying that traffic shaping may randomly have benefits (because some ppl claim that there are benefits, but obviously noone is able to show the benefits in practise) --- or not.  All the while its disadvantages remain excluded from the consideration.

This means that everyone needs to do their own testing and decide if they think it has benefits or not, and they can claim whatever they want.  That's a kind of mysticism, and it doesn't provide valid answers.

I get why you are skeptical — it really sounds like "slowing yourself down on purpose" would not help. But the key is that traffic shaping is not about speed, it is about control.

Without shaping, your router sends and receives as fast as it can until something upstream (your ISP) gets overwhelmed. Packets then pile up in the ISP's queue, latency goes through the roof, and the sender (like a website or download server) does not notice the congestion until a packet finally gets dropped far away. That delay in feedback is what causes the up-and-down "pumping" effect you see.

I'm not seeing a pumping effect.  It can be expected that there's a spike at the start of the test before the parties involved settle on a bandwidth, and I'm not seeing any disadvantages from that.  I see big advantages because I get about 10% higher bandwidth though.  So I disabled it and am not seeing any disadvantages, either.

When you enable shaping, your router makes sure your own queue is the first place that fills up — not the ISP's. It starts dropping or delaying packets slightly earlier, so the congestion signals (packet delay or loss) happen right at your end. TCP reacts much faster, keeping the link steady and latency low.

I think you have a misconception about latency.  Reducing the bandwidth through shaping only makes it so that the packets get delayed at a different place, i. e. at the router.  That means that they are being sent later and arrive later.  Without traffic shaping, the packets are being sent earlier and are then delayed at a different place, so they are being sent earlier and arrive later.

The outcome is the same.  The disadvantage is that you can not use the available bandwidth but loose about 10%.  Why would I want that?

That is easy to see on the upload side, but it actually helps downloads too.

Upload latency according to the speedtest is between 1 and 6ms.  Traffic shaping has no influence on that.  I have doubts that these values are correct, but that's what the test says.

TCP slow start and congestion control work based on the acknowledgments (ACKs) your system sends back upstream. If your upload queue gets bloated, those ACKs get delayed — which makes the sender think the link is slower than it really is, then it overshoots and backs off again. It is a mess of ramp-ups and stalls. By shaping your outgoing traffic, you keep those ACKs flowing smoothly, which makes your downloads smoother too.

That's not what the test shows.  It shows a spike at the start and after that, it settles on a bandwidth.  Acknowledging the packets seems to work fine.  That is what is to be expected.

And if you shape downstream directly (some routers can), you do the same thing on incoming packets: you hold them just before they hit your LAN instead of letting the ISP's buffer clog up. That way, you control the bottleneck and the TCP sender sees early feedback during slow start, preventing those massive bursts that cause latency spikes.

I have not seen or experienced latency spikes with or without shaping.

You can see this in action: Do a ping test or play an online game while maxing out your connection — watch the latency jump without shaping. Then turn TS on and do it again. Pings stay steady,

What do you suggest I ping?

downloads are smoother, and everything just feels "snappier". That is basically what most test sites proposed at bufferbloat.net do.

If anything, things are snappier with traffic shaping disabled.

So yes — it might sound like limiting yourself for no reason, but shaping actually moves the bottleneck to where you can control it. The physics of TCP feedback do not care where the limit is, just how quickly the sender learns about it.

That's what I've been saying, the packets are being delayed one way or another.  When the bottleneck is closer to the sender, I can assume that the sender learns sooner where the limit is.  Creating the bottlneck the furthest away from the sender as it possibly can would be the worst setup.

After all experimentation I've done, I still can only say that when you have issues that would seem to make traffic shaping useful, the issues don't go away with traffic shaping, and the only solution is to get more bandwidth.

Also, some ISPs already do a good work at adressing this. If your measurements are fine without traffic shaping enabled, then leave it be.

This one suggests I enable traffic shaping.  They didn't say that before I upgraded to more bandwidth.  I wouldn't have bothered with it if they hadn't suggested it and if I hadn't been curious.

Other than that, my suggestion is: Try it first, measure the pings, then argue. 😄

I'll try that, but what should I ping?

I'm curious about it.  In practise, it won't make a difference because I'm the only user of this connection, so there's noone else who would use all the bandwidth and cause delays for me.  Reducing the usable bandwidth for myself is more disadvantageous than anything else.

P.S.: Do you really think so many people would think about this, fine-tune it and/or use it, if it were useless?

People say lots of things.  So far, no people have actually shown any benefit of traffic shaping to me.  My own experimentations always have shown no benefits of traffic shaping.

I've even tried it with a 348Kbit connection.  You would think that traffic shaping makes all the difference, but no, it doesn't.  The congestion on a connection with so little bandwidth overcomes all traffic shaping, and it's not really usable nowadays, shaping or not.

Now, that is not the way it works. Of course you are seeing changing numbers without traffic shaping. That is exactly the pumping effect that traffic shaping tries to avoid.

Are you referring to the bandwidth or to the latency?  Bandwidth settles in after the spike at the start.  The latency continues to go up and down during the test no matter what.  When the bandwith usage is over, the last number that showed up during the test continues to be displayed.

By dropping packets earlier via traffic shaping, you can avoid the pumping effect over high BDP (bandwidth * delay product) links. If the other side does not get notified early about the buffer being overrun, it will pump data into the pipe which will subsequently get discarded. Then TCP relies on the retry mechanism to reget the dropped packets. Over a high-latency connection, it simply takes too long to accomodate for small buffers at the target.

How does the other side get notified about the buffers being overrun?  The traffic shaping on my router doesn't have any information about the conditions of the buffers that may be between the router and the target and thus has no way to inform the target about their conditions.

What does it matter where the packets are being dropped?  One way or another, some are being dropped somewhere.

Thus, it is better if the target (i.e.: you) signals early when it it gets overwhelmed by the amount of data the source (i.e.: any website you open) can pump into its uplink, which is often 10x or more than what your downlink can chew.

My router doesn't get overwhelmed.  The bandwidth is limited somewhere else before it ever reaches my router.  The ISP allows only so much bandwidth to go through my connection, and my router doesn't have any influence over that.  Same goes for upstream, it doesn't overwhelm the router but the ISP limits it somewhere else.

The only thing my router can do is limit the bandwidth even further.  And why would I want it to do that?

It still doesn't make sense.

As far as I can tell, the latency shown by the ookla speedtest shows the last value that showed up while the test was running.  So that value is some more or less random number and not even an average.  I'd expect the test to run for a couple weeks at least to provide a good average and meaningful results.

So no, I don't think I'm reading the output incorrectly.

Why should there be a problem with latency at the start of the test when no traffic shaping on the router is in place?  It's not like the transmitted data would arrive later because the buffers are full.  A single packet may arrive later when a buffer is full, but if there were traffic shaping in place, such a packet would be either dropped and later retransmitted, and/or it would be sent later.  So the latency might be not higher, or not lower.

What would explain that there would be higher latency in regards to the data that is transmitted whithout traffic shaping?  When the transfer is sustained, the bandwidth settles at the limit anyway.

I can imagine that certain types (i. e. marked) packets could be made to experience less latency by giving those a preference over other packets, but the basic traffic shaping that limits the usage of the overall bandwidth doesn't do that -- or at least I didn't configure any such preferences myself.

As said, speed suffers a little because of the reserve for control packets. To actually see the benefits of traffic shaping, you should try https://www.waveform.com/tools/bufferbloat or https://speed.cloudflare.com/ and compare the latency before and after applying these settings.

The first one gets stuck at "Warming up" in the "Active" part.  The second one only produces a message "Application error: a client-side exception has occurred (see the browser console for more information).".  They need to fix their tests ...

That is expected.

When you try to control bufferbloat, even if you would set your full BW for example 100Mbit as is your contracted speed, you will never get 100Mbit. FQ_C takes some % from the total throughput (I think 5% by default).

I thought it would adjust to the numbers I'm giving it, and I'd use appropriate numbers.

The point of all of this is to have control over the possible congestion; prevention, handling & management.

To achieve this in first place, you need to take the control from the ISP. This is done by limiting the BW in order to not trigger ISP based queue management which is usually terrible.

Why is the limiting by the ISP so terrible?

Tradeoff is you will have a lower Throughput, benefit is usually a total mitigation or/and control over the latency during congestion state.

P.S. The docs contain pictures with examples, one of them is for ookla speeds test where it show where to look for the latency during load.

During that speedtest, the numbers for the latency keep changing all the time, so I can't really tell what it is.  Upload latency seems to be much lower than the download latency.

When I set the pipe numbers to half the bandwiths I'm supposed to have, bandwith is lower and the latency seems the same.  When I set the numbers to double the bandwidth, latency is like half and I'm getting about 10% more bandwidth than I'm supposed to get.  When I set the numbers to 10% over what I'm supposed to get --- and that is more than the limits the ISP suggests --- I'm getting better bandwidth and lower latency.  I can only guess that shaping on the router doesn't kick in because I'm not reaching the bandwith.  So I guess I could as well delete all the settings because they don't give any measurable benefits and only lower the usable bandwidth.

So what is the point of this traffic shaping?  It seems to only lower the bandwith I'm getting and to increase the latency.  This doesn't make sense.

What do you suggest?  Should I just delete the settings?

BTW, there's one difference with no traffic shaping: at the start of the test, download bandwidth may spike up to about over 2.5 times of what I'm supposed to get before it goes down.  With traffic shaping, that doesn't happen.  But does it even matter?

As @meyergru said.

Both of those guides were tested and are used by several users. So following them should yield proper results.

One comment; do not use the Interface2 option in the Shaper Rules. This is only used for more granular control. 99% of all use cases you do not need it, you only need the Interface1 with the proper direction related to the Pipe/Queue config.

Regards,
S.

Well, I followed https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html and am not really happy with the results.  Basically, according to speedtest.net, I'm getting a little less bandwidth than I should when I'm using the numbers for the bandwidth I got from the ISP.  When I increase the numbers, I get more bandwidth.  I can't really tell what the latency is, though.

When I use way lower numbers (like half the bandwith I have) for the pipes, then I'm not getting as much bandwidth as I'd expect but less than the numbers would allow.

Does that mean that something isn't working right or is this to be expected?

I do not understand what your problem is. If you follow the docs for bufferbloat and control plane priorization, you do not need any IP addresses (i.e. you put "any" where you would have an IP in the rules, which is literally what the docs show).

I just left the entry field blank and it kept saying I need to specify an IP address.  There is no indication that you have to put 'any'.  The GUI should just assume 'any' when the field is left blank.  Otherwise the users have to assume that they can't set up a rule without an address.

I don't expect it to do anything, see above.  But the ISP must have some reasons to tell customers to use certain bandwith limitations, and it's a learning experience for me to try it out, and I'm curious to see what happens.

I assume that fighting bufferbloat is their aim and I am 99% sure that this is what your ISP refers to, because of the mentioning of the limits. It happens to be what most people - including me - use traffic shaping for. Read this, point 26 for an explanation.

You're probably right about that.  I downloaded a 7.9GB video today and was watching the pretty graphs and could see the bandwith usage go up and down.  So it may be a good idea to use traffic shaping.

But how could I get that to work without specifing IPs in the rules?  I tried that and the window to edit the rule kept saying that an IP address is required no matter if I was setting a second interface or not.  So I removed all the pipes, queues and rules yesterday because I didn't want a broken setup to get in the way that doesn't work anyway because I can't specify an IPv6 address range.

I didn't want to follow https://forum.opnsense.org/index.php?topic=46990.0 because I'm not seeing packet loss and rather start with a simple, basic approach and go from there.  If there is packet loss, I can always 'upgrade'.

I think it's a VOIP Desk Phone that is talked about.

It's a WIFI phone.  It has a little charging station you can it plug into.  For normal use you carry it around.  It works fine when connected via WIFI to a local vlan.

I once a few years back used a Yealink T46 as OpenVPN client, but the firmware was buggy and the Yealink support gave me custom firmwares to try but it always sucked.

A VOIP phone shouldn't need VPN, it should just use SIP/RTP directly, or encrypt it via SIP(S) and RTP(S) if required. Or use H.323 which also supports encryption.

I don't want to make the asterisk instance publicly available over the internet for everyone.

Maybe not many customers use the VPN option and the firmware is buggy.

I don't have a client log.  The phone only shows a message that it couldn't connect.

I used the OpenVPN connect app on iPhone and Android as many others and it writes a log, which you can inspect and share.

That I don't have.

The log file on the router only has entries like:


TLS Error: cannot locate HMAC in incoming packet from [AF_INET]47.237.115.221:49386
TLS Error: incoming packet authentication failed from [AF_INET]47.237.115.221:45287


Those are not from IP addresses used by the place the phone was taken to for testing, and from times when the phone wasn't tested.  There is no indication at all that the phone tried to connect, and I don't know if there should be any.  This is all I got.

At least this prooves that the VPN server is accessible from outside.

So we have to assume, that the problem is on the client side.
Was the phone connected to the internet and outside of your local network / wifi, while you tried to connect?
Did it connect to your public IP?

Yes, the phone was taken to another place to make sure it would connect via an unlreated internet connection to avoid possible interferences from trying it behind the very router that the phone is supposed to connect to.  It got a different private IPv4, and I don't think the port was blocked by the foreign router, but I can't be 100% sure.

I can't tell if it connected to the public IP or not.  I have no indication as to wether it did or didn't.  If it did connect successfully, that might not create log entries.

It seems obvious to me that "in" is like "in" on firewall rules for the first interface.

That isn't at all obvious to me.  It could be 'in to the WAN interface (1st interface) out of the LAN interface (2nd interface, or it's network if an IP address is used instead of an interface)'.  I could also be 'out of the 1st interface to the internet' or 'out of the 2nd interface in to the 1st interface'.  Or if I don't specify a 2nd interface or a network, it could mean 'in to the 1st interface from anywhere' or 'out of the 1st interface (i. e. to the internet in this case or to any interface)" or whatever.  It's entirely unclear to me.

So you would only need "in" and "out" rules for WAN with any/any as src/dst with no second interface at all if you do not want it to be significant.

Is that even possible?  For a rule that is supposed to limit the upload bandwidth to the ISP (internet), it would make sense because a rule limited to the LAN interface wouldn't limit the other interfaces.  The guide seems to require an IP address ...

You normally only want to limit in and out directions for WAN, for which the limit exists in the first place and you know what bandwidth it has.

If I have a guest network, for example, I might want to use different limits for that.  In that case I'd have to set up rules which, for example, allow only 20% of the upload bandwidth for the guest network and 80% for the LAN.  When you have a bunch of interfaces you might want to limit, like a VOIP interface for a VOIP vlan and some others, that gets tedious to set up.

That is no different that with the documentation example.

The difference would be that all traffic (from all interfaces) that goes out to the internet rather than only the traffic from then LAN interface would be limited.  The latter is what the guide does.  I actually should use a rule that limits all traffic from all interfaces that wants to go out to the internet.

The sole thing that did not exist back when that was written is the in/out distinction and the second interface, such that nextmasks were used instead.

That was a great improvement.

While you could use the same approach for different VLANs as well, it is more like a bandwitdh restriction for specific VLANs then, and yes, if you want that, you will have to set it up.

right

Apart from that: what are you trying to achieve? Fair distribution amongst your clients

I'm merely simply trying out the feature after I upgraded my connection to a bit more bandwidth, and my ISP suggested to put specific limits for upload and download.  It never hurts to learn something, and I'm curious if it would make any difference.

From what I've seen so far, OPNsense does a great job of distributing the bandwidth between the clients by default without any extra traffic shaping.

Also, I'm not a fan of traffic shaping.  The problem with that is that you can't really accomplish anything other than dropping packets.  Having to drop packets because you don't have enough bandwidth doesn't improve anything because packets inevitably get dropped anyway.  You can decide which packets to drop, but when you don't have enough bandwidth, the line is still plugged and the solution is to get more bandwidth. --- There may be cases in which you might be able to achieve improvements because you have better control over the traffic, like through the switches on your own network, but for your internet connection, the bottleneck is out of your control, and traffic shaping is pointless.

or bufferbloat optimization because of asymmetric up/downstream? For the latter, there is another documentation section that has most recently been reworked and that does actually work.

What does it do?

Unless you have clients that are far more powerful than others, I would expect this distribution not to do anything much visible. For bufferbloat issues, that is another story.

I don't expect it to do anything, see above.  But the ISP must have some reasons to tell customers to use certain bandwith limitations, and it's a learning experience for me to try it out, and I'm curious to see what happens.

So what I'm trying to accomplish is to limit upload and download to the numbers my ISP gave me, for both IPv4 and IPv6.  But there isn't even an option to set up a rule for both IPv4 and IPv6 at the same time, which I guess would be required.  And since I can't get a static IPv6 prefix, it seems I'm limited to use interfaces instead of networks when setting up rules.

Since there is no option to make a rule for both IPv4 and IPv6 at the same time, how am I supposed to set up a limit?  Do I just make two rules each for each IPv4 and IPv6 and hope that the pipe will still take care of the limit?

Now let's see if I can just use a single interface for a rule ...

Hm, what does 'ip' mean in a rule?  Does that mean both IPv4 and IPv6 or something else?

When I make a rule for upload with only the WAN interface, what does the direction mean?  I don't want to limit traffic that goes out of the WAN interface into the router but only to the ISP (internet).

And here we go: it won't let me apply the rules when I don't specify an IP address and not a 2nd interface either.  It only says "Error reconfiguring service. 200".

Why do I need queues when I can use a pipe as target for the rule?

PS: Apparently I got logged out of OPNsense, which is why it wouldn't let me apply the rules.

However, I can't make a rule without specifing an IP address.  What's the 2nd interface for then?

And that takes me back to the question: How do I specify an IPv6 address in the rule when I don't have a static prefix?