IPv6 Control Plane with FQ_CoDel Shaping

[OPNenthu]

In regards of security applications. ICMPv6 for IPv6 functionality needs to be allowed. By design any control for any protocol needs to be allowed in both ways.

Understood, but, ICMPv6 has many types.  In the default OPNsense ruleset, is my router allowed to send/respond RAs and NDs on the open internet?

EDIT: I did forget for a moment that IPv6 is meant to be globally routeable, although still not sure.

I missed this earlier, too.  Makes sense.
"[...] But to make such rule more tighter, the source or destination depending on the rule direction should be the FW/GW itself"

Can you try it?

Sure, will do some testing.

[Seimus]

Yes. I cleared the respective parts. I am at a loss what difference is actually causing the problem. Maybe it is easier to try to break my working setup by changing towards your suggested setup step-by-step to find the root cause if it is not that casual glitch both you and @OPNenthu saw.

I must say your setup is a mystery to me of why this is happening.
It could be the glitch, when changing playing with Shaper sometimes its just janked. Even thou in CLI when you verify if the config is correct (in ipfw) which it is still the results may not be as expected.

In on paper, the Control Plane, is an addition if there is already a Pipe existing, so that means there should be no changes needed to an already established Pipe flow other than changing the BW (subtraction of BW from one Pipe to a New one)


Regards,
S.

[Seimus]

Understood, but, ICMPv6 has many types.  In the default OPNsense ruleset, is my router allowed to send/respond RAs and NDs on the open internet?

EDIT: I did forget for a moment that IPv6 is meant to be globally routeable, although still not sure.

I missed this earlier, too.  Makes sense.
"[...] But to make such rule more tighter, the source or destination depending on the rule direction should be the FW/GW itself"

It would have to be specified by the IPv6 RFC4890. Similar as is in the default rules I believe.

Sure, will do some testing.

In advance many thanks!

Regards,
S.

[meyergru]

So I tested further starting from my working setup.

1. Removed FQ-Codel parameters from the pipes. Test was O.K., but results are worse (B) than before (A+). Switching back and forth broke my connection completely once.
2. Removing the masks from the pipes changed nothing, tests went O.K., still A+ grading.
3. Enlarging the bandwith from 1 to 10 MBit/s on the control plane pipes changed nothing.
4. Removing the masks from the up- and downstream queues changed nothing.
5. Disabling the icmp (v4) rules changed nothing.
6. Creating queues for the control plane and point the control plane rules for icmp-ipv6 to those changed nothing.

So I arrived almost at the recommended setup, with the only difference of the FQ-Codel parameters and PIE on the pipes enabled (I also tried with no PIE, which changed nothing).

Then I reduced the up- and downstream bandwidth (the old ones were optimized for attainable speed) to 900/600 to verify that the shaper had to do anything at all. This worked as well and got this result. Note that there is no more latency increase with either up- or download.

For reference, this is the relevant config section now:

Code Select Expand

    <TrafficShaper version="1.0.3">
      <pipes>
        <pipe uuid="bbe0a667-ed41-4f7b-b47e-8ab22286a1fb">
          <number>10000</number>
          <enabled>1</enabled>
          <bandwidth>600</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue/>
          <mask>none</mask>
          <buckets/>
          <scheduler>fq_codel</scheduler>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>1</codel_ecn_enable>
          <pie_enable>1</pie_enable>
          <fqcodel_quantum>1500</fqcodel_quantum>
          <fqcodel_limit>20480</fqcodel_limit>
          <fqcodel_flows>65535</fqcodel_flows>
          <origin>TrafficShaper</origin>
          <delay/>
          <description>Upstream Pipe</description>
        </pipe>
        <pipe uuid="020a34ef-cd71-4081-9161-286926ee00cc">
          <number>10001</number>
          <enabled>1</enabled>
          <bandwidth>900</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue/>
          <mask>none</mask>
          <buckets/>
          <scheduler>fq_codel</scheduler>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>1</codel_ecn_enable>
          <pie_enable>1</pie_enable>
          <fqcodel_quantum>1500</fqcodel_quantum>
          <fqcodel_limit>20480</fqcodel_limit>
          <fqcodel_flows>65535</fqcodel_flows>
          <origin>TrafficShaper</origin>
          <delay/>
          <description>Downstream Pipe</description>
        </pipe>
        <pipe uuid="fb829d32-e950-4026-a2ee-3663104a355b">
          <number>10003</number>
          <enabled>1</enabled>
          <bandwidth>10</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue/>
          <mask>none</mask>
          <buckets/>
          <scheduler/>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <pie_enable>0</pie_enable>
          <fqcodel_quantum/>
          <fqcodel_limit/>
          <fqcodel_flows/>
          <origin>TrafficShaper</origin>
          <delay/>
          <description>Upload-Control</description>
        </pipe>
        <pipe uuid="883ed783-df03-4109-9364-a6c387f5954f">
          <number>10004</number>
          <enabled>1</enabled>
          <bandwidth>10</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue/>
          <mask>none</mask>
          <buckets/>
          <scheduler/>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <pie_enable>0</pie_enable>
          <fqcodel_quantum/>
          <fqcodel_limit/>
          <fqcodel_flows/>
          <origin>TrafficShaper</origin>
          <delay/>
          <description>Download-Control</description>
        </pipe>
      </pipes>
      <queues>
        <queue uuid="0db3f4e6-daf8-4349-a46f-b67fdde17c98">
          <number>10000</number>
          <enabled>1</enabled>
          <pipe>020a34ef-cd71-4081-9161-286926ee00cc</pipe>
          <weight>100</weight>
          <mask>none</mask>
          <buckets/>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>1</codel_ecn_enable>
          <pie_enable>0</pie_enable>
          <description>Downstream Queue</description>
          <origin>TrafficShaper</origin>
        </queue>
        <queue uuid="d846a66a-a668-4db8-9c92-55d5c172e7af">
          <number>10001</number>
          <enabled>1</enabled>
          <pipe>bbe0a667-ed41-4f7b-b47e-8ab22286a1fb</pipe>
          <weight>100</weight>
          <mask>none</mask>
          <buckets/>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>1</codel_ecn_enable>
          <pie_enable>0</pie_enable>
          <description>Upstream Queue</description>
          <origin>TrafficShaper</origin>
        </queue>
        <queue uuid="6c535ef5-1aa5-4760-a94e-b6f72af55dd8">
          <number>10002</number>
          <enabled>1</enabled>
          <pipe>883ed783-df03-4109-9364-a6c387f5954f</pipe>
          <weight>100</weight>
          <mask>none</mask>
          <buckets/>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <pie_enable>0</pie_enable>
          <description>Control-plane-IPv6-Queue-Download</description>
          <origin>TrafficShaper</origin>
        </queue>
        <queue uuid="a71074a0-e387-4ff6-8203-1f7e08ef7b32">
          <number>10003</number>
          <enabled>1</enabled>
          <pipe>fb829d32-e950-4026-a2ee-3663104a355b</pipe>
          <weight>100</weight>
          <mask>none</mask>
          <buckets/>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <pie_enable>0</pie_enable>
          <description>Control-plane-IPv6-Queue-Upload</description>
          <origin>TrafficShaper</origin>
        </queue>
      </queues>
      <rules>
        <rule uuid="9eba5117-ad2e-450a-96ed-8416f5f278da">
          <enabled>1</enabled>
          <sequence>3</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ip</proto>
          <iplen/>
          <source>any</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>any</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <dscp/>
          <direction>in</direction>
          <target>0db3f4e6-daf8-4349-a46f-b67fdde17c98</target>
          <description>Downstream Rule</description>
          <origin>TrafficShaper</origin>
        </rule>
        <rule uuid="3c347909-3afd-4a14-b1e2-8eb105ff99a0">
          <enabled>1</enabled>
          <sequence>4</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ip</proto>
          <iplen/>
          <source>any</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>any</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <dscp/>
          <direction>out</direction>
          <target>d846a66a-a668-4db8-9c92-55d5c172e7af</target>
          <description>Upstream Rule</description>
          <origin>TrafficShaper</origin>
        </rule>
        <rule uuid="844829a2-ece6-4d34-ab2c-27c2ba8cef76">
          <enabled>1</enabled>
          <sequence>1</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ipv6-icmp</proto>
          <iplen/>
          <source>any</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>any</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <dscp/>
          <direction>out</direction>
          <target>a71074a0-e387-4ff6-8203-1f7e08ef7b32</target>
          <description>Upload-Control Rule ICMPv6</description>
          <origin>TrafficShaper</origin>
        </rule>
        <rule uuid="16503037-a658-438c-8be5-7274cece9dde">
          <enabled>1</enabled>
          <sequence>2</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ipv6-icmp</proto>
          <iplen/>
          <source>any</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>any</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <dscp/>
          <direction>in</direction>
          <target>6c535ef5-1aa5-4760-a94e-b6f72af55dd8</target>
          <description>Download-Control Rule ICMPv6</description>
          <origin>TrafficShaper</origin>
        </rule>
      </rules>
    </TrafficShaper>

So maybe it really is a glitch were playing around with the params does sometimes break things...

As for the bandwith limits: there seems to be a tradeoff between maximum attainable speed, which can be reached when you actually add 5% to your maximum speed without traffic shaping, at the expense of some increased latency which still gives an A+ rating. If you want no latency increase at all, you will have to sacrifice on attainable speed.

[OPNenthu]

Testing with intermediary queues seems just as good/stable as without.  Actually the home internet is busy currently and I still am posting good pings while under additional load from speedtest:

Code Select Expand

C:\>ping -6 -n 10 2606:4700:4700::1111

Pinging 2606:4700:4700::1111 with 32 bytes of data:
Reply from 2606:4700:4700::1111: time=13ms
Reply from 2606:4700:4700::1111: time=13ms
Reply from 2606:4700:4700::1111: time=11ms
Reply from 2606:4700:4700::1111: time=14ms
Reply from 2606:4700:4700::1111: time=12ms
Reply from 2606:4700:4700::1111: time=14ms
Reply from 2606:4700:4700::1111: time=13ms
Reply from 2606:4700:4700::1111: time=10ms
Reply from 2606:4700:4700::1111: time=15ms
Reply from 2606:4700:4700::1111: time=11ms

Ping statistics for 2606:4700:4700::1111:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 10ms, Maximum = 15ms, Average = 12ms

Bufferbloat remains A+.

speedtest.net result:

You cannot view this attachment.

This is with QFQ for control pipes and FQ_CoDel+ECN on default pipes.  No masks, PIE, or CoDel params.  All queue weights 100.  IPv4 icmp and other rules/queues disabled (only testing ipv6-icmp on control plane, everything else to default rules).

Actually I have a question about this: if I want to add back ipv4 icmp to the control plane, I will need to create 2 more queues.  What weights should they get?  Are they also 100, or do we split the difference (50-50) with the ipv6-icmp queues?

[Seimus]

@meyergru
Once again many thanks. So basically this confirms the config in doc works as suggested.

Few comments from my side.

So I arrived almost at the recommended setup, with the only difference of the FQ-Codel parameters and PIE on the pipes enabled (I also tried with no PIE, which changed nothing).

Actually i would not call it a difference, as mentioned previously if you already have a Pipe created, this should remain as is only BW should be subtracted. The main point of Control Plane class is to allocate it its own BW, and to take out the potential back-pressure caused by the sojourn time FQ_C for examples relies on.

That config provided looks correct to me. In your original configuration you had masks, queue(value) in Pipe. This actually doesn't do anything if you have a manually created queue connected to it. It is only applicable in case you attach a rule directly to the Pipe. ECN in queue applies only for CoDel not FQ_C.

As for the bandwith limits: there seems to be a tradeoff between maximum attainable speed, which can be reached when you actually add 5% to your maximum speed without traffic shaping, at the expense of some increased latency which still gives an A+ rating. If you want no latency increase at all, you will have to sacrifice on attainable speed.

Believe or not but this is expected :D. The reason behind it is I think the FQ that FQ_C and FQ_P use. Fair Queue has problems to provide consistent rate, or maximum rate so it takes away like 3%-5% of the BW set in Pipe. DRR for example is better in regards of this but can create insane latency due to the deficit calculation. QFQ should be able as well, but problem is FQ_C and FQ_P implementation are only available in FQ not QFQ.

Regards,
S.

[OPNenthu]

3. Enlarging the bandwith from 1 to 10 MBit/s on the control plane pipes changed nothing.

I too am observing no benefit from increasing the download control pipe.

Maybe for servers this is a good rule of thumb?  As a home internet user with an asymmetrical data plan, should I reasonably expect to have proportionally higher control traffic on ingress than on egress?

As for the bandwith limits: there seems to be a tradeoff between maximum attainable speed, which can be reached when you actually add 5% to your maximum speed without traffic shaping, at the expense of some increased latency which still gives an A+ rating

Interesting.  I always have been leaving some bandwidth on the table to optimize for latency (as per the Bufferbloat guide) but my speed without shaping measures above 900Mbit/s (sometimes bursts up to 1Gbps), even though it's advertised at only 800. 

Setting the Download pipe to 910Mbit/s gets me this: https://www.waveform.com/tools/bufferbloat?test-id=477bf3a2-4b43-4f61-91bd-f9c0f44c668a

+5 on D/L latency, but still A+.

Though I wonder if that will start to break down during peak use when all the neighbors are online.

[Seimus]

@OPNenthu many thanks for testing!

I am glad you could provide similar results as @meyergru did.

This is with QFQ for control pipes and FQ_CoDel+ECN on default pipes. 

Perfect! > this is exactly how I would like to have it tested.

QFQ overall should provide more consistent rates vs WFQ. So its always worth to try the one or another. Yet keep in mind guys this is only affecting the Control Plane nothing else, as rest of the traffic is in different Pipes with different schedulers.


In regards of your question

Actually I have a question about this: if I want to add back ipv4 icmp to the control plane, I will need to create 2 more queues.  What weights should they get?  Are they also 100, or do we split the difference (50-50) with the ipv6-icmp queues?

Yes, keep in mind, we want to to keep control planes of different protocols separated, yet utilize the BW dedicated for Control Plane as such. The weight depends on you, or rather depends on the rate of the specific control plane, how much BW each of them you want to give.

So if you set the weights 50 & 50 in theory during saturation each of them will get 500Kbit/s if BW is 1Mbit/s.

[Seimus]

I too am observing no benefit from increasing the download control pipe.

Maybe for servers this is a good rule of thumb?  As a home internet user with an asymmetrical data plan, should I reasonably expect to have proportionally higher control traffic on ingress than on egress?

Actually no, because control plane usually is a consistent rate it should be okay with very low BW values set to the specification minimum.
We need to keep in mind that the rules for Control Plane Shaper do not only involve the control plane. It as well catches pings. Thus the 1Mbit is optimal to a certain degree.

Interesting.  Interesting.  I always have been leaving some bandwidth on the table to optimize for latency (as per the Bufferbloat guide) but my speed without shaping measures above 900Mbit/s (sometimes bursts up to 1Gbps), even though it's advertised at only 800.

Setting the Download pipe to 910Mbit/s gets me this: https://www.waveform.com/tools/bufferbloat?test-id=477bf3a2-4b43-4f61-91bd-f9c0f44c668a

+5 on D/L latency, but still A+.

Though I wonder if that will start to break down during peak use when all the neighbors are online.

What you see is actually correct. And purely depends on how your ISP divides the BW to its customers. Its possible they have overprovisioning. Its as well possible during peak hours when more users from that ISP are online, it will cause on an aggregation point a stall and you may not reach those speeds which could cause additional latency.

Also if you over-provision your BW on one Pipe, the moment that BW is not available it will start to eat into other Pipes affecting the Control Plane.

Its always better to create a BW buffer, cause if you dont you will be on the mercy of your ISP to handle the bufferbloat.

Regards,
S.

One fun fact; When using FQ_C even if you set higher BW than you really have, FQ_C due to its algorithm is still capable somewhat to manage the latency. It will not be so good as when you have a BW buffer created, but its 10x better compared to not have any FQ_C at all.

[OPNenthu]

Thanks @Seimus and @meyergru for all the inputs so far.  I'm learning a lot.

I went back to re-enable all my previous queues & rules for things like TCP-ACK and DNS.  In the process I renamed my objects to normalize them.  I am now again seeing the glitch that we talked about where Bufferbloat test is stalled.  Also, speedtest.net is showing reduced bandwidth.

So there really is truth to this.  There is some issue that crops up when you are adjusting the Shaper objects.

I won't reboot anything this time.  Will wait to see if it clears on its own.

[OPNenthu]

Adding to the previous post-

I observe that UDP traffic is impeded.  The screenshot here shows only outgoing, but it's actually in both directions.  TCP seems fine.

Not sure if these are queue drops or bad firewall state?

I've been connected to a VPN provider over UDP the whole time (on a different client/VM) and when I run speedtest these red lines appear in the FW log now.  The VPN endpoint is the destination.

Meanwhile, D/L speeds have degraded further.  VPN remains connected.  Packet drops only observed when running speedtest (I guess pointing to a queue issue).

Will post back when something changes; still holding off on reboot.

[Seimus]

Not sure if these are queue drops or bad firewall state?

Queue drops would not generate those logs in Live.
You can see if a Shaper is dropping via cli sommand.

Code Select Expand

ipfw queue show
ipfw sched show
ipfw pipe show

However those live logs if real, are blocking sessions.

Also have a look as well on Interface drop count, parent interface LAN and WAN.

Question is 
is it the same Source and Destination? Or only one specific?
when you click the "i" on the blocked one what additional info does it tells you?

Regards,
S.

[OPNenthu]

Still happening hours later.  It may be time finally for a reboot.

Code Select Expand

ipfw queue show
ipfw sched show
ipfw pipe show

Code Select Expand

root@firewall:~ # ipfw sched show
10002:   1.000 Mbit/s    0 ms burst 0
 sched 10002 type QFQ flags 0x0 0 buckets 0 active
   Children flowsets: 10009 10007
10003:   1.000 Mbit/s    0 ms burst 0
 sched 10003 type QFQ flags 0x0 0 buckets 0 active
   Children flowsets: 10008 10006
10000: 849.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 10004 10002 10000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       82     5431  0    0   0
10001:  39.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 10001 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 10005 10003 10001
  0 ip           0.0.0.0/0             0.0.0.0/0     13301 19218727 23 34500  99

If I'm reading correctly, there were 99 drops on upstream here (39Mbit/s CoDel).

Code Select Expand

root@firewall:~ # ipfw queue show
q10006  50 sl. 0 flows (1 buckets) sched 10003 weight 50 lmax 1500 pri 0 droptail
q10007  50 sl. 0 flows (1 buckets) sched 10002 weight 50 lmax 1500 pri 0 droptail
q10004  50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0 droptail
q10005  50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0 droptail
q10002  50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0 droptail
q10003  50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0 droptail
q10000  50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0 droptail
q10001  50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0 droptail
q10008  50 sl. 0 flows (1 buckets) sched 10003 weight 50 lmax 1500 pri 0 droptail
q10009  50 sl. 0 flows (1 buckets) sched 10002 weight 50 lmax 1500 pri 0 droptail


Code Select Expand

root@firewall:~ # ipfw pipe show
10002:   1.000 Mbit/s    0 ms burst 0
q141074  50 sl. 0 flows (1 buckets) sched 75538 weight 0 lmax 0 pri 0 droptail
 sched 75538 type FIFO flags 0x0 0 buckets 0 active
10003:   1.000 Mbit/s    0 ms burst 0
q141075  50 sl. 0 flows (1 buckets) sched 75539 weight 0 lmax 0 pri 0 droptail
 sched 75539 type FIFO flags 0x0 0 buckets 0 active
10000: 849.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 75536 type FIFO flags 0x0 0 buckets 0 active
10001:  39.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 75537 type FIFO flags 0x0 0 buckets 0 active
root@firewall:~ #


Also have a look as well on Interface drop count, parent interface LAN and WAN.

LAN is on a LAGG group (2 x 2.5Gbps)

The 'Output Errors: 13' count was there from before.  I've noticed that for a long time on the LAGG IF.

You cannot view this attachment. You cannot view this attachment.

Question is 
is it the same Source and Destination? Or only one specific?

The speedtest is still abnormal and Bufferbloat test is stalled as before, but the only blocks in the F/W log are from those specific src/dest addresses which is the VPN connection.  Other traffic seems to be getting passed as normal.

when you click the "i" on the blocked one what additional info does it tells you?

It pops up rule info for the in-built 'force gw' rule:

You cannot view this attachment.

[OPNenthu]

The router reboot did the trick, but there was some settling needed as well.  Immediately following the reboot the latencies were high on the bufferbloat test and the FW log was still showing blocked traffic.  Several minutes later that all cleared up and now the tests are back to normal.

I do see tail drops on the upload data pipe/scheduler, only during the upload portion of the speed tests.  I think this is expected, though.  This is probably CoDel/AQM doing its job.

I also rebooted the VM where the VPN was connected and made sure that was again active / working during the tests.

Code Select Expand

root@firewall:~ # ipfw sched show
10002:   1.000 Mbit/s    0 ms burst 0
 sched 10002 type QFQ flags 0x0 0 buckets 0 active
   Children flowsets: 10009 10007
10003:   1.000 Mbit/s    0 ms burst 0
 sched 10003 type QFQ flags 0x0 0 buckets 0 active
   Children flowsets: 10008 10006
10000: 849.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 10004 10002 10000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       19     1140  0    0   0
10001:  39.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 10001 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 10005 10003 10001
  0 ip           0.0.0.0/0             0.0.0.0/0     51935 74618246 26 38507 395

So, long story short, messing with shaping can in some instances cause initial instability that needs a reboot + settling time.

[Seimus]

I do see tail drops on the upload data pipe/scheduler, only during the upload portion of the speed tests.  I think this is expected, though.  This is probably CoDel/AQM doing its job.

Yes, this is basically FQ_C taking care of packets that are too long in the Flow queue. FQ_C will drop "if their sojourn times exceed the target setting for longer than the interval". Sadly because those are dynamic and under scheduler, we dont see specific Flows only as whole thats why there is 0.0.0.0/0.

So, long story short, messing with shaping can in some instances cause initial instability that needs a reboot + settling time.

Agree, as several peoples were able to experience this and its possible to replicate.

Regards,
S.