fq_codel latency spikes

[Curdie]

I have a 10 megabit MetroE through Frontier and I'm sharing it among too many users. I've been using fq_codel to try to keep latency low for VoIP and gaming while providing reasonable bandwidth for streamers and downloads. I'm failing miserably.

I've set up fq_codel and it mostly works (I get an A+ for bufferbloat from dslreports) but when new streamers/downloads/tests start latency will momentarily spike by around 100 ms. This kills the gamers and happens quite often during peak time.

I've no idea what's wrong with my setup. Please help!

# ipfw sched show

Code: [Select]

10000:   8.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 50ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 10000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        4      669  0    0   0
10001:   8.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 10001 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 50ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 10001
  0 ip           0.0.0.0/0             0.0.0.0/0        3      457  0    0   0

# ipfw pipe show

Code: [Select]

10000:   8.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 75536 type FIFO flags 0x0 0 buckets 0 active
10001:   8.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 75537 type FIFO flags 0x0 0 buckets 0 active


Code: [Select]

60001   48276  17525072 queue 10001 esp from any to *ipsec site* via igb0 // wan: upload queue
60002   61714  76825600 queue 10000 esp from *ipsec site* to any via igb0 // wan: download queue
60003  426694 548488221 queue 10000 ip from any to *lan subnet* via igb0 // wan: download queue
60004  317134  43959271 queue 10001 ip from *lan subnet* to any via igb0 // wan: upload queue


[muchacha_grande]

Hi... I think that you will need to put some rules to differentiate traffic and pass it through different queues.
You may generate two uptream queues with different weights. The one with a higher weight will be used for voip and the one with the lower one for the rest of the traffic. Then, in the rules you just put a rule that passes voip traffic through the voip queue and the rest through other one.
Recently OPNSense has implemented DSCP filtering on the advanced rules. DSCP is a tag that works at IP packet level that voip traffic uses to identify itself as a lattency sensitive traffic. You just have to make a rule like this at the beggining of your list:
Enable Advanced Mode
INTERFACE: WAN
PROTO: IP
SOURCE: ANY
SRC-PORT: ANY
DESTINATION: ANY
DST-PORT: ANY
DSCP: (here you put the DSCP number used by your voip traffic)
DIRECTION: OUT
TARGET: VOIP UPSTREAM QUEUE (the one with a higher weight)

Note that if your voip traffic does not uses DSCP to identify itself, you will need to change the rule to filter the traffic using another method, for example, the source IP address or the port, but try to configure DSCP on your voip traffic to use the rules as I said before.

Bye

[Curdie]

I could certainly do that for VoIP but I will still need to resolve this issue with the pipe and scheduler. Latency should not be spiking when new downloads begin. It appears I'm exceeding the bandwidth set in my pipe by quite a lot; traffic graph shows 10.5-11 when my download begins with an 8 megabit bandwidth limit on my pipe. I'm not sure how this can be. Maybe my rules are incomplete? They seem very simple and air-tight to me, which leads me to believe that I'm missing some really foundational concept.

[Curdie]

Code: [Select]

# ipfw show
00100      0         0 allow pfsync from any to any
00110      0         0 allow carp from any to any
00120      0         0 allow ip from any to any layer2 mac-type 0x0806,0x8035
00130      0         0 allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140      0         0 allow ip from any to any layer2 mac-type 0x8863,0x8864
00150      0         0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200      0         0 skipto 60000 ip6 from ::1 to any
00201   2472   3379472 skipto 60000 ip4 from 127.0.0.0/8 to any
00202      0         0 skipto 60000 ip6 from any to ::1
00203      0         0 skipto 60000 ip4 from any to 127.0.0.0/8
06000 678786 579395407 skipto 60000 tcp from any to any out
06199 805279 667974243 skipto 60000 ip from any to any
30000      0         0 count ip from any to any
60000      0         0 return ip from any to any
60001  23016  12858000 queue 10003 esp from any to *ipsec site* via igb0 // wan: ipsec upload
60002  21873  19908104 queue 10002 esp from *ipsec site* to any via igb0 // wan: ipsec download
60003 402163 550469553 queue 10000 ip from any to *lan subnet* via igb0 // wan: download queue
60004 269819  31212155 queue 10001 ip from *lan subnet* to any via igb0 // wan: upload queue
65533 769666 636301310 allow ip from any to any
65534      0         0 deny ip from any to any
65535    106    102938 allow ip from any to any


[Curdie]

I'm seeing *some* traffic in the scheduler:

Code: [Select]

# ipfw sched show
10000:   8.500 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 1ms interval 3ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 10002 10000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       58    85552  2 3000   1
10001:   9.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 10001 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 75ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 10001 10003
  0 ip           0.0.0.0/0             0.0.0.0/0        8      320  0    0   0


[Curdie]

The ethernet link to my Metro-E switch port was set to auto-negotiate on both sides and had negotiated half-duplex 100 Mbps. Switched to 100 Mbps full-duplex. Frontier engineer also removed shaping on my link. Problems seem to have gone away. Wish I knew which change made the difference, but I am extremely happy to have a resolution. I now get the best results with  the recommended settings for a 10 megabit link, and an 8.5 megabit pipe limit:

Code: [Select]

10000:   8.500 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 1ms interval 3ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 10002 10000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1     1500  0    0   0
10001:   8.500 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 10001 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 100ms interval 5ms quantum 300 limit 600 flows 1024 NoECN
   Children flowsets: 10001 10003

Additional practical testing will be required to verify that it is indeed fixed, but I am optimistic!

muchacha_grande, thank you so much for taking the time to help me. Having a second set of eyes look my rules over made me a lot more confident about calling Frontier for help.

[muchacha_grande]

You're wellcome...

Glad to see it's solved.

Cheers...

PS: In my case, I have 50 Mbit downstream/10 Mbit upstream fiber. I had to set limiter about 7.5 Mbit upload to qualify A+. I think it's because with a home class connection you can't get full speed and high quality at the same time.