OPNsense Forum

English Forums => General Discussion => Topic started by: Curdie on May 21, 2019, 05:31:59 pm

Title: fq_codel latency spikes
Post by: Curdie on May 21, 2019, 05:31:59 pm: I have a 10 megabit MetroE through Frontier and I'm sharing it among too many users. I've been using fq_codel to try to keep latency low for VoIP and gaming while providing reasonable bandwidth for streamers and downloads. I'm failing miserably.

I've set up fq_codel and it mostly works (I get an A+ for bufferbloat from dslreports) but when new streamers/downloads/tests start latency will momentarily spike by around 100 ms. This kills the gamers and happens quite often during peak time.

I've no idea what's wrong with my setup. Please help!

# ipfw sched show
Code: [Select]
10000: 8.000 Mbit/s 0 ms burst 0 q75536 50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 50ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 10000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 4 669 0 0 0 10001: 8.000 Mbit/s 0 ms burst 0 q75537 50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail sched 10001 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 50ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 10001 0 ip 0.0.0.0/0 0.0.0.0/0 3 457 0 0 0
# ipfw pipe show
Code: [Select]
10000: 8.000 Mbit/s 0 ms burst 0 q75536 50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail sched 75536 type FIFO flags 0x0 0 buckets 0 active 10001: 8.000 Mbit/s 0 ms burst 0 q75537 50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail sched 75537 type FIFO flags 0x0 0 buckets 0 active
Code: [Select]
60001 48276 17525072 queue 10001 esp from any to *ipsec site* via igb0 // wan: upload queue 60002 61714 76825600 queue 10000 esp from *ipsec site* to any via igb0 // wan: download queue 60003 426694 548488221 queue 10000 ip from any to *lan subnet* via igb0 // wan: download queue 60004 317134 43959271 queue 10001 ip from *lan subnet* to any via igb0 // wan: upload queue
Title: Re: fq_codel latency spikes
Post by: muchacha_grande on May 22, 2019, 12:33:02 pm: Hi... I think that you will need to put some rules to differentiate traffic and pass it through different queues.
You may generate two uptream queues with different weights. The one with a higher weight will be used for voip and the one with the lower one for the rest of the traffic. Then, in the rules you just put a rule that passes voip traffic through the voip queue and the rest through other one.
Recently OPNSense has implemented DSCP filtering on the advanced rules. DSCP is a tag that works at IP packet level that voip traffic uses to identify itself as a lattency sensitive traffic. You just have to make a rule like this at the beggining of your list:
Enable Advanced Mode
INTERFACE: WAN
PROTO: IP
SOURCE: ANY
SRC-PORT: ANY
DESTINATION: ANY
DST-PORT: ANY
DSCP: (here you put the DSCP number used by your voip traffic)
DIRECTION: OUT
TARGET: VOIP UPSTREAM QUEUE (the one with a higher weight)

Note that if your voip traffic does not uses DSCP to identify itself, you will need to change the rule to filter the traffic using another method, for example, the source IP address or the port, but try to configure DSCP on your voip traffic to use the rules as I said before.

Bye
Title: Re: fq_codel latency spikes
Post by: Curdie on May 22, 2019, 07:50:28 pm: I could certainly do that for VoIP but I will still need to resolve this issue with the pipe and scheduler. Latency should not be spiking when new downloads begin. It appears I'm exceeding the bandwidth set in my pipe by quite a lot; traffic graph shows 10.5-11 when my download begins with an 8 megabit bandwidth limit on my pipe. I'm not sure how this can be. Maybe my rules are incomplete? They seem very simple and air-tight to me, which leads me to believe that I'm missing some really foundational concept.
Title: Re: fq_codel latency spikes
Post by: Curdie on May 22, 2019, 08:04:02 pm: Code: [Select]
# ipfw show 00100 0 0 allow pfsync from any to any 00110 0 0 allow carp from any to any 00120 0 0 allow ip from any to any layer2 mac-type 0x0806,0x8035 00130 0 0 allow ip from any to any layer2 mac-type 0x888e,0x88c7 00140 0 0 allow ip from any to any layer2 mac-type 0x8863,0x8864 00150 0 0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd 00200 0 0 skipto 60000 ip6 from ::1 to any 00201 2472 3379472 skipto 60000 ip4 from 127.0.0.0/8 to any 00202 0 0 skipto 60000 ip6 from any to ::1 00203 0 0 skipto 60000 ip4 from any to 127.0.0.0/8 06000 678786 579395407 skipto 60000 tcp from any to any out 06199 805279 667974243 skipto 60000 ip from any to any 30000 0 0 count ip from any to any 60000 0 0 return ip from any to any 60001 23016 12858000 queue 10003 esp from any to *ipsec site* via igb0 // wan: ipsec upload 60002 21873 19908104 queue 10002 esp from *ipsec site* to any via igb0 // wan: ipsec download 60003 402163 550469553 queue 10000 ip from any to *lan subnet* via igb0 // wan: download queue 60004 269819 31212155 queue 10001 ip from *lan subnet* to any via igb0 // wan: upload queue 65533 769666 636301310 allow ip from any to any 65534 0 0 deny ip from any to any 65535 106 102938 allow ip from any to any
Title: Re: fq_codel latency spikes
Post by: Curdie on May 22, 2019, 08:07:53 pm: I'm seeing *some* traffic in the scheduler:

Code: [Select]
# ipfw sched show 10000: 8.500 Mbit/s 0 ms burst 0 q75536 50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 1ms interval 3ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 10002 10000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 58 85552 2 3000 1 10001: 9.000 Mbit/s 0 ms burst 0 q75537 50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail sched 10001 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 75ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 10001 10003 0 ip 0.0.0.0/0 0.0.0.0/0 8 320 0 0 0
Title: Re: fq_codel latency spikes
Post by: Curdie on May 22, 2019, 10:31:34 pm: The ethernet link to my Metro-E switch port was set to auto-negotiate on both sides and had negotiated half-duplex 100 Mbps. Switched to 100 Mbps full-duplex. Frontier engineer also removed shaping on my link. Problems seem to have gone away. Wish I knew which change made the difference, but I am extremely happy to have a resolution. I now get the best results with the recommended settings for a 10 megabit link, and an 8.5 megabit pipe limit:

Code: [Select]
10000: 8.500 Mbit/s 0 ms burst 0 q75536 50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail sched 10000 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 1ms interval 3ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 10002 10000 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 1 1500 0 0 0 10001: 8.500 Mbit/s 0 ms burst 0 q75537 50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail sched 10001 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 100ms interval 5ms quantum 300 limit 600 flows 1024 NoECN Children flowsets: 10001 10003
Additional practical testing will be required to verify that it is indeed fixed, but I am optimistic!

muchacha_grande, thank you so much for taking the time to help me. Having a second set of eyes look my rules over made me a lot more confident about calling Frontier for help.
Title: Re: fq_codel latency spikes
Post by: muchacha_grande on May 23, 2019, 04:15:07 am: You're wellcome...

Glad to see it's solved.

Cheers...

PS: In my case, I have 50 Mbit downstream/10 Mbit upstream fiber. I had to set limiter about 7.5 Mbit upload to qualify A+. I think it's because with a home class connection you can't get full speed and high quality at the same time.