TCP BBR congestion control algorithm. Anyone using it? how to enable it?

[TheLinuxGuy]

I have been experimenting with BBR congestion algorithm from Google. It's widely available in Linux. Apparently FreeBSD may have had it committed in: https://reviews.freebsd.org/rS352657 (I am not sure how to check)

*edit: https://svnweb.freebsd.org/base?view=revision&revision=352657 * perhaps the feature request for OPNsense is to enable the build flags mentioned here?

This
is a completely separate TCP stack (tcp_bbr.ko) that will be built only if
you add the make options WITH_EXTRA_TCP_STACKS=1 and also include the option
TCPHPTS

In linux I have seen an improved performance with it and wanted to enable it for my OPNsense at home, does anyone have any guidance on how TCP BRR congestion algorithm is enabled for use in the system?

Found this but it isn't relevant to opnsense (https://fasterdata.es.net/host-tuning/freebsd/)

Code: [Select]

[root@fw ~]# ls /boot/kernel/cc_* | grep -v symbols
/boot/kernel/cc_cdg.ko
/boot/kernel/cc_chd.ko
/boot/kernel/cc_cubic.ko
/boot/kernel/cc_dctcp.ko
/boot/kernel/cc_hd.ko
/boot/kernel/cc_htcp.ko
/boot/kernel/cc_vegas.ko

If BRR is news to you, some references:
https://github.com/google/bbr
https://atoonk.medium.com/tcp-bbr-exploring-tcp-congestion-control-84c9c11dc3a9
https://www.cyberciti.biz/cloud-computing/increase-your-linux-server-internet-speed-with-tcp-bbr-congestion-control/

[dinguz]

I have experimented with alternate congestion control algorithms as well, but I found its usefulness quite limited. Enabling it on the opnsense box only tweaks performance for tcp-based services running on the box itself, but not for other machines in your own network for which the firewall is forwarding traffic.

[bolmsted]

Can the developers comment when/if we will see support for BBR TCP Congestion Control in pfSense/OPNsense?   I understand that it may be a feature of FreeBSD 13.x and if HardenedBSD is based on the same release level we should expect this to show up in HBSD 13.x as well when OPNsense integrates HBSD 13.x?

There is nothing official on this but I came across links like this
https://fasterdata.es.net/host-tuning/freebsd/
https://lists.freebsd.org/pipermail/freebsd-current/2020-April/075930.html

The FreeBSD list doesn't really refer to which version of FreeBSD with the non CC method.


Linux has this integrated for quite some time but there is no elegant front end interface like pfSense or OPNsense for Linux - too bad it wasn't ported to Linux as well with nftables backend.   If I build on linux I have to do all the setup from scratch link DNS, DHCP, firewalls, etc and not a big fan of the debian model for network configs and would much prefer the RHEL model.

I have a gigabit internet connection and they sync the GPON at 2.5Gbps which means that on upload I can actually exceed the provisioned upload rate causing packet loss, packet retransmissions, and performance degradation - thus not getting the provisioned upload rate.

I have 1Gbps/750Mbps but I should be able to get 1G/1G since they overprovision the line and right now I get 930/660 usually but if I switch to 1Gbps sync then I get 700/930 so it flips around so it seems realistic I should be able to get 1G/1G if I could prevent flooding using BBR on FreeBSD/HBSD under pfSense/OPNsense.  I may consider 1.5G/1G package one day but 1G plan is good for me especially if I could get full upload rate.


Some folks on a forum I hang out show results using Linux with BBR when syncing at 1Gbps and 2.5Gbps on a 1.5Gbps/1000Gbps plan.   

This is what it looks like for me under various conditions. performance loss is affected by latency.
1000 (No Throttle)      Latency   Down   Up
Bell Alliant - Halifax (3907)      24   792   919
Bell Canada - Montreal (17567)      1   901   933
Bell Canada - Toronto (17394)      8   765   933
Bell Mobility - Winnepeg (17395)   30   874   913
Bell Mobility - Calgary (17399)      47   782   898

2500 (No Throttle)      Latency   Down   Up
Bell Alliant - Halifax (3907)      24   1564   358
Bell Canada - Montreal (17567)      1   1638   933
Bell Canada - Toronto (17394)      8   1619   771
Bell Mobility - Winnepeg (17395)   30   1569   324
Bell Mobility - Calgary (17399)      47   1570   236

2500 (Throttled 1000 up)      Latency   Down   Up
Bell Alliant - Halifax (3907)      24   1558   915
Bell Canada - Montreal (17567)      1   1650   931
Bell Canada - Toronto (17394)      8   1620   931
Bell Mobility - Winnepeg (17395)   30   1569   885
Bell Mobility - Calgary (17399)      47   1600   890

2500 (Throttled 1000 up +bbr )      Latency   Down   Up
Bell Alliant - Halifax (3907)      24   1541   922
Bell Canada - Montreal (17567)      1   1639   940
Bell Canada - Toronto (17394)      8   1583   936
Bell Mobility - Winnepeg (17395)   30   1617   928
Bell Mobility - Calgary (17399)      47   1547   916

[Patrick M. Hausen]

In which way would this be relevant for a packet filtering firewall? OPNsense does not initiate TCP connections. The systems on both sides of the firewall do. They need to support that algorithm.

[bolmsted]

ok then how can I throttle the upload to 1000Mbps like they are doing with Linux as it seems the BBR is negligible or just doing nothing?

I've tried to use pipes/queues/rules but has a detrimental effect.   The FQ_CODEL doesn't seem to help.   

I'm trying to understand why my desktop which is connected through switch to my OPNsense gets 930/660 with a 2.5Gbps sync on the GPON module in my broadcom bxe/bnx2x but when its a 1Gbps sync on the GPON module in my broadcom bxe/bnx2x I get 700/930.

The project I'm referring to is on dslreports.com in the Bell Canada forums where people are syncing the card at 2.5Gbps to get the full 1.5+Gbps/1Gbps and passing it onto their 10Gbps LAN if they have it.  In my case I'm trying to get 1G/1G at this point before I explore that.

The folks in the dslreports.com form an a private slack forum seem to think that we are flooding the ONT on upload and hence why they are trying to use BBR and throttling the egress traffic on the Broadcom card up to the GPON network.

[Patrick M. Hausen]

I think you are confusing two things here. The TCP congestion control algorithm is part of the TCP stack on the end nodes of any communication.

https://en.wikipedia.org/wiki/TCP_congestion_control

It is not a traffic shaping method that is applied on any intermediate node like a firewall.

Am I correct that it is really traffic shaping on the firewall that you are looking for? If yes, sorry - I personally have no experience with that. Others will have to help then.

But again, using the BBR or any other algorithm takes place on the end nodes.


HTH - at least a bit
Patrick