OPNsense Forum
Archive => 21.1 Legacy Series => Topic started by: TheLinuxGuy on March 01, 2021, 05:46:25 am
-
I have been experimenting with BBR congestion algorithm from Google. It's widely available in Linux. Apparently FreeBSD may have had it committed in: https://reviews.freebsd.org/rS352657 (I am not sure how to check)
*edit: https://svnweb.freebsd.org/base?view=revision&revision=352657 * perhaps the feature request for OPNsense is to enable the build flags mentioned here?
This
is a completely separate TCP stack (tcp_bbr.ko) that will be built only if
you add the make options WITH_EXTRA_TCP_STACKS=1 and also include the option
TCPHPTS
In linux I have seen an improved performance with it and wanted to enable it for my OPNsense at home, does anyone have any guidance on how TCP BRR congestion algorithm is enabled for use in the system?
Found this but it isn't relevant to opnsense (https://fasterdata.es.net/host-tuning/freebsd/)
[root@fw ~]# ls /boot/kernel/cc_* | grep -v symbols
/boot/kernel/cc_cdg.ko
/boot/kernel/cc_chd.ko
/boot/kernel/cc_cubic.ko
/boot/kernel/cc_dctcp.ko
/boot/kernel/cc_hd.ko
/boot/kernel/cc_htcp.ko
/boot/kernel/cc_vegas.ko
If BRR is news to you, some references:
https://github.com/google/bbr
https://atoonk.medium.com/tcp-bbr-exploring-tcp-congestion-control-84c9c11dc3a9
https://www.cyberciti.biz/cloud-computing/increase-your-linux-server-internet-speed-with-tcp-bbr-congestion-control/
-
I have experimented with alternate congestion control algorithms as well, but I found its usefulness quite limited. Enabling it on the opnsense box only tweaks performance for tcp-based services running on the box itself, but not for other machines in your own network for which the firewall is forwarding traffic.
-
Can the developers comment when/if we will see support for BBR TCP Congestion Control in pfSense/OPNsense? I understand that it may be a feature of FreeBSD 13.x and if HardenedBSD is based on the same release level we should expect this to show up in HBSD 13.x as well when OPNsense integrates HBSD 13.x?
There is nothing official on this but I came across links like this
https://fasterdata.es.net/host-tuning/freebsd/
https://lists.freebsd.org/pipermail/freebsd-current/2020-April/075930.html
The FreeBSD list doesn't really refer to which version of FreeBSD with the non CC method.
Linux has this integrated for quite some time but there is no elegant front end interface like pfSense or OPNsense for Linux - too bad it wasn't ported to Linux as well with nftables backend. If I build on linux I have to do all the setup from scratch link DNS, DHCP, firewalls, etc and not a big fan of the debian model for network configs and would much prefer the RHEL model.
I have a gigabit internet connection and they sync the GPON at 2.5Gbps which means that on upload I can actually exceed the provisioned upload rate causing packet loss, packet retransmissions, and performance degradation - thus not getting the provisioned upload rate.
I have 1Gbps/750Mbps but I should be able to get 1G/1G since they overprovision the line and right now I get 930/660 usually but if I switch to 1Gbps sync then I get 700/930 so it flips around so it seems realistic I should be able to get 1G/1G if I could prevent flooding using BBR on FreeBSD/HBSD under pfSense/OPNsense. I may consider 1.5G/1G package one day but 1G plan is good for me especially if I could get full upload rate.
Some folks on a forum I hang out show results using Linux with BBR when syncing at 1Gbps and 2.5Gbps on a 1.5Gbps/1000Gbps plan.
This is what it looks like for me under various conditions. performance loss is affected by latency.
1000 (No Throttle) Latency Down Up
Bell Alliant - Halifax (3907) 24 792 919
Bell Canada - Montreal (17567) 1 901 933
Bell Canada - Toronto (17394) 8 765 933
Bell Mobility - Winnepeg (17395) 30 874 913
Bell Mobility - Calgary (17399) 47 782 898
2500 (No Throttle) Latency Down Up
Bell Alliant - Halifax (3907) 24 1564 358
Bell Canada - Montreal (17567) 1 1638 933
Bell Canada - Toronto (17394) 8 1619 771
Bell Mobility - Winnepeg (17395) 30 1569 324
Bell Mobility - Calgary (17399) 47 1570 236
2500 (Throttled 1000 up) Latency Down Up
Bell Alliant - Halifax (3907) 24 1558 915
Bell Canada - Montreal (17567) 1 1650 931
Bell Canada - Toronto (17394) 8 1620 931
Bell Mobility - Winnepeg (17395) 30 1569 885
Bell Mobility - Calgary (17399) 47 1600 890
2500 (Throttled 1000 up +bbr ) Latency Down Up
Bell Alliant - Halifax (3907) 24 1541 922
Bell Canada - Montreal (17567) 1 1639 940
Bell Canada - Toronto (17394) 8 1583 936
Bell Mobility - Winnepeg (17395) 30 1617 928
Bell Mobility - Calgary (17399) 47 1547 916
-
In which way would this be relevant for a packet filtering firewall? OPNsense does not initiate TCP connections. The systems on both sides of the firewall do. They need to support that algorithm.
-
ok then how can I throttle the upload to 1000Mbps like they are doing with Linux as it seems the BBR is negligible or just doing nothing?
I've tried to use pipes/queues/rules but has a detrimental effect. The FQ_CODEL doesn't seem to help.
I'm trying to understand why my desktop which is connected through switch to my OPNsense gets 930/660 with a 2.5Gbps sync on the GPON module in my broadcom bxe/bnx2x but when its a 1Gbps sync on the GPON module in my broadcom bxe/bnx2x I get 700/930.
The project I'm referring to is on dslreports.com in the Bell Canada forums where people are syncing the card at 2.5Gbps to get the full 1.5+Gbps/1Gbps and passing it onto their 10Gbps LAN if they have it. In my case I'm trying to get 1G/1G at this point before I explore that.
The folks in the dslreports.com form an a private slack forum seem to think that we are flooding the ONT on upload and hence why they are trying to use BBR and throttling the egress traffic on the Broadcom card up to the GPON network.
-
I think you are confusing two things here. The TCP congestion control algorithm is part of the TCP stack on the end nodes of any communication.
https://en.wikipedia.org/wiki/TCP_congestion_control
It is not a traffic shaping method that is applied on any intermediate node like a firewall.
Am I correct that it is really traffic shaping on the firewall that you are looking for? If yes, sorry - I personally have no experience with that. Others will have to help then.
But again, using the BBR or any other algorithm takes place on the end nodes.
HTH - at least a bit ;)
Patrick