Flow control - best practices

[devhunter55]

Dear all

i read a lot about advantages about having flow control disabled - but i'm not sure if this might be some kind of mandatory for some use cases (like suricata or ISP WAN IP) ?

I read a forum advice from a member that Flowcontrol enabled on some device solved his problem, so i'm unsure, what to do (let flowcontrol enabled on all nics or not).

"Turns out it was a problem with flow control. Once I enabled flow control for only my LAN SFP+ connection on my firewall to the switch and left flow control off for the WAN interface on the firewall both 1G and 10G clients were getting proper WAN speeds on the switch.

Remember ethernet speed mismatch can cause problems, use flow control to resolve them!
"

my current opnsense settings:

root@opnsense:~ # sysctl -a | grep dev.igb| grep -i control
dev.igb.5.rx_control: 4194304
dev.igb.5.device_control: 136053313
dev.igb.5.eee_control: 1
dev.igb.4.rx_control: 4194304
dev.igb.4.device_control: 136053313
dev.igb.4.eee_control: 1
dev.igb.3.rx_control: 4194304
dev.igb.3.device_control: 135791169
dev.igb.3.eee_control: 1
dev.igb.2.rx_control: 4194304
dev.igb.2.device_control: 136053313
dev.igb.2.eee_control: 1
dev.igb.1.rx_control: 71598082
dev.igb.1.device_control: 1075577409
dev.igb.1.eee_control: 1
dev.igb.0.rx_control: 71598082
dev.igb.0.device_control: 1075577409
dev.igb.0.eee_control: 1

Thx all for help & ideas .. and Happy Easter 

[opnfwb]

The tunables that you have posted will only adjust EEE (energy efficient ethernet). However, it's often recommended to set this to 0 as well to avoid any kind of port instability or ethernet speed negotiation issues.

To adjust flow control on igb cards within FreeBSD you'll want this tunable where X is the interface number:

Code: [Select]

sysctl dev.igb.X.fc
By default it's set to 3, you can successfully set this to 0 to disable FC on the interface. Be warned, in some cases disabling FC has left the interface unresponsive. The router is still online but traffic may stop passing in some instances. If this happens, just press the power button on the router and allow it to gracefully shutdown (it make take a minute or so to soft shutdown). Once you boot it back up, FC will be disabled and all interfaces should be working fine. I would not recommend playing with this setting unless you're physically present next to the router (don't do it for a remote office location or something like that).

I usually disable EEE and FC on all of my igb NICs and I've never had a problem afterwards (unless you include the occasional traffic freeze encountered when first toggling the setting). In my case, disabling FC actually can improve upload speed ramp up. I find if I leave FC enabled and I run a speedtest, my upload speeds slowly climb to finally reach the max speed of my connection (500/500 fibre). If I disable FC, the upload tests ramp more quickly to the maximum speed and stay there. Results may vary but that's my 2c on the matter. Let us know how it works and what options you decide on.

[devhunter55]

Thx opnfwb for this very good advice !

So, yes.. the environment is very productive and thx also the good hint to be very careful, when enabling them the very 1rst time (to be prepared when the interface(s) is/are unresponsive).

I think - i've got two opnsense hw, i could test it on the one - if it would completely fail, i could switch to the 2ncd at least.

Best
Mike

[srepper]

my current opnsense settings:

root@opnsense:~ # sysctl -a | grep dev.igb| grep -i control
dev.igb.5.rx_control: 4194304
dev.igb.5.device_control: 136053313
dev.igb.5.eee_control: 1
dev.igb.4.rx_control: 4194304
dev.igb.4.device_control: 136053313
dev.igb.4.eee_control: 1
dev.igb.3.rx_control: 4194304
dev.igb.3.device_control: 135791169
dev.igb.3.eee_control: 1
dev.igb.2.rx_control: 4194304
dev.igb.2.device_control: 136053313
dev.igb.2.eee_control: 1
dev.igb.1.rx_control: 71598082
dev.igb.1.device_control: 1075577409
dev.igb.1.eee_control: 1
dev.igb.0.rx_control: 71598082
dev.igb.0.device_control: 1075577409
dev.igb.0.eee_control: 1

Thx all for help & ideas .. and Happy Easter 

is this config still recommended?

[Siarap]

If you have problems with bufferbloat enabling or leaving flow control at default + fq_codel shaper solves the bufferbloat issue. When i disabled flow control bufferbloat test results was bad.