Bridge bandwidth shared?

[tuxevil]

Hi everyone, I just installed an opnsense firewall on a supermicro server with 6 10gbps nics
https://www.ebay.com/itm/175222506227

I added a 1gbps nic for WAN and bridge all 6 10gbps ports together as documentation and tutorials says.

Everything is working "fine" but I did a few iperf3 tests and I saw my bridge bandwidht being shared.

I mean, I have 4 servers with 1gbps nics connected directly to each 10gbps port in the bridge.

When I do one iperf3 test between server 1 and server 2 it runs at almost 1gbps, but when I start a second iperf3 test between server 3 and server 4, both tests goes down to around 500mbps.

Why is this happening, shouldnt it have to work as a virtual direct link between the 2 ports so it may run at full speed (1gbps in this case)?

What can i be missing?

I will be upgrading server's nics to 2.5gbps soon, before going to 10gbps, but i would like to solve this issue first.

Thanks for any tips you can provide!

[Patrick M. Hausen]

FreeBSD is not a switch. Bridging is done in software, so depending on your CPU the resulting bandwidth will be lower than your interface speed. There have been improvements in FreeBSD 13, but for anything beyond 1G I would recommend using a real switch.

[tuxevil]

FreeBSD is not a switch. Bridging is done in software, so depending on your CPU the resulting bandwidth will be lower than your interface speed. There have been improvements in FreeBSD 13, but for anything beyond 1G I would recommend using a real switch.

oh i get it, im using an i5-4570T low power cpu, when i ran those 2 speed test, cpu usage increase to around 70%, maybe if i get an i7 with more power you think network performance will increase?

is there another way to take advantage of the 10gbps nics?

[Patrick M. Hausen]

I'll go and ask the author of the bridge code if there are any real world measurements. Give me a few days, please.

[tuxevil]

thanks a lot!

[Patrick M. Hausen]

That went way quicker than expected.

First some links:
https://freebsdfoundation.org/blog/500-if_bridge-performance-improvement/
https://www.youtube.com/watch?v=KNMW1_g5Hxk
https://issue.freebsdfoundation.org/publication/?i=660151&ver=html5&p=27

Second Kristof's explanation:

Benchmark results will vary wildly depending on setup and hardware. There is a reason the universal advice about performance is "benchmark your workload".

My old numbers seem to suggest ~18 million packets per second, or about 8Gbps in 64-byte packets. It should easily saturate a 10Gb link with a more realistic packet size mix.

[...]

Secondly, if_bridge is sufficiently fast that using iperf3 you're no longer measuring what if_bridge can do, but what the socket layer can do. iperf is the wrong tool for high-performance network benchmarking. It spends most of its time copying data in and out of userspace. It's simply unable to generate sufficient load to produce useful results. Use pkt-gen/netmap or DPDK or some hardware packet generator.

Kind regards,
Patrick

[tuxevil]

Great info, im getting into it
Thanks for the fast response!