Single thread routing throughput / Router CPU recommendations?

[stefan00]

Hi,

I run a DEC3850 as firewall and inter-vlan router connected via 10G sfp+. No complicated / many rules.

For comparison, I am interested in SINGLE thread routing throughput numbers.

Doing some tests I noticed that inter-vlan (routing) throughput is about 3Gbit/s, using a single iperf3 thread. Tests run through the firewall / router, both client and server are not CPU saturated.

Since I put together another custom router box and want to use OpnSense as platform, I do some comparisons at the moment. My goal is to have as much single thread throughput as possible. Just for the sake of it.

My questions are:

- how much GBit/s routing performance can I expect single thread on a DEC3850?

- If more single thread routing performance is needed, what should I update (CPU?) and what numbers can I expect on (which?) a CPU?

best & thanks,
Stefan

[meyergru]

You should definitely re-think if that is really the right question.

I do not know if you know much about IP traffic and how it works. Measuring real throughput with iperf3 is debatable at best. When you look at the documentation, you will find may obscure options that reference IP stack history, like '-C congestion_algorithm', which in fact alters the result considerably.

The first thing to say is take a look at "bandwidth * delay product". Depending on the protocol, delay and buffer sizes, this will influence your results quite hefty. This is because if you send only one packet and wait for an ACK, this will have a pause of DELAY during which no data is transferred. Imagine a DELAY of 1ms: During that time, another 1000 bytes could have been transferred at 1000000 bytes/s (i.e. 10 GBit/s) if you had a big enough buffer "in flight". TCP (which is iperf3's default protocol) does that, but on WAN links, the delays are much larger and the maximum buffer size for iperf3 is quite limited (option -w).

That is why more than one thread is used for measuring instead.

You can try for yourself: On my machine (a DEC750), the througput between two VLANs is much higher with UDP (option -u). Also, there are other options to improve routing speeds like RSS.

You see: "Just for the sake of it" is a weird approach. You will only get so far with the limitations of iperf3. Otherwise, you would have to overcome those limits first by modifying options, only to hit an imaginary goal.

A more sensible approach was if you have a specific application in mind for which you want to improve speeds and look at what you need to achieve it with the given limitations.
For inter-VLAN transfers at high speeds, you will have to use large buffers or multiple connections at once (bbcp, bcopy or lftp come to mind).

With WAN connections, all that should realistically matter is the round-trip delay and the buffer size that you need. That must be solved by the application, not the router.

I once had such a problem which illustrates the principle very well. The real background to the story was that the company I worked for had two locations, one of which piled up big stacks of unprocessed data while the other one (configured identically) worked fine. The culprit turned out to be the roundtrip delay: One site had 25ms, the other one 4ms.

I wish that more people would focus on their goals before jumping to conclusions about the means to achieve it. As an analogy: If you want to transport ten tons of material from city A to city B, it may turn out that buying a faster car is the wrong approach. Use a slow truck (or a few of them instead), it will turn out faster.

[stefan00]

@meyergru : First of all, thank you for the detailed reply

As stated before, I want to build a new router box. Single stream testing is - at least to me - a good starting point for getting base / raw numbers on routing 1) performance.

I like to go single stream testing first because everything that comes afterwards is somehow based on it (buffers, RSS, cpu binding, congestion - whatever). Additionally, I always assume that I have no control of the client / server protocols passing the router.

On the machine I used for testing right now, single stream routing throughput limits to ~3GBit/s. One core is maxed out while testing. Unless something would be terribly misconfigured, I therefore assume that the bottleneck is the CPU. 2)

However, I started this thread because I could not find any benchmark sources on how much routing performance I can expect "per stream per core", based on different CPU models.

Also, it would be nice to hear what the specs on this are for the DEC3850 (v1).

I would be glad if someone has rough numbers / resources on that.



--

1) Testsing: iperf3 single stream, Linux client / server. Switching only runs ~9.5Gbit/s both directions, 2 switches between the boxes, no errors / drops / retr.

2) depending on configuration, it varies. Enabling RSS it's mostly to irq, but overall single stream throughput does not change noticeably.

[stefan00]

For anyone curious about the initial question:

The CPU is the limiting factor for routing on standard hardware. I went the big way and plugged a recent i5-14600K into the board: Voilá, ~8-9Gbit/s routing single stream between vlans / networks.