DEC3840 - very slow throughput

[Berzerker]

How are you measuring that?

IPerf plug-in on FW or IPerf.server-DEC3840-IPerf.client?

I believe just the iperf plugin on the firewall.

[NW4FUN]

I see...

Directly plugged into DEC3840 or via Switch (if so, can you name make, model and SW version pls)?

[Cerberus]

* pppoe wan
* 10g

In my experience that did not work well with high bandwith, no matter if FreeBSD or Linux, pppoe has way to much overhead for that.

[NW4FUN]

* pppoe wan
* 10g

In my experience that did not work well with high bandwith, no matter if FreeBSD or Linux, pppoe has way to much overhead for that.

I don't disagree with that, however, as part of my testing, I've had my ISP to provide me with DHCP instead of PPPoE and guess what? Situation remains unchanged...

[Berzerker]

I see...

Directly plugged into DEC3840 or via Switch (if so, can you name make, model and SW version pls)?

I'm actually going through multiple switches, mostly unifi equipment, Enterprise-8-PoE then a Switch Aggregation, the up to the DEC.

Latest versions on everything.

* pppoe wan
* 10g

In my experience that did not work well with high bandwith, no matter if FreeBSD or Linux, pppoe has way to much overhead for that.

* pppoe wan
* 10g

In my experience that did not work well with high bandwith, no matter if FreeBSD or Linux, pppoe has way to much overhead for that.

I don't disagree with that, however, as part of my testing, I've had my ISP to provide me with DHCP instead of PPPoE and guess what? Situation remains unchanged...

In my case, I'm not even using PPPoE and this is just within different VLAN networks.

With some tuning, I was able to get it up to about 7Gbps, but still much lower than the expected 14+ advertised.

[NW4FUN]

Can you what tuning have you applied to increase performances?

[Berzerker]

Can you what tuning have you applied to increase performances?

Turn on VLAN Hardware Offload, enable the parent interface (no IPs or anything, just assign it and enable), also enabled HT interrupts as per https://wiki.freebsd.org/Networking/10GbE/Router#Allow_interrupts_on_HTT_logical_CPUs this BSD 10Gb tuning article

[NW4FUN]

Turn on VLAN Hardware Offload, enable the parent interface (no IPs or anything, just assign it and enable), also enabled HT interrupts as per https://wiki.freebsd.org/Networking/10GbE/Router#Allow_interrupts_on_HTT_logical_CPUs this BSD 10Gb tuning article

I'm not sure how to enable HTT interrupts, however, would it make any sense at all since DEC3840 is AMD based whilst HT instructions are Intel's?

[Berzerker]

Turn on VLAN Hardware Offload, enable the parent interface (no IPs or anything, just assign it and enable), also enabled HT interrupts as per https://wiki.freebsd.org/Networking/10GbE/Router#Allow_interrupts_on_HTT_logical_CPUs this BSD 10Gb tuning article

I'm not sure how to enable HTT interrupts, however, would it make any sense at all since DEC3840 is AMD based whilst HT instructions are Intel's?

I assumed it may have had something to do with general multi-core enhancement, rather than *specifically* HT, I did see about a 1Gbps performance increase adding that, so /shrug.

[Berzerker]

Looks like 22.1.8 was just released, anyone update and see if there's any improvements?

[franco]

I don't think there are significant changes on 22.1.x and I'm not sure why people sometimes wonder if there are... release notes state what is being worked on and source repositories have annotated changes too.

As a general principle: performance gain is nice, but stability is much nicer still.


Cheers,
Franco

[Berzerker]

I don't think there are significant changes on 22.1.x and I'm not sure why people sometimes wonder if there are... release notes state what is being worked on and source repositories have annotated changes too.

As a general principle: performance gain is nice, but stability is much nicer still.


Cheers,
Franco

All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.

Or am I looking at the specifications of this firewall incorrectly. It mentions 14.6Gbps throughput, does that include inbound *and* outbound or should I reasonably expect full near 10Gb routing performance for traffic going both in and out of the box?

[opnfwb]

Maybe try these tunables and see if they are any help? The IBRS especially has helped me get more throughput on VMware.

Code: [Select]

hw.ibrs_disable:1
net.inet.rss.enabled:1
net.isr.maxthreads:-1

[franco]

All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.

Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.


Cheers,
Franco

[Berzerker]

All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.

Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.


Cheers,
Franco

Well, Deciso are the ones that posted the numbers, are they not? If the numbers are saying that "14.6Gbps" is a total of 7Gbps coming into the box, and 7Gbps going out of the box from source to destination, then that is *incredibly* misleading especially for a 1000 euro+ piece of equipment. If I'm misinterpreting those numbers and I should, realistically, see a full near 10Gb inter-VLAN routing performance from this box (given I can get this on other routers, my switching hardware and clients aren't the problem), then there's something either wrong with my config or there's some tuning that needs to be involved.

Or perhaps is there a special version of OPNsense that these are supposed to run pre-tuned to properly achieve the advertised performance numbers?