OPNsense Forum

English Forums => Hardware and Performance => Topic started by: patient0 on February 04, 2022, 05:23:13 pm

Title: OPNsense DEC700 AMD 10GbE drivers FreeBSD/Linux
Post by: patient0 on February 04, 2022, 05:23:13 pm
Hi,

Please move this post if not the right place or remove if necessary ... I'm aware it's probably offtopic since not really OPNsense related, but at least OPNsense hardware related  :)

Last week I got the DEC740 I ordered and I'm playing around with it, especially with the 10GbE interfaces.

Since OPNsense got newer drivers for the 10GbE than FreeBSD 13 do you have any idea when they get included in FreeBSD? I couldn't find anything related to this driver. But then I may just looked at the wrong place.

I did compile a custom FreeBSD stable/13 kernel with the code diff for the axgbe from OPNsense and it works well. Having it upstream would of course be even easier.

What I did see though is a difference of cpu load when receiving packages. The cpu load while receiving (iperf3 -R) is about 2 to 3 times higher than sending on a fresh installation of FreeBSD 13. Using iperf3 with one client (-P1) and on the other end is an Intel X520 I can max out the 10GbE easily.
Playing around with net.isr.dispatch, net.isr.maxthreads and net.isr.bindthreads didn't change much.


And now to the completely off-topic: The Linux shipped kernel 5.13 and also the lastest 5.17-rc2 do not recognise the AMD 10GbE interfaces. Are you in contact with the developer(s) from AMD and have any idea what the plans are in regard to the Linux kernel and the amd-xgbe driver? From the OPNsense axgbe code diff is seems like it would not be too much work for the developers to have it working on Linux.

/Thomas
Title: Re: OPNsense DEC700 AMD 10GbE drivers FreeBSD/Linux
Post by: AdSchellevis on February 04, 2022, 08:21:25 pm
Hi Thomas,

Your post is in the right place, no problem at all.

Let me try to answer some questions.

Quote
Since OPNsense got newer drivers for the 10GbE than FreeBSD 13 do you have any idea when they get included in FreeBSD? I couldn't find anything related to this driver. But then I may just looked at the wrong place.

I did compile a custom FreeBSD stable/13 kernel with the code diff for the axgbe from OPNsense and it works well. Having it upstream would of course be even easier.

It's not very hard to move the drivers over indeed. During development of the driver we communicated a lot with someone from AMD to test and improve the code, later on we fixed some issues ourselves but haven't found time to upstream any of the changes yet.

The current state of the driver seems to be pretty good, so if new bugs don't surface on short notice, we should try to upstream our changes.

Quote
What I did see though is a difference of cpu load when receiving packages. The cpu load while receiving (iperf3 -R) is about 2 to 3 times higher than sending on a fresh installation of FreeBSD 13. Using iperf3 with one client (-P1) and on the other end is an Intel X520 I can max out the 10GbE easily.
Playing around with net.isr.dispatch, net.isr.maxthreads and net.isr.bindthreads didn't change much.

If performance differs a lot (using OPNsense 22.1), I would expect there are some system tunables in play, best place to start looking are the ones described in https://docs.opnsense.org/troubleshooting/hardening.html

Quote
And now to the completely off-topic: The Linux shipped kernel 5.13 and also the lastest 5.17-rc2 do not recognise the AMD 10GbE interfaces. Are you in contact with the developer(s) from AMD and have any idea what the plans are in regard to the Linux kernel and the amd-xgbe driver? From the OPNsense axgbe code diff is seems like it would not be too much work for the developers to have it working on Linux.

As far as I know AMD is still involved in driver development on Linux, but you probably want to build the latest driver yourself. If I'm not mistaken you can find the latest source here https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/amd/xgbe


Best regards,

Ad
Title: Re: OPNsense DEC700 AMD 10GbE drivers FreeBSD/Linux
Post by: patient0 on February 05, 2022, 08:29:02 am
Hi Ad,

Thanks for your answers, that cleared it up on the FreeBSD side.

I'll post to the Linux kernel bug mailing list about the 10GbE AMD cards. Even with the latest code from yesterday I had no luck.

/Thomas