OPNsense DEC700 AMD 10GbE drivers FreeBSD/Linux

[patient0]

Hi,

Please move this post if not the right place or remove if necessary ... I'm aware it's probably offtopic since not really OPNsense related, but at least OPNsense hardware related  :)

Last week I got the DEC740 I ordered and I'm playing around with it, especially with the 10GbE interfaces.

Since OPNsense got newer drivers for the 10GbE than FreeBSD 13 do you have any idea when they get included in FreeBSD? I couldn't find anything related to this driver. But then I may just looked at the wrong place.

I did compile a custom FreeBSD stable/13 kernel with the code diff for the axgbe from OPNsense and it works well. Having it upstream would of course be even easier.

What I did see though is a difference of cpu load when receiving packages. The cpu load while receiving (iperf3 -R) is about 2 to 3 times higher than sending on a fresh installation of FreeBSD 13. Using iperf3 with one client (-P1) and on the other end is an Intel X520 I can max out the 10GbE easily.
Playing around with net.isr.dispatch, net.isr.maxthreads and net.isr.bindthreads didn't change much.

• Does anyone have a DEC740/750 box and did some benchmarking with FreeBSD or on OPNsense with the firewall disabled?
• Are there any recommendations for the AMD 10GbE drivers settings?
  

And now to the completely off-topic: The Linux shipped kernel 5.13 and also the lastest 5.17-rc2 do not recognise the AMD 10GbE interfaces. Are you in contact with the developer(s) from AMD and have any idea what the plans are in regard to the Linux kernel and the amd-xgbe driver? From the OPNsense axgbe code diff is seems like it would not be too much work for the developers to have it working on Linux.

/Thomas

[AdSchellevis]

Hi Thomas,

Your post is in the right place, no problem at all.

Let me try to answer some questions.

Since OPNsense got newer drivers for the 10GbE than FreeBSD 13 do you have any idea when they get included in FreeBSD? I couldn't find anything related to this driver. But then I may just looked at the wrong place.

I did compile a custom FreeBSD stable/13 kernel with the code diff for the axgbe from OPNsense and it works well. Having it upstream would of course be even easier.

It's not very hard to move the drivers over indeed. During development of the driver we communicated a lot with someone from AMD to test and improve the code, later on we fixed some issues ourselves but haven't found time to upstream any of the changes yet.

The current state of the driver seems to be pretty good, so if new bugs don't surface on short notice, we should try to upstream our changes.

What I did see though is a difference of cpu load when receiving packages. The cpu load while receiving (iperf3 -R) is about 2 to 3 times higher than sending on a fresh installation of FreeBSD 13. Using iperf3 with one client (-P1) and on the other end is an Intel X520 I can max out the 10GbE easily.
Playing around with net.isr.dispatch, net.isr.maxthreads and net.isr.bindthreads didn't change much.

If performance differs a lot (using OPNsense 22.1), I would expect there are some system tunables in play, best place to start looking are the ones described in https://docs.opnsense.org/troubleshooting/hardening.html

And now to the completely off-topic: The Linux shipped kernel 5.13 and also the lastest 5.17-rc2 do not recognise the AMD 10GbE interfaces. Are you in contact with the developer(s) from AMD and have any idea what the plans are in regard to the Linux kernel and the amd-xgbe driver? From the OPNsense axgbe code diff is seems like it would not be too much work for the developers to have it working on Linux.

As far as I know AMD is still involved in driver development on Linux, but you probably want to build the latest driver yourself. If I'm not mistaken you can find the latest source here https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/amd/xgbe


Best regards,

Ad

[patient0]

Hi Ad,

Thanks for your answers, that cleared it up on the FreeBSD side.

I'll post to the Linux kernel bug mailing list about the 10GbE AMD cards. Even with the latest code from yesterday I had no luck.

/Thomas