I've been struggling with this too. Not sure if its the same issue, but I feel its helpful to share. My hardware is a Qotom with the C3808 Atom and 4 10G SFP ports, so maybe this isn't related, but it sure sounds like it might be.
Where I see slowdown is in receiving on the interface. Just a simple iperf test, I can get 9.4Gb sending firewall->server. But all sending from server->firewall are 1G-1.5G. Between servers i get 9+Gb both direction, on same switch. Like others, testing with direct connection, connect to switch, using fiber , DAC, or CAT6a, different SFP modules, different ports, different server, different NIC's, same results. And tried all kinds of tunnables. But I'm seeing some more in other posts I'd like to test, and I'll double check my bios.
@AdSchellevis, Sorry, I understand me creating a new account just to chime in isn't too much help, but I find it helpful. ;) If there's anything I can test I'd be happy to report back.
Edit:
I wonder if this has something to do with this? Link PCIe speed is 1x?
pciconf -lbcevV pci0:11:0:0
cap 10[a0] = PCI-Express 2 endpoint max data 128(128) FLR RO
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
Where I see slowdown is in receiving on the interface. Just a simple iperf test, I can get 9.4Gb sending firewall->server. But all sending from server->firewall are 1G-1.5G. Between servers i get 9+Gb both direction, on same switch. Like others, testing with direct connection, connect to switch, using fiber , DAC, or CAT6a, different SFP modules, different ports, different server, different NIC's, same results. And tried all kinds of tunnables. But I'm seeing some more in other posts I'd like to test, and I'll double check my bios.
@AdSchellevis, Sorry, I understand me creating a new account just to chime in isn't too much help, but I find it helpful. ;) If there's anything I can test I'd be happy to report back.
Edit:
I wonder if this has something to do with this? Link PCIe speed is 1x?
pciconf -lbcevV pci0:11:0:0
cap 10[a0] = PCI-Express 2 endpoint max data 128(128) FLR RO
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)