Internet Bandwidth can't reach 1Gbps and CPU AES-NI crypto missing on OpenVPN

[opnsrcfw]

Can anyone @OPNSENSE or anyone from forum knows how to resolve issue that I'm having will be great appreciated. Advance thanks.

Issue:
1. any of my wired LAN devices can't reach no where near to 1Gpbs down/up speed to/from internet.
2. OpenVPN server config can't detect CPU AES-NI cryto chip which is enabled by default.

Note:
suricata is not heavily configured.

Current firewall setup:
WAN bandwidth speed is 1Gbps In/Out
LAN 1Gpbs for all devices connected through 24port switch
NO VLANs configured or exists.
NAT firewall rules: 2
Services Running:
acme, clamd, configd, dhcpd, dyndns, flowd_aggregate, freshclam, iperf, login, ntpd, openssh, openvpn, pf, samplicate, suricata, syslog, unbound

Interfaces: Settings:
Hardware CRC: Checked    #Disable hardware checksum offload
Hardware TSO: Checked    #Disable hardware TCP segmentation offload
Hardware LRO: Checked    #Disable hardware large receive offload

Currently Running OpnSense Info:
Versions: OPNsense 18.7.2-amd64
FreeBSD: 11.1-RELEASE-p13
OpenSSL: 1.0.2p 14 Aug 2018

Current CPU hardware info:
hw.model: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz
hw.machine: amd64
hw.ncpu: 8

Current NIC hardware info:
Intel Ethernet 10-Gigabit X540-AT2 (2 Ports)
Intel NetXtreme II BCM5716 Gigabit (2 Ports)

IF ANY INFORMATION IS REQUIRED TO INVESTIGATE FURTHER, WILLING TO PROVIDE IT .

Thank you,
Mahesh

[mimugmail]

Suricata and NAT will break such high throughput

[opnsrcfw]

I understand it will decrease the network performance but I even tried turning suricata off and NAT is pretty basic. I believe OPNSense needs either kernel or nic tunning for ixgbe drivers and igb drivers. Not sure what tunning settings to apply yet.

I'm currently testing various tunning setings to see if that helps.

[mimugmail]

You test without VPN? Are you sure your test servers at WAN support full GB?

[elfrom]

Hi Mahesh
I think it can be of interest which NIC is connected to WAN and which is connected LAN.
I don't want to be picky but "no where near to 1Gpbs" is not an exact measure, what are we talking about?
Please note that the NetXtreme II BCM5716 is NOT based on an Intel chipset but rather a chipset from Broadcom.

As many details as possible will get you the best and fastest assistance.

[opnsrcfw]

@mimugmail
all test was done without vpn and yes source speedtest supported upto 10G. My WAN cap is 1G. But overall I manage to resolve the issue.

@elfrom
you brought up good point which chipset is WAN and LAN. BCM5716 is being used for WAN and X540 for LAN.
WAN speed was getting upto 500-600Mbps but not more than that.

[Resolved Internet bandwidth issue]
After investigating with FreeBSD system and nic tunning settings, I had to add following items to OPNSENSE Tunables page.
hw.bce.tso_enable = 0   
hw.pci.enable_msix = 0

Added following to /etc/sysctl.conf
kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144