OPNsense Forum

English Forums => Hardware and Performance => Topic started by: 134 on May 02, 2021, 06:15:20 pm

Title: Intel X710 woes
Post by: 134 on May 02, 2021, 06:15:20 pm
Recently I upgraded the link between Opnsense router and switch to 10G, the NIC is X710-DA2. Since then I've had bad network experience: Kicked out of online games, "broken pipe" SSH connection to local server on another VLAN, local streaming and file operations from NAS are interupted.....

Occurrence of these drops can vary from 0 in whole day to consistent disconnect every several seconds.

I believe the switch is not issue because i tried replacing it with another 10G switch laying around, so it must be Opnsense router.

I'm not sure if this is 21.1.5 issue because I had this NIC briefly before upgrading OpnSense to 21.1.5.

Is there any thing I can do to fix this? System turntables? Another driver instead of default one? Or just buy another NIC (X520 ???).

Thanks!
Title: Re: Intel X710 woes
Post by: Supermule on May 02, 2021, 06:22:11 pm
I40e driver on VmWare for the X710 is heavily flawed when offloading and using jumbo frames.

Disable all offload and run 1500MTU, then its rocksolid.
Title: Re: Intel X710 woes
Post by: 134 on May 02, 2021, 06:31:03 pm
Quote from: Supermule on May 02, 2021, 06:22:11 pm
I40e driver on VmWare for the X710 is heavily flawed when offloading and using jumbo frames.

Disable all offload and run 1500MTU, then its rocksolid.

My Opnsense in on bare metal, not virtualized. This 10G link is trunked with multiple VLANs.

I already have all offload disabled by default.
Title: Re: Intel X710 woes
Post by: Supermule on May 03, 2021, 06:03:26 am
What about jumbo frames?
Title: Re: Intel X710 woes
Post by: 134 on May 03, 2021, 03:47:33 pm
Quote from: Supermule on May 03, 2021, 06:03:26 am
What about jumbo frames?

i'm not using jumbo frames.
Title: Re: Intel X710 woes
Post by: 134 on May 04, 2021, 01:51:09 pm
Strangely, this exactly same hardware has been running pfsense fine without any problem for 2 days. Once i pop the disk with Opnsense installed on, problem appears again. My X710 NIC or Mikrotik switches might not be the cause.

AFAIK Opnsense had flow control disabled by default and tried to match same setting on my switch. But the problem persists.

I'm thinking about downgrading to 21.1 and see how it goes in few days, but i doubt 21.1.5 is issue. Frustrating, SSH into local server to do some works with and i'm kicked out after between 5 seconds or 1 min for "broken pipe" error. This is my 2nd attempt to get a stable install with Opnsense. 2 years ago i tried Opnsense and had to go back to pfsense after series of weird issues.

I notice that connection drops seem to happen a lot more frequent during the evening, i don't know how, we aren't pushing a lot of bandwidth. Perhaps more devices at the home?

Title: Re: Intel X710 woes
Post by: binaryanomaly on May 05, 2021, 12:37:28 pm
Do you have the latest firmware?

Can you find anything in the logs?

I have an X710-T4 which runs fine so far (virtualized), except for suricata because of the jumbo frames. https://forum.opnsense.org/index.php?topic=22942.0
Title: Re: Intel X710 woes
Post by: 134 on May 06, 2021, 11:39:03 am
Quote from: binaryanomaly on May 05, 2021, 12:37:28 pm
Do you have the latest firmware?

Can you find anything in the logs?

I have an X710-T4 which runs fine so far (virtualized), except for suricata because of the jumbo frames. https://forum.opnsense.org/index.php?topic=22942.0

i'm on 21.1.5.

What kind of logs i should be looking at if it's hardware issue? If it was firewall mishandling the packets that caused 'broken pipe' error in when SSH to another VLAN , where should i look at for confirmation?

Thanks !
Title: Re: Intel X710 woes
Post by: binaryanomaly on May 06, 2021, 11:48:56 am
Sorry, I meant the NIC firmware itself.

You should probably see something in System -> Logfiles -> General if there are some hardware/driver issues.
Title: Re: Intel X710 woes
Post by: 134 on May 06, 2021, 11:58:09 am
Quote from: binaryanomaly on May 06, 2021, 11:48:56 am
Sorry, I meant the NIC firmware itself.

You should probably see something in System -> Logfiles -> General if there are some hardware/driver issues.

I don't know about firmware, this NIC should be working out of the box with FreeBSD and everything based on FreeBSD, which include pfsense, HardenedBSD and Opnsense.

There is nothing about hardware issue in System > Log files > General.
Title: Re: Intel X710 woes
Post by: 134 on May 06, 2021, 12:25:41 pm
Looks like this NIC is on firmware version 6.0.48442:

Code: [Select]
ixl1: fw 6.0.48442 api 1.7 nvm 6.01 etid 800035cf oem 1.262.0
I'll try to upgrade this NIC to latest fw which is 8.30 according to Intel:

https://downloadcenter.intel.com/product/83964/Intel-Ethernet-Converged-Network-Adapter-X710-DA2

However here's the strange bit, sysctl -a | grep fc shows the flow control is disabled, probably by default as i didn't touch the system turnables:

Code: [Select]
dev.ixl.1.fc: 0
...but dmesg and ifconfig ixl1 show Flow control is Full ????

Code: [Select]
ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: Full
Code: [Select]
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
Edit: After upgrading X710 fw to version 8.30, the NIC showed up with Flow control being None. It was 'Full' prioer to upgrade and i still didn't touch the tunables, more weird stuffs.