i40e errors on boot

[dcol]

Did some tweaking with my Intel X710's because I was having Internet connectivity issues. Updated firmware, added some tunables such as
dev.ixl.0.eee.enable=0
dev.ixl.0.fc=0    
dev.ixl.0.fw_lldp=0

Maybe disabling LLDP does this. It was recommended to do this when using LACP
Using lagg with 2 ports on the X710's. Upon bootup, I get these errors. Still having Internet drop outs. Works fine for a while then hangs and then comes back. Only happens using LACP.
Is this something I need to look at. Help point me in the right direction. Thanks

[mimugmail]

Do you have hw filtering disabled? Do you use the shaper?

[dcol]

No VLAN's used. All offload disabled. Do not use the shaper . Basically a default install with only IDS and the tunables I mentioned. One LAN and one WAN. This is a testing environment to be put into production after all the issues are worked out.

Started with a standard install. Updated through the console and reverted Suricata back to 6.0.8_1 Then added 4 rulesets , enabled IDS and added the tunables. Added two plugins, dmidecode and ddclient.

Here is the general log from this morning. I received another batch of errors on the console last night
Looks like lagg0 detached itself until I started up the computer on the LAN. Normally a switch would be connected to the LAN, but I have it running directly to the computer, so when I turned off the computer, lagg0 detached. That may be normal action, who knows. But those errors listed in the first post appeared on the console again when lagg0  reattached.

2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: The WAN_DHCP monitor address is empty, skipping.   
2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: The WAN_DHCP6 monitor address is empty, skipping.   
2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway <my IPv6>'   
2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv6 default route to <my IPv6>   
2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: ROUTING: IPv6 default gateway set to wan   
2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway '<my IPv4>'   
2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv4 default route to <my IPv4>   
2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: ROUTING: IPv4 default gateway set to wan   
2023-01-03T08:09:23-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'   
2023-01-03T08:09:22-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: On (IP address: <my IPv6>) (interface: WAN[wan]) (real interface: igc0).   
2023-01-03T08:09:22-07:00   Error   opnsense   /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'igc0'   
2023-01-03T08:09:22-07:00   Error   opnsense   /usr/local/etc/rc.linkup: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on lagg0   
2023-01-03T08:09:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: The WAN_DHCP monitor address is empty, skipping.   
2023-01-03T08:09:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: The WAN_DHCP6 monitor address is empty, skipping.   
2023-01-03T08:09:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route   
2023-01-03T08:09:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: IPv6 default gateway set to wan   
2023-01-03T08:09:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route   
2023-01-03T08:09:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan   
2023-01-03T08:09:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: ROUTING: entering configure using 'lan'   
2023-01-03T08:09:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic lan(lagg0)   
2023-01-02T17:10:21-07:00   Error   opnsense   /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic lan(lagg0)

The internet worked at first, but now the internet port is dead. iperf3 is just hanging. Only a restart of opnsense will fix it. I am convinced this issue has something to do with lagg0. My production opnsense box has been running lagg0 on the LAN for over a year, with the same model NIC. All the X710's are running the latest firmware NVM 8.4

[dcol]

The console errors shown in the first post are a result of upgrading the firmware on the X710-DA2. Obviously the X710 NIC driver in FreeBSD is missing something that the new firmware is requesting. Other than that, after I reinstalled OPNsense again and did the updates, everything seems to work fine. The only time these errors pop up on the console is when the NIC goes down or up.

So for those updating NIC firmware, be advised. Please share if anyone has anything to add.

[dcol]

For those interested, I updated the firmware to NVM 9.1 and it cleared the errors.

Here is a guide to update the Intel X710-DA2
Get the firmware update package from Intel
https://downloadmirror.intel.com/759052/700Series_NVMUpdatePackage_v9_10.zip
Unzip to a temp folder.
Run (in Windows) -  nvmupdatew64e -i -l -o inventory.xml
If this gives error, reinstall the latest drivers for the NIC and try again.
Get the ETrackID from the generated list
Overwrite the attached nvmupdate.cfg and add the ETrackID to the REPLACES list
then run
nvmupdatew64e -u -l -o results.xml -b -c nvmupdate.cfg
and wait for it to complete.

This works for all OEM versions of the card, DELL,HP,Lenovo,etc. May work Retail as well. I have done this on Dell and Lenovo cards. I am using these NIC's as LACP on lagg0 and teaming in Windows 10. Probably best to match NVM versions on both NIC's

Also for max performance use these settings
Windows
Flow control - Off
Interrupt moderation rate - Low
Log link state event - Off
Receive buffers - 1024
Transmit buffers - 1024

OPNSense tunables
kern.ipc.nmbclusters=1000000
kern.ipc.nmbjumbop=524288
hw.intr_storm_threshold=10000
net.inet.tcp.tso=0
net.isr.dispatch: deferred
net.isr.maxthreads: -1
net.isr.bindthreads: 1
dev.ixl.0.fc=0
dev.ixl.1.fc=0
dev.ixl.0.eee.enable=0
dev.ixl.1.eee.enable=0
dev.ixl.0.iflib.override_qs_enable=1
dev.ixl.1.iflib.override_qs_enable=1
dev.ixl.0.iflib.override_nrxqs=128
dev.ixl.1.iflib.override_nrxqs=128
dev.ixl.0.iflib.override_nrxqs=128
dev.ixl.1.iflib.override_nrxqs=128
dev.ixl.0.iflib.override_ntxqs=128
dev.ixl.1.iflib.override_ntxqs=128
dev.ixl.0.iflib.override_nrxds=128
dev.ixl.1.iflib.override_nrxds=128
dev.ixl.0.iflib.override_ntxds=128
dev.ixl.1.iflib.override_ntxds=128

And if you are using LAGG LACP then these are recommended.
dev.ixl.0.fw_lldp=0
dev.ixl.1.fw_lldp=0