Has there been a change to the Intel X520 driver in the last few months

[IsaacFL]

I am getting hard crashes just by applying any change to the WAN interface.

My interface is a dual 10gb SFP+ interface:

Code Select Expand

# sysctl -a | grep -E 'dev.(ix).*.%desc:'
dev.ix.1.%desc: Intel(R) X520 82599ES (SFI/SFP+)
dev.ix.0.%desc: Intel(R) X520 82599ES (SFI/SFP+)


My WAN interface is ix0, I have setup ix1 as opt1 with multiple vlans.

I don't change the WAN often, but when crowdsec came out, I installed the plugin and it worked fine. Later, I changed something on the WAN interface and it had a hard crash. So I blamed crowdsec and removed it.

Later, I had the same issue, and I thought it was something to do with NUT and the USB driver, so I removed NUT.

Lately I have been testing "Block bogons" and that is when I have noticed that I can change anything on the WAN, that when I apply changes it crashes with a wall of text on the console that ends with:

cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 2

At that point only thing to do is hold power switch. Even ctl-alt-del doesn't work. And after it boots, I send the crash report in.

This is a Dell OptiPlex 5050 connect via SFP+ to a Brocade ICX6450.

I have noted that applying changes to a vlan interface riding on top of the ix1 interface does not cause a crash.

Could this be a hardware failure starting? Or has there been a change in the driver or firmware?

[IsaacFL]

The last thing my upstream syslog server captured was the following:

Code Select Expand

Jul 26 08:14:17 OPNsense.redacted.com radvd exiting, 1 sigterm(s) received
Jul 26 08:14:17 OPNsense.redacted.com radvd sending stop adverts
Jul 26 08:14:17 OPNsense.redacted.com radvd removing /var/run/radvd.pid
Jul 26 08:14:17 OPNsense.redacted.com radvd returning from radvd main
Jul 26 08:14:17 OPNsense.redacted.com dhcp6c restarting
Jul 26 08:14:17 OPNsense.redacted.com dhcp6c Bypassing address release because of -n flag
Jul 26 08:14:17 OPNsense.redacted.com dhcp6c remove an IA: NA-0
Jul 26 08:14:17 OPNsense.redacted.com dhcp6c remove an address 2605:e000:XXXX:XXXX:XXXX:3660:da:50fa
Jul 26 08:16:13 OPNsense.redacted.com kernel ---<<BOOT>>---
Jul 26 08:16:13 OPNsense.redacted.com kernel Copyright (c) 1992-2021 The FreeBSD Project.
Jul 26 08:16:13 OPNsense.redacted.com kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jul 26 08:16:13 OPNsense.redacted.com kernel The Regents of the University of California. All rights reserved.
Jul 26 08:16:13 OPNsense.redacted.com kernel FreeBSD is a registered trademark of The FreeBSD Foundation.
Jul 26 08:16:13 OPNsense.redacted.com kernel FreeBSD 13.1-RELEASE-p8 stable/23.1-n250461-cf0d42d1ca7 SMP amd64
Jul 26 08:16:13 OPNsense.redacted.com kernel FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
Jul 26 08:16:13 OPNsense.redacted.com kernel VT(efifb): resolution 1024x768
Jul 26 08:16:13 OPNsense.redacted.com kernel CPU: Intel(R) Core(TM) i5-7600 CPU @ 3.50GHz (3500.00-MHz K8-class CPU)
Jul 26 08:16:13 OPNsense.redacted.com kernel   Origin="GenuineIntel"  Id=0x906e9  Family=0x6  Model=0x9e  Stepping=9
Jul 26 08:16:13 OPNsense.redacted.com kernel   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jul 26 08:16:13 OPNsense.redacted.com kernel   Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Jul 26 08:16:13 OPNsense.redacted.com kernel   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Jul 26 08:16:13 OPNsense.redacted.com kernel   AMD Features2=0x121<LAHF,ABM,Prefetch>
Jul 26 08:16:13 OPNsense.redacted.com kernel   Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Jul 26 08:16:13 OPNsense.redacted.com kernel   Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Jul 26 08:16:13 OPNsense.redacted.com kernel   XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Jul 26 08:16:13 OPNsense.redacted.com kernel   IA32_ARCH_CAPS=0xc04<RSBA>
Jul 26 08:16:13 OPNsense.redacted.com kernel   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Jul 26 08:16:13 OPNsense.redacted.com kernel   TSC: P-state invariant, performance statistics
Jul 26 08:16:13 OPNsense.redacted.com kernel real memory  = 8589934592 (8192 MB)
Jul 26 08:16:13 OPNsense.redacted.com kernel avail memory = 8133185536 (7756 MB)

[IsaacFL]

I figured out these were all false leads. 

I don't know what it is but logs show something after dhcp6c remove the wan external ipv6 address.

Code Select Expand

Jul 26 11:55:41 OPNsense.redacted.com root Test 1155
Jul 26 11:55:42 OPNsense.redacted.com audit user root@2603:redacted:a074:97fa changed configuration to /conf/backup/config-1690397742.9008.xml in /interfaces.php?if=wan [/interfaces.php made changes]
Jul 26 11:55:42 OPNsense.redacted.com config config-event: new_config /conf/backup/config-1690397742.9008.xml
Jul 26 11:55:43 OPNsense.redacted.com filterlog 623,,,897fe8cf5a7f2bbe18d6a5cd3fa8107f,vlan01,match,pass,in,6,0x00,0x8826c,64,tcp,6,32,2603:redactedlocalip6:a074:97fa,2603:redactedlocalip6:baff:fe0c:f909,64203,443,0,S,1086884918,,64800,,mss;nop;wscale;nop;nop;sackOK
Jul 26 11:55:43 OPNsense.redacted.com configctl event @ 1690397742.91 msg: Jul 26 11:55:42 OPNsense.redacted.com config[31669]: config-event: new_config /conf/backup/config-1690397742.9008.xml 
Jul 26 11:55:43 OPNsense.redacted.com configctl event @ 1690397742.91 exec: system event config_changed
Jul 26 11:55:44 OPNsense.redacted.com opnsense /interfaces.php: plugins_configure dhcp (,inet6,[Array,Array,Array,Array,Array,Array,Array])
Jul 26 11:55:45 OPNsense.redacted.com opnsense /interfaces.php: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6,[Array,Array,Array,Array,Array,Array,Array]))
Jul 26 11:55:45 OPNsense.redacted.com devd Processing event '!system=VFS subsystem=FS type=REMOUNT mount-point="/var/dhcpd/dev" mount-dev="devfs" mount-type="devfs" fsid=0x01ff007171000000 owner=0 flags="local;multilabel;"'
Jul 26 11:55:45 OPNsense.redacted.com devd Pushing table
Jul 26 11:55:45 OPNsense.redacted.com devd Processing notify event
Jul 26 11:55:45 OPNsense.redacted.com devd Popping table
Jul 26 11:55:50 OPNsense.redacted.com radvd exiting, 1 sigterm(s) received
Jul 26 11:55:50 OPNsense.redacted.com radvd sending stop adverts
Jul 26 11:55:50 OPNsense.redacted.com radvd removing /var/run/radvd.pid
Jul 26 11:55:50 OPNsense.redacted.com radvd returning from radvd main
Jul 26 11:55:50 OPNsense.redacted.com dhcp6c restarting
Jul 26 11:55:50 OPNsense.redacted.com dhcp6c Bypassing address release because of -n flag
Jul 26 11:55:50 OPNsense.redacted.com dhcp6c remove an IA: NA-0
Jul 26 11:55:50 OPNsense.redacted.com dhcp6c remove an address 2605:redactedispip6:19e7:3660:da:50fa
Jul 26 11:57:53 OPNsense.redacted.com kernel ---<<BOOT>>---
Jul 26 11:57:53 OPNsense.redacted.com kernel Copyright (c) 1992-2021 The FreeBSD Project.


I thought was  because I am using bogonsv6 alias inside of an alias I created that so that I could exceptions, and inadvertently removed all of the block bogons.

Code Select Expand

BOGONS_LOCAL_V6  Network(s) bogonsv6,!fe80::/64,!fdxx:abcd:ef::/48

But even when I verified that block bogons was active on the WAN interface, just reloading the interface by clicking save, then apply changes causes a hard crash.