Interfaces randomly go down/unroutable

[franco]

Hmm, so it might be an issue with the recent Suricata update patch for Netmap v14. To confirm this:

# opnsense-revert -r 21.7.5 suricata

Although 6.0.4 could be also at play here I doubt there was substantial changes on their end from 6.0.3.


Cheers,
Franco

[franco]

And a follow up question: who is using Suricata IPS mode on the unstable interface? There's really a lack of that basic info.

[chemlud]

And a follow up question: who is using Suricata IPS mode on the unstable interface? There's really a lack of that basic info.

Hi franco, thanks for reply, what do you mean by "who"? The opnsense of course...

What "basic info" is missing that is not in dmesg or the snip from system log above?

PS: will do revert in the evening, don't want a queue of angry users (again...)

[jt-socal]

I am not using Suricata and have an unstable WAN interface (dmesg above).  I'm not certain my issue is the same, but the unstable interface is new and not seeing complaints of similar issues in the dslreports forum for my ISP.

I am now getting this new messages in dmesg
arpresolve: can't allocate llinfo for redacted on ixl0
arpresolve: can't allocate llinfo for redacted on ixl0

[chemlud]

Hmm, so it might be an issue with the recent Suricata update patch for Netmap v14. To confirm this:

# opnsense-revert -r 21.7.5 suricata

Although 6.0.4 could be also at play here I doubt there was substantial changes on their end from 6.0.3.


Cheers,
Franco

Reverted, restarted suricata, enabled the interface having problems since yesterday for IPS. Stable for the moment.

[n1nja]

I think my thread's been jacked here, but I'm not even using suricata.  I've disabled sensei, but I still get this problem.  Sometimes LAN interfaces pingable, sometimes not.  Sometimes when they are pingable the traffic goes into the firewall and vanishes, I don't know where to look beyond that.  Here's my dmesg output, and the last up/down in the log was me swapping for a different cable.  But I strongly suspect that won't do a thing because simply rebooting the box fixes it every time.

Code: [Select]

---<<BOOT>>---
Copyright (c) 2013-2019 The HardenedBSD Project.
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p21-HBSD #0  1c99b63a2ba(stable/21.7)-dirty: Wed Nov 10 11:17:14 CET 2021
    root@sensey:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(efifb): resolution 800x600
HardenedBSD: initialize and check features (__HardenedBSD_version 1200059 __FreeBSD_version 1201000).
CPU: Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz (2400.10-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x806e9  Family=0x6  Model=0x8e  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 12884901888 (12288 MB)
avail memory = 12339802112 (11768 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119 on motherboard
Launching APs: 1 2 3
Timecounter "TSC-low" frequency 1200050100 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff812947f0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
000.000054 [4344] netmap_init               netmap: loaded module
[ath_hal] loaded
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Sunrise Point-LP USB 3.0 controller> mem 0xdf600000-0xdf60ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Sunrise Point-LP AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xdf614000-0xdf615fff,0xdf618000-0xdf6180ff,0xdf617000-0xdf6177ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 3 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) 82583V> port 0xe000-0xe01f mem 0xdf500000-0xdf51ffff,0xdf520000-0xdf523fff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:e0:67:21:c0:a4
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) 82583V> port 0xd000-0xd01f mem 0xdf400000-0xdf41ffff,0xdf420000-0xdf423fff irq 17 at device 0.0 on pci2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:e0:67:21:c0:a5
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) 82583V> port 0xc000-0xc01f mem 0xdf300000-0xdf31ffff,0xdf320000-0xdf323fff irq 18 at device 0.0 on pci3
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:e0:67:21:c0:a6
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) 82583V> port 0xb000-0xb01f mem 0xdf200000-0xdf21ffff,0xdf220000-0xdf223fff irq 19 at device 0.0 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:e0:67:21:c0:a7
em3: netmap queues/slots: TX 1/1024, RX 1/1024
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
em4: <Intel(R) 82583V> port 0xa000-0xa01f mem 0xdf100000-0xdf11ffff,0xdf120000-0xdf123fff irq 16 at device 0.0 on pci5
em4: Using 1024 TX descriptors and 1024 RX descriptors
em4: Using an MSI interrupt
em4: Ethernet address: 00:e0:67:21:c0:a8
em4: netmap queues/slots: TX 1/1024, RX 1/1024
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
em5: <Intel(R) 82583V> port 0x9000-0x901f mem 0xdf000000-0xdf01ffff,0xdf020000-0xdf023fff irq 17 at device 0.0 on pci6
em5: Using 1024 TX descriptors and 1024 RX descriptors
em5: Using an MSI interrupt
em5: Ethernet address: 00:e0:67:21:c0:a9
em5: netmap queues/slots: TX 1/1024, RX 1/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Hoodisk SSD SBFMBBA3> ACS-4 ATA SATA 3.x device
ada0: Serial Number L9MLCCC11295650
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 30533MB (62533296 512 byte sectors)
ugen0.1: <0x8086 XHCI root HUB> at usbus0
Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]...
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub0: 18 ports with 18 removable, self powered
em0: link state changed to UP
em1: link state changed to UP
em2: link state changed to UP
lo0: link state changed to UP
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
lagg0: IPv6 addresses on em4 have been removed before adding it as a member to prevent IPv6 address scope violation.
lagg0: link state changed to DOWN
lagg0: IPv6 addresses on em5 have been removed before adding it as a member to prevent IPv6 address scope violation.
em1: link state changed to DOWN
vlan0: changing name to 'em1_vlan30'
vlan1: changing name to 'em1_vlan10'
vlan2: changing name to 'em1_vlan35'
tun1: changing name to 'ovpns1'
tun2: changing name to 'ovpnc2'
em2: link state changed to DOWN
em1: link state changed to UP
em1_vlan35: link state changed to UP
em1_vlan10: link state changed to UP
em1_vlan30: link state changed to UP
em2: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
pflog0: permanently promiscuous mode enabled
ovpns1: link state changed to UP
ovpns1: link state changed to DOWN
ovpns1: link state changed to UP
997.257986 [ 853] iflib_netmap_config       txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
em1: permanently promiscuous mode enabled
997.358747 [ 853] iflib_netmap_config       txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
em1: link state changed to DOWN
em1_vlan35: link state changed to DOWN
em1_vlan10: link state changed to DOWN
em1_vlan30: link state changed to DOWN
em1: link state changed to UP
em1_vlan35: link state changed to UP
em1_vlan10: link state changed to UP
em1_vlan30: link state changed to UP
arp: 10.0.0.250 moved from b0:b8:67:c9:d4:56 to b0:b8:67:c9:bd:a8 on em1
arp: 10.0.0.250 moved from b0:b8:67:c9:bd:a8 to b0:b8:67:c9:d4:56 on em1
em1: link state changed to DOWN
em1_vlan35: link state changed to DOWN
em1_vlan10: link state changed to DOWN
em1_vlan30: link state changed to DOWN
em1: link state changed to UP
em1_vlan35: link state changed to UP
em1_vlan10: link state changed to UP
em1_vlan30: link state changed to UP



[allebone]

Ninja in your case check arp table of oth the client you are pinging from and the arp table on opnsense during the issue. Hopefully you can attach a screen and keyboard to it while its happening. Check the arp table is good as the lan is not pingable this indicates a possible issue here.

[madj42]

I had this problem this morning.  I rebooted because I wasn't sure what was going on as it was just one of my networks on the router that had the issue.  I couldn't ping or reach anything on the network
  This network is on a BCE adapter.  I am running suricata on both networks.  I didn't think anything of it until now because I made some firewall rule changes last night and thought it was that.  I'll troubleshoot more next time.

[chemlud]

Ninja in your case check arp table of oth the client you are pinging from and the arp table on opnsense during the issue. Hopefully you can attach a screen and keyboard to it while its happening. Check the arp table is good as the lan is not pingable this indicates a possible issue here.

In my case there are two dumb switches on the network which should handle ARP, or? I could not reach LAN clients from same LAN, which makes no sense at all...

[Patrick M. Hausen]

ARP is not handled by the infrastructure, i.e. switches, but by the hosts involved. So checking won't hurt.

[guest31184]

Same issue with latest update to 21.7.6 on weekend.
My (internal) interfaces are unreachable. I can log-in through VPN via WAN. After reboot, it runs some time (on Monday full day, Tuesday only 1 h). I've disabled IDS then and it was stable since.
So I suspect an issue with
*ports: suricata 6.0.4[9] with Netmap API version 14 enabled
So internet / WAN interaface is still working, however internal interfaces (on which suricata runs) were unreachable.

[chemlud]

ARP is not handled by the infrastructure, i.e. switches, but by the hosts involved. So checking won't hurt.

yepp, switches apparently use MAC...

https://serverfault.com/questions/597221/why-arp-cache-is-stored-in-both-machine-and-switch

[Patrick M. Hausen]

ARP - Address Resolution Protocol - is the way IPv4 nodes on IEEE 802.3 networks map layer 3 to layer 2 addresses ...
A "switch" is simply a learning bridge in that context, so it keeps a MAC address to port cache.

[Sparkey]

I had the same issue as Sushifish

Same issue with latest update to 21.7.6 on weekend.
My (internal) interfaces are unreachable. I can log-in through VPN via WAN. After reboot, it runs some time (on Monday full day, Tuesday only 1 h). I've disabled IDS then and it was stable since.
So I suspect an issue with
*ports: suricata 6.0.4[9] with Netmap API version 14 enabled
So internet / WAN interaface is still working, however internal interfaces (on which suricata runs) were unreachable.

[Taomyn]

I had this a couple of times last week, also using Suricata, but it was fine until today. Now I can force the problem by simply manually doing a configuration backup, including the RRD data - the file is sometimes generated but then fails to download, the GUI loses the connection as does the machine I am browsing from, none of the machines on the LAN can access the firewall, if I physically connect or use a VPN from external all looks fine but it cannot see the LAN even though it shows the LAN port is active. Via VPN the GUI can be accessed but it's slow which I think is because it's trying to make an LDAP connection to my domain which it can no longer see.

Only a firewall reboot resolves the problem, but I can simply do the backup again from the browser on a LAN machine and bang, it's all offline again.