OPNsense Forum
Archive => 21.7 Legacy Series => Topic started by: n1nja on November 26, 2021, 11:12:38 pm
-
I have this issue where every so often (a bit on the spontaneous side, unfortunately) I lose internet. I can't ping my IPv4 LAN facing gateway. I can't ping my management port (which I created for troubleshooting this problem). Looking at /var/log/system.log, I see this:
Nov 26 13:08:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:08:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:08:27 OPNsense dhclient[52573]: Creating resolv.conf
Nov 26 13:08:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:13:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:13:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:13:27 OPNsense dhclient[88047]: Creating resolv.conf
Nov 26 13:13:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:16:00 OPNsense root[28797]: reload filter for configured schedules
Nov 26 13:18:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:18:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:18:27 OPNsense dhclient[87230]: Creating resolv.conf
Nov 26 13:18:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:23:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:23:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:23:27 OPNsense dhclient[74715]: Creating resolv.conf
Nov 26 13:23:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:28:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:28:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:28:27 OPNsense dhclient[65189]: Creating resolv.conf
Nov 26 13:28:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:31:00 OPNsense root[30800]: reload filter for configured schedules
Nov 26 13:33:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
Nov 26 13:33:27 OPNsense dhclient[46336]: DHCPACK from 208.110.116.101
Nov 26 13:33:27 OPNsense dhclient[84129]: Creating resolv.conf
Nov 26 13:33:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
Nov 26 13:33:32 OPNsense kernel: em1: link state changed to DOWN
Nov 26 13:33:32 OPNsense kernel: em1_vlan35: link state changed to DOWN
Nov 26 13:33:32 OPNsense kernel: em1_vlan10: link state changed to DOWN
Nov 26 13:33:32 OPNsense kernel: em1_vlan30: link state changed to DOWN
Nov 26 13:33:32 OPNsense kernel: em2: link state changed to DOWN
Nov 26 13:33:32 OPNsense opnsense[93736]: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for lan
Nov 26 13:33:33 OPNsense opnsense[63600]: /usr/local/etc/rc.linkup: Hotplug event detected for Ooma(opt2) but ignoring since interface is configured with static IP (10.35.0.254 ::)
Nov 26 13:33:33 OPNsense opnsense[75016]: /usr/local/etc/rc.linkup: Hotplug event detected for WirelessGuest(opt3) but ignoring since interface is configured with static IP (10.10.0.254 ::)
Nov 26 13:33:34 OPNsense opnsense[80883]: /usr/local/etc/rc.linkup: Hotplug event detected for WirelessTrust(opt1) but ignoring since interface is configured with static IP (10.30.0.254 ::)
Nov 26 13:33:34 OPNsense opnsense[91909]: /usr/local/etc/rc.linkup: Hotplug event detected for MGMT(opt6) but ignoring since interface is configured with static IP (10.255.255.254 ::)
Nov 26 13:36:03 OPNsense kernel: em1: link state changed to UP
Nov 26 13:36:03 OPNsense kernel: em1_vlan35: link state changed to UP
Nov 26 13:36:03 OPNsense kernel: em1_vlan10: link state changed to UP
Nov 26 13:36:03 OPNsense kernel: em1_vlan30: link state changed to UP
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for lan
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface lan
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: ROUTING: entering configure using 'lan'
Nov 26 13:36:03 OPNsense kernel: em2: link state changed to UP
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan
Nov 26 13:36:03 OPNsense opnsense[52126]: /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure ipsec (,lan)
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure ipsec (execute task : ipsec_configure_do(,lan))
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dhcp ()
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dns ()
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dns (execute task : dnsmasq_configure_do())
Nov 26 13:36:03 OPNsense opnsense[52126]: plugins_configure dns (execute task : unbound_configure_do())
Nov 26 13:36:06 OPNsense opnsense[191]: /usr/local/etc/rc.linkup: Hotplug event detected for Ooma(opt2) but ignoring since interface is configured with static IP (10.35.0.254 ::)
Nov 26 13:36:06 OPNsense opnsense[6297]: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'em1_vlan35'
I found another forum post that says marking their interface as the gateway solved their problem, but mines already set this way.
For now I've disabled gateway monitoring to see if it makes any difference, but I'm not sure why I'd lose my LAN facing stuff in this case.
-
Same here after upgrade to 21.7.6.
Thinking of rolling back to previous version.
-
Don't now what exactly the problem was....
But rolling back to 21.7.5 from a console resolved the issue.
opnsense-revert -r 21.7.5 opnsensehttps://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert
So far so good....
-
Hey guys, just reading up on this. I just reverted to. Been having issues with interfaces this way to, after upgrading to 21.7.6.
Did you guys perhaps have suricata running on those interfaces? My issues seem to be resolved when i disable suricata or taking the interface out of the config.
I was experiencing this on the lan side btw, since i dont have suricata on wan side. My lagg interface seems to be doing just fine with suricata enabled.
Sidenote: rss enabled. intel i210.
I suspect this issue to have something to do with.
Suricata 6.0.4 with an additional change for the Netmap API version 14. not sure :-X
-
Hello confirmation bias, my old friend...
I don't see any hints in the release notes and also can't recall anything that drastic. Reverting 21.7.5 core package obviously would leave newer Suricata in place so it's not that specifically or the newer ET rulesets give you trouble (easy enough to test if enough info was given).
What seems like a joke is:
Nov 26 13:08:27 OPNsense dhclient[46336]: bound to 208.110.116.102 -- renewal in 300 seconds.
300 seconds is 5 minutes... then...
Nov 26 13:13:27 OPNsense dhclient[46336]: DHCPREQUEST on em0 to 208.110.116.101 port 67
That is EXACTLY 5 minutes later so I am unsure how this is thought to be not what it should be?
I get that something doesn't work, but the report doesn't say and jumps to conclusions. Never a good combo.
Cheers,
Franco
-
Hey guys, just reading up on this. I just reverted to. Been having issues with interfaces this way to, after upgrading to 21.7.6.
Did you guys perhaps have suricata running on those interfaces? My issues seem to be resolved when i disable suricata or taking the interface out of the config.
I was experiencing this on the lan side btw, since i dont have suricata on wan side. My lagg interface seems to be doing just fine with suricata enabled.
Sidenote: rss enabled. intel i210.
I suspect this issue to have something to do with.
Suricata 6.0.4 with an additional change for the Netmap API version 14. not sure :-X
Yep. I have Suricata on LAN side interfaces.
-
Yep. I have Suricata on LAN side interfaces.
After upgrade to 21.7.6 i was facing issues where some interfaces became unreachable, also via setting a static ip. Gateway did not respond at all, dhcp did seem to reach the server. verified by the log. But thats was all. There seemed to be, something stuck. :o
I noticed this on my management interface.
After removing the management interface from suricata it worked again.
After putting the interface back in the config, it worked like it did before, but did not survive a reboot.
Yesterday i removed all rules from suricata and disabled suricata for ids/ips.
After downloading all rules and enabling ids/ips my issue has been solved!
-
I've been mucking with this issue for weeks. I had it on both mentioned versions of the software. I initially blamed Sensei but even disabling it I had that problem. To be clean, yes my wan DHCP stuff seems weird in the log but I'm not concerned about that right now because my problem is lan facing. I can't ping my lan side. I can't ping my management port. I can't ssh to either. The thing seems dead. As soon as I initiate shutdown with hardware button press, there's a brief moment where about 12 pings make it through before it shuts down. In my case I also lose DHCP because I'm using OPNsense as a DHCP server
-
Yep. I have Suricata on LAN side interfaces.
After upgrade to 21.7.6 i was facing issues where some interfaces became unreachable, also via setting a static ip. Gateway did not respond at all, dhcp did seem to reach the server. verified by the log. But thats was all. There seemed to be, something stuck. :o
I noticed this on my management interface.
After removing the management interface from suricata it worked again.
After putting the interface back in the config, it worked like it did before, but did not survive a reboot.
Yesterday i removed all rules from suricata and disabled suricata for ids/ips.
After downloading all rules and enabling ids/ips my issue has been solved!
i have this behaivor too, IDS crashes from time to time didnt know the cause.
i've followed your steps seems to works for 30 min and after it crashes.
i am back to the old version.
-
dmesg please
-
I updated to 21.7.6 yesterday in the evening, this morning hell broke loose on the most important interface (suricata in IPS mode running on a total of 3 interfaces). All of the sudden no internet connection, dhcp delivers addresses, but devices are not reachable on the same LAN.
I rebooted, changed the physical interface (network card), I moved the whole network to another 21.7.6 install, I changed ALL dumb switches on the network, it helps for an hour and then the terror starts again. I have no idea where to start, nothing remarkable in the system log, dhcp log, unbound log, suricata log.
dmesg appended....
-
Disabled suricata on the interface and traffic started flowing. Does this make sense at all for traffic an the SAME interface?
PS: changing the interface switched from em to igb, without any success....
-
I've had the exact same experience.
I updated to 21.7.6 yesterday in the evening, this morning hell broke loose on the most important interface (suricata in IPS mode running on a total of 3 interfaces). All of the sudden no internet connection, dhcp delivers addresses, but devices are not reachable on the same LAN.
I rebooted, changed the physical interface (network card), I moved the whole network to another 21.7.6 install, I changed ALL dumb switches on the network, it helps for an hour and then the terror starts again. I have no idea where to start, nothing remarkable in the system log, dhcp log, unbound log, suricata log.
dmesg appended....
-
I'm also having problems with WAN link going down and up. This has not happened in the past, though I cannot be certain it is not my ISP. dmesg attached
-
For testing I enabled suricata IPS on the interface with problems yesterday, it took about 2 min to blow away the WAN interface
2021-11-30T15:03:59 opnsense[97742] /usr/local/etc/rc.linkup: Clearing states for stale wan route on em2
2021-11-30T15:03:59 dhclient[11453] exiting.
2021-11-30T15:03:59 dhclient[11453] connection closed
2021-11-30T15:03:59 opnsense[97742] /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
2021-11-30T15:03:55 opnsense[48849] plugins_configure hosts (execute task : unbound_hosts_generate())
2021-11-30T15:03:55 opnsense[48849] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2021-11-30T15:03:55 opnsense[48849] plugins_configure hosts ()
2021-11-30T15:03:54 opnsense[48849] /usr/local/etc/rc.newwanip: On (IP address: 10.110.122.1) (interface: Drack[opt2]) (real interface: igb0).
2021-11-30T15:03:54 opnsense[48849] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'igb0'
2021-11-30T15:03:53 opnsense[14602] /usr/local/etc/rc.linkup: Hotplug event detected for Drack(opt2) but ignoring since interface is configured with static IP (10.110.122.1 ::)
2021-11-30T15:03:52 opnsense[22399] /usr/local/etc/rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (10.11.12.1 ::)
2021-11-30T15:03:51 opnsense[54788] /usr/local/etc/rc.linkup: Hotplug event detected for iNET(opt1) but ignoring since interface is configured with static IP (10.157.11.1 ::)
2021-11-30T15:03:49 opnsense[76900] /usr/local/etc/rc.linkup: Hotplug event detected for Drack(opt2) but ignoring since interface is configured with static IP (10.20.22.1 ::)
Why the heck
/usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'igb0'
igb0 is the problematic LAN interface, not WAN (em2)....
-
Hmm, so it might be an issue with the recent Suricata update patch for Netmap v14. To confirm this:
# opnsense-revert -r 21.7.5 suricata
Although 6.0.4 could be also at play here I doubt there was substantial changes on their end from 6.0.3.
Cheers,
Franco
-
And a follow up question: who is using Suricata IPS mode on the unstable interface? There's really a lack of that basic info.
-
And a follow up question: who is using Suricata IPS mode on the unstable interface? There's really a lack of that basic info.
Hi franco, thanks for reply, what do you mean by "who"? The opnsense of course...
What "basic info" is missing that is not in dmesg or the snip from system log above?
PS: will do revert in the evening, don't want a queue of angry users (again...)
-
I am not using Suricata and have an unstable WAN interface (dmesg above). I'm not certain my issue is the same, but the unstable interface is new and not seeing complaints of similar issues in the dslreports forum for my ISP.
I am now getting this new messages in dmesg
arpresolve: can't allocate llinfo for redacted on ixl0
arpresolve: can't allocate llinfo for redacted on ixl0
-
Hmm, so it might be an issue with the recent Suricata update patch for Netmap v14. To confirm this:
# opnsense-revert -r 21.7.5 suricata
Although 6.0.4 could be also at play here I doubt there was substantial changes on their end from 6.0.3.
Cheers,
Franco
Reverted, restarted suricata, enabled the interface having problems since yesterday for IPS. Stable for the moment.
-
I think my thread's been jacked here, but I'm not even using suricata. I've disabled sensei, but I still get this problem. Sometimes LAN interfaces pingable, sometimes not. Sometimes when they are pingable the traffic goes into the firewall and vanishes, I don't know where to look beyond that. Here's my dmesg output, and the last up/down in the log was me swapping for a different cable. But I strongly suspect that won't do a thing because simply rebooting the box fixes it every time.
---<<BOOT>>---
Copyright (c) 2013-2019 The HardenedBSD Project.
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p21-HBSD #0 1c99b63a2ba(stable/21.7)-dirty: Wed Nov 10 11:17:14 CET 2021
root@sensey:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(efifb): resolution 800x600
HardenedBSD: initialize and check features (__HardenedBSD_version 1200059 __FreeBSD_version 1201000).
CPU: Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz (2400.10-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x806e9 Family=0x6 Model=0x8e Stepping=9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Structured Extended Features3=0xc000000<IBPB,STIBP>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics
real memory = 12884901888 (12288 MB)
avail memory = 12339802112 (11768 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119 on motherboard
Launching APs: 1 2 3
Timecounter "TSC-low" frequency 1200050100 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff812947f0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
000.000054 [4344] netmap_init netmap: loaded module
[ath_hal] loaded
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Sunrise Point-LP USB 3.0 controller> mem 0xdf600000-0xdf60ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Sunrise Point-LP AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xdf614000-0xdf615fff,0xdf618000-0xdf6180ff,0xdf617000-0xdf6177ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 3 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) 82583V> port 0xe000-0xe01f mem 0xdf500000-0xdf51ffff,0xdf520000-0xdf523fff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:e0:67:21:c0:a4
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) 82583V> port 0xd000-0xd01f mem 0xdf400000-0xdf41ffff,0xdf420000-0xdf423fff irq 17 at device 0.0 on pci2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:e0:67:21:c0:a5
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) 82583V> port 0xc000-0xc01f mem 0xdf300000-0xdf31ffff,0xdf320000-0xdf323fff irq 18 at device 0.0 on pci3
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:e0:67:21:c0:a6
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) 82583V> port 0xb000-0xb01f mem 0xdf200000-0xdf21ffff,0xdf220000-0xdf223fff irq 19 at device 0.0 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:e0:67:21:c0:a7
em3: netmap queues/slots: TX 1/1024, RX 1/1024
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
em4: <Intel(R) 82583V> port 0xa000-0xa01f mem 0xdf100000-0xdf11ffff,0xdf120000-0xdf123fff irq 16 at device 0.0 on pci5
em4: Using 1024 TX descriptors and 1024 RX descriptors
em4: Using an MSI interrupt
em4: Ethernet address: 00:e0:67:21:c0:a8
em4: netmap queues/slots: TX 1/1024, RX 1/1024
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
em5: <Intel(R) 82583V> port 0x9000-0x901f mem 0xdf000000-0xdf01ffff,0xdf020000-0xdf023fff irq 17 at device 0.0 on pci6
em5: Using 1024 TX descriptors and 1024 RX descriptors
em5: Using an MSI interrupt
em5: Ethernet address: 00:e0:67:21:c0:a9
em5: netmap queues/slots: TX 1/1024, RX 1/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Hoodisk SSD SBFMBBA3> ACS-4 ATA SATA 3.x device
ada0: Serial Number L9MLCCC11295650
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 30533MB (62533296 512 byte sectors)
ugen0.1: <0x8086 XHCI root HUB> at usbus0
Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]...
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub0: 18 ports with 18 removable, self powered
em0: link state changed to UP
em1: link state changed to UP
em2: link state changed to UP
lo0: link state changed to UP
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
lagg0: IPv6 addresses on em4 have been removed before adding it as a member to prevent IPv6 address scope violation.
lagg0: link state changed to DOWN
lagg0: IPv6 addresses on em5 have been removed before adding it as a member to prevent IPv6 address scope violation.
em1: link state changed to DOWN
vlan0: changing name to 'em1_vlan30'
vlan1: changing name to 'em1_vlan10'
vlan2: changing name to 'em1_vlan35'
tun1: changing name to 'ovpns1'
tun2: changing name to 'ovpnc2'
em2: link state changed to DOWN
em1: link state changed to UP
em1_vlan35: link state changed to UP
em1_vlan10: link state changed to UP
em1_vlan30: link state changed to UP
em2: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
pflog0: permanently promiscuous mode enabled
ovpns1: link state changed to UP
ovpns1: link state changed to DOWN
ovpns1: link state changed to UP
997.257986 [ 853] iflib_netmap_config txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
em1: permanently promiscuous mode enabled
997.358747 [ 853] iflib_netmap_config txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
em1: link state changed to DOWN
em1_vlan35: link state changed to DOWN
em1_vlan10: link state changed to DOWN
em1_vlan30: link state changed to DOWN
em1: link state changed to UP
em1_vlan35: link state changed to UP
em1_vlan10: link state changed to UP
em1_vlan30: link state changed to UP
arp: 10.0.0.250 moved from b0:b8:67:c9:d4:56 to b0:b8:67:c9:bd:a8 on em1
arp: 10.0.0.250 moved from b0:b8:67:c9:bd:a8 to b0:b8:67:c9:d4:56 on em1
em1: link state changed to DOWN
em1_vlan35: link state changed to DOWN
em1_vlan10: link state changed to DOWN
em1_vlan30: link state changed to DOWN
em1: link state changed to UP
em1_vlan35: link state changed to UP
em1_vlan10: link state changed to UP
em1_vlan30: link state changed to UP
-
Ninja in your case check arp table of oth the client you are pinging from and the arp table on opnsense during the issue. Hopefully you can attach a screen and keyboard to it while its happening. Check the arp table is good as the lan is not pingable this indicates a possible issue here.
-
I had this problem this morning. I rebooted because I wasn't sure what was going on as it was just one of my networks on the router that had the issue. I couldn't ping or reach anything on the network
This network is on a BCE adapter. I am running suricata on both networks. I didn't think anything of it until now because I made some firewall rule changes last night and thought it was that. I'll troubleshoot more next time.
-
Ninja in your case check arp table of oth the client you are pinging from and the arp table on opnsense during the issue. Hopefully you can attach a screen and keyboard to it while its happening. Check the arp table is good as the lan is not pingable this indicates a possible issue here.
In my case there are two dumb switches on the network which should handle ARP, or? I could not reach LAN clients from same LAN, which makes no sense at all...
-
ARP is not handled by the infrastructure, i.e. switches, but by the hosts involved. So checking won't hurt.
-
Same issue with latest update to 21.7.6 on weekend.
My (internal) interfaces are unreachable. I can log-in through VPN via WAN. After reboot, it runs some time (on Monday full day, Tuesday only 1 h). I've disabled IDS then and it was stable since.
So I suspect an issue with
*ports: suricata 6.0.4[9] with Netmap API version 14 enabled
So internet / WAN interaface is still working, however internal interfaces (on which suricata runs) were unreachable.
-
ARP is not handled by the infrastructure, i.e. switches, but by the hosts involved. So checking won't hurt.
yepp, switches apparently use MAC...
https://serverfault.com/questions/597221/why-arp-cache-is-stored-in-both-machine-and-switch
-
ARP - Address Resolution Protocol - is the way IPv4 nodes on IEEE 802.3 networks map layer 3 to layer 2 addresses ...
A "switch" is simply a learning bridge in that context, so it keeps a MAC address to port cache.
-
I had the same issue as Sushifish
Same issue with latest update to 21.7.6 on weekend.
My (internal) interfaces are unreachable. I can log-in through VPN via WAN. After reboot, it runs some time (on Monday full day, Tuesday only 1 h). I've disabled IDS then and it was stable since.
So I suspect an issue with
*ports: suricata 6.0.4[9] with Netmap API version 14 enabled
So internet / WAN interaface is still working, however internal interfaces (on which suricata runs) were unreachable.
-
I had this a couple of times last week, also using Suricata, but it was fine until today. Now I can force the problem by simply manually doing a configuration backup, including the RRD data - the file is sometimes generated but then fails to download, the GUI loses the connection as does the machine I am browsing from, none of the machines on the LAN can access the firewall, if I physically connect or use a VPN from external all looks fine but it cannot see the LAN even though it shows the LAN port is active. Via VPN the GUI can be accessed but it's slow which I think is because it's trying to make an LDAP connection to my domain which it can no longer see.
Only a firewall reboot resolves the problem, but I can simply do the backup again from the browser on a LAN machine and bang, it's all offline again.
-
I have experienced similar issue as some here. LAN interface is not reachable. Has happened with Suricata turned on and enabled on DMZ and LAN interface. I was able to reach DMZ side, not LAN. Turning off Suricata has kept my firewall working.
-
Although the status here is inconclusive we're walking back on the new Netmap API for a 21.7 to see if this is actually the issue or not.
https://github.com/opnsense/tools/commit/43679e8b1894
Expect this with 21.7.7 later this week.
Cheers,
Franco
-
Also having random issues with routing to WAN interface. Restarting all services using SSH console restores routing. Issue solved after disabling IDS/IPS and Suricata. Only disabling Suricata did not solve the issue. IDS/IPS was active on WAN and Suricata was active on all LAN/VLANs.
-
Also having random issues with routing to WAN interface. Restarting all services using SSH console restores routing. Issue solved after disabling IDS/IPS and Suricata. Only disabling Suricata did not solve the issue. IDS/IPS was active on WAN and Suricata was active on all LAN/VLANs.
Hmmm, IDS/IPS is actually suricata, what else have you disabled?
-
It seems the actual issue is that non-physical interfaces are enabled in Suricata IPS mode where the new libnetmap/Suricata combination will move these previously defunct setups (IPS doesn't work but traffic goes through) from defunct setup (IPS will be enabled on these interfaces in emulation mode causing traffic drops eventually due to partial support).
We will revert the behaviour for 21.7.x, but in 22.1 and up we would ask users to take more care in verifying their setups beforehand.
None of this was ever an issue on using IPS mode for physical interfaces or VLAN parents in promiscuous mode when VLAN scanning is necessary.
Cheers,
Franco
-
I don't have VLANs.
-
Is that good or bad?
-
Sorry to ask, I just have my two LAN interfaces in ID and was getting problems until I disabled it, but I was thinking the other day of adding my WAN which is PPPoE, so what's the best way to do this or do I have to stick with just the ones I already have?
-
It seems the actual issue is that non-physical interfaces are enabled in Suricata IPS mode where the new libnetmap/Suricata combination will move these previously defunct setups (IPS doesn't work but traffic goes through) from defunct setup (IPS will be enabled on these interfaces in emulation mode causing traffic drops eventually due to partial support).
We will revert the behaviour for 21.7.x, but in 22.1 and up we would ask users to take more care in verifying their setups beforehand.
None of this was ever an issue on using IPS mode for physical interfaces or VLAN parents in promiscuous mode when VLAN scanning is necessary.
Cheers,
Franco
I have the following set-up:
WAN PPoE
LAN running several VLANS
Scurita running on WAN and LAN, with promiscous mode
Some weeks (months?) ago I had the issue that IPS was running, traffic working but I did not get any alerts (it was around when the policy /rules set-up was changed in IPS). This resolved at some point (can't remember if by updated FW or some change in my config). Then was running OK until the recent update.
I think I run IPS only on the physical interfaces, not on the VLAN itself. So what should I change or what is wrong in my set-up?
-
By now my motivation to provide community support for relentless setup issues regarding IPS is almost zero, sorry.
I suggest switching to IDS or find an expert who can spend the time to look at the setup and give a recommendation on how to solve the identified issue reasonably.
Cheers,
Franco
-
Franco, I fully understand that you cannot resolve my issue ad-hoc and for free.
Just from your sentence
"We will revert the behaviour for 21.7.x, but in 22.1 and up we would ask users to take more care in verifying their setups beforehand."
I understood that - if things got broken in 21.7.6 - it has to do with a faulty set-up - which I cannot see from my settings, as I used recommended settings (IPS on physical devices). But I confess I'm not too much of an expert and quite new to OpnSense...
But many Thanks anyway for taking up the topic and submitting for a fix / change. Appreaciate!
-
It might not be that particular issue. I think in any version update, even the ones that have no relevant changes IPS comes up as "unreliable". This can be a configuration issue, could be hardware related (actual chipsets, RAM considerations, CPU power, driver quality), could be software related (mainly netmap/iflib framework in FreeBSD or Suricata itself), could be traffic spikes encountered.
There are some easy things to check: NIC driver name? "dmesg" output on the console? Problem same as previous major version? Or actually minor version issue? Trying to see if IDS works fine or reduce the number of interfaces inspected (most of the time LAN or DMZ is the one you need, not more).
This is just the tip of the iceberg. Half an hour of dedicated support is not uncommon here just to make sure we have a clear picture.
Cheers,
Franco
-
I have been chasing after this since upgrading to 21.7.6. No issues with 21.7.5 running the same configuration.
My hardware specs:
- Qotom Q355G4
- Intel Core i5-5300U
- 8GB memory - 31% used
- Load avg: 0.27, 0.19, 0.16
- igb NIC driver
- igb1 - LAN
- igb2_vlan2 - IoT VLAN
- igb2_vlan3 - Guest VLAN
- Traffic shaping is configured
- On a 1Gb/35Mb circuit
Symptom:
- Issue comes up around every 2 days
- DHCP clients on igb1 do not receive IP address - shows 169.254.0.0/16 APIPA
- tcpdump on igb1 shows repeated DHCPDISCOVER and DHCPOFFER packets from/to the same client MAC
- /usr/local/etc/rc.d/suricata stop and start fixes the problem with no other change
- nothing in dmesg stands out. There are several lines of iflib_netmap_config but I read that is normal.
- Suricata log shows rules update via cron, then no entries until service restart.
Settings:
- Enabled, IPS mode, and Promiscuous mode - CHECKED
- Syslog alerts, Eve syslog output, and Log package payload - UNCHECKED
- Pattern matcher - Hyperscan
- Detect profile - Medium
- Interfaces - LAN, [igb2]
- Default packet size - BLANK
Troubleshooting attempts:
- Completely disable IDS - ran for 1 week with no issues
- Deleted all policies and disabled all rules except for opnsense.test.rules - still issues
My next step is to set Interfaces to just LAN and see if the situation improves. This system is in production so I can only gather info and quickly restart Suricata. I am not looking for or expecting support, just submitting another data point.
2021-12-15 1700Z update - still ran into issues after removing [igb2] from Interfaces. I also noticed this affects DHCPREQUEST and DHCPACK as well. Running Wireshark on a client in igb1, I see REQUESTs go out but no ACKs received even though dhcpd logged sending these ACKs. Another observation; an IoT device in igb2_vlan2 was having issues (2 clients in igb1 flagged it offline) even after I removed [igb2] from the Suricata interface list. The device was immediately flagged online after the Suricata restart. I will upgrade to 21.7.7 and put [igb2] back in.
In case this is related, here are my non-default tunables. I also have Hardware CRC, TSO, LRO, and VLAN Hardware Filtering all DISABLED.
- hw.ibrs_disable = 1
- hw.igb.rx_process_limit = -1
- hw.igb.tx_process_limit = -1
- legal.intel_igb.license_ack = 1
- net.inet.icmp.drop_redirect = 1
- vm.pmap.pti = 0
I also noticed differences in the Suricata log. This may be related to the netmap change, but I want to put it out there:
21.7.6 - 03:18 is the ids rule update cron
2021-12-15T10:39:36 suricata[60953] [100520] <Notice> -- all 4 packet processing threads, 4 management threads initialized, engine started.
2021-12-15T10:39:34 suricata[58549] [100680] <Notice> -- This is Suricata version 6.0.4 RELEASE running in SYSTEM mode
2021-12-15T10:39:33 suricata[36929] [100447] <Notice> -- Signal Received. Stopping engine.
2021-12-15T03:18:01 suricata[36929] [100447] <Notice> -- rule reload complete
2021-12-15T03:18:01 suricata[36929] [100447] <Notice> -- rule reload starting
2021-12-14T23:24:18 suricata[36929] [100447] <Notice> -- all 4 packet processing threads, 4 management threads initialized, engine started.
2021-12-14T23:24:16 suricata[27169] [100448] <Notice> -- This is Suricata version 6.0.4 RELEASE running in SYSTEM mode
2021-12-14T23:24:15 suricata[88364] [100551] <Notice> -- Signal Received. Stopping engine.
21.7.7 - [igb2] added back to interface list
2021-12-15T11:28:24 suricata[29857] [100447] <Notice> -- all 4 packet processing threads, 4 management threads initialized, engine started.
2021-12-15T11:28:24 suricata[29857] [100888] <Notice> -- opened netmap:igb2/T from igb2: 0x8c35aa2300
2021-12-15T11:28:24 suricata[29857] [100888] <Notice> -- opened netmap:igb2^ from igb2^: 0x8c35aa2000
2021-12-15T11:28:23 suricata[29857] [100879] <Notice> -- opened netmap:igb2^ from igb2^: 0x8c0b5f7300
2021-12-15T11:28:23 suricata[29857] [100879] <Notice> -- opened netmap:igb2/R from igb2: 0x8c0b5f7000
2021-12-15T11:28:23 suricata[29857] [100878] <Notice> -- opened netmap:igb1/T from igb1: 0x8bcbdfd300
2021-12-15T11:28:23 suricata[29857] [100878] <Notice> -- opened netmap:igb1^ from igb1^: 0x8bcbdfd000
2021-12-15T11:28:22 suricata[29857] [100869] <Notice> -- opened netmap:igb1^ from igb1^: 0x8bb6ca3300
2021-12-15T11:28:22 suricata[29857] [100869] <Notice> -- opened netmap:igb1/R from igb1: 0x8bb6ca3000
2021-12-15T11:28:22 suricata[25946] [100432] <Notice> -- This is Suricata version 6.0.4 RELEASE running in SYSTEM mode
2021-12-15T11:28:21 suricata[92259] [100589] <Notice> -- Stats for 'igb1^': pkts: 51783, drop: 0 (0.00%), invalid chksum: 0
2021-12-15T11:28:21 suricata[92259] [100589] <Notice> -- Stats for 'igb1': pkts: 12554, drop: 0 (0.00%), invalid chksum: 0
2021-12-15T11:28:21 suricata[92259] [100589] <Notice> -- Signal Received. Stopping engine.
2021-12-22 1700Z update - it has been one week, and all has been working and stable since the upgrade to 21.7.7. No configuration changes were made during that time and Suricata logs only show rule updates. Although others still report issues, disabling netmap v14 fixed mine. I am going to start reenabling rules but I do not expect any more issues.
2022-05-25 2200Z update - to anyone still running into issues or holding back on upgrading, the solution I found is to set "VLAN Hardware Filtering" under Interfaces > Settings > Network Interfaces section to "Leave default". I ran several versions of 22.1.x without issues - currently on 22.1.6. This might be the fix for you as well.
-
Yes, my issue is more or less the same as Allan's. I also doubt a hardware issue, it's two intel NICs in a x86 platform, supported without any extras by stock firmware.
-
Has 21.7.7 solved this problem?
-
It has not been fixed in 21.7.7 for me.
Also I have only enabled suricata on my WAN interface which has no virtual interfaces, so I don't understand why I am impacted. For now disabling suricata has solved the issue.
-
For what it's worth, I have only had the single instance of the issue I had and I'm still running suricata. Not sure but I think mine was just a hiccup.
-
It has not been fixed in 21.7.7 for me.
Also I have only enabled suricata on my WAN interface which has no virtual interfaces, so I don't understand why I am impacted. For now disabling suricata has solved the issue.
Did you try rolling back the suricata version to the one before 21.7.6? It works wonderfully for me after a rollback.
-
Did you try rolling back the suricata version to the one before 21.7.6? It works wonderfully for me after a rollback.
Nope. I tried to do it, but couldn't figure it out, and didn't care that much... Just wanted to report that it still seemed to be an issue.
-
Has 21.7.7 solved this problem?
Since 21.7.7. it's running stable as with .5 before. I saw no issues so far, scuritata enabled again.
-
21.7.7 running stable for me too. Suricata enabled once again on LAN and DMZ, no VLANS being used.