Interface lost with 18.7.10

[chemlud]

Hy!

Updated this morning to 18.7.10 amd64 with LibreSSL, some minutes ago I lost one interface, which came up again after plugging out/in the RJ45:

Code: [Select]

Jan 11 11:36:40 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.45.23.1) (interface: iNET[opt1]) (real interface: igb1).
Jan 11 11:36:40 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'igb1'
Jan 11 11:36:40 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for iNET(opt1) but ignoring since interface is configured with static IP (10.45.23.1 ::)
Jan 11 11:36:40 kernel: igb1: link state changed to UP
Jan 11 11:36:20 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for iNET(opt1) but ignoring since interface is configured with static IP (10.45.23.1 ::)
Jan 11 11:36:20 kernel: igb1: link state changed to DOWN

[guest19757]

Hello there,

Plan to do a snapshot and upgrade to 18.7.10, but I don't seem to understand what's happening there in your post? It was hotplugged?

Regards

[chemlud]

I didn't change anything, but suddenly I couldn't reach one of my local subnets anymore. Only thing I could do was to pull the RJ45 and stick it in again. Afterwards the subnet was back. Never had this with the opnsense before.

Then I had a look in the logs and found what I posted above...

[guest19757]

Instead of logs, did you look at 'dmesg' before you unplug the cable, was there a 'link' light on?

[chemlud]

Here the complete dmesg since the reboot this morning after upgrading to 18.7.10:

Code: [Select]

Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p18  4b57161bcff(stable/18.7) amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
VT(vga): resolution 640x480
[HBSD ASLR (compat)] status: opt-out
[HBSD ASLR (compat)] mmap: 14 bit
[HBSD ASLR (compat)] exec base: 14 bit
[HBSD ASLR (compat)] stack: 14 bit
[HBSD ASLR (compat)] vdso: 8 bit
[HBSD LOG] logging to system: enabled
[HBSD LOG] logging to user: disabled
[HBSD HARDENING] procfs hardening: enabled
[HBSD ASLR] status: opt-out
[HBSD ASLR] mmap: 30 bit
[HBSD ASLR] exec base: 30 bit
[HBSD ASLR] stack: 42 bit
[HBSD ASLR] vdso: 28 bit
[HBSD ASLR] map32bit: 18 bit
[HBSD ASLR] disallow MAP_32BIT mode mmap: opt-in
[HBSD SEGVGUARD] status: opt-out
[HBSD SEGVGUARD] expiry: 120 sec
[HBSD SEGVGUARD] suspension: 600 sec
[HBSD SEGVGUARD] maxcrashes: 5
CPU: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz (3093.04-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLF>
  Features2=0x1d9ae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8137867264 (7760 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <DELL   CBX3   >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1546521568 Hz quality 1000
random: entropy device external interface
wlan: mac acl policy registered
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff810b3240, 0) error 19
kbd1 at kbdmux0
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <DELL CBX3   > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x4
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x4020-0x403f mem 0xe34200001
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 6c:b3:11:yy:yy:y
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x4000-0x401f mem 0xe34000001
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 6c:b3:11:yy:yy:yy
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
vgapci0: <VGA-compatible display> port 0x5000-0x503f mem 0xe0c00000-0xe0ffffff,0xd0000000-0xdffff0
vgapci0: Boot video device
pci0: <simple comms> at device 22.0 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0x5080-0x509f mem 0xe3f00000-0xe3f1ffff,0
em0: Using an MSI interrupt
em0: Ethernet address: 18:03:73:yy:yy:yy
em0: netmap queues/slots: TX 1/1024, RX 1/1024
ehci0: <Intel Cougar Point USB 2.0 controller> mem 0xe3f70000-0xe3f703ff irq 16 at device 26.0 on0
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
hdac0: <Intel Cougar Point HDA Controller> mem 0xe3f60000-0xe3f63fff irq 22 at device 27.0 on pci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pcib3: [GIANT-LOCKED]
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib4
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x2020-0x203f mem 0xe20200003
igb2: Using MSIX interrupts with 5 vectors
igb2: Ethernet address: 6c:b3:11:yy:yy:yy
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
igb2: netmap queues/slots: TX 4/1024, RX 4/1024
igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x2000-0x201f mem 0xe20000003
igb3: Using MSIX interrupts with 5 vectors
igb3: Ethernet address: 6c:b3:11:yy:yy:yy
igb3: Bound queue 0 to cpu 0
igb3: Bound queue 1 to cpu 1
igb3: Bound queue 2 to cpu 2
igb3: Bound queue 3 to cpu 3
igb3: netmap queues/slots: TX 4/1024, RX 4/1024
ehci1: <Intel Cougar Point USB 2.0 controller> mem 0xe3f50000-0xe3f503ff irq 17 at device 29.0 on0
usbus1: EHCI version 1.0
usbus1 on ehci1
usbus1: 480Mbps High Speed USB v2.0
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci4: <ACPI PCI bus> on pcib5
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0x50d0-0x50d7,0x50c0-0x50c3,0x50b0-0x50b7,00
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich2: <AHCI channel> at channel 2 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
acpi_button0: <Power Button> on acpi0
acpi_syscontainer0: <System Container> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
orm0: <ISA Option ROMs> at iomem 0xce800-0xcf7ff,0xcf800-0xd07ff,0xd0800-0xd17ff,0xd1800-0xd27ff 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
Timecounters tick every 1.000 msec
nvme cam probe device init
hdacc0: <Realtek ALC269 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC269 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC269 (Analog)> at nid 20 and 24 on hdaa0
pcm1: <Realtek ALC269 (Analog 2.0+HP/2.0)> at nid 27,33 and 25 on hdaa0
hdacc1: <Intel Cougar Point HDA CODEC> at cad 3 on hdac0
hdaa1: <Intel Cougar Point Audio Function Group> at nid 1 on hdacc1
pcm2: <Intel Cougar Point (HDMI/DP 8ch)> at nid 6 on hdaa1
ugen1.1: <Intel EHCI root HUB> at usbus1
ugen0.1: <Intel EHCI root HUB> at usbus0
ses0 at ahciem0 bus 0 scbus1 target 0 lun 0
uhub0: ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
<Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub1: ada0 at ahcich2 bus 0 scbus0 target 0 lun 0
<Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ada0: <TS64GSSD370S P1225CE> ACS-2 ATA SATA 3.x device
ada0: Serial Number E029431460
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors)
Trying to mount root from ufs:/dev/ufs/OPNsense [rw]...
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen1.2: <vendor 0x8087 product 0x0024> at usbus1
uhub2 on uhub0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
ugen0.2: <vendor 0x8087 product 0x0024> at usbus0
uhub3 on uhub1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
uhub3: 6 ports with 6 removable, self powered
uhub2: 8 ports with 8 removable, self powered
igb0: link state changed to UP
igb1: link state changed to UP
em0: link state changed to UP
igb2: link state changed to UP
aesni0: No AESNI support.
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
tun1: changing name to 'ovpns1'
tun5: changing name to 'ovpns5'
tun2: changing name to 'ovpns2'
igb2: link state changed to DOWN
igb0: link state changed to DOWN
em0: link state changed to DOWN
em0: link state changed to UP
igb2: link state changed to UP
igb0: link state changed to UP
igb1: link state changed to DOWN
pflog0: promiscuous mode enabled
igb1: link state changed to UP
ovpns1: link state changed to UP
ovpns5: link state changed to UP
ovpns1: link state changed to DOWN
ovpns1: link state changed to UP
ovpns5: link state changed to DOWN
ovpns5: link state changed to UP
pid 94772 (unbound), uid 59: exited on signal 11
em0: permanently promiscuous mode enabled
em0: link state changed to DOWN
igb0: permanently promiscuous mode enabled
igb0: link state changed to DOWN
igb1: permanently promiscuous mode enabled
igb1: link state changed to DOWN
igb2: permanently promiscuous mode enabled
igb2: link state changed to DOWN
em0: link state changed to UP
igb0: link state changed to UP
igb1: link state changed to UP
igb2: link state changed to UP
ovpns1: link state changed to DOWN
ovpns1: link state changed to UP
ovpns5: link state changed to DOWN
ovpns5: link state changed to UP
pid 36455 (unbound), uid 59: exited on signal 11
[HBSD SEGVGUARD] [unbound (36455)] Suspension expired.
 -> pid: 36455 ppid: 1 p_pax: 0x850<SEGVGUARD,ASLR,NODISALLOWMAP32BIT>
pid 36965 (unbound), uid 59: exited on signal 11
igb3: link state changed to UP
igb3: link state changed to DOWN
igb1: link state changed to DOWN
igb1: link state changed to UP

..didn't check the lights on the network card, sorry...


PS: re. the unbound error see

https://forum.opnsense.org/index.php?topic=7811.msg50139#msg50139

I downgraded unbound to 1.8.1, as described in the thread above.

[franco]

Strange about the Unbound issue. There was another thread regarding lost IP, but there were no changes in interface code.

Can you try flipping this commit instead?

https://github.com/opnsense/core/commit/3d8fd0088a

# opnsense-patch 3d8fd0088a


Cheers,
Franco

[franco]

PS: Unbound crashing would imply some sort of manual DoH or TLS implementation?

[chemlud]

Hi Franco!

See link above, using unbound with TLS (LibreSSL) results in these crashes, that's why I stayed at 1.8.1 until today, until I updated to 18.7.10 this morning. Now I'm back to 1.8.1

RE the patch: I will update to latest unbound and then apply the patch, correct? :-)

[franco]

Patch with 18.7.10, Unbound version shouldn't matter.... 1.8 has been weird so far... :/

[guest19757]

Hello,

How does userspace service 'breaks' ethernet traffic? I don't understand privileges programs in freebsb, but you saying a bug in unbound will break ethernet?

[guest19757]

I say this because this seem a very easy dos attack? no

[chemlud]

OK, patched and unbound restarted. Wait'n see

[chemlud]

OK, the interface is down again. Nothing in the General log, nothing in dmesg. Lights are on at the physical interface.

However, this time pulling RJ45 and plugging in again doesn't bring up the interface:

Code: [Select]

Jan 11 17:56:07 sudo: ASDFGH : TTY=ttyu0 ; PWD=/home/ASDFG ; USER=root ; COMMAND=/sbin/dmesg
Jan 11 17:55:59 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.45.23.1) (interface: iNET[opt1]) (real interface: igb1).
Jan 11 17:55:59 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'igb1'
Jan 11 17:55:59 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for iNET(opt1) but ignoring since interface is configured with static IP (10.45.23.1 ::)
Jan 11 17:55:59 kernel: igb1: link state changed to UP
Jan 11 17:55:55 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for iNET(opt1) but ignoring since interface is configured with static IP (10.45.23.1 ::)
Jan 11 17:55:55 kernel: igb1: link state changed to DOWN
Jan 11 17:55:18 sudo: ASDFGH : TTY=ttyu0 ; PWD=/home/ASDFGH ; USER=root ; COMMAND=/sbin/dmesg
OK, pulled the RJ45 a second time, this time the interface came back. Does this make sense at all?

[franco]

> How does userspace service 'breaks' ethernet traffic?

I'm merely saying it could meddle with interface reload if e.g. the service hangs while we try to reconfigure an interface.

There is no Ethernet breakage. If you try to go through a door but the door is stuck you're not crashing the door but you don't get anywhere also.


Cheers,
Franco

[franco]

Any VIPs configured? IPS on that interface? I have no idea why the NIC goes down.


Cheers,
Franco