netmap_transmit bce0 drop mbuf that needs checksum offload

[andreaslink]

So this is driver related and nothing I can manipulate via setup adjustments? Else let me know, if there is something I can do to test any suggestions.

[mb]

Hi @andreas,

It looks like this is more like a wireguard problem. This is why I wonder if there's any other people having this -maybe- with other ethernet adapters?

[andreaslink]

Just updated to OPNsense v20.7.4 and this error is still existing and can be reproduced exactly the same way .

When rebooting, I ping my wireguard side2side VPN peer and get a working wg handshake. I do not do anything to the network towards the other side for around ~5 minutes (or more). Then the peer pings me or a host in my net (~ at 21:00:20) and I get the following error right in the moment I receive the ping (read bottom up, see timestamp):

2020-10-23T21:02:55   kernel   775.438456 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:50   kernel   770.065512 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:45   kernel   764.944924 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:40   kernel   759.823284 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:34   kernel   754.446545 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:29   kernel   749.073050 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:23   kernel   743.694746 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:18   kernel   738.574607 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:13   kernel   733.199104 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:08   kernel   728.080889 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:02   kernel   722.702687 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:00   configctl[65140]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-10-23T21:02:00   configctl[97151]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-10-23T21:01:57   kernel   717.330560 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:52   kernel   712.208508 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:46   kernel   706.830221 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:41   kernel   701.710273 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:36   kernel   696.334982 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:31   kernel   690.959233 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:25   kernel   685.582285 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:20   kernel   680.206269 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:15   kernel   675.088935 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:09   kernel   669.711789 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:04   kernel   664.334168 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:00:59   kernel   659.213843 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:00:54   kernel   653.966996 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:00:49   kernel   648.882188 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T20:28:15   configctl[79183]   event @ 1603477695.12 exec: system event config_changed
2020-10-23T20:28:15   configctl[79183]   event @ 1603477695.12 msg: Oct 23 20:28:15 OPNsense.lan config[42107]: config-event: new_config /conf/backup/config-1603477695.1182.xml
2020-10-23T20:28:02   configctl[79183]   event @ 1603477682.01 exec: system event config_changed
2020-10-23T20:28:02   configctl[79183]   event @ 1603477682.01 msg: Oct 23 20:28:02 OPNsense.lan config[42107]: config-event: new_config /conf/backup/config-1603477682.0029.xml
2020-10-23T20:23:28   opnsense[97415]   plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
2020-10-23T20:23:28   opnsense[97415]   plugins_configure dhcp ()

So this problem is still existing and could not be fixed with new drivers so far or likewise.

[mb]

Hi @andreas,

We think that this is not related to netmap and has something to do with wireguard's resetting interface flags; but did not find much time to have a look in detail.

I'll post an update once we have some more information.

[kpiq]

Greetings,

Getting same message on 20.7.5, except there is no VPN tunnel, the system just starts increasing its memory use, all of a sudden name resolution breaks a few minutes after rebooting.   

Have disabled all hardware checksum offloading and VLAN filtering.   It is now more stable.   Let's see how long it lasts. 

System Information:

RAM: 4G
Versions    OPNsense 20.7.5-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1h 22 Sep 2020
Updates    Click to check for updates.
CPU Type    Intel(R) Atom(TM) CPU E3845 @ 1.91GHz (4 cores)
CPU usage    
Load average    0.12, 0.32, 0.45
Uptime    00:20:39
Current date/time    Wed Nov 25 19:45:52 +04 2020
Last config change    Wed Nov 25 19:25:38 +04 2020
State table size    
0 % ( 94/394000 )
MBUF Usage    
0 % ( 2160/242796 )
Memory usage    
70 % ( 2799/3947 MB )
SWAP usage    
0 % ( 0/8192 MB )
0 % ( 0/2048 MB )
Disk usage    
9% / [ufs] (4.2G/49G)


Log from previous failure follows:

---<<BOOT>>---
Copyright (c) 2013-2019 The HardenedBSD Project.
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p10-HBSD #0  6e16e28f1bf(stable/20.7)-dirty: Tue Oct 20 13:30:19 CEST 2020
    root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(efifb): resolution 800x600
HardenedBSD: initialize and check features (__HardenedBSD_version 1200059 __FreeBSD_version 1201000).
CPU: Intel(R) Atom(TM) CPU  E3845  @ 1.91GHz (1916.71-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30679  Family=0x6  Model=0x37  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3960811520 (3777 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20181213/tbfadt-748)
ioapic0 <Version 2.0> irqs 0-86 on motherboard
Launching APs: 2 1 3
Timecounter "TSC" frequency 1916713096 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
kbd0 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff8128e7c0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
000.000054 [4335] netmap_init               netmap: loaded module
[ath_hal] loaded
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 10a11fff vs 10a12000
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xe080-0xe087 mem 0x90000000-0x903fffff,0x80000000-0x8fffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
ahci0: <AHCI SATA controller> port 0xe070-0xe077,0xe060-0xe063,0xe050-0xe057,0xe040-0xe043,0xe020-0xe03f mem 0x90a11000-0x90a117ff irq 19 at device 19.0 on pci0
ahci0: AHCI v1.30 with 2 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
xhci0: <Intel BayTrail USB 3.0 controller> mem 0x90a00000-0x90a0ffff irq 20 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection> port 0xd000-0xd01f mem 0x90900000-0x9091ffff,0x90920000-0x90923fff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:e0:67:1f:6e:a0
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) PRO/1000 Network Connection> port 0xc000-0xc01f mem 0x90800000-0x9081ffff,0x90820000-0x90823fff irq 17 at device 0.0 on pci2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:e0:67:1f:6e:a1
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) PRO/1000 Network Connection> port 0xb000-0xb01f mem 0x90700000-0x9071ffff,0x90720000-0x90723fff irq 18 at device 0.0 on pci3
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:e0:67:1f:6e:a2
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) PRO/1000 Network Connection> port 0xa000-0xa01f mem 0x90600000-0x9061ffff,0x90620000-0x90623fff irq 19 at device 0.0 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:e0:67:1f:6e:a3
em3: netmap queues/slots: TX 1/1024, RX 1/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Sleep Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
uart0: console (115200,n,8,1)
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ugen0.1: <0x8086 XHCI root HUB> at usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Protectli 64GB mSATA SBFMBB.3> ACS-4 ATA SATA 3.x device
ada0: Serial Number Bxxxxxxxxxxxxxxxxxxx
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors)
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Mounting filesystems...
tunefs: soft updates remains unchanged as enabled
tunefs: file system reloaded
tunefs: issue TRIM to the disk remains unchanged as enabled
tunefs: file system reloaded
** /dev/gpt/rootfs
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 11755881 free (9577 frags, 1468288 blocks, 0.1% fragmentation)
uhub0: 7 ports with 7 removable, self powered
Setting hostuuid: xxxxxxxxxxxxxxxxxxxxxxxxxx
Setting hostid: 0xxxxxxxxx.
Configuring vt: blanktime.
ugen0.2: <Microsoft Microsoft Nano Transceiver v2.0> at usbus0
ukbd0 on uhub0
ukbd0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
kbd1 at ukbd0
Configuring crash dump device: /dev/gpt/swapfs
swapon: adding /dev/gpt/swapfs as swap device
.ELF ldconfig path: /lib /usr/lib /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path:
done.
>>> Invoking early script 'update'
>>> Invoking early script 'configd'
Starting configd.
>>> Invoking early script 'templates'
Generating configuration: OK
>>> Invoking early script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
Cannot 'stop' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
Cannot 'start' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.
>>> Invoking backup script 'rrd'
>>> Invoking early script 'carp'
CARP event system: OK
Launching the init system...done.
Initializing...........done.
em0: link state changed to UP
em1: link state changed to UP
em2: link state changed to UP
em3: link state changed to UP
Starting device manager...
ums0 on uhub0
ums0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
ums0: 5 buttons and [XYZT] coordinates ID=26
ums0: 0 buttons and [T] coordinates ID=0
uhid0 on uhub0
uhid0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
done.
Configuring login behaviour...done.
Configuring loopback interface...
lo0: link state changed to UP
done.
Configuring kernel modules...
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
done.
Setting up extended sysctls...done.
Setting timezone...done.
Writing firmware setting...done.
Writing trust files...done.
Setting hostname: psfwl01.lan
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...
em1: link state changed to DOWN
lagg0: IPv6 addresses on em1 have been removed before adding it as a member to prevent IPv6 address scope violation.
lagg0: link state changed to DOWN
em2: link state changed to DOWN
lagg0: IPv6 addresses on em2 have been removed before adding it as a member to prevent IPv6 address scope violation.
done.
Configuring VLAN interfaces...done.
Configuring LAN interface...done.
Configuring OPT2_OOBM interface...
em3: link state changed to DOWN
done.
Configuring WAN interface...
em0: link state changed to DOWN
done.
Creating IPsec VTI instances...done.
Creating OpenVPN instances...done.
Generating /etc/resolv.conf...done.
Configuring firewall........done.
Starting PFLOG...done.
pflog0: promiscuous mode enabled
Configuring OpenSSH...done.
Starting web GUI...done.
Configuring CRON...done.
Setting up routes...
em1: link state changed to UP
done.
Generating /etc/hosts...done.
Starting DHCPv4 service...done.
em2: link state changed to UP
Starting Unbound DNS...
em3: link state changed to UP
em0: link state changed to UP
done.
Setting up gateway monitors...done.
Configuring firewall........done.
Starting PFLOG...
pflog0: promiscuous mode disabled
done.
pflog0: promiscuous mode enabled
lagg0: link state changed to UP
Syncing OpenVPN settings...done.
Starting NTP service...deferred.
Starting Unbound DNS...done.
Generating RRD graphs...done.
Configuring system logging...done.
>>> Invoking start script 'newwanip'
>>> Invoking start script 'freebsd'
Starting suricata.
\^[[32m25/11/2020 -- 11:03:01\^[[0m - <\^[[33mInfo\^[[0m> - Including configuration file installed_rules.yaml.\^[[0m
\^[[32m25/11/2020 -- 11:03:01\^[[0m - <\^[[33mInfo\^[[0m> - Configuration node 'rule-files' redefined.\^[[0m
\^[[32m25/11/2020 -- 11:03:01\^[[0m - <\^[[33mInfo\^[[0m> - Including configuration file custom.yaml.\^[[0m
>>> Invoking start script 'syslog-ng'
Stopping syslog_ng.
Waiting for PIDS: 76975.
Starting syslog_ng.
>>> Invoking start script 'frr'
Checking zebra.conf
2020/11/25 11:03:02 ZEBRA: [EC 4043309110] Disabling MPLS support (no kernel support)
OK
Starting zebra.
2020/11/25 11:03:02 ZEBRA: [EC 4043309110] Disabling MPLS support (no kernel support)
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK
>>> Invoking start script 'beep'
Root file system: /dev/gpt/rootfs
Wed Nov 25 11:03:04 AST 2020

*** psfwl01.lan: OPNsense 20.7.5 (amd64/OpenSSL) ***

 LAN (lagg0)     -> v4: 10.254.0.1/29
 OPT2_OOBM (em3) -> v4: 10.252.0.1/24
 WAN (em0)       -> v4: 192.168.1.2/24

 HTTPS: SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 SSH:   SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (ECDSA)
 SSH:   SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (ED25519)
 SSH:   SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (RSA)
em3: link state changed to DOWN
em0: link state changed to DOWN
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
295.605088 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
296.000171 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
299.663500 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
300.711246 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
301.944716 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
307.152032 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
317.161617 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
319.942421 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
326.090014 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
327.171306 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
329.157861 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
332.230135 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
335.301443 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
337.180975 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
341.444752 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
341.444752 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
344.517496 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
347.190720 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
349.634838 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
353.732195 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
356.125718 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
357.200401 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
359.875731 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
361.055369 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
362.055363 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
363.055326 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
366.019362 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
366.019362 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
367.210064 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
369.091586 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
372.163588 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
375.235404 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
377.219926 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
381.378750 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
386.311801 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
387.229683 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
392.897270 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
397.239161 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
401.451326 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
407.248862 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
417.258537 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
423.265873 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
425.120266 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
427.268229 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
428.055273 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
430.055338 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
431.264016 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
432.184853 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
434.335220 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
437.277935 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
440.478657 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
444.510706 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
447.288637 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
457.176310 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
467.296949 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
477.306677 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
481.444369 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
487.316747 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
496.054340 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
497.054339 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
498.055333 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
499.054362 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
507.325814 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
510.829153 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
511.441303 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
517.336104 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
526.025186 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
527.345187 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
537.355094 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
547.364585 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
551.441303 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
556.083072 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
557.374335 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
562.055312 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
564.055322 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
565.055323 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
567.383876 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
577.394167 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
587.403621 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
591.436833 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
596.110595 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
597.413115 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
607.423465 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
617.442410 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
627.102088 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
631.055359 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
632.055328 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
633.055327 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
634.055313 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
637.452339 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
647.461424 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
648.027429 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
649.044572 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
650.062587 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
651.107827 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
652.126261 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
653.144827 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
654.155589 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
657.471179 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
661.431557 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
667.480892 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
676.141464 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
677.490796 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
683.349199 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
687.500302 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
697.509950 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
698.055279 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
700.055308 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
701.055359 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
707.519618 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
717.529263 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
727.538982 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
737.548714 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
741.423212 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
747.558355 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
757.568210 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
766.055284 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
767.577969 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
768.055289 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
770.055322 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
776.075766 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
777.587462 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
781.419079 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
787.597092 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
797.606852 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
807.616503 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
817.626504 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
821.419447 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
827.635923 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
834.055306 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
836.055317 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
837.055314 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
844.166790 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
847.645304 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
857.654964 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
861.406558 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
867.664716 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
em0: link state changed to DOWN
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
em0: link state changed to UP
em0: link state changed to DOWN
pflog0: promiscuous mode disabled
pid 98156 (syslog-ng), jid 0, uid 0: exited on signal 6 (core dumped)
em0: link state changed to UP
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 8 8 0 0 done
Waiting (max 60 seconds) for system thread `bufdaemon' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-0' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-1' to stop... done
All buffers synced.
Uptime: 22m34s
---<<BOOT>>---
Copyright (c) 2013-2019 The HardenedBSD Project.
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p10-HBSD #0  6e16e28f1bf(stable/20.7)-dirty: Tue Oct 20 13:30:19 CEST 2020
    root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(efifb): resolution 800x600
HardenedBSD: initialize and check features (__HardenedBSD_version 1200059 __FreeBSD_version 1201000).
CPU: Intel(R) Atom(TM) CPU  E3845  @ 1.91GHz (1916.71-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30679  Family=0x6  Model=0x37  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3960811520 (3777 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20181213/tbfadt-748)
ioapic0 <Version 2.0> irqs 0-86 on motherboard
Launching APs: 2 1 3
Timecounter "TSC" frequency 1916712774 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
kbd0 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff8128e7c0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
000.000054 [4335] netmap_init               netmap: loaded module
[ath_hal] loaded
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 10a11fff vs 10a12000
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xe080-0xe087 mem 0x90000000-0x903fffff,0x80000000-0x8fffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
ahci0: <AHCI SATA controller> port 0xe070-0xe077,0xe060-0xe063,0xe050-0xe057,0xe040-0xe043,0xe020-0xe03f mem 0x90a11000-0x90a117ff irq 19 at device 19.0 on pci0
ahci0: AHCI v1.30 with 2 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
xhci0: <Intel BayTrail USB 3.0 controller> mem 0x90a00000-0x90a0ffff irq 20 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection> port 0xd000-0xd01f mem 0x90900000-0x9091ffff,0x90920000-0x90923fff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:e0:67:1f:6e:a0
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) PRO/1000 Network Connection> port 0xc000-0xc01f mem 0x90800000-0x9081ffff,0x90820000-0x90823fff irq 17 at device 0.0 on pci2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:e0:67:1f:6e:a1
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) PRO/1000 Network Connection> port 0xb000-0xb01f mem 0x90700000-0x9071ffff,0x90720000-0x90723fff irq 18 at device 0.0 on pci3
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:e0:67:1f:6e:a2
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) PRO/1000 Network Connection> port 0xa000-0xa01f mem 0x90600000-0x9061ffff,0x90620000-0x90623fff irq 19 at device 0.0 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:e0:67:1f:6e:a3
em3: netmap queues/slots: TX 1/1024, RX 1/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Sleep Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
uart0: console (115200,n,8,1)
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ugen0.1: <0x8086 XHCI root HUB> at usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Protectli 64GB mSATA SBFMBB.3> ACS-4 ATA SATA 3.x device
ada0: Serial Number Bxxxxxxxxxxxxxxxxx
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors)
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Mounting filesystems...
tunefs: soft updates remains unchanged as enabled
tunefs: file system reloaded
tunefs: issue TRIM to the disk remains unchanged as enabled
tunefs: file system reloaded
** /dev/gpt/rootfs
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 11754518 free (9558 frags, 1468120 blocks, 0.1% fragmentation)
uhub0: 7 ports with 7 removable, self powered
Setting hostuuid: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Setting hostid: 0xxxxxxxxx.
Configuring vt: blanktime.
ugen0.2: <Microsoft Microsoft Nano Transceiver v2.0> at usbus0
ukbd0 on uhub0
ukbd0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
kbd1 at ukbd0
Configuring crash dump device: /dev/gpt/swapfs
swapon: adding /dev/gpt/swapfs as swap device
.ELF ldconfig path: /lib /usr/lib /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path:
done.
>>> Invoking early script 'update'
>>> Invoking early script 'configd'
Starting configd.
>>> Invoking early script 'templates'
Generating configuration: OK
>>> Invoking early script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
Cannot 'stop' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
Cannot 'start' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.
>>> Invoking backup script 'rrd'
>>> Invoking early script 'carp'
CARP event system: OK
Launching the init system...done.
Initializing...........done.
em0: link state changed to UP
em1: link state changed to UP
em2: link state changed to UP
Starting device manager...
ums0 on uhub0
ums0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
ums0: 5 buttons and [XYZT] coordinates ID=26
ums0: 0 buttons and [T] coordinates ID=0
uhid0 on uhub0
uhid0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
done.
Configuring login behaviour...done.
Configuring loopback interface...
lo0: link state changed to UP
done.
Configuring kernel modules...
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
done.
Setting up extended sysctls...done.
Setting timezone...done.
Writing firmware setting...done.
Writing trust files...done.
Setting hostname: psfwl01.lan
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...
em1: link state changed to DOWN
lagg0: IPv6 addresses on em1 have been removed before adding it as a member to prevent IPv6 address scope violation.
lagg0: link state changed to DOWN
em2: link state changed to DOWN
lagg0: IPv6 addresses on em2 have been removed before adding it as a member to prevent IPv6 address scope violation.
done.
Configuring VLAN interfaces...done.
Configuring LAN interface...done.
Configuring OPT2_OOBM interface...done.
Configuring WAN interface...
em0: link state changed to DOWN
done.
Creating IPsec VTI instances...done.
Creating OpenVPN instances...done.
Generating /etc/resolv.conf...done.
Configuring firewall........done.
Starting PFLOG...done.
pflog0: promiscuous mode enabled
Configuring OpenSSH...done.
Starting web GUI...done.
Configuring CRON...done.
Setting up routes...done.
Generating /etc/hosts...done.
em1: link state changed to UP
Starting DHCPv4 service...
em2: link state changed to UP
done.
Starting Unbound DNS...
em0: link state changed to UP
done.
Setting up gateway monitors...done.
Configuring firewall........done.
Starting PFLOG...
pflog0: promiscuous mode disabled
done.
pflog0: promiscuous mode enabled
lagg0: link state changed to UP
Syncing OpenVPN settings...done.
Starting NTP service...deferred.
Starting Unbound DNS...done.
Generating RRD graphs...done.
Configuring system logging...done.
>>> Invoking start script 'newwanip'
>>> Invoking start script 'freebsd'
Starting suricata.
\^[[32m25/11/2020 -- 19:26:01\^[[0m - <\^[[33mInfo\^[[0m> - Including configuration file installed_rules.yaml.\^[[0m
\^[[32m25/11/2020 -- 19:26:01\^[[0m - <\^[[33mInfo\^[[0m> - Configuration node 'rule-files' redefined.\^[[0m
\^[[32m25/11/2020 -- 19:26:01\^[[0m - <\^[[33mInfo\^[[0m> - Including configuration file custom.yaml.\^[[0m
>>> Invoking start script 'syslog-ng'
Stopping syslog_ng.
Waiting for PIDS: 65881.
Starting syslog_ng.
>>> Invoking start script 'frr'
Checking zebra.conf
2020/11/25 19:26:03 ZEBRA: [EC 4043309110] Disabling MPLS support (no kernel support)
OK
Starting zebra.
2020/11/25 19:26:03 ZEBRA: [EC 4043309110] Disabling MPLS support (no kernel support)
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK
>>> Invoking start script 'beep'
Root file system: /dev/gpt/rootfs
Wed Nov 25 19:26:05 +04 2020

*** psfwl01.lan: OPNsense 20.7.5 (amd64/OpenSSL) ***

 LAN (lagg0)     -> v4: 10.254.0.1/29
 OPT2_OOBM (em3) -> v4: 10.252.0.1/24
 WAN (em0)       -> v4: 192.168.1.2/24

 HTTPS: SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
               xxxxxxxxxxxxxxxxxxxxxxxxxx
 SSH:   SHA256 0xxxxxxxxxxxxxxxxxxxxxxxxxxxxx (ECDSA)
 SSH:   SHA256 xxxxxxxxxxxxxxxxxxxxxx (ED25519)
 SSH:   SHA256 xxxxxxxxxxxxxxxxxx (RSA)
em0: link state changed to DOWN
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
em0: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled


Regards

Pedro Serrano

[mb]

Hi @kpiq, are you also on bce driver?

[kpiq]

Apparently not.

root@psfwl01:~ # kldstat
Id Refs Address                Size Name
 1   35                0x0  268d060 kernel
 2    1                0x0    10250 carp.ko
 3    1                0x0     f998 if_bridge.ko
 4    2                0x0     72a8 bridgestp.ko
 5    1                0x0     3e78 if_enc.ko
 6    1                0x0     b1c0 if_gre.ko
 7    1                0x0    16008 if_lagg.ko
 8    1                0x0     8b60 if_tap.ko
 9    3                0x0    582f0 pf.ko
10    1                0x0     2af8 pflog.ko
11    1                0x0     ebd0 pfsync.ko
12    1                0x0     18a0 uhid.ko
13    1                0x0     2928 ums.ko
14    1                0x0     1aa0 wmt.ko
15    1                0x0     8d50 aesni.ko

[kpiq]

Here's my hardware:

root@psfwl01:~ # pciconf -lv
hostb0@pci0:0:0:0:   class=0x060000 card=0x22128086 chip=0x0f008086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series SoC Transaction Register'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0:   class=0x030000 card=0x22128086 chip=0x0f318086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Graphics & Display'
    class      = display
    subclass   = VGA
ahci0@pci0:0:19:0:   class=0x010601 card=0x0f238086 chip=0x0f238086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series SATA AHCI Controller'
    class      = mass storage
    subclass   = SATA
xhci0@pci0:0:20:0:   class=0x0c0330 card=0x0f358086 chip=0x0f358086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx, Celeron N2000 Series USB xHCI'
    class      = serial bus
    subclass   = USB
none0@pci0:0:26:0:   class=0x108000 card=0x0f188086 chip=0x0f188086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine'
    class      = encrypt/decrypt
pcib1@pci0:0:28:0:   class=0x060400 card=0x0f488086 chip=0x0f488086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:28:1:   class=0x060400 card=0x0f4a8086 chip=0x0f4a8086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 2'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:28:2:   class=0x060400 card=0x0f4c8086 chip=0x0f4c8086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 3'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:28:3:   class=0x060400 card=0x0f4e8086 chip=0x0f4e8086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 4'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:   class=0x060100 card=0x0f1c8086 chip=0x0f1c8086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Power Control Unit'
    class      = bridge
    subclass   = PCI-ISA
none1@pci0:0:31:3:   class=0x0c0500 card=0x0f128086 chip=0x0f128086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series SMBus Controller'
    class      = serial bus
    subclass   = SMBus
em0@pci0:1:0:0:   class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em1@pci0:2:0:0:   class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em2@pci0:3:0:0:   class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em3@pci0:4:0:0:   class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet

[kpiq]

I haven't seen this "drop mbuf" situation happen again after disabling all checksum offloading.  But the high memory use, the failing name resolution, and general sluggish performance is still a problem.  I see Suricata using lots of memory.   I configured it to use  IDS with HyperScan.   Here is a top snapshot.

last pid: 97141;  load averages:  0.45,  0.47,  0.59    up 0+00:22:31  23:42:11
41 processes:  1 running, 40 sleeping
CPU:  4.1% user,  0.0% nice,  0.7% system,  0.1% interrupt, 95.1% idle
Mem: 2042M Active, 99M Inact, 702M Wired, 209M Buf, 941M Free
Swap: 10G Total, 10G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
55943 root          7  20    0  4432M  1998M nanslp   0  11:56   1.40% suricata
 6906 root          1  20    0    40M    22M accept   1   0:01   1.27% php-cgi
20515 root          1  52    0    77M    47M accept   1   0:18   0.53% python3.
22285 root          1  20    0    21M    13M kqread   0   0:05   0.11% lighttpd
85104 root          1  20    0  1038M  3988K CPU0     0   0:00   0.07% top
44995 root          2  20    0    26M    10M kqread   3   0:01   0.04% syslog-n
57851 root          1  20    0  1042M  3396K select   2   0:01   0.03% syslogd
77086 root          1  20    0    27M    17M select   0   0:03   0.03% python3.
76484 root          1  20    0  1037M  3256K bpf      0   0:00   0.02% filterlo
 2818 root          1  20    0  1050M  7088K select   3   0:00   0.02% ntpd
73550 root          1  20    0    20M    10M select   2   0:00   0.02% python3.
20354 root          1  20    0    21M    11M select   0   0:00   0.01% python3.
61391 root          1  20    0  1048M  7904K select   1   0:00   0.01% sshd
67620 root          1  20    0    40M    22M accept   3   0:03   0.00% php-cgi
39280 root          1  52    0    40M    22M accept   1   0:03   0.00% php-cgi
79766 root          1  52    0    40M    22M accept   1   0:03   0.00% php-cgi
 4785 root          1  52    0    40M    22M accept   0   0:02   0.00% php-cgi
58740 root          1  52    0    31M    19M wait     1   0:02   0.00% python3.
11160 unbound       4  20    0    74M    33M kqread   0   0:00   0.00% unbound
 4429 root          1  52    0  1037M  3324K wait     3   0:00   0.00% sh
21869 root          1  52    0    40M    22M accept   0   0:00   0.00% php-cgi
79316 root          1  20    0    39M    18M wait     2   0:00   0.00% php-cgi
75327 root          1  52    0    39M    18M wait     0   0:00   0.00% php-cgi
75337 root          1  52    0  1036M  3264K nanslp   3   0:00   0.00% cron
21315 root          1  20    0  1044M  4592K pause    2   0:00   0.00% csh
70892 root          1  20    0    14M  5016K select   2   0:00   0.00% devd
  230 dhcpd         1  20    0    22M    10M select   2   0:00   0.00% dhcpd
86589 root          3  20    0  1061M  8648K select   3   0:00   0.00% zebra
67491 root          1  52    0  1037M  3296K wait     2   0:00   0.00% sh

[andreaslink]

Hi @andreas,

We think that this is not related to netmap and has something to do with wireguard's resetting interface flags; but did not find much time to have a look in detail.

I'll post an update once we have some more information.

I just want to bring this topic up again. I doubt you found the time @mb to look further into this, but since this is still existing with the exact same situation as before and in my case triggered via a foreign wireguard ping, I thought I bring it up again and ask again.
So the same problem still existing in 21.1.3_3 so far (did not go to 20.1.4 yet, due to major wireguard implementation changes).

Any news here or plan to progress?

[MenschAergereDichNicht]

I am not sure that it is related. But i read something about Wireguard and checksum errors inside this thread and just wanted to add that i also have some problems in that area.
I turned off all hardware offload features and still see UDP checksum errors. In my case it is a PPPoE interface on top of a VLAN (for WAN connection).

I am using OPNsense 21.7.6.

I am sure that the problem is specific to Wireguard as other UDP traffic on this interface does not create invalid packets.