OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: andreaslink on September 14, 2020, 10:58:00 pm

Title: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 14, 2020, 10:58:00 pm

I'm running OPNsense (20.7.2-amd64) with one Broadcom NetXtreme II BCM5709 for WAN (bce0) and one for LAN (bce1), further on I have 4x Intel 82580, which I use for other LANs like IoT (igb1) and Guests (igb0) etc.

I have "some" traffic on WAN with quite constantly 60 to 100MBit (mainly due to IP cam streams), which I consider as handeable with my setup. I also have IDS/IPS up and running as well as Sensei.

After "a while" (usually only minutes after reboot) of traffic I get the following error in the log, multiple times per second:

2020-09-10T00:28:10   kernel   490.690419 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:05   kernel   485.572543 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:00   kernel   480.194945 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:00   kernel   479.940436 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:54   kernel   474.761838 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:49   kernel   469.475112 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:44   kernel   464.324372 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:39   kernel   459.205033 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:33   kernel   453.830080 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:28   kernel   448.126626 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:23   kernel   443.431391 [ 320] generic_netmap_register Emulated adapter for bce0 activated
2020-09-10T00:27:23   kernel   443.431259 [1130] generic_netmap_attach Emulated adapter for bce0 created (prev was NULL)
2020-09-10T00:27:23   kernel   bce0: permanently promiscuous mode enabled
2020-09-10T00:27:23   kernel   443.407436 [1035] generic_netmap_dtor Emulated netmap adapter for bce0 destroyed
2020-09-10T00:27:23   kernel   443.407409 [1130] generic_netmap_attach Emulated adapter for bce0 created (prev was NULL)

As you can see on the attached screenshot, the MBUF usage is at 0% and with ~9720 way below the limit of 1.271.626, so there should be plenty of MBUF available.

So what triggers this error?

I can get rid of it, when deactivating IDS/IPS, and since I'm testing it, the error did not show up again. So is it somehow IPS throughput related? Nonetheless, I would like to turn IDS/IPS on again :).

How can I tune my system, so the "netmap_transmit" can handle the load? (BTW: What process/step ist it, what does it do here?)
And whay does the mbuf "need checksum offload"? What does that exactly mean?

Some more config details:

I have all three hooks set, so all of these three are disabled:
- Hardware CRC
- Hardware TSO
- Hardware LRO


root@OPNsense:~ # sysctl -a | grep nmbclusters
kern.ipc.nmbclusters: 1271626

root@OPNsense:~ # sysctl -a | grep msi
hw.sdhci.enable_msi: 1
hw.puc.msi_disable: 0
hw.pci.honor_msi_blacklist: 1
hw.pci.msix_rewrite_table: 0
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1
hw.mfi.msi: 1
hw.malo.pci.msi_disable: 0
hw.ix.enable_msix: 1
hw.bce.msi_enable: 1
hw.aac.enable_msi: 1
machdep.disable_msix_migration: 0
machdep.num_msi_irqs: 512
dev.igb.3.iflib.disable_msix: 0
dev.igb.2.iflib.disable_msix: 0
dev.igb.1.iflib.disable_msix: 0
dev.igb.0.iflib.disable_msix: 0


BTW: I also experimented with following values, which did not bring any change:

kern.ipc.nmbclusters="2543660"
hw.bce.tso_enable="0"
hw.pci.enable_msix="0"

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on September 16, 2020, 04:23:02 pm

Hi @andreaslink, do you have offloadings and vlan hardware filtering set to disabled? See Interfaces -> Settings

If so, please try the official netmap test kernel which will be announced today

opnsense-update -kr 20.7.2-netmap

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 16, 2020, 08:52:34 pm

Awesome @mb, thank you! I have done that and rebooted:

root@OPNsense:~ # opnsense-update -kr 20.7.2-netmap
Fetching kernel-20.7.2-netmap-amd64.txz: ....... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-20.7.2-netmap-amd64.txz... done
Please reboot.

I've also activated IDS/IPS again to monitor it now. 5 mins later no problems yet, so still monitoring.
I keep you posted!

PS: And as requested, all offloadings and vlan hardware filtering were already set to disabled.

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 17, 2020, 10:56:17 pm

Just to return some feedback here, I'm testing now for 24h under "full load" incl. IDS/IPS and Sensei and the messages did not appear anymore. So I consider this issue as solved with the new kernel "kernel-20.7.2-netmap-amd64.txz"!

Thank you very much :)!

PS: I assume, my preloading to test before next official update is not an issue for the upcoming release aka official update or will I get in troube with this kernel now?

Title: Re: (SOLVED) netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on September 17, 2020, 11:42:16 pm

Hi Andreas,

That's great to hear. All welcome and thanks for the update.

No, you're fine. 20.7.3 will just install it's own kernel.

Title: Re: (SOLVED) netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: franco on September 18, 2020, 08:37:57 am

We will have a new kernel with 20.7.3 from the looks of it, but we will give netmap another test round so it's a later 20.7.x for sure.


Cheers,
Franco

Title: Re: (SOLVED) netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 18, 2020, 11:41:15 am

Hmm, Ok understandable and good to know, so hoping, I will not expect these errors again then, but then I know at least why :). Thanks for making me aware.

Title: Re: (SOLVED) netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 22, 2020, 07:47:48 pm

Some bad news :(, router was running 5 days, nearly 6 days now, without this issue, but today - out of a sudden - the error messages returned:
[4006] netmap_transmit bce0 drop mbuf that needs checksum offload

MBUF usage is slightly higher than normal, but far (!) away from the ciritcal maximum:

MBUF Usage  0% (10432/1271498)

So I'm not sure, if I can consider this still as solved, but at least as remarkably better.

And I said "out of a sudden" but I'm afraid the trigger might be somehow in relation to my wireguard side2side VPN, it started briefly after the counter part send a test ping after a quite long period of data silence between the two locations. I'm not sure, if this might be related, it could be coincidence, but I think, I should mention it.

Title: Re: (SOLVED) netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on September 22, 2020, 08:36:53 pm

Hi @andreas,

Thanks for the mention. That might be related. It looks like somehow HW checksum offload was enabled on the interface. Netmap requires all HW offloading be disabled.

What does ifconfig bce0 tell?

Title: Re: (SOLVED) netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 24, 2020, 07:36:14 pm

Sorry for the delay, I had to provoke it first.
I can now clearly destroy it with this approach:

• Reboot OPNsense --> Everything is fine
• Let a ping go constanly from my network over into the other net of the wireguard tunnel --> Everything is fine
• Stop my ping  --> Everything is fine
• Let other wireguard side ping one device of my local network --> MBUF error appears nearly immediatly with the first ping

So when everything is working fine it looks like this:

root@OPNsense:~ # ifconfig bce0
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
   ether 00:21:5e:c8:be:88
   inet6 fe80::221:5eff:fec8:be88%bce0 prefixlen 64 scopeid 0x1
   inet6 2a02:2f4:xxxx:xxx0:221:5eff:fec8:be88 prefixlen 64 autoconf
   inet6 fd00:0:cafe:affe:221:5eff:fec8:be88 prefixlen 64 autoconf
   inet 192.168.0.100 netmask 0xffffff00 broadcast 192.168.0.255
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


After wireguard ping trigger from other side partner:

root@OPNsense:~ # ifconfig bce0
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
   ether 00:21:5e:c8:be:88
   inet6 fe80::221:5eff:fec8:be88%bce0 prefixlen 64 scopeid 0x1
   inet6 2a02:2f4:xxxx:xxx0:221:5eff:fec8:be88 prefixlen 64 autoconf
   inet6 fd00:0:cafe:affe:221:5eff:fec8:be88 prefixlen 64 autoconf
   inet 192.168.0.100 netmask 0xffffff00 broadcast 192.168.0.255
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

I cannot see an immediate difference here ???.

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on September 25, 2020, 02:16:15 am

Hi Andreas,

I thought this might be related to netmap offloads being enabled. But it looks like this is different.

Can you share the exact mbuf error message? May be a screenshot?

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 25, 2020, 07:56:35 am

Ok, I see.
So I hope, this helps, for a screenshot I need to provoke it later again, these are messages from v20.7.2 (see dates):


2020-09-10T00:28:10   kernel   490.690419 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:05   kernel   485.572543 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:00   kernel   480.194945 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:00   kernel   479.940436 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:54   kernel   474.761838 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:49   kernel   469.475112 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:44   kernel   464.324372 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:39   kernel   459.205033 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:33   kernel   453.830080 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:28   kernel   448.126626 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:23   kernel   443.431391 [ 320] generic_netmap_register Emulated adapter for bce0 activated
2020-09-10T00:27:23   kernel   443.431259 [1130] generic_netmap_attach Emulated adapter for bce0 created (prev was NULL)
2020-09-10T00:27:23   kernel   bce0: permanently promiscuous mode enabled
2020-09-10T00:27:23   kernel   443.407436 [1035] generic_netmap_dtor Emulated netmap adapter for bce0 destroyed
2020-09-10T00:27:23   kernel   443.407409 [1130] generic_netmap_attach Emulated adapter for bce0 created (prev was NULL)


Another case:

2020-09-09T23:42:03   kernel   723.581121 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-09T23:41:58   kernel   718.205255 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-09T23:41:53   kernel   713.085191 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-09T23:41:48   kernel   707.965228 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-09T23:41:42   kernel   702.589255 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-09T23:41:37   kernel   697.337566 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-09T23:02:00   configctl[8592]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-09-09T23:02:00   configctl[53856]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-09-09T22:43:00   /update_tables.py[68067]   fetch alias url https://www.spamhaus.org/drop/drop.txt (lines: 944)
2020-09-09T22:02:00   /update_tables.py[62598]   fetch alias url https://www.spamhaus.org/drop/dropv6.txt (lines: 39)
2020-09-09T22:02:00   configctl[9337]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-09-09T22:02:00   configctl[46137]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-09-09T21:59:09   /flowd_aggregate.py[5222]   vacuum done

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on September 26, 2020, 04:00:19 am

Hi @andreas,

Do these checksum offloading errors start just after you start the ping from the other side of the vpn tunnel?

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 26, 2020, 10:37:11 am

Yes, exactly in that moment (or +~2secs).

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on September 28, 2020, 11:35:46 pm

Understood, thanks for the update.

Weird. Then it seems wireguard somehow manages to enable offloadings on the bce adapter....

Anyone who has a similar problem with wireguard + Suricata / Sensei? I wonder if this is a common problem?

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on September 29, 2020, 08:56:00 am

So this is driver related and nothing I can manipulate via setup adjustments? Else let me know, if there is something I can do to test any suggestions.

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on September 29, 2020, 03:12:02 pm

Hi @andreas,

It looks like this is more like a wireguard problem. This is why I wonder if there's any other people having this -maybe- with other ethernet adapters?

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on October 23, 2020, 09:24:46 pm

Just updated to OPNsense v20.7.4 and this error is still existing and can be reproduced exactly the same way :-\.

When rebooting, I ping my wireguard side2side VPN peer and get a working wg handshake. I do not do anything to the network towards the other side for around ~5 minutes (or more). Then the peer pings me or a host in my net (~ at 21:00:20) and I get the following error right in the moment I receive the ping (read bottom up, see timestamp):

2020-10-23T21:02:55   kernel   775.438456 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:50   kernel   770.065512 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:45   kernel   764.944924 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:40   kernel   759.823284 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:34   kernel   754.446545 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:29   kernel   749.073050 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:23   kernel   743.694746 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:18   kernel   738.574607 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:13   kernel   733.199104 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:08   kernel   728.080889 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:02   kernel   722.702687 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:02:00   configctl[65140]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-10-23T21:02:00   configctl[97151]   error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-10-23T21:01:57   kernel   717.330560 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:52   kernel   712.208508 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:46   kernel   706.830221 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:41   kernel   701.710273 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:36   kernel   696.334982 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:31   kernel   690.959233 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:25   kernel   685.582285 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:20   kernel   680.206269 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:15   kernel   675.088935 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:09   kernel   669.711789 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:01:04   kernel   664.334168 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:00:59   kernel   659.213843 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:00:54   kernel   653.966996 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T21:00:49   kernel   648.882188 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-10-23T20:28:15   configctl[79183]   event @ 1603477695.12 exec: system event config_changed
2020-10-23T20:28:15   configctl[79183]   event @ 1603477695.12 msg: Oct 23 20:28:15 OPNsense.lan config[42107]: config-event: new_config /conf/backup/config-1603477695.1182.xml
2020-10-23T20:28:02   configctl[79183]   event @ 1603477682.01 exec: system event config_changed
2020-10-23T20:28:02   configctl[79183]   event @ 1603477682.01 msg: Oct 23 20:28:02 OPNsense.lan config[42107]: config-event: new_config /conf/backup/config-1603477682.0029.xml
2020-10-23T20:23:28   opnsense[97415]   plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
2020-10-23T20:23:28   opnsense[97415]   plugins_configure dhcp ()

So this problem is still existing and could not be fixed with new drivers so far or likewise.

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on October 23, 2020, 11:48:15 pm

Hi @andreas,

We think that this is not related to netmap and has something to do with wireguard's resetting interface flags; but did not find much time to have a look in detail.

I'll post an update once we have some more information.

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: kpiq on November 25, 2020, 04:36:17 pm

Greetings,

Getting same message on 20.7.5, except there is no VPN tunnel, the system just starts increasing its memory use, all of a sudden name resolution breaks a few minutes after rebooting.   

Have disabled all hardware checksum offloading and VLAN filtering.   It is now more stable.   Let's see how long it lasts. 

System Information:

RAM: 4G
Versions    OPNsense 20.7.5-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1h 22 Sep 2020
Updates    Click to check for updates.
CPU Type    Intel(R) Atom(TM) CPU E3845 @ 1.91GHz (4 cores)
CPU usage    
Load average    0.12, 0.32, 0.45
Uptime    00:20:39
Current date/time    Wed Nov 25 19:45:52 +04 2020
Last config change    Wed Nov 25 19:25:38 +04 2020
State table size    
0 % ( 94/394000 )
MBUF Usage    
0 % ( 2160/242796 )
Memory usage    
70 % ( 2799/3947 MB )
SWAP usage    
0 % ( 0/8192 MB )
0 % ( 0/2048 MB )
Disk usage    
9% / [ufs] (4.2G/49G)


Log from previous failure follows:

---<<BOOT>>---
Copyright (c) 2013-2019 The HardenedBSD Project.
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p10-HBSD #0  6e16e28f1bf(stable/20.7)-dirty: Tue Oct 20 13:30:19 CEST 2020
    root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(efifb): resolution 800x600
HardenedBSD: initialize and check features (__HardenedBSD_version 1200059 __FreeBSD_version 1201000).
CPU: Intel(R) Atom(TM) CPU  E3845  @ 1.91GHz (1916.71-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30679  Family=0x6  Model=0x37  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3960811520 (3777 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20181213/tbfadt-748)
ioapic0 <Version 2.0> irqs 0-86 on motherboard
Launching APs: 2 1 3
Timecounter "TSC" frequency 1916713096 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
kbd0 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff8128e7c0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
000.000054 [4335] netmap_init               netmap: loaded module
[ath_hal] loaded
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 10a11fff vs 10a12000
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xe080-0xe087 mem 0x90000000-0x903fffff,0x80000000-0x8fffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
ahci0: <AHCI SATA controller> port 0xe070-0xe077,0xe060-0xe063,0xe050-0xe057,0xe040-0xe043,0xe020-0xe03f mem 0x90a11000-0x90a117ff irq 19 at device 19.0 on pci0
ahci0: AHCI v1.30 with 2 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
xhci0: <Intel BayTrail USB 3.0 controller> mem 0x90a00000-0x90a0ffff irq 20 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection> port 0xd000-0xd01f mem 0x90900000-0x9091ffff,0x90920000-0x90923fff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:e0:67:1f:6e:a0
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) PRO/1000 Network Connection> port 0xc000-0xc01f mem 0x90800000-0x9081ffff,0x90820000-0x90823fff irq 17 at device 0.0 on pci2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:e0:67:1f:6e:a1
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) PRO/1000 Network Connection> port 0xb000-0xb01f mem 0x90700000-0x9071ffff,0x90720000-0x90723fff irq 18 at device 0.0 on pci3
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:e0:67:1f:6e:a2
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) PRO/1000 Network Connection> port 0xa000-0xa01f mem 0x90600000-0x9061ffff,0x90620000-0x90623fff irq 19 at device 0.0 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:e0:67:1f:6e:a3
em3: netmap queues/slots: TX 1/1024, RX 1/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Sleep Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
uart0: console (115200,n,8,1)
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ugen0.1: <0x8086 XHCI root HUB> at usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Protectli 64GB mSATA SBFMBB.3> ACS-4 ATA SATA 3.x device
ada0: Serial Number Bxxxxxxxxxxxxxxxxxxx
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors)
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Mounting filesystems...
tunefs: soft updates remains unchanged as enabled
tunefs: file system reloaded
tunefs: issue TRIM to the disk remains unchanged as enabled
tunefs: file system reloaded
** /dev/gpt/rootfs
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 11755881 free (9577 frags, 1468288 blocks, 0.1% fragmentation)
uhub0: 7 ports with 7 removable, self powered
Setting hostuuid: xxxxxxxxxxxxxxxxxxxxxxxxxx
Setting hostid: 0xxxxxxxxx.
Configuring vt: blanktime.
ugen0.2: <Microsoft Microsoft Nano Transceiver v2.0> at usbus0
ukbd0 on uhub0
ukbd0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
kbd1 at ukbd0
Configuring crash dump device: /dev/gpt/swapfs
swapon: adding /dev/gpt/swapfs as swap device
.ELF ldconfig path: /lib /usr/lib /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path:
done.
>>> Invoking early script 'update'
>>> Invoking early script 'configd'
Starting configd.
>>> Invoking early script 'templates'
Generating configuration: OK
>>> Invoking early script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
Cannot 'stop' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
Cannot 'start' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.
>>> Invoking backup script 'rrd'
>>> Invoking early script 'carp'
CARP event system: OK
Launching the init system...done.
Initializing...........done.
em0: link state changed to UP
em1: link state changed to UP
em2: link state changed to UP
em3: link state changed to UP
Starting device manager...
ums0 on uhub0
ums0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
ums0: 5 buttons and [XYZT] coordinates ID=26
ums0: 0 buttons and [T] coordinates ID=0
uhid0 on uhub0
uhid0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
done.
Configuring login behaviour...done.
Configuring loopback interface...
lo0: link state changed to UP
done.
Configuring kernel modules...
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
done.
Setting up extended sysctls...done.
Setting timezone...done.
Writing firmware setting...done.
Writing trust files...done.
Setting hostname: psfwl01.lan
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...
em1: link state changed to DOWN
lagg0: IPv6 addresses on em1 have been removed before adding it as a member to prevent IPv6 address scope violation.
lagg0: link state changed to DOWN
em2: link state changed to DOWN
lagg0: IPv6 addresses on em2 have been removed before adding it as a member to prevent IPv6 address scope violation.
done.
Configuring VLAN interfaces...done.
Configuring LAN interface...done.
Configuring OPT2_OOBM interface...
em3: link state changed to DOWN
done.
Configuring WAN interface...
em0: link state changed to DOWN
done.
Creating IPsec VTI instances...done.
Creating OpenVPN instances...done.
Generating /etc/resolv.conf...done.
Configuring firewall........done.
Starting PFLOG...done.
pflog0: promiscuous mode enabled
Configuring OpenSSH...done.
Starting web GUI...done.
Configuring CRON...done.
Setting up routes...
em1: link state changed to UP
done.
Generating /etc/hosts...done.
Starting DHCPv4 service...done.
em2: link state changed to UP
Starting Unbound DNS...
em3: link state changed to UP
em0: link state changed to UP
done.
Setting up gateway monitors...done.
Configuring firewall........done.
Starting PFLOG...
pflog0: promiscuous mode disabled
done.
pflog0: promiscuous mode enabled
lagg0: link state changed to UP
Syncing OpenVPN settings...done.
Starting NTP service...deferred.
Starting Unbound DNS...done.
Generating RRD graphs...done.
Configuring system logging...done.
>>> Invoking start script 'newwanip'
>>> Invoking start script 'freebsd'
Starting suricata.
\^[[32m25/11/2020 -- 11:03:01\^[[0m - <\^[[33mInfo\^[[0m> - Including configuration file installed_rules.yaml.\^[[0m
\^[[32m25/11/2020 -- 11:03:01\^[[0m - <\^[[33mInfo\^[[0m> - Configuration node 'rule-files' redefined.\^[[0m
\^[[32m25/11/2020 -- 11:03:01\^[[0m - <\^[[33mInfo\^[[0m> - Including configuration file custom.yaml.\^[[0m
>>> Invoking start script 'syslog-ng'
Stopping syslog_ng.
Waiting for PIDS: 76975.
Starting syslog_ng.
>>> Invoking start script 'frr'
Checking zebra.conf
2020/11/25 11:03:02 ZEBRA: [EC 4043309110] Disabling MPLS support (no kernel support)
OK
Starting zebra.
2020/11/25 11:03:02 ZEBRA: [EC 4043309110] Disabling MPLS support (no kernel support)
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK
>>> Invoking start script 'beep'
Root file system: /dev/gpt/rootfs
Wed Nov 25 11:03:04 AST 2020

*** psfwl01.lan: OPNsense 20.7.5 (amd64/OpenSSL) ***

 LAN (lagg0)     -> v4: 10.254.0.1/29
 OPT2_OOBM (em3) -> v4: 10.252.0.1/24
 WAN (em0)       -> v4: 192.168.1.2/24

 HTTPS: SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 SSH:   SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (ECDSA)
 SSH:   SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (ED25519)
 SSH:   SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (RSA)
em3: link state changed to DOWN
em0: link state changed to DOWN
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
295.605088 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
296.000171 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
299.663500 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
300.711246 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
301.944716 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
307.152032 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
317.161617 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
319.942421 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
326.090014 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
327.171306 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
329.157861 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
332.230135 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
335.301443 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
337.180975 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
341.444752 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
341.444752 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
344.517496 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
347.190720 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
349.634838 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
353.732195 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
356.125718 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
357.200401 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
359.875731 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
361.055369 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
362.055363 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
363.055326 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
366.019362 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
366.019362 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
367.210064 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
369.091586 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
372.163588 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
375.235404 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
377.219926 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
381.378750 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
386.311801 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
387.229683 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
392.897270 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
397.239161 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
401.451326 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
407.248862 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
417.258537 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
423.265873 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
425.120266 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
427.268229 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
428.055273 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
430.055338 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
431.264016 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
432.184853 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
434.335220 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
437.277935 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
440.478657 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
444.510706 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
447.288637 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
457.176310 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
467.296949 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
477.306677 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
481.444369 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
487.316747 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
496.054340 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
497.054339 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
498.055333 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
499.054362 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
507.325814 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
510.829153 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
511.441303 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
517.336104 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
526.025186 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
527.345187 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
537.355094 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
547.364585 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
551.441303 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
556.083072 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
557.374335 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
562.055312 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
564.055322 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
565.055323 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
567.383876 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
577.394167 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
587.403621 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
591.436833 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
596.110595 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
597.413115 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
607.423465 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
617.442410 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
627.102088 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
631.055359 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
632.055328 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
633.055327 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
634.055313 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
637.452339 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
647.461424 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
648.027429 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
649.044572 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
650.062587 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
651.107827 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
652.126261 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
653.144827 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
654.155589 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
657.471179 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
661.431557 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
667.480892 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
676.141464 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
677.490796 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
683.349199 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
687.500302 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
697.509950 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
698.055279 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
700.055308 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
701.055359 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
707.519618 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
717.529263 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
727.538982 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
737.548714 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
741.423212 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
747.558355 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
757.568210 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
766.055284 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
767.577969 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
768.055289 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
770.055322 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
776.075766 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
777.587462 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
781.419079 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
787.597092 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
797.606852 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
807.616503 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
817.626504 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
821.419447 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
827.635923 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
834.055306 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
836.055317 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
837.055314 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
844.166790 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
847.645304 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
857.654964 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
861.406558 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
867.664716 [4006] netmap_transmit           em0 drop mbuf that needs checksum offload
em0: link state changed to DOWN
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
em0: link state changed to UP
em0: link state changed to DOWN
pflog0: promiscuous mode disabled
pid 98156 (syslog-ng), jid 0, uid 0: exited on signal 6 (core dumped)
em0: link state changed to UP
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 8 8 0 0 done
Waiting (max 60 seconds) for system thread `bufdaemon' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-0' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-1' to stop... done
All buffers synced.
Uptime: 22m34s
---<<BOOT>>---
Copyright (c) 2013-2019 The HardenedBSD Project.
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p10-HBSD #0  6e16e28f1bf(stable/20.7)-dirty: Tue Oct 20 13:30:19 CEST 2020
    root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(efifb): resolution 800x600
HardenedBSD: initialize and check features (__HardenedBSD_version 1200059 __FreeBSD_version 1201000).
CPU: Intel(R) Atom(TM) CPU  E3845  @ 1.91GHz (1916.71-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30679  Family=0x6  Model=0x37  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3960811520 (3777 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
Firmware Warning (ACPI): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20181213/tbfadt-748)
ioapic0 <Version 2.0> irqs 0-86 on motherboard
Launching APs: 2 1 3
Timecounter "TSC" frequency 1916712774 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
kbd0 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff8128e7c0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
000.000054 [4335] netmap_init               netmap: loaded module
[ath_hal] loaded
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 10a11fff vs 10a12000
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xe080-0xe087 mem 0x90000000-0x903fffff,0x80000000-0x8fffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
ahci0: <AHCI SATA controller> port 0xe070-0xe077,0xe060-0xe063,0xe050-0xe057,0xe040-0xe043,0xe020-0xe03f mem 0x90a11000-0x90a117ff irq 19 at device 19.0 on pci0
ahci0: AHCI v1.30 with 2 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
xhci0: <Intel BayTrail USB 3.0 controller> mem 0x90a00000-0x90a0ffff irq 20 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <encrypt/decrypt> at device 26.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection> port 0xd000-0xd01f mem 0x90900000-0x9091ffff,0x90920000-0x90923fff irq 16 at device 0.0 on pci1
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:e0:67:1f:6e:a0
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) PRO/1000 Network Connection> port 0xc000-0xc01f mem 0x90800000-0x9081ffff,0x90820000-0x90823fff irq 17 at device 0.0 on pci2
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using an MSI interrupt
em1: Ethernet address: 00:e0:67:1f:6e:a1
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) PRO/1000 Network Connection> port 0xb000-0xb01f mem 0x90700000-0x9071ffff,0x90720000-0x90723fff irq 18 at device 0.0 on pci3
em2: Using 1024 TX descriptors and 1024 RX descriptors
em2: Using an MSI interrupt
em2: Ethernet address: 00:e0:67:1f:6e:a2
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) PRO/1000 Network Connection> port 0xa000-0xa01f mem 0x90600000-0x9061ffff,0x90620000-0x90623fff irq 19 at device 0.0 on pci4
em3: Using 1024 TX descriptors and 1024 RX descriptors
em3: Using an MSI interrupt
em3: Ethernet address: 00:e0:67:1f:6e:a3
em3: netmap queues/slots: TX 1/1024, RX 1/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Sleep Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
uart0: console (115200,n,8,1)
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ugen0.1: <0x8086 XHCI root HUB> at usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Protectli 64GB mSATA SBFMBB.3> ACS-4 ATA SATA 3.x device
ada0: Serial Number Bxxxxxxxxxxxxxxxxx
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors)
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Mounting filesystems...
tunefs: soft updates remains unchanged as enabled
tunefs: file system reloaded
tunefs: issue TRIM to the disk remains unchanged as enabled
tunefs: file system reloaded
** /dev/gpt/rootfs
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 11754518 free (9558 frags, 1468120 blocks, 0.1% fragmentation)
uhub0: 7 ports with 7 removable, self powered
Setting hostuuid: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Setting hostid: 0xxxxxxxxx.
Configuring vt: blanktime.
ugen0.2: <Microsoft Microsoft Nano Transceiver v2.0> at usbus0
ukbd0 on uhub0
ukbd0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
kbd1 at ukbd0
Configuring crash dump device: /dev/gpt/swapfs
swapon: adding /dev/gpt/swapfs as swap device
.ELF ldconfig path: /lib /usr/lib /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path:
done.
>>> Invoking early script 'update'
>>> Invoking early script 'configd'
Starting configd.
>>> Invoking early script 'templates'
Generating configuration: OK
>>> Invoking early script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
Cannot 'stop' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
Cannot 'start' flowd_aggregate. Set flowd_aggregate_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.
>>> Invoking backup script 'rrd'
>>> Invoking early script 'carp'
CARP event system: OK
Launching the init system...done.
Initializing...........done.
em0: link state changed to UP
em1: link state changed to UP
em2: link state changed to UP
Starting device manager...
ums0 on uhub0
ums0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
ums0: 5 buttons and [XYZT] coordinates ID=26
ums0: 0 buttons and [T] coordinates ID=0
uhid0 on uhub0
uhid0: <Microsoft Microsoft Nano Transceiver v2.0, class 0/0, rev 2.00/9.44, addr 1> on usbus0
done.
Configuring login behaviour...done.
Configuring loopback interface...
lo0: link state changed to UP
done.
Configuring kernel modules...
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
done.
Setting up extended sysctls...done.
Setting timezone...done.
Writing firmware setting...done.
Writing trust files...done.
Setting hostname: psfwl01.lan
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...
em1: link state changed to DOWN
lagg0: IPv6 addresses on em1 have been removed before adding it as a member to prevent IPv6 address scope violation.
lagg0: link state changed to DOWN
em2: link state changed to DOWN
lagg0: IPv6 addresses on em2 have been removed before adding it as a member to prevent IPv6 address scope violation.
done.
Configuring VLAN interfaces...done.
Configuring LAN interface...done.
Configuring OPT2_OOBM interface...done.
Configuring WAN interface...
em0: link state changed to DOWN
done.
Creating IPsec VTI instances...done.
Creating OpenVPN instances...done.
Generating /etc/resolv.conf...done.
Configuring firewall........done.
Starting PFLOG...done.
pflog0: promiscuous mode enabled
Configuring OpenSSH...done.
Starting web GUI...done.
Configuring CRON...done.
Setting up routes...done.
Generating /etc/hosts...done.
em1: link state changed to UP
Starting DHCPv4 service...
em2: link state changed to UP
done.
Starting Unbound DNS...
em0: link state changed to UP
done.
Setting up gateway monitors...done.
Configuring firewall........done.
Starting PFLOG...
pflog0: promiscuous mode disabled
done.
pflog0: promiscuous mode enabled
lagg0: link state changed to UP
Syncing OpenVPN settings...done.
Starting NTP service...deferred.
Starting Unbound DNS...done.
Generating RRD graphs...done.
Configuring system logging...done.
>>> Invoking start script 'newwanip'
>>> Invoking start script 'freebsd'
Starting suricata.
\^[[32m25/11/2020 -- 19:26:01\^[[0m - <\^[[33mInfo\^[[0m> - Including configuration file installed_rules.yaml.\^[[0m
\^[[32m25/11/2020 -- 19:26:01\^[[0m - <\^[[33mInfo\^[[0m> - Configuration node 'rule-files' redefined.\^[[0m
\^[[32m25/11/2020 -- 19:26:01\^[[0m - <\^[[33mInfo\^[[0m> - Including configuration file custom.yaml.\^[[0m
>>> Invoking start script 'syslog-ng'
Stopping syslog_ng.
Waiting for PIDS: 65881.
Starting syslog_ng.
>>> Invoking start script 'frr'
Checking zebra.conf
2020/11/25 19:26:03 ZEBRA: [EC 4043309110] Disabling MPLS support (no kernel support)
OK
Starting zebra.
2020/11/25 19:26:03 ZEBRA: [EC 4043309110] Disabling MPLS support (no kernel support)
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK
>>> Invoking start script 'beep'
Root file system: /dev/gpt/rootfs
Wed Nov 25 19:26:05 +04 2020

*** psfwl01.lan: OPNsense 20.7.5 (amd64/OpenSSL) ***

 LAN (lagg0)     -> v4: 10.254.0.1/29
 OPT2_OOBM (em3) -> v4: 10.252.0.1/24
 WAN (em0)       -> v4: 192.168.1.2/24

 HTTPS: SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
               xxxxxxxxxxxxxxxxxxxxxxxxxx
 SSH:   SHA256 0xxxxxxxxxxxxxxxxxxxxxxxxxxxxx (ECDSA)
 SSH:   SHA256 xxxxxxxxxxxxxxxxxxxxxx (ED25519)
 SSH:   SHA256 xxxxxxxxxxxxxxxxxx (RSA)
em0: link state changed to DOWN
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
em0: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled


Regards

Pedro Serrano

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: mb on November 25, 2020, 04:44:20 pm

Hi @kpiq, are you also on bce driver?

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: kpiq on November 25, 2020, 05:03:40 pm

Apparently not.

root@psfwl01:~ # kldstat
Id Refs Address                Size Name
 1   35                0x0  268d060 kernel
 2    1                0x0    10250 carp.ko
 3    1                0x0     f998 if_bridge.ko
 4    2                0x0     72a8 bridgestp.ko
 5    1                0x0     3e78 if_enc.ko
 6    1                0x0     b1c0 if_gre.ko
 7    1                0x0    16008 if_lagg.ko
 8    1                0x0     8b60 if_tap.ko
 9    3                0x0    582f0 pf.ko
10    1                0x0     2af8 pflog.ko
11    1                0x0     ebd0 pfsync.ko
12    1                0x0     18a0 uhid.ko
13    1                0x0     2928 ums.ko
14    1                0x0     1aa0 wmt.ko
15    1                0x0     8d50 aesni.ko

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: kpiq on November 25, 2020, 05:05:49 pm

Here's my hardware:

root@psfwl01:~ # pciconf -lv
hostb0@pci0:0:0:0:   class=0x060000 card=0x22128086 chip=0x0f008086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series SoC Transaction Register'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0:   class=0x030000 card=0x22128086 chip=0x0f318086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Graphics & Display'
    class      = display
    subclass   = VGA
ahci0@pci0:0:19:0:   class=0x010601 card=0x0f238086 chip=0x0f238086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series SATA AHCI Controller'
    class      = mass storage
    subclass   = SATA
xhci0@pci0:0:20:0:   class=0x0c0330 card=0x0f358086 chip=0x0f358086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx, Celeron N2000 Series USB xHCI'
    class      = serial bus
    subclass   = USB
none0@pci0:0:26:0:   class=0x108000 card=0x0f188086 chip=0x0f188086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine'
    class      = encrypt/decrypt
pcib1@pci0:0:28:0:   class=0x060400 card=0x0f488086 chip=0x0f488086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:28:1:   class=0x060400 card=0x0f4a8086 chip=0x0f4a8086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 2'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:28:2:   class=0x060400 card=0x0f4c8086 chip=0x0f4c8086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 3'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:28:3:   class=0x060400 card=0x0f4e8086 chip=0x0f4e8086 rev=0x11 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series PCI Express Root Port 4'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:   class=0x060100 card=0x0f1c8086 chip=0x0f1c8086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor Z36xxx/Z37xxx Series Power Control Unit'
    class      = bridge
    subclass   = PCI-ISA
none1@pci0:0:31:3:   class=0x0c0500 card=0x0f128086 chip=0x0f128086 rev=0x11 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor E3800 Series SMBus Controller'
    class      = serial bus
    subclass   = SMBus
em0@pci0:1:0:0:   class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em1@pci0:2:0:0:   class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em2@pci0:3:0:0:   class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em3@pci0:4:0:0:   class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: kpiq on November 26, 2020, 08:43:21 pm

I haven't seen this "drop mbuf" situation happen again after disabling all checksum offloading.  But the high memory use, the failing name resolution, and general sluggish performance is still a problem.  I see Suricata using lots of memory.   I configured it to use  IDS with HyperScan.   Here is a top snapshot.

last pid: 97141;  load averages:  0.45,  0.47,  0.59    up 0+00:22:31  23:42:11
41 processes:  1 running, 40 sleeping
CPU:  4.1% user,  0.0% nice,  0.7% system,  0.1% interrupt, 95.1% idle
Mem: 2042M Active, 99M Inact, 702M Wired, 209M Buf, 941M Free
Swap: 10G Total, 10G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
55943 root          7  20    0  4432M  1998M nanslp   0  11:56   1.40% suricata
 6906 root          1  20    0    40M    22M accept   1   0:01   1.27% php-cgi
20515 root          1  52    0    77M    47M accept   1   0:18   0.53% python3.
22285 root          1  20    0    21M    13M kqread   0   0:05   0.11% lighttpd
85104 root          1  20    0  1038M  3988K CPU0     0   0:00   0.07% top
44995 root          2  20    0    26M    10M kqread   3   0:01   0.04% syslog-n
57851 root          1  20    0  1042M  3396K select   2   0:01   0.03% syslogd
77086 root          1  20    0    27M    17M select   0   0:03   0.03% python3.
76484 root          1  20    0  1037M  3256K bpf      0   0:00   0.02% filterlo
 2818 root          1  20    0  1050M  7088K select   3   0:00   0.02% ntpd
73550 root          1  20    0    20M    10M select   2   0:00   0.02% python3.
20354 root          1  20    0    21M    11M select   0   0:00   0.01% python3.
61391 root          1  20    0  1048M  7904K select   1   0:00   0.01% sshd
67620 root          1  20    0    40M    22M accept   3   0:03   0.00% php-cgi
39280 root          1  52    0    40M    22M accept   1   0:03   0.00% php-cgi
79766 root          1  52    0    40M    22M accept   1   0:03   0.00% php-cgi
 4785 root          1  52    0    40M    22M accept   0   0:02   0.00% php-cgi
58740 root          1  52    0    31M    19M wait     1   0:02   0.00% python3.
11160 unbound       4  20    0    74M    33M kqread   0   0:00   0.00% unbound
 4429 root          1  52    0  1037M  3324K wait     3   0:00   0.00% sh
21869 root          1  52    0    40M    22M accept   0   0:00   0.00% php-cgi
79316 root          1  20    0    39M    18M wait     2   0:00   0.00% php-cgi
75327 root          1  52    0    39M    18M wait     0   0:00   0.00% php-cgi
75337 root          1  52    0  1036M  3264K nanslp   3   0:00   0.00% cron
21315 root          1  20    0  1044M  4592K pause    2   0:00   0.00% csh
70892 root          1  20    0    14M  5016K select   2   0:00   0.00% devd
  230 dhcpd         1  20    0    22M    10M select   2   0:00   0.00% dhcpd
86589 root          3  20    0  1061M  8648K select   3   0:00   0.00% zebra
67491 root          1  52    0  1037M  3296K wait     2   0:00   0.00% sh

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: andreaslink on March 31, 2021, 12:22:48 pm

Quote from: mb on October 23, 2020, 11:48:15 pm

Hi @andreas,

We think that this is not related to netmap and has something to do with wireguard's resetting interface flags; but did not find much time to have a look in detail.

I'll post an update once we have some more information.

I just want to bring this topic up again. I doubt you found the time @mb to look further into this, but since this is still existing with the exact same situation as before and in my case triggered via a foreign wireguard ping, I thought I bring it up again and ask again.
So the same problem still existing in 21.1.3_3 so far (did not go to 20.1.4 yet, due to major wireguard implementation changes).

Any news here or plan to progress?

Title: Re: netmap_transmit bce0 drop mbuf that needs checksum offload
Post by: MenschAergereDichNicht on December 09, 2021, 10:35:13 pm

I am not sure that it is related. But i read something about Wireguard and checksum errors inside this thread and just wanted to add that i also have some problems in that area.
I turned off all hardware offload features and still see UDP checksum errors. In my case it is a PPPoE interface on top of a VLAN (for WAN connection).

I am using OPNsense 21.7.6.

I am sure that the problem is specific to Wireguard as other UDP traffic on this interface does not create invalid packets.