Archive > 20.7 Legacy Series
netmap_transmit bce0 drop mbuf that needs checksum offload
andreaslink:
I'm running OPNsense (20.7.2-amd64) with one Broadcom NetXtreme II BCM5709 for WAN (bce0) and one for LAN (bce1), further on I have 4x Intel 82580, which I use for other LANs like IoT (igb1) and Guests (igb0) etc.
I have "some" traffic on WAN with quite constantly 60 to 100MBit (mainly due to IP cam streams), which I consider as handeable with my setup. I also have IDS/IPS up and running as well as Sensei.
After "a while" (usually only minutes after reboot) of traffic I get the following error in the log, multiple times per second:
2020-09-10T00:28:10 kernel 490.690419 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:05 kernel 485.572543 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:00 kernel 480.194945 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:28:00 kernel 479.940436 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:54 kernel 474.761838 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:49 kernel 469.475112 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:44 kernel 464.324372 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:39 kernel 459.205033 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:33 kernel 453.830080 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:28 kernel 448.126626 [4006] netmap_transmit bce0 drop mbuf that needs checksum offload
2020-09-10T00:27:23 kernel 443.431391 [ 320] generic_netmap_register Emulated adapter for bce0 activated
2020-09-10T00:27:23 kernel 443.431259 [1130] generic_netmap_attach Emulated adapter for bce0 created (prev was NULL)
2020-09-10T00:27:23 kernel bce0: permanently promiscuous mode enabled
2020-09-10T00:27:23 kernel 443.407436 [1035] generic_netmap_dtor Emulated netmap adapter for bce0 destroyed
2020-09-10T00:27:23 kernel 443.407409 [1130] generic_netmap_attach Emulated adapter for bce0 created (prev was NULL)
As you can see on the attached screenshot, the MBUF usage is at 0% and with ~9720 way below the limit of 1.271.626, so there should be plenty of MBUF available.
So what triggers this error?
I can get rid of it, when deactivating IDS/IPS, and since I'm testing it, the error did not show up again. So is it somehow IPS throughput related? Nonetheless, I would like to turn IDS/IPS on again :).
How can I tune my system, so the "netmap_transmit" can handle the load? (BTW: What process/step ist it, what does it do here?)
And whay does the mbuf "need checksum offload"? What does that exactly mean?
Some more config details:
I have all three hooks set, so all of these three are disabled:
- Hardware CRC
- Hardware TSO
- Hardware LRO
root@OPNsense:~ # sysctl -a | grep nmbclusters
kern.ipc.nmbclusters: 1271626
root@OPNsense:~ # sysctl -a | grep msi
hw.sdhci.enable_msi: 1
hw.puc.msi_disable: 0
hw.pci.honor_msi_blacklist: 1
hw.pci.msix_rewrite_table: 0
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1
hw.mfi.msi: 1
hw.malo.pci.msi_disable: 0
hw.ix.enable_msix: 1
hw.bce.msi_enable: 1
hw.aac.enable_msi: 1
machdep.disable_msix_migration: 0
machdep.num_msi_irqs: 512
dev.igb.3.iflib.disable_msix: 0
dev.igb.2.iflib.disable_msix: 0
dev.igb.1.iflib.disable_msix: 0
dev.igb.0.iflib.disable_msix: 0
BTW: I also experimented with following values, which did not bring any change:
kern.ipc.nmbclusters="2543660"
hw.bce.tso_enable="0"
hw.pci.enable_msix="0"
mb:
Hi @andreaslink, do you have offloadings and vlan hardware filtering set to disabled? See Interfaces -> Settings
If so, please try the official netmap test kernel which will be announced today
opnsense-update -kr 20.7.2-netmap
andreaslink:
Awesome @mb, thank you! I have done that and rebooted:
root@OPNsense:~ # opnsense-update -kr 20.7.2-netmap
Fetching kernel-20.7.2-netmap-amd64.txz: ....... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-20.7.2-netmap-amd64.txz... done
Please reboot.
I've also activated IDS/IPS again to monitor it now. 5 mins later no problems yet, so still monitoring.
I keep you posted!
PS: And as requested, all offloadings and vlan hardware filtering were already set to disabled.
andreaslink:
Just to return some feedback here, I'm testing now for 24h under "full load" incl. IDS/IPS and Sensei and the messages did not appear anymore. So I consider this issue as solved with the new kernel "kernel-20.7.2-netmap-amd64.txz"!
Thank you very much :)!
PS: I assume, my preloading to test before next official update is not an issue for the upcoming release aka official update or will I get in troube with this kernel now?
mb:
Hi Andreas,
That's great to hear. All welcome and thanks for the update.
No, you're fine. 20.7.3 will just install it's own kernel.
Navigation
[0] Message Index
[#] Next page
Go to full version