Sensei on OPNsense - Application based filtering

[mb]

Thanks for the heads-up. Looking into that.

[mb]

@marci, @mr.x.y, @denvertech, can you try re-installing the package and see if this fixes the problem:

Code: [Select]

# pkg install -f -y os-sensei

[mr.yx]

i have the package reinstalled several times since 1.6 release because i thought something was broken with mailreports, is there any change to the package since then?

Version        : 1.6
Installed on   : Thu Sep 17 13:40:13 2020 CEST

also mellanox connectx3 LAN (no surricata) + vlans are still not working, defaults to emulated netmap driver, non vlan traffic flows, vlan traffic gets blocked/denied.

sys.device.mlx4_core0.hw.fw_version: 2.42.5000 (newest firmware)
dev.mlx4_core.0.%desc: Mellanox driver (3.5.1)

[GreenMatter]

2020-09-09T04:31:07   kernel: 667.875025 [1180] netmap_grab_packets bad pkt at 390 len 0
2020-09-09T04:31:07   kernel: 667.875016 [1180] netmap_grab_packets bad pkt at 389 len 0
2020-09-09T04:31:07   kernel: 667.875008 [1180] netmap_grab_packets bad pkt at 388 len 0
2020-09-09T04:31:07   kernel: 667.875001 [1180] netmap_grab_packets bad pkt at 387 len 0
2020-09-09T04:31:07   kernel: 667.874992 [1180] netmap_grab_packets bad pkt at 386 len 0
2020-09-09T04:31:07   kernel: 667.874306 [ 277] vmxnet3_netmap_rxsync 130 skipped! idx 46
2020-09-09T04:31:07   kernel: vmx1: watchdog timeout on queue 0
2020-09-09T04:31:02   eastpect[8308]: nm1::vmx1^: permanently promiscuous mode enabled
2020-09-09T04:31:02   eastpect[8308]: nm0::vmx1: permanently promiscuous mode enabled
What surprising me is that all has been working fine for months, I had done no changes in setup, no new packages were installed and all of sudden this problem appears. I know it's net map but could it be triggered somehow by Sensei which inspects parent interface vmx1?
Shall I reinstall Sensei, would it help?

@mb just to let you know that above issue must be caused or triggered by Sensei. I can reinstate LAN communication by simply stopping Sensei Packet Engine (fyi OPNsense is still on 20.1.9 and Sensei 1.5.2).
I'm writing this to ask whether this issue has been addressed in new release?

[mb]

i have the package reinstalled several times since 1.6 release because i thought something was broken with mailreports, is there any change to the package since then?

Yep, we've uploaded a new package.

also mellanox connectx3 LAN (no surricata) + vlans are still not working, defaults to emulated netmap driver, non vlan traffic flows, vlan traffic gets blocked/denied.

It's expected that vlan interfaces use emulated driver.

Question: If you do not run anything on the parent mlx interface, are you still experiencing problems with the child interfaces?

[mb]

@mb just to let you know that above issue must be caused or triggered by Sensei. I can reinstate LAN communication by simply stopping Sensei Packet Engine (fyi OPNsense is still on 20.1.9 and Sensei 1.5.2).
I'm writing this to ask whether this issue has been addressed in new release?

20.7.2-netmap kernel looks fine. I've just seen your correspondance with our support team. I guess you'll be waiting for the release kernel

[mb]

Dear Sensei users,

As some of you might have noticed, Sensei now has a dedicated board on the OPNsense forum. This thread has been moved under the new main board: https://forum.opnsense.org/index.php?board=38.0

We'll be following up with all of the discussions here. Feel free to join the discussions.

We'd like to thank OPNsense team for this. It'll help a lot in the sense that new conversations around Sensei will be better organized.

[DenverTech]

@marci, @mr.x.y, @denvertech, can you try re-installing the package and see if this fixes the problem:

Code: [Select]

# pkg install -f -y os-sensei

Looks to be working. Thanks again!

[marcri]

@marci, @mr.x.y, @denvertech, can you try re-installing the package and see if this fixes the problem:

Code: [Select]

# pkg install -f -y os-sensei

that worked for me, thank you!

[GreenMatter]

20.7.2-netmap kernel looks fine. I've just seen your correspondance with our support team. I guess you'll be waiting for the release kernel

Exactly  , I would have tried out test kernel but in such a case I need physical access to router. Will all changes be included in 20.7.3 or rather later updates?
And for now I removed completely Sensei and have it reinstalled. Maybe DB had been corrupted during unsuccessful update to 20.7 and following VM's snapshot restoration?

[mb]

Will all changes be included in 20.7.3 or rather later updates?

It looks like, OPNsense will land them on 20.7.4 at the earliest. I've heard from @franco that there'll be another netmap test kernel based on 20.7.3. So, not for 20.7.3 for sure.

And for now I removed completely Sensei and have it reinstalled. Maybe DB had been corrupted during unsuccessful update to 20.7 and following VM's snapshot restoration?

Yes, this is the reason. Both Mongo and Elastic do lots of buffered I/O for performance reasons. In case of an abrupt shutdown, they have no way of recovering in-memory data which is not yet written to disk.

Sensei -> Configuration -> Reporting & Data -> Reset Reporting will try to recover broken indexes, if not, they'll reset broken indexes.

[GreenMatter]

It looks like, OPNsense will land them on 20.7.4 at the earliest. I've heard from @franco that there'll be another netmap test kernel based on 20.7.3. So, not for 20.7.3 for sure.

Thus it means all users with vmx interfaces must wait at least until 20.7.4 is released?

Both Mongo and Elastic do lots of buffered I/O for performance reasons. In case of an abrupt shutdown, they have no way of recovering in-memory data which is not yet written to disk.
Sensei -> Configuration -> Reporting & Data -> Reset Reporting will try to recover broken indexes, if not, they'll reset broken indexes.

Since I did complete reinstallation, I don't need to reset DB once again?

[mb]

Thus it means all users with vmx interfaces must wait at least until 20.7.4 is released?

Unfortunately, yes; if you do not want to use the beta kernels.

Since I did complete reinstallation, I don't need to reset DB once again?

Yes, if you're fine. No need. If you still have problems, I'd suggest resetting the DB again.

[mb]

A good discussion on why you might consider offloading Elasticsearch reporting:

https://forum.opnsense.org/index.php?topic=19266.msg88593#msg88593

[GreenMatter]

A good discussion on why you might consider offloading Elasticsearch reporting:
https://forum.opnsense.org/index.php?topic=19266.msg88593#msg88593

In some scenarios it makes sense to offload DB, but also would be very convenient if Sensei runs periodical checks of DB and if required, some basic auto repair plus reports any inconsistency...