Call for testing: netmap on 20.7

[EHRETic]

To be a little fair the change log opening sentence seems a bit ambiguous.  I don't think it was intended that way but the first time I read it I thought it was saying there were other changes that weren't noted in the log.

Yep, me this morning: "yeah, finally a fix", snapshot, upgrade... and Meeeeeeh, revert snapshot!

I might have read to fast, my bad!

[Archanfel80]

@mb

Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?

No, read the changelog. That is not fix the netmap issues.

"From the reported issues we still have more logging quirks to investigate and especially Netmap support (used in IPS and Sensei) is lacking in some areas that were previously working. Patches are being worked on already so we shall get there soon enough. Stay tuned."

That could mean anything...

Yes but one thing is certain. The fix is still not ready

[binaryanomaly]

Yes but one thing is certain. The fix is still not ready

Seems not. But then it could also have been - that test kernel runs better than any production one so far here...

Anyway I installed the test kernel again after upgrading to 20.7.1 which runs smooth so far.

[mb]

Hi @heresjody, sure, we'll post a new one based on 20.7.1 this week. -- and hopefully with a final patch for vmx issue.

[mb]

A quick update:

I hear that OPNsense will ship an official netmap test kernel in the coming week.

For the impatient, here is a new test kernel which is based on 20.7.1 stock kernel:

Code: [Select]

[root@20gw /root]# cd /boot/
[root@20gw:/boot # fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0815-2.tar.gz
kernel-12.1-0815-2.tar.gz                           45 MB 4980 kBps    10s
[root@20gw /boot]# mv kernel kernel.stock.save
[root@20gw /boot]# tar zxf kernel-12.1-0815-2.tar.gz
[root@20gw /boot]# reboot

After the reboot, you should be able to see this kernel information:

Code: [Select]

root@20gw:~ # uname -a
FreeBSD 20gw.local 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #3  87f253a0d(master)-dirty: Sat Aug 15 09:29:08 PDT 2020     root@bsd12_openssl:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64
root@20gw:~ #
To restore stock OPNsense kernel:

Code: [Select]

# cd /boot
# rm -rf kernel
# mv kernel.stock.save kernel
# reboot

[bunchofreeds]

I have run up the following kernel on my Proxmox vtnet pppoe setup

FreeBSD ... 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #3  87f253a0d(master)-dirty: Sat Aug 15 09:29:08 PDT 2020     root@bsd12_openssl:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64

I was getting hi CPU usage from flowd_aggregate which settled down after about 5 mins

Code: [Select]

39226 root 101 0 38M 29M CPU0 0 0:56 98.97% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.7)
11 root 155 ki31 0 32K RUN 0 8:24 50.98% [idle{idle: cpu0}]
syslog_ng starts then stops (I manually started this up again although I do not do remote logging so not sure if I need this for any local OPNsense logging?)

Sensei is running successfully on vtnet LAN interface
Suricata enables on vtnet/PPPoE but does not work

Looking forward to testing out some PPPoE netmap updates

[scream]

I hear that OPNsense will ship an official netmap test kernel in the coming week.

Is this test kernel already available over opnsense update feature?
Is the vmx bug fixed and sensei working with this test kernel?

[mb]

Good morning. I have some good news

Finally we have a test kernel which addresses a range of netmap/iflib issues on OPNsense 20.7.x/BSD 12:

• em+vlan stall
• vtnet crash
• vmx crash
• vpn - tun interface support : We've confirmed that this kernel also works for Sensei on vpn - tun(4) interfaces.
• some other minor problems

Here are the steps to give it a try; and  please do test and provide feedback. Sooner we provide some feedback to OPNsense, sooner they can make this kernel available for general use.

Code: [Select]

[root@20gw /root]# cd /boot/
[root@20gw:/boot # fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0822-1.tar.gz
kernel-12.1-0822-1.tar.gz                           45 MB 4980 kBps    10s
[root@20gw /boot]# mv kernel kernel.stock.save
[root@20gw /boot]# tar zxf kernel-12.1-0822-1.tar.gz
[root@20gw /boot]# reboot
After the reboot, you should be able to see this kernel information:

Code: [Select]

root@20gw:~ # uname -a
FreeBSD 20gw.local 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6  39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020     root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64
root@20gw:~ #
To restore stock OPNsense kernel:

Code: [Select]

# cd /boot
# rm -rf kernel
# mv kernel.stock.save kernel
# reboot

[bunchofreeds]

Firstly, thanks for the awesome effort so far in getting these netmap issues sorted out.

Confirmed running
FreeBSD 20gw.local 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6  39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020     root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64

Unfortunately I am having trouble making my PPPoE interface available for Sensei.
I have gone as far as adding an additional vtnet interface to my Proxmox OPNsense guest at OPT1. This was to ensure I could move Suricata to an unused interface (Suricata is disabled).

My only available interfaces in Sensei are:

LAN vtnet0 - Currently running on this interface
(Unassigned) vtnet1 - this is the interface that PPPoE resides on
OPT1 (vtnet2) - The new interface I added and moved Suricata to. To be sure it was not conflicting.

Should I be expecting to see PPPoE as an available interface?

[Quetschwalze]

Tested with igb interfaces and pppoe on wan (removed VLAN for testing)
Suricata seems to start fine:

Code: [Select]

2020-08-23T09:02:28 suricata[76340] [100106] <Notice> -- all 2 packet processing threads, 4 management threads initialized, engine started.
2020-08-23T09:02:28 suricata[76340] [100805] <Notice> -- opened netmap:pppoe1/T from pppoe1: 0x172871fd300
2020-08-23T09:02:28 suricata[76340] [100805] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x172871fd000
2020-08-23T09:02:28 suricata[76340] [100798] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x17236be4300
2020-08-23T09:02:28 suricata[76340] [100798] <Notice> -- opened netmap:pppoe1/R from pppoe1: 0x17236be4000

However, it doesn't alert or block on anything.
Then I tried Sensei on the WAN Interface. It starts, but afterwards Internet is gone.
Reports do not show any sessions or blocks.

[scream]

@mb

I installed like described in your post yesterday.
Looks like it works for my vmx interfaces on opnsense vm.

But I see some issues:
- can’t see any interface on „interface list“ to select.
- can‘t see tun interfaces

My vm only having vmx and tun interfaces.

Do I need to do some additional steps?

Edit:

Is there a way to show unsupported interfaces in sensei configuration? I think I do not see any interface as I only have vmx0-7 and tun0. So there is just no supported interface.
To test vmx and the tun I need to reconfigure the used interfaces.
At the moment (old config before upgrading) sensei was active on 7 of 8 vmx interfaces and is now running fine since around 2h. Filter/Blocking is working as expected.

Edit2:
Found this here... after "commenting" vmx and ovpns1 I can now see the interfaces

https://forum.opnsense.org/index.php?topic=9521.msg84199#msg84199

If I add ovpns1 to "protected interfaces" Sensei is creashing. So no luck with OpenVPN tun interface.

Additional info: All interfaces do not use vlan. VLAN tagging is done on ESXi level (dvSwitch).

[mb]

Should I be expecting to see PPPoE as an available interface?

Hi @bunchofreeds, thanks and all wellcome.

PPPoE interfaces are filtered on 1.5.2 release, and Sensei is meant to be running in inner interfaces. This is why you can't see them. 

Any chances that you can create a small pcap trace (e.g. tcpdump -s0 -n -i pppoe0 -c 100 -w pppoe.pcap) and PM it to me?

Suricata should be ok working with PPPoE now. I would like to see if anything is different from vpn interfaces.

[mb]

Tested with igb interfaces and pppoe on wan (removed VLAN for testing)
Suricata seems to start fine:

..
..

However, it doesn't alert or block on anything.
Then I tried Sensei on the WAN Interface. It starts, but afterwards Internet is gone.
Reports do not show any sessions or blocks.

Hi @Quetschwalze, thanks. Any chances that you can also send a pcap trace? - Sensei is not meant for WAN right now.

[mb]

At the moment (old config before upgrading) sensei was active on 7 of 8 vmx interfaces and is now running fine since around 2h. Filter/Blocking is working as expected.

Edit2:
Found this here... after "commenting" vmx and ovpns1 I can now see the interfaces

https://forum.opnsense.org/index.php?topic=9521.msg84199#msg84199

If I add ovpns1 to "protected interfaces" Sensei is creashing. So no luck with OpenVPN tun interface.

Hi @scream, thanks for confirming vmx. Glad to hear that it's working.

For tun, I need to provide a 1.6 beta to you since Sensei needs to tweak interface initialization parameters for tun(4) interfaces. It's not done on 1.5.

Stay tuned, I'll provide the 1.6 txz link today.

[scream]

For tun, I need to provide a 1.6 beta to you since Sensei needs to tweak interface initialization parameters for tun(4) interfaces. It's not done on 1.5.

Stay tuned, I'll provide the 1.6 txz link today.

Thanks a lot! I‘m looking forward to test tun :-) Let me know how I can install the beta.
vmx works like a charm on my opnsense vm since upgrade to this kernel.

The only thing I want to mention is that CPU usage is higher than before.

20.1 with sensei used around 15-20% of each of the two vCPU.
20.7 with sensei is now using around 25-27% of each of the two vCPU.