Call for testing: netmap on 20.7

[madj42]

Just reporting that I've installed the last test kernel and enabled IPS on my PPPoE connection with IGB drivers.  All appears to be good and haven't had issues over the weekend.

[mb]

@madj42, that's great news!. Thanks for letting us know.

[binaryanomaly]

@binaryanomaly, thanks for the feedback. vtnet seems to be doing even better than 20.1.

I can definitely confirm that, had lots of page faults already on 20.1. Running smooth and steady since 3 days - great job, well done!

[bunchofreeds]

Running Proxmox with vtnet drivers on WAN (PPPoE) and LAN

Kernel
12.1-RELEASE-p7-HBSD FreeBSD 12.1-RELEASE-p7-HBSD #3  5742b25c4(master)-dirty: Thu Aug  6 16:17:42 PDT 2020     root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64

Sensei is running successfully on LAN

I'm unsure if IPS is correctly running on WAN (PPPoE)
Have enabled and set IPS and Promiscuous (Have not selected Syslog or Eve)

Log Output via GUI is:

2020-08-11T08:25:46   suricata[15476]: [100117] <Notice> -- all 2 packet processing threads, 4 management threads initialized, engine started.
2020-08-11T08:25:46   suricata[15476]: [101288] <Notice> -- opened netmap:pppoe1/T from pppoe1: 0x51b7a149300
2020-08-11T08:25:46   suricata[15476]: [101288] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x51b7a149000
2020-08-11T08:25:46   suricata[15476]: [100335] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x51b78319300
2020-08-11T08:25:46   suricata[15476]: [100335] <Notice> -- opened netmap:pppoe1/R from pppoe1: 0x51b78319000
2020-08-11T08:24:50   suricata[15476]: [100117] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'is_proto_irc' is checked but not set. Checked in 2002029 and 1 other sigs
2020-08-11T08:24:50   suricata[15476]: [100117] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.gadu.loggedin' is checked but not set. Checked in 2807836 and 0 other sigs
2020-08-11T08:24:50   suricata[15476]: [100117] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.eduphish' is checked but not set. Checked in 2025114 and 0 other sigs
2020-08-11T08:24:30   suricata[89429]: [100183] <Notice> -- This is Suricata version 5.0.3 RELEASE running in SYSTEM mode

However it does not appear to block/drop P2P traffic as a test.
Downloading ubuntu via bit torrent
Ruleset set to drop ET telemetry/emerging-p2p

My Log seems to be broken into two parts. One being the original style logs as shown above. These always appear at the top of the log view.
The second a more like 'stats' that cycles continuously every few seconds. There are LOTS of these being generated.

   flow.memuse | Total | 7154304
    tcp.reassembly_memuse | Total | 196608
    tcp.memuse | Total | 1146880
    flow_mgr.rows_skipped | Total | 65536
    flow_mgr.rows_checked | Total | 65536
    flow.spare | Total | 10000
    ------------------------------------------------------------------------------------
    Counter | TM Name | Value
    ------------------------------------------------------------------------------------
    Date: 8/11/2020 -- 08:29:14 (uptime: 0d, 00h 04m 43s)
    ------------------------------------------------------------------------------------

I do not think that Suricata is operational on my WAN (PPPoE) interface, although it has enabled without errors.

Any ideas how to test this further?

[mb]

I do not think that Suricata is operational on my WAN (PPPoE) interface, although it has enabled without errors.

Hmm, tun(4) implementation is adding an ethernet header (since tun does not have); and mac addresses are all zeros. That might be creating a problem.

Let us have a look deeper; and I'll post updates.

[lewald]

@mb

@lewald, that's great to hear, though I wouldn't expect netmap work might have contributed to the vpn speed. It could be virtio that you can use it with 8 queues.

Maybee. But without the Testkernel i have a lot dequeues on netmap. Now this deueues are gone.

[binaryanomaly]

@mb

Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?

[Archanfel80]

@mb

Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?

No, read the changelog. That is not fix the netmap issues.

[binaryanomaly]

@mb

Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?

No, read the changelog. That is not fix the netmap issues.

"From the reported issues we still have more logging quirks to investigate and especially Netmap support (used in IPS and Sensei) is lacking in some areas that were previously working. Patches are being worked on already so we shall get there soon enough. Stay tuned."

That could mean anything...

[FullyBorked]

@mb

Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?

No, read the changelog. That is not fix the netmap issues.

To be a little fair the change log opening sentence seems a bit ambiguous.  I don't think it was intended that way but the first time I read it I thought it was saying there were other changes that weren't noted in the log.

Small update here with security advisories, multicast fixes and logging reliability patches amongst others.

Also I think he is asking if it will revert his changes, not if it has been fixed in the release. 

[franco]

"amongst others" references the full change log below. It's intentionally ambiguous in the sense that the actual changes are listed below. If you don't see your issue there it's probably just that.

The second paragraph is more loose in terms of content from release to release. It is meant to hint at past and future events. In this case it unambiguously states that Sensei and IPS issues are not yet resolved in the release.

I'm not sure how to make this any clearer other than: don't panic and use 20.1 if you must.


Cheers,
Franco

[FullyBorked]

"amongst others" references the full change log below. It's intentionally ambiguous in the sense that the actual changes are listed below. If you don't see your issue there it's probably just that.

The second paragraph is more loose in terms of content from release to release. It is meant to hint at past and future events. In this case it unambiguously states that Sensei and IPS issues are not yet resolved in the release.

I'm not sure how to make this any clearer other than: don't panic and use 20.1 if you must.


Cheers,
Franco

No panic here   I just read it wrong, initially.  Thought maybe someone else did too.  No need to rewrite anything at all.  I'm patiently waiting on 20.7 and figure it'll work when it works and it will be great as usual. 

[franco]

Thanks, next week will have progress on this front for sure. If we can confirm quickly we may even start adding patches to 20.7.2 already. Just want to keep the changes small and impact big... in the positive sense of course.


Cheers,
Franco

[binaryanomaly]

I'm not sure how to make this any clearer other than: don't panic and use 20.1 if you must.

vtnet page faults on anything besides the test kernel from mb here. So I'll stay with the test kernel until the fix finds its way upstream

[heresjody]

@sorano, it does not seem to be related.

Please follow below steps and see if this kernel is of help:

Code: [Select]

[root@20gw /root]# cd /boot/
[root@20gw:/boot # fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0805-2.tar.gz
kernel-12.1-0805-2.tar.gz                           45 MB 4980 kBps    10s
[root@20gw /boot]# mv kernel kernel.stock.save
[root@20gw /boot]# tar zxf kernel-12.1-0805-2.tar.gz 
[root@20gw /boot]# reboot
After the reboot, you should be able to see this kernel information:

Code: [Select]

root@20gw:~ # uname -a
FreeBSD 20gw.local 12.1-RELEASE-p7-HBSD FreeBSD 12.1-RELEASE-p7-HBSD #2  5742b25c4(master)-dirty: Wed Aug  5 22:20:24 PDT 2020     root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64
root@20gw:~ #

To restore stock OPNsense kernel:

Code: [Select]

# cd /boot
# rm -rf kernel
# mv kernel.stock.save kernel
# reboot

Is it possible to post a test kernel based on the 20.7.1 kernel? Then I can test Surricata on PPPoE WAN without my girlfriend complaining about not being able to watch IPTV