OPNsense Forum
Archive => 20.7 Legacy Series => Topic started by: mb on May 23, 2020, 02:32:10 am
-
Dear OPNsense community,
As promised, we've[1] kicked off another project which focuses on killing remaining netmap bugs on HardenedBSD 12 (FreeBSD 12).
Any help in testing the upcoming OPNsense 20.7 with Suricata (IPS mode) and/or Sensei and providing bug reports would be much appreciated.
We'll get them prioritized, fixed, and committed to the upstream Operating System as soon as possible.
We hope to help provide a release quality netmap implementation for the upcoming OPNsense 20.7 release.
Make sure you update to the latest 20.7 beta after the ISO installation, since latest 20.7 includes some important patches with regard to interface drivers. Kernel should read 12.1-RELEASE-p5 or later:
12.1-RELEASE-p5-HBSD FreeBSD 12.1-RELEASE-p5-HBSD #0 d8b850736ba(master)-dirty
[1] Sunny Valley Networks
-
You mean taking latest beta image, install on hardware and run IPS mode is sufficient?
-
How could we join this testing?
May i just use the beta update from the GUI (20.1.7 with Sensei here)?
-
Hmms, I can only use my sensei license on one device that is somehow unfortunate... :-/
Edit:
20.7 is still 11.2 isn't it?
OPNsense 20.7.b_157-amd64
FreeBSD 11.2-RELEASE-p20-HBSD
OpenSSL 1.1.1g 21 Apr 2020
-
The development branch doesnt upgrade OS yet. You have to install via ISO or IMG
-
The development branch doesnt upgrade OS yet. You have to install via ISO or IMG
Ah bummer, thx.
Edit:
Where can I find a development ISO of OPNsense with FreeBSD 12. Seems not that easy to find?
-
20.7 beta ISO:
https://pkg.opnsense.org/FreeBSD:12:amd64/snapshots/
-
You mean taking latest beta image, install on hardware and run IPS mode is sufficient?
Yes, install 20.7 beta ISO, and run Suricata in IPS mode (IDS won't help since it uses pcap interface) and/or Sensei
-
@mb
Thx for the link to the isos.
Are there any sensei dev licenses available? Can't re-use my prod one... and in the end I will route prod traffic over the test-instance as well so I may want to have the full capabilities.
-
Hi @binaryanomaly,
You're all welcome. Please reach out to us via Sensei UI -> Contact Team. Let's see what we can do about this.
-
Cool, will do, thanks.
-
Stuck at installing it...
Can't get past "Firmware status check was aborted internally. Please try again." after adding os-sunnyvalley-devel.
-
Hmm, that could be something we need to discuss with OPNsense team.
You can still install Sensei from the command line
pkg install os-sensei
-
Unfortunately not.
It doesn't find the prod packages:
root@OPNsense-DEV:~ # pkg install os-sensei
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'os-sensei' have been found in the repositories
root@OPNsense-DEV:~ # pkg install os-sunnyvalley
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'os-sunnyvalley' have been found in the repositories
And can't install the dev packages:
root@OPNsense-DEV:~ # pkg install os-sunnyvalley-devel
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
os-sunnyvalley-devel: 1.2
Number of packages to be installed: 1
Proceed with this action? [y/N]: y
[1/1] Installing os-sunnyvalley-devel-1.2...
[1/1] Extracting os-sunnyvalley-devel-1.2: 100%
root@OPNsense-DEV:~ # pkg update
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
pkg: https://updates.sunnyvalley.io/opnsense/FreeBSD:12:amd64/20.7/OpenSSL/latest/meta.txz: Not Found
repository SunnyValley has no meta file, using default settings
pkg: https://updates.sunnyvalley.io/opnsense/FreeBSD:12:amd64/20.7/OpenSSL/latest/packagesite.txz: Not Found
Unable to update repository SunnyValley
Error updating repositories!
-
By design 20.7-BETA does not include anything but development packages.
Cheers,
Franco
-
By design 20.7-BETA does not include anything but development packages.
Ok so this means that testing sensei with 20.7 will only be possible once sunnyvalley provides the respective working development package for sensei?
-
Can you try once more?
pkg install os-sensei
-
Not working but no surprise because according to Franco:
By design 20.7-BETA does not include anything but development packages.
root@OPNsense-DEV:~ # pkg install os-sensei
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'os-sensei' have been found in the repositories
root@OPNsense-DEV:~ # pkg install os-sensei-devel
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'os-sensei-devel' have been found in the repositories
root@OPNsense-DEV:~ #
root@OPNsense-DEV:~ # pkg install os-sunnyvalley
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'os-sunnyvalley' have been found in the repositories
That explains why the production release can't be installed.
Further installing the Development packages fails:
root@OPNsense-DEV:~ # pkg install os-sunnyvalley-devel
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
os-sunnyvalley-devel: 1.2
Number of packages to be installed: 1
Proceed with this action? [y/N]: y
[1/1] Installing os-sunnyvalley-devel-1.2...
[1/1] Extracting os-sunnyvalley-devel-1.2: 100%
root@OPNsense-DEV:~ # pkg update
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
Fetching meta.txz: 100% 948 B 1.0kB/s 00:01
Fetching packagesite.txz: 100% 12 KiB 12.0kB/s 00:01
Processing entries: 0%
pkg: wrong architecture: FreeBSD:11:* instead of FreeBSD:12:amd64
pkg: repository SunnyValley contains packages with wrong ABI: FreeBSD:11:*
Processing entries: 100%
Unable to update repository SunnyValley
Error updating repositories!
Guess you need to fix this first
pkg: wrong architecture: FreeBSD:11:* instead of FreeBSD:12:amd64
pkg: repository SunnyValley contains packages with wrong ABI: FreeBSD:11:*
-
Got it, sorry about that. It looks like out team tried the 20.7 through upgrading via UI.
Give us a few days and we'll ship 12.x packages.
In the meantime, you can also test Suricata if you like.
-
No problem. Keep us posted.
I'll start re-routing some client traffic once I can replicate the prod setup and achieve the same level of control and protection - so I'll wait for the update.
-
The development branch doesnt upgrade OS yet. You have to install via ISO or IMG
Did i get this correct? 20.7 in devel branch via GUI is not yet the same as the downloadable iso?
As i have a pc-engines apu4, i disabled suricata, i just got on 75mbps. Just tried it now (on 20.7 devel branch, not the iso) and got up to 140 mbps. CPU-Load is about 50% whilst 2 gb download
btw. using aho-corasick "Ken Steele variant"
Thx!
Ruggerio
-
20.7 development exists on HBSD 11.2 via 20.1 and also on HBSD 12.1 via 20.7-BETA.
The difference is the underlying operating system depending on what image you installed indeed. It gives us the extra opportunity to test core changes independently from the OS update. :)
Cheers,
Franco
-
20.7 development exists on HBSD 11.2 via 20.1 and also on HBSD 12.1 via 20.7-BETA.
The difference is the underlying operating system depending on what image you installed indeed. It gives us the extra opportunity to test core changes independently from the OS update. :)
Cheers,
Franco
Ahaaaaa :) this means, i just profited from suricata 5 and not from the mentionned netmap-bufixing, isn't it? the new netmap-thingy is just in the iso, not in the devel-branch from 20.1.
Thx!
-
Yep. :)
-
Hi,
OK, changing firewall back zu ISO, updating to Beta .108 - but not finding "Ken Steele variant" in Suricata, as it is in 1.7 Devel-Branch.
-
Please, slow down. We have not published a new 20.7-BETA snapshot yet with the latest packages...
-
;) thx for information.
-
Sensei packages for OPNsense 20.7 (amd64/OpenSSL) is out and available for testing.
If you test OPNsense 20.7, as a bonus, you get to access to the latest Sensei (1.5.1.rc1) which is yet to be released ;)
PS: Make sure you update to the latest 20.7 beta after the ISO installation, since latest 20.7 includes some important patches with regard to interface drivers. Kernel should read 12.1-RELEASE-p5 or later:
12.1-RELEASE-p5-HBSD FreeBSD 12.1-RELEASE-p5-HBSD #0 d8b850736ba(master)-dirty
-
Works now 👍🏻
-
Perfect.
This is the first one we'll be fixing:
1. If you have VLANs on em(4), you cannot have em(4) parent interface for Suricata/Sensei. https://github.com/luigirizzo/netmap/issues/703.
Anyone out there experiencing the same problem with other drivers (igb, ix, ixl, and.... vtnet?)
-
Im not able to update repoL
root@SRV190520:~ # pkg install os-sensei
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating SunnyValley repository catalogue...
pkg: https://updates.sunnyvalley.io/opnsense/FreeBSD:12:amd64/OpenSSL/repo/meta.txz: Not Found
repository SunnyValley has no meta file, using default settings
pkg: https://updates.sunnyvalley.io/opnsense/FreeBSD:12:amd64/OpenSSL/repo/packagesite.txz: Not Found
Unable to update repository SunnyValley
Error updating repositories!
-
Hi @actionhenkt,
I guess you're installing from getsensei script.
Try this:
rm -f /usr/local/etc/pkg/repos/SunnyValley.conf
pkg install os-sunnyvalley-devel
pkg install os-sensei
In the meantime, we'll update getsensei script to handle 20.7
-
Not sure if this would be the right place to post, but netmap is reporting this for my virtio interfaces:
vtnet0: <VirtIO Networking Adapter> on virtio_pci1
vtnet0: Ethernet address: 00:00:00:00:00:00
vtnet0: netmap queues/slots: TX 4/256, RX 4/128
000.001082 [ 503] vtnet_netmap_attach vtnet attached txq=4, txd=256 rxq=4, rxd=128
txd and rxd aren't supposed to be the same? This is an OPNsense VM running in KVM (Proxmox). Ring parameters for the interface in the host are set to 2048:
Ring parameters for ens4f2:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 2048
RX Mini: 0
RX Jumbo: 0
TX: 2048
And multiqueue is set to 4, which OPNsense correctly applies, but I read in the netmap github that both rxd and txd values must be same to avoid issues. Any thoughts?
-
Hi @robsewca,
It not a big deal. But performance wise, it is advisable that they both have the same number of descriptors.
I do not expect that you see any side effects since normal Internet use is not symmetrical (i.e. download/upload usage is not identical).
But if you do have a fairly symmetrical Internet usage, you'll be limited to the lower descriptor number.
-
Thank you @mb,
Any reason why Netmap may be applying two different number of descriptors? Is this something that depends on the hypervisor or OPNsense to decide?
The reason why I'm trying to troubleshoot this is that my internet connection is symmetrical gigabit and while my upload speeds reach 940Mbps, my download speeds are capped at 700-900Mbps, something definitely seems wrong. All hardware offload settings/features are disabled for both OPNsense and the host, but I can't find a way to force higher txd/rxd parameters.
-
Hi @robsewca
Actually netmap does not mangle with interface rx/tx descriptors; instead it inherits the values from the OS.
As you suspected lower download speeds are related to lower RX count.
I wouldn't be surprised if hypervisor would automatically adjust guest vtnet rx descriptors if you also increased the host interface RX descriptor sizes (i.e. set both RX/TX to 4096).
-
Hi,
I need PPPoE on my WAN which has known issue's for running IPS. I believe this is due to FreeBSD and its NetMap support of virtual interfaces like PPPoE.
Also my OPNsense is virtual on Proxmox using VirtIO network interfaces.
With a move to HardenedBSD 12.1 and Suricata 5, can I expect to see some change now, or could me testing this help to get this resolved?
Thanks
An earlier post of mine under IPS https://forum.opnsense.org/index.php?topic=16462.0
I have installed a virtual instance of 20.7 although it is not terminating PPPoE currently. I can switch this over if the above testing scope is possible. Install was really easy by the way and has updated and performing quite well. IPS enabled and operational on WAN (although not PPPoE)
-
Hi @bunchofreeds,
PPPoE runs over Ethernet. So I see no reason why it should not work provided that Ethernet interface used for the PPPoE connection has no issues with netmap.
Let's have a look at this. Can you do a PPPoE configuration on the latest OPNsense 20.7 beta (kernel 12.1-p5), and try Suricata on it.
Please let me know if this works or not. If it does not work, provide an ifconfig -a output (you can PM me).
-
Ok,
Below file keeps the last status for the Ethernet Drivers <-> netmap compatibility.
https://docs.google.com/spreadsheets/d/1RVj8K3XOzWi-Bkjq6hUxWudu7Cxd8FFTqjLiBMzZWEM/edit#gid=0
This page also explains how you can easily test OPNsense 20.7 netmap.
Feel free to grab a driver, test and provide test results. You should be able to leave comments on the Google Sheets file.
It looks like there's some work to do. We're here to fix.
Just test and provide feedback.
-
Thanks for looking into this @mb
I have put the 20.7 proxmox virtual instance of OPNsense inline with PPPoE by backing up my 20.1 config and restoring to 20.7. Really easy to do and was up and running quickly.
Suricata runs successfully and alerts on the LAN interface (Virtio)
Switched to run on the WAN interface and am not receiving alerts. The new log view with v5 Suricata is different and seems to cycle with this when trying to establish IPS on the PPPoE WAN
Counter | TM Name | Value
------------------------------------------------------------------------------------
Date: 6/14/2020 -- 08:49:09 (uptime: 0d, 00h 08m 17s)
------------------------------------------------------------------------------------
flow.memuse | Total | 7154304
tcp.reassembly_memuse | Total | 196608
tcp.memuse | Total | 1146880
flow_mgr.rows_skipped | Total | 65536
flow_mgr.rows_checked | Total | 65536
flow.spare | Total | 10000
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
Date: 6/14/2020 -- 08:49:01 (uptime: 0d, 00h 08m 09s)
------------------------------------------------------------------------------------
flow.memuse | Total | 7154304
tcp.reassembly_memuse | Total | 196608
tcp.memuse | Total | 1146880
flow_mgr.rows_skipped | Total | 65536
flow_mgr.rows_checked | Total | 65536
flow.spare | Total | 10000
------------------------------------------------------------------------------------
I am getting the ifconfig -a to you via PM
Here is the log when it successfully runs on the LAN interface
flow.memuse | Total | 7177096
tcp.reassembly_memuse | Total | 231424
tcp.memuse | Total | 1146880
flow_mgr.rows_maxlen | Total | 1
flow_mgr.rows_skipped | Total | 65534
flow_mgr.rows_checked | Total | 65536
flow_mgr.flows_notimeout | Total | 2
flow_mgr.flows_checked | Total | 2
flow.spare | Total | 10000
flow_mgr.new_pruned | Total | 9
app_layer.flow.failed_udp | Total | 40
app_layer.tx.dns_udp | Total | 20
app_layer.flow.dns_udp | Total | 6
app_layer.flow.failed_tcp | Total | 4
app_layer.tx.dhcp | Total | 2
app_layer.flow.dhcp | Total | 1
app_layer.flow.tls | Total | 3
tcp.overlap | Total | 1
tcp.rst | Total | 16
tcp.synack | Total | 9
tcp.syn | Total | 9
tcp.sessions | Total | 9
flow.udp | Total | 47
flow.tcp | Total | 39
decoder.max_pkt_size | Total | 1514
decoder.avg_pkt_size | Total | 474
decoder.udp | Total | 320
decoder.tcp | Total | 988
decoder.ethernet | Total | 1594
decoder.ipv6 | Total | 2
decoder.ipv4 | Total | 1306
decoder.bytes | Total | 756423
decoder.pkts | Total | 1594
capture.kernel_packets | Total | 1594
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
Date: 6/14/2020 -- 09:10:57 (uptime: 0d, 00h 01m 52s)
-----------------------------------------------------------------------------------
-
Hi @bunchofreeds - thanks for the ifconfig output.
I must have guessed it: pppoe is a tun(4) interface. It does not work with netmap(4).
This was the bad news.
Good news is, tun(4) support along with bridge(4) and lagg(4) netmap support in our current scope. Having fixed bugs, this is the first item we'll implement.
-
@mb thanks and good luck with the bug fixing!
The support of PPPoE with Netmap would be great for me. I could then run IPS on my WAN and Sensei on my LAN.
Please let me know if I can help with testing any fixes.
Oh and 20.7 is running really well on my Proxmox host. Install was smooth and all services appear to be functioning properly.
Just need some qemu guest support now 😉
All the best.
-
@mb thanks and good luck with the bug fixing!
Oh and 20.7 is running really well on my Proxmox host. Install was smooth and all services appear to be functioning properly.
Just need some qemu guest support now 😉
Kudos to the OPNsense team. Our experience is the same.
Work in em(4) and vtnet(4) has been started:
https://docs.google.com/spreadsheets/d/1RVj8K3XOzWi-Bkjq6hUxWudu7Cxd8FFTqjLiBMzZWEM/edit#gid=0
Tests/feedback with other drivers is welcome and will be much appreciated.
-
Will the netmap kernel changes also be available for the coming 20.7 stable release ?
I would like to not install the beta branch. 20.7 stable should be released next week ?
-
No netmap test kernel for after 20.7 planned, but it's likely we start netmap improvement efforts for 21.1 release.
20.7-RC1 will be out today, final 20.7 next week, yep.
Cheers,
Franco
-
Okay thanks, im looking forward to the netmap merge 21.1/21.7. Running suricata in a opnsense kvm would be awesome.
-
We'll be posting a status update about the upstream project - hopefully this week.
-
I just read your blog post, gives a nice insight, im gonna link it for others: https://www.sunnyvalley.io/post/status-on-the-netmap-improvement-efforts-for-opnsense-20-7/
Btw in your plans page there is a typo "DnyDNS" should be "DynDNS" ?
-
Hi @Voodoo, thanks for the heads-up.
Yes, here's the latest updated netmap status:
https://www.sunnyvalley.io/post/status-on-the-netmap-improvement-efforts-for-opnsense-20-7/
Speaking with @franco, some good news: it looks like OPNsense team will be able to provide a test kernel and start landing the bug-fixes with 20.7.1 or 20.7.2.
As mentioned in the blog post, we need more testing with regard to some drivers. Any help in that regard would be much appreciated.
We can't start fixing a problem if we don't know there is a problem.
-
I really love OPNsense and Sensei, absolutely great work!
I am wondering that are there any possibilities that we will get em driver support for the upcoming 20.7.1 or 20.7.2 releases?
I am running a 20.1 setup with igb and em NICs and using also VLANS. I am wondering how long should I postpone the upgrade due to em driver issues especially since from my understanding we are not receiving updates to the 20.1 anymore.
I would believe that the em issue affects quite many setups.
-
Hi @hfvk,
Thanks for your feedback. Much appreciated.
The issue is specific to em(4). If yours is igb(4), than you're on the safe side, you can go ahead and upgrade to 20.7 ;)
-
Hi @mb,
Thank you for the reply.
How about systems running on em(4) NICs? Is there any estimate at this point when we could expect to get the fix to OPNsense 20.7? I am just planning how to push the updates to the systems.
I have both igb(4) and em(4) systems ;)
-
Hi @hvfk,
We have witnessed two em(4) systems behaving differently.
Run pciconf -l
If your Chip ID is 0x100e8086, you should be fine. We also have at least one other report that it's working.
If on the other hand, the chip id is, 0x10d38086 than it's a no go.
Speaking with @franco, I would expect 20.7.1 or 20.7.2 introducing improvements.
-
Hi again,
Well, my chip seems to be chip=0x156f8086.
So, I think I’ll just select one test system with this chip and see what happens. I will report you back :)
-
So, I think I’ll just select one test system with this chip and see what happens. I will report you back :)
Yup, sounds good :D
-
But wait, should I have had this problem already on 20.1 since it is running FreeBSD 11.2?
-
if you've been using it with VLANs, yes I would expect so
-
I upgraded my test system and it seems that on 0x156f8086 em(4) is working fine.
OPNsense 20.7-amd64
FreeBSD 12.1-RELEASE-p7-HBSD
OpenSSL 1.1.1g 21 Apr 2020
I will continue monitoring the system and I will post back if I detect any issues.
-
I just upgraded from 20.1.9_1 to 20.7. OPNsense is running as a VM within Proxmox, network driver is vtnet. WAN is PPPoE on VLAN, IPTV_WAN is on VLAN. Surricata is enabled (IPS). Ovpnc1 is a VPN.
Networks are working fine, just a couple of messages which don't seem to have a negative impact.
2020-07-31T23:08:03 kernel: 683.727786 [ 320] generic_netmap_register Emulated adapter for pppoe4 activated
2020-07-31T23:08:03 kernel: 683.727620 [1130] generic_netmap_attach Emulated adapter for pppoe4 created (prev was NULL)
2020-07-31T23:08:03 kernel: pppoe4: permanently promiscuous mode enabled
2020-07-31T23:08:03 kernel: 683.725852 [1035] generic_netmap_dtor Emulated netmap adapter for pppoe4 destroyed
2020-07-31T23:08:03 kernel: 683.725821 [1130] generic_netmap_attach Emulated adapter for pppoe4 created (prev was NULL)
2020-07-31T23:08:03 kernel: 683.708925 [ 320] generic_netmap_register Emulated adapter for ovpnc1 activated
2020-07-31T23:08:03 kernel: 683.708692 [1130] generic_netmap_attach Emulated adapter for ovpnc1 created (prev was NULL)
2020-07-31T23:08:03 kernel: ovpnc1: permanently promiscuous mode enabled
2020-07-31T23:08:03 kernel: 683.701541 [1035] generic_netmap_dtor Emulated netmap adapter for ovpnc1 destroyed
2020-07-31T23:08:03 kernel: 683.701510 [1130] generic_netmap_attach Emulated adapter for ovpnc1 created (prev was NULL)
2020-07-31T23:08:03 kernel: 683.623629 [ 83] vtnet_free_used 1 sgs dequeued from TX-0 (netmap=0)
2020-07-31T23:08:03 kernel: vtnet0: permanently promiscuous mode enabled
If you need any more information, or need me to do something to help troubleshoot, please let me know.
-
Hi again,
Well, my chip seems to be chip=0x156f8086.
So, I think I’ll just select one test system with this chip and see what happens. I will report you back :)
Are the following supported?
em0@pci0:1:0:0: class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
em1@pci0:2:0:0: class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
em2@pci0:3:0:0: class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
em3@pci0:4:0:0: class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
em4@pci0:5:0:0: class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
em5@pci0:6:0:0: class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
-
@hfvk, @heresjody, thanks for letting us know.
@guyp2k, we have not heard of any feedback about that.
As of now, chip ids that are reported to be working:
0x156f8086
0x100e8086
Problematic chip id:
0x10d38086
-
My system has
em0@pci0:0:31:6: class=0x020000 card=0x00008086 chip=0x156f8086 rev=0x21 hdr=0x00
used on the WAN, the rest are igb.
All looks ok, except WAN does not appear in the sensei list (but i think it didn't in 20.1)
-
Hi @aimdev, thanks. Do you have Suricata (in IPS mode) on WAN? If so, that's good. Yours is another testimony that this particular chipset works.
-
No not yet. I don't fully understand the ramifications of disabling the hardware options.
-
Hi @aimdev, thanks. Do you have Suricata (in IPS mode) on WAN? If so, that's good. Yours is another testimony that this particular chipset works.
@mb just a FYI, I do have Suricata running in IPS mode on the WAN interface however, I had to add the WAN IP to the "Home Networks in order to start logging and blocking, chipset em0@pci0:1:0:0: class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00.
Question, I subscribe to Sensei and still confused on the road map, is the ultimate goal is to use Suricata on both internal and external interface to take the place of Suricata or use both? Under Sensei Configuration, I do not have an option to select the external/WAN interface, just LAN and and VLANs.
Again, just a little confused on this entire Sensei, Suricata, and netmap effort.....
TIA
-
Thanks for looking into this @mb
I have put the 20.7 proxmox virtual instance of OPNsense inline with PPPoE by backing up my 20.1 config and restoring to 20.7. Really easy to do and was up and running quickly.
Suricata runs successfully and alerts on the LAN interface (Virtio)
Switched to run on the WAN interface and am not receiving alerts. The new log view with v5 Suricata is different and seems to cycle with this when trying to establish IPS on the PPPoE WAN
Counter | TM Name | Value
------------------------------------------------------------------------------------
Date: 6/14/2020 -- 08:49:09 (uptime: 0d, 00h 08m 17s)
------------------------------------------------------------------------------------
flow.memuse | Total | 7154304
tcp.reassembly_memuse | Total | 196608
tcp.memuse | Total | 1146880
flow_mgr.rows_skipped | Total | 65536
flow_mgr.rows_checked | Total | 65536
flow.spare | Total | 10000
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
Date: 6/14/2020 -- 08:49:01 (uptime: 0d, 00h 08m 09s)
------------------------------------------------------------------------------------
flow.memuse | Total | 7154304
tcp.reassembly_memuse | Total | 196608
tcp.memuse | Total | 1146880
flow_mgr.rows_skipped | Total | 65536
flow_mgr.rows_checked | Total | 65536
flow.spare | Total | 10000
------------------------------------------------------------------------------------
I am getting the ifconfig -a to you via PM
Here is the log when it successfully runs on the LAN interface
flow.memuse | Total | 7177096
tcp.reassembly_memuse | Total | 231424
tcp.memuse | Total | 1146880
flow_mgr.rows_maxlen | Total | 1
flow_mgr.rows_skipped | Total | 65534
flow_mgr.rows_checked | Total | 65536
flow_mgr.flows_notimeout | Total | 2
flow_mgr.flows_checked | Total | 2
flow.spare | Total | 10000
flow_mgr.new_pruned | Total | 9
app_layer.flow.failed_udp | Total | 40
app_layer.tx.dns_udp | Total | 20
app_layer.flow.dns_udp | Total | 6
app_layer.flow.failed_tcp | Total | 4
app_layer.tx.dhcp | Total | 2
app_layer.flow.dhcp | Total | 1
app_layer.flow.tls | Total | 3
tcp.overlap | Total | 1
tcp.rst | Total | 16
tcp.synack | Total | 9
tcp.syn | Total | 9
tcp.sessions | Total | 9
flow.udp | Total | 47
flow.tcp | Total | 39
decoder.max_pkt_size | Total | 1514
decoder.avg_pkt_size | Total | 474
decoder.udp | Total | 320
decoder.tcp | Total | 988
decoder.ethernet | Total | 1594
decoder.ipv6 | Total | 2
decoder.ipv4 | Total | 1306
decoder.bytes | Total | 756423
decoder.pkts | Total | 1594
capture.kernel_packets | Total | 1594
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
Date: 6/14/2020 -- 09:10:57 (uptime: 0d, 00h 01m 52s)
-----------------------------------------------------------------------------------
I'm having the same result on my Qotom Installation with intel interfaces. I'm using pppoe over VLAN as WAN Interface. It would be really awesome to have netmap support for setups like that :)
-
A quick update:
I think this vmx bug has been resolved on FreeBSD 12-STABLE:
https://svnweb.freebsd.org/base?view=revision&revision=363163
Let's do some tests and I'll post some results.
If anyone out there is able to test 12/STABLE kernel with an ESX vmx interface, that'd also be great :)
-
A quick update:
I think this vmx bug has been resolved on FreeBSD 12-STABLE:
https://svnweb.freebsd.org/base?view=revision&revision=363163
Let's do some tests and I'll post some results.
If anyone out there is able to test 12/STABLE kernel with an ESX vmx interface, that'd also be great :)
I have ESXi 7.0 with Netmap crashing using vmxnet3 with Intrusion Detection and Sensei enabled. So if you think this test would fix it, I could give it a try. Let me know how to test.
-
netmap with 20.7 release for vnet driver (virtio) is working, the kernel panic is gone.
-
No sure if this is netmap related, but I can't seem to get any alerts showing for IPS anymore?
They we're showing initially after upgrading to 20.7 rc1
My test is to initiate to P2P traffic from a client behind OPNsense.
I can usually catch this with 'ET telemetry/emerging-p2p' set to drop and its associated alerts.
Running:
OPNsense 20.7-amd64
FreeBSD 12.1-RELEASE-p7-HBSD
OpenSSL 1.1.1g 21 Apr 2020
Currently IPS is enabled on my LAN interface which is vtnet through Proxmox
I have tried em through Proxmox also with no success
IPS Enabled
Promiscuous Enabled
Syslog Dsiabled
Eve Disabled
I don't use remote logging, but as an FYI I have disabled circular logging so am using syslog-ng.
IPS appears to be running successfully when checking the IPS log from the GUI
The log appears to have returned to its original format...
Have enabled 'drop' for all rulesets including ET Telemetry in an attempt to 'kick' it into life.
Any ideas of how to test this further and progress would be great.
-
netmap with 20.7 release for vnet driver (virtio) is working, the kernel panic is gone.
Hi @Vodoo, what happens when you go out of netmap mode ? (e.g. shut down suricata/sensei) ? virtio bug is related to that. entering netmap mode seems fine.
-
netmap with 20.7 release for vnet driver (virtio) is working, the kernel panic is gone.
I still do observe pagefaults with virtio vtnet interfaces...
-
netmap with 20.7 release for vnet driver (virtio) is working, the kernel panic is gone.
I still do observe pagefaults with virtio vtnet interfaces...
Yes, this is expected as of now. Fix is on upstream.
-
For those who
- are using Suricata/Sensei on VLANs on em(4)
- experiencing vmx crash
- experiencing vtnet crash
- want to have Suricata on PPPoE / OpenVPN interfaces
I have an (unofficial) test kernel ready. Please PM me if you'd like to give it a try.
PS: vmx patch fixes the kernel crash, but has an outstanding issue. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248494
-
For those who
- are using Suricata/Sensei on VLANs on em(4)
- experiencing vmx crash
- experiencing vtnet crash
- want to have Suricata on PPPoE / OpenVPN interfaces
I have an (unofficial) test kernel ready. Please PM me if you'd like to give it a try.
PS: vmx patch fixes the kernel crash, but has an outstanding issue. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248494
I would like to try out the test kernel, (and see if I get duplicates).
Just out of curiosity, are you certain that the outstanding issue with duplicates is not related to the following: https://kb.vmware.com/s/article/59235
Cause I was seeing lots of duplicates when running CARP & Sensei until I applied the mac-learning workaround on the dswitch.
-
@sorano, it does not seem to be related.
Please follow below steps and see if this kernel is of help:
[root@20gw /root]# cd /boot/
[root@20gw:/boot # fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0805-2.tar.gz
kernel-12.1-0805-2.tar.gz 45 MB 4980 kBps 10s
[root@20gw /boot]# mv kernel kernel.stock.save
[root@20gw /boot]# tar zxf kernel-12.1-0805-2.tar.gz
[root@20gw /boot]# reboot
After the reboot, you should be able to see this kernel information:
root@20gw:~ # uname -a
FreeBSD 20gw.local 12.1-RELEASE-p7-HBSD FreeBSD 12.1-RELEASE-p7-HBSD #2 5742b25c4(master)-dirty: Wed Aug 5 22:20:24 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
root@20gw:~ #
To restore stock OPNsense kernel:
# cd /boot
# rm -rf kernel
# mv kernel.stock.save kernel
# reboot
-
@mb before 20.7 enabling ips mode with suricata/sensei crashed the kvm virtio opnsense with a kernel panic within 5 seconds, now i can enable it without any issues everything is working, also disabling/enabling.
-
@Voodoo, that's great to hear, thanks for sharing.
-
Is the kernel above the test kernel that includes the PPPoE fixes, or should we still PM?
-
Hi @madj42, yes. Can you test and provide feedback?
You can also use a newer kernel: https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0806-1.tar.gz
-
I have run up the latest two kernels offered here but cannot enable either IPS or Sensei on my LAN vtnet0 interface.
Setup is Proxmox
PPPoE on vtnet1 for WAN (VLAN 10 on Host Proxmox interface)
vtnet0 for LAN
I reset the logs from System>Settings>Logging to reset Intrusion Detection logs as the button within that view does not work for the 'stats' logs that are displayed.
When I enable IPS on LAN I get cycling of the log within Services>Intrusion Detection>Log File. Cycles every few seconds so am assuming it is the application of IPS failing? This is the' stats' log, not the older detailed log.
I cannot enable Sensei as the LAN interface is not available for selection. Only the underlying vtnet1 of the WAN PPPoE is available.
-
I cannot enable Sensei as the LAN interface is not available for selection. Only the underlying vtnet1 of the WAN PPPoE is available.
@buchoffreeds, you can use this hack to have Sensei on vtnet1:
https://forum.opnsense.org/index.php?topic=9521.msg84199#msg84199
Your feedback is much appreciated.
NOTE: We'll remove this check once we have the test kernel in production.
-
Thanks @mb
That allowed me to select my LAN interface within Sensei.
Sensei is now running successfully on my Proxmox vtnet0 LAN Interface!
It first alerted that Suricata was in use on the LAN Interface, so I moved Suricata to WAN Interface to resolve this.
Just Disabling Intrusion Detection was not enough.
-
I'm on chip=0x150e8086 and my graphs don't work with IPS enabled. Also having some very poor throughput with or without IPS. If this helps at all.
Edit: intel nic using the igb (i think that designates the drivers being used?)
-
Hi @bunchofreeds, thanks for the feedback. Glad to hear that vtnet is now fine.
It first alerted that Suricata was in use on the LAN Interface, so I moved Suricata to WAN Interface to resolve this.
Just Disabling Intrusion Detection was not enough.
Yes, this is done on purpose, since people might enable Suricata on LAN in a future time forgetting that Sensei is running there.
-
Hi @FullyBorked, yours might be related to a different issue. Generally netmap problems generally appear in cases where you have total packet flow problems.
-
I see there is progress on PPPoE when looking at the google drive sheet.
Let me know if you want this tested.
I am running Proxmox and vtnet drivers.
Have Sensei successfully running on the LAN interface currently.
-
netmap with 20.7 release for vnet driver (virtio) is working, the kernel panic is gone.
I still do observe pagefaults with virtio vtnet interfaces...
Yes, this is expected as of now. Fix is on upstream.
That seems to have massively improved or even be fixed completely with the new test kernel. 👍🏻
-
Thanks to the new test kernel, I now have almost the max. over OpenVPN what I can transmit with my line. Instead of 25 Mbit it is now 45 MBit. :) I run the opnsense on both sides as VM within Proxmox. Network in Proxmox virtio with 8 queues.
PS: Sensei and Suricata enabled. And Suricata works now in VM :)
-
@bunchofreeds, yes that'd be perfect if you can have Suricata on WAN (pppoe) and see how it goes. Our early tests showed good results with our test tools.
@binaryanomaly, thanks for the feedback. vtnet seems to be doing even better than 20.1.
@lewald, that's great to hear, though I wouldn't expect netmap work might have contributed to the vpn speed. It could be virtio that you can use it with 8 queues.
-
Just reporting that I've installed the last test kernel and enabled IPS on my PPPoE connection with IGB drivers. All appears to be good and haven't had issues over the weekend.
-
@madj42, that's great news!. Thanks for letting us know.
-
@binaryanomaly, thanks for the feedback. vtnet seems to be doing even better than 20.1.
I can definitely confirm that, had lots of page faults already on 20.1. Running smooth and steady since 3 days - great job, well done!
-
Running Proxmox with vtnet drivers on WAN (PPPoE) and LAN
Kernel
12.1-RELEASE-p7-HBSD FreeBSD 12.1-RELEASE-p7-HBSD #3 5742b25c4(master)-dirty: Thu Aug 6 16:17:42 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
Sensei is running successfully on LAN
I'm unsure if IPS is correctly running on WAN (PPPoE)
Have enabled and set IPS and Promiscuous (Have not selected Syslog or Eve)
Log Output via GUI is:
2020-08-11T08:25:46 suricata[15476]: [100117] <Notice> -- all 2 packet processing threads, 4 management threads initialized, engine started.
2020-08-11T08:25:46 suricata[15476]: [101288] <Notice> -- opened netmap:pppoe1/T from pppoe1: 0x51b7a149300
2020-08-11T08:25:46 suricata[15476]: [101288] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x51b7a149000
2020-08-11T08:25:46 suricata[15476]: [100335] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x51b78319300
2020-08-11T08:25:46 suricata[15476]: [100335] <Notice> -- opened netmap:pppoe1/R from pppoe1: 0x51b78319000
2020-08-11T08:24:50 suricata[15476]: [100117] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'is_proto_irc' is checked but not set. Checked in 2002029 and 1 other sigs
2020-08-11T08:24:50 suricata[15476]: [100117] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.gadu.loggedin' is checked but not set. Checked in 2807836 and 0 other sigs
2020-08-11T08:24:50 suricata[15476]: [100117] <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.eduphish' is checked but not set. Checked in 2025114 and 0 other sigs
2020-08-11T08:24:30 suricata[89429]: [100183] <Notice> -- This is Suricata version 5.0.3 RELEASE running in SYSTEM mode
However it does not appear to block/drop P2P traffic as a test.
Downloading ubuntu via bit torrent
Ruleset set to drop ET telemetry/emerging-p2p
My Log seems to be broken into two parts. One being the original style logs as shown above. These always appear at the top of the log view.
The second a more like 'stats' that cycles continuously every few seconds. There are LOTS of these being generated.
flow.memuse | Total | 7154304
tcp.reassembly_memuse | Total | 196608
tcp.memuse | Total | 1146880
flow_mgr.rows_skipped | Total | 65536
flow_mgr.rows_checked | Total | 65536
flow.spare | Total | 10000
------------------------------------------------------------------------------------
Counter | TM Name | Value
------------------------------------------------------------------------------------
Date: 8/11/2020 -- 08:29:14 (uptime: 0d, 00h 04m 43s)
------------------------------------------------------------------------------------
I do not think that Suricata is operational on my WAN (PPPoE) interface, although it has enabled without errors.
Any ideas how to test this further?
-
I do not think that Suricata is operational on my WAN (PPPoE) interface, although it has enabled without errors.
Hmm, tun(4) implementation is adding an ethernet header (since tun does not have); and mac addresses are all zeros. That might be creating a problem.
Let us have a look deeper; and I'll post updates.
-
@mb
@lewald, that's great to hear, though I wouldn't expect netmap work might have contributed to the vpn speed. It could be virtio that you can use it with 8 queues.
Maybee. But without the Testkernel i have a lot dequeues on netmap. Now this deueues are gone.
-
@mb
Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?
-
@mb
Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?
No, read the changelog. That is not fix the netmap issues.
-
@mb
Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?
No, read the changelog. That is not fix the netmap issues.
"From the reported issues we still have more logging quirks to investigate and especially Netmap support (used in IPS and Sensei) is lacking in some areas that were previously working. Patches are being worked on already so we shall get there soon enough. Stay tuned."
That could mean anything...
-
@mb
Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?
No, read the changelog. That is not fix the netmap issues.
To be a little fair the change log opening sentence seems a bit ambiguous. I don't think it was intended that way but the first time I read it I thought it was saying there were other changes that weren't noted in the log.
Small update here with security advisories, multicast fixes and logging reliability patches amongst others.
Also I think he is asking if it will revert his changes, not if it has been fixed in the release.
-
"amongst others" references the full change log below. It's intentionally ambiguous in the sense that the actual changes are listed below. If you don't see your issue there it's probably just that.
The second paragraph is more loose in terms of content from release to release. It is meant to hint at past and future events. In this case it unambiguously states that Sensei and IPS issues are not yet resolved in the release.
I'm not sure how to make this any clearer other than: don't panic and use 20.1 if you must. ;)
Cheers,
Franco
-
"amongst others" references the full change log below. It's intentionally ambiguous in the sense that the actual changes are listed below. If you don't see your issue there it's probably just that.
The second paragraph is more loose in terms of content from release to release. It is meant to hint at past and future events. In this case it unambiguously states that Sensei and IPS issues are not yet resolved in the release.
I'm not sure how to make this any clearer other than: don't panic and use 20.1 if you must. ;)
Cheers,
Franco
No panic here :) I just read it wrong, initially. Thought maybe someone else did too. No need to rewrite anything at all. I'm patiently waiting on 20.7 and figure it'll work when it works and it will be great as usual.
-
Thanks, next week will have progress on this front for sure. If we can confirm quickly we may even start adding patches to 20.7.2 already. Just want to keep the changes small and impact big... in the positive sense of course.
Cheers,
Franco
-
I'm not sure how to make this any clearer other than: don't panic and use 20.1 if you must. ;)
vtnet page faults on anything besides the test kernel from mb here. So I'll stay with the test kernel until the fix finds its way upstream :)
-
@sorano, it does not seem to be related.
Please follow below steps and see if this kernel is of help:
[root@20gw /root]# cd /boot/
[root@20gw:/boot # fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0805-2.tar.gz
kernel-12.1-0805-2.tar.gz 45 MB 4980 kBps 10s
[root@20gw /boot]# mv kernel kernel.stock.save
[root@20gw /boot]# tar zxf kernel-12.1-0805-2.tar.gz
[root@20gw /boot]# reboot
After the reboot, you should be able to see this kernel information:
root@20gw:~ # uname -a
FreeBSD 20gw.local 12.1-RELEASE-p7-HBSD FreeBSD 12.1-RELEASE-p7-HBSD #2 5742b25c4(master)-dirty: Wed Aug 5 22:20:24 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
root@20gw:~ #
To restore stock OPNsense kernel:
# cd /boot
# rm -rf kernel
# mv kernel.stock.save kernel
# reboot
Is it possible to post a test kernel based on the 20.7.1 kernel? Then I can test Surricata on PPPoE WAN without my girlfriend complaining about not being able to watch IPTV ::)
-
To be a little fair the change log opening sentence seems a bit ambiguous. I don't think it was intended that way but the first time I read it I thought it was saying there were other changes that weren't noted in the log.
Yep, me this morning: "yeah, finally a fix", snapshot, upgrade... and Meeeeeeh, revert snapshot! ::)
I might have read to fast, my bad! ;)
-
@mb
Is 20.7.1 fixing the netmap issues adressed in the test kernel or would it set me back to the state before?
No, read the changelog. That is not fix the netmap issues.
"From the reported issues we still have more logging quirks to investigate and especially Netmap support (used in IPS and Sensei) is lacking in some areas that were previously working. Patches are being worked on already so we shall get there soon enough. Stay tuned."
That could mean anything...
Yes but one thing is certain. The fix is still not ready :)
-
Yes but one thing is certain. The fix is still not ready :)
Seems not. But then it could also have been - that test kernel runs better than any production one so far here...
Anyway I installed the test kernel again after upgrading to 20.7.1 which runs smooth so far.
-
Hi @heresjody, sure, we'll post a new one based on 20.7.1 this week. -- and hopefully with a final patch for vmx issue.
-
A quick update:
I hear that OPNsense will ship an official netmap test kernel in the coming week.
For the impatient, here is a new test kernel which is based on 20.7.1 stock kernel:
[root@20gw /root]# cd /boot/
[root@20gw:/boot # fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0815-2.tar.gz
kernel-12.1-0815-2.tar.gz 45 MB 4980 kBps 10s
[root@20gw /boot]# mv kernel kernel.stock.save
[root@20gw /boot]# tar zxf kernel-12.1-0815-2.tar.gz
[root@20gw /boot]# reboot
After the reboot, you should be able to see this kernel information:
root@20gw:~ # uname -a
FreeBSD 20gw.local 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #3 87f253a0d(master)-dirty: Sat Aug 15 09:29:08 PDT 2020 root@bsd12_openssl:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
root@20gw:~ #
To restore stock OPNsense kernel:
# cd /boot
# rm -rf kernel
# mv kernel.stock.save kernel
# reboot
-
I have run up the following kernel on my Proxmox vtnet pppoe setup
FreeBSD ... 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #3 87f253a0d(master)-dirty: Sat Aug 15 09:29:08 PDT 2020 root@bsd12_openssl:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
I was getting hi CPU usage from flowd_aggregate which settled down after about 5 mins
39226 root 101 0 38M 29M CPU0 0 0:56 98.97% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.7)
11 root 155 ki31 0 32K RUN 0 8:24 50.98% [idle{idle: cpu0}]
syslog_ng starts then stops (I manually started this up again although I do not do remote logging so not sure if I need this for any local OPNsense logging?)
Sensei is running successfully on vtnet LAN interface
Suricata enables on vtnet/PPPoE but does not work
Looking forward to testing out some PPPoE netmap updates :)
-
I hear that OPNsense will ship an official netmap test kernel in the coming week.
Is this test kernel already available over opnsense update feature?
Is the vmx bug fixed and sensei working with this test kernel?
-
Good morning. I have some good news :)
Finally we have a test kernel which addresses a range of netmap/iflib issues on OPNsense 20.7.x/BSD 12:
- em+vlan stall
- vtnet crash
- vmx crash
- vpn - tun interface support : We've confirmed that this kernel also works for Sensei on vpn - tun(4) interfaces.
- some other minor problems
Here are the steps to give it a try; and please do test and provide feedback. Sooner we provide some feedback to OPNsense, sooner they can make this kernel available for general use.
[root@20gw /root]# cd /boot/
[root@20gw:/boot # fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0822-1.tar.gz
kernel-12.1-0822-1.tar.gz 45 MB 4980 kBps 10s
[root@20gw /boot]# mv kernel kernel.stock.save
[root@20gw /boot]# tar zxf kernel-12.1-0822-1.tar.gz
[root@20gw /boot]# reboot
After the reboot, you should be able to see this kernel information:
root@20gw:~ # uname -a
FreeBSD 20gw.local 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6 39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
root@20gw:~ #
To restore stock OPNsense kernel:
# cd /boot
# rm -rf kernel
# mv kernel.stock.save kernel
# reboot
-
Firstly, thanks for the awesome effort so far in getting these netmap issues sorted out.
Confirmed running
FreeBSD 20gw.local 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6 39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
Unfortunately I am having trouble making my PPPoE interface available for Sensei.
I have gone as far as adding an additional vtnet interface to my Proxmox OPNsense guest at OPT1. This was to ensure I could move Suricata to an unused interface (Suricata is disabled).
My only available interfaces in Sensei are:
LAN vtnet0 - Currently running on this interface
(Unassigned) vtnet1 - this is the interface that PPPoE resides on
OPT1 (vtnet2) - The new interface I added and moved Suricata to. To be sure it was not conflicting.
Should I be expecting to see PPPoE as an available interface?
-
Tested with igb interfaces and pppoe on wan (removed VLAN for testing)
Suricata seems to start fine:
2020-08-23T09:02:28 suricata[76340] [100106] <Notice> -- all 2 packet processing threads, 4 management threads initialized, engine started.
2020-08-23T09:02:28 suricata[76340] [100805] <Notice> -- opened netmap:pppoe1/T from pppoe1: 0x172871fd300
2020-08-23T09:02:28 suricata[76340] [100805] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x172871fd000
2020-08-23T09:02:28 suricata[76340] [100798] <Notice> -- opened netmap:pppoe1^ from pppoe1^: 0x17236be4300
2020-08-23T09:02:28 suricata[76340] [100798] <Notice> -- opened netmap:pppoe1/R from pppoe1: 0x17236be4000
However, it doesn't alert or block on anything.
Then I tried Sensei on the WAN Interface. It starts, but afterwards Internet is gone.
Reports do not show any sessions or blocks.
-
@mb
I installed like described in your post yesterday.
Looks like it works for my vmx interfaces on opnsense vm.
But I see some issues:
- can’t see any interface on „interface list“ to select.
- can‘t see tun interfaces
My vm only having vmx and tun interfaces.
Do I need to do some additional steps?
Edit:
Is there a way to show unsupported interfaces in sensei configuration? I think I do not see any interface as I only have vmx0-7 and tun0. So there is just no supported interface.
To test vmx and the tun I need to reconfigure the used interfaces.
At the moment (old config before upgrading) sensei was active on 7 of 8 vmx interfaces and is now running fine since around 2h. Filter/Blocking is working as expected.
Edit2:
Found this here... after "commenting" vmx and ovpns1 I can now see the interfaces :)
https://forum.opnsense.org/index.php?topic=9521.msg84199#msg84199
If I add ovpns1 to "protected interfaces" Sensei is creashing. So no luck with OpenVPN tun interface.
Additional info: All interfaces do not use vlan. VLAN tagging is done on ESXi level (dvSwitch).
-
Should I be expecting to see PPPoE as an available interface?
Hi @bunchofreeds, thanks and all wellcome.
PPPoE interfaces are filtered on 1.5.2 release, and Sensei is meant to be running in inner interfaces. This is why you can't see them.
Any chances that you can create a small pcap trace (e.g. tcpdump -s0 -n -i pppoe0 -c 100 -w pppoe.pcap) and PM it to me?
Suricata should be ok working with PPPoE now. I would like to see if anything is different from vpn interfaces.
-
Tested with igb interfaces and pppoe on wan (removed VLAN for testing)
Suricata seems to start fine:
..
..
However, it doesn't alert or block on anything.
Then I tried Sensei on the WAN Interface. It starts, but afterwards Internet is gone.
Reports do not show any sessions or blocks.
Hi @Quetschwalze, thanks. Any chances that you can also send a pcap trace? - Sensei is not meant for WAN right now.
-
At the moment (old config before upgrading) sensei was active on 7 of 8 vmx interfaces and is now running fine since around 2h. Filter/Blocking is working as expected.
Edit2:
Found this here... after "commenting" vmx and ovpns1 I can now see the interfaces :)
https://forum.opnsense.org/index.php?topic=9521.msg84199#msg84199
If I add ovpns1 to "protected interfaces" Sensei is creashing. So no luck with OpenVPN tun interface.
Hi @scream, thanks for confirming vmx. Glad to hear that it's working.
For tun, I need to provide a 1.6 beta to you since Sensei needs to tweak interface initialization parameters for tun(4) interfaces. It's not done on 1.5.
Stay tuned, I'll provide the 1.6 txz link today.
-
For tun, I need to provide a 1.6 beta to you since Sensei needs to tweak interface initialization parameters for tun(4) interfaces. It's not done on 1.5.
Stay tuned, I'll provide the 1.6 txz link today.
Thanks a lot! I‘m looking forward to test tun :-) Let me know how I can install the beta.
vmx works like a charm on my opnsense vm since upgrade to this kernel.
The only thing I want to mention is that CPU usage is higher than before.
20.1 with sensei used around 15-20% of each of the two vCPU.
20.7 with sensei is now using around 25-27% of each of the two vCPU.
-
Tested with igb interfaces and pppoe on wan (removed VLAN for testing)
Suricata seems to start fine:
..
..
However, it doesn't alert or block on anything.
Then I tried Sensei on the WAN Interface. It starts, but afterwards Internet is gone.
Reports do not show any sessions or blocks.
Hi @Quetschwalze, thanks. Any chances that you can also send a pcap trace? - Sensei is not meant for WAN right now.
Thanks @mb
Yes, No Problem. Just to make sure, you'll need a pcap of suricata/pppoe wan interface traffic?
Gesendet von meinem MI 9 mit Tapatalk
-
Yes, No Problem. Just to make sure, you'll need a pcap of suricata/pppoe wan interface traffic?
Yep.
-
@mb
One more thing that happens with sensei on vmx:
If on 20.7 and sensei is active I see a performance degration to 100-150 Mbit/s. Without sensei I can reach about 350-400 Mbit/s (on same device connected to WiFi).
Before upgrade I was able to reach 450 Mbit/s with sensei running.
-
Hi @scream, are you doing the tests with speedtest.net ? If so, can you repeat the test with scp ? Try scp'ing a (large) file to one of the IP addresses on sensei protected interfaces on the firewall (you'll basically copy a file to the firewall.)
Run this test with and without sensei and see how much it differs.
e.g. scp 1gbfile root@fw-sensei-protected-interface-ip:/dev/null
-
Hi @scream, are you doing the tests with speedtest.net ? If so, can you repeat the test with scp ? Try scp'ing a (large) file to one of the IP addresses on sensei protected interfaces on the firewall (you'll basically copy a file to the firewall.)
Run this test with and without sensei and see how much it differs.
e.g. scp 1gbfile root@fw-sensei-protected-interface-ip:/dev/null
No. I‘m using iperf on a VM on another subnet and my iPhone.
-
@mb
I did iperf testing now on two ubuntu vms on esx itself.
So...
vm1 (subnet1) -> opnsense vmx0 -> opnsense vmx1 -> vm2 (subnet2). Everything on esx. Sensei configured to be active on both vmx interfaces.
I did 4 tests:
1. opnsense 20.7 with sensei => 126 Mbit/s
2. opnsense 20.7 without sensei (stopped packet engine) => 904 Mbit/s
3. opnsense 20.1.9 with sensei => 918 Mbit/s
4. opnsense 20.1.9 without sensei (stopped packet engine) => 921 Mbit/s
Detail results:
https://paste.ubuntu.com/p/Vjqmrr5Z8m/
-
@scream, thanks. that looks interesting. let us try to reproduce this here. currently we can attain 450-500 Mbps with or without sensei in our lab.
-
@scream, thanks. that looks interesting. let us try to reproduce this here. currently we can attain 450-500 Mbps with or without sensei in our lab.
Probably just an issue with vmx interfaces?
May I did a mistake on the installation steps? I didn‘t reset the config, just updated from 20.1.9 to 20.7.
Then updated packages. After that I patched kernel an rebooted opnsense completly. After that I just startet elasticsearch and packet engine.
After that I saw that I can‘t see any Interface in configuration tab... so I patched also php file as described.
A simple revert to the snapshot I‘ve created before upgrading to 20.7 brings back the „wirespeed“ performance.
I can also try 1.6 beta, may it is fixed there?
-
It is mostly related to the kernel. I would not expect 1.6 would do any difference.
What happens if you put sensei into bypass mode?
-
It is mostly related to the kernel. I would not expect 1.6 would do any difference.
What happens if you put sensei into bypass mode?
20.1.9 sensei bypass mode => 855 Mbit/s
20.7 sensei bypass mode => 205 Mbit/s
Note that there is now some other traffic in the networks so 855 Mbit/s is normal.
https://paste.ubuntu.com/p/35v3HxmJrT/
-
20.1.9 sensei bypass mode => 855 Mbit/s
20.7 sensei bypass mode => 205 Mbit/s
Note that there is now some other traffic in the networks so 855 Mbit/s is normal.
https://paste.ubuntu.com/p/35v3HxmJrT/
@scream, these tests are very helpful, thanks.
sensei in bypass mode does nothing than simply bridging packets back and forth. Hence, this looks like netmap performance.
However, in our labs we can attain 450-500 Mbps throughput between VMware guests (vmx).
Any chances that you can reach out? Send a PR via "Report Bug" menu on the upper right hand side of the screen. We would like to have a closer look.
-
Any chances that you can reach out? Send a PR via "Report Bug" menu on the upper right hand side of the screen. We would like to have a closer look.
Done. :)
Any chance for the 1.6 link to test tun devices?
-
Hi @scream,
Sure, please find it below :)
# fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/os-sensei-1.6.beta1.txz
os-sensei-1.6.beta1.txz 25 MB 4688 kBps 05s
# pkg add os-sensei-1.6.beta1.txz
Please be noted, although this has been thoroughly tested and reached beta stage, it's still not meant for production use. Use carefully.
-
Hi @scream,
Sure, please find it below :)
# fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/os-sensei-1.6.beta1.txz
os-sensei-1.6.beta1.txz 25 MB 4688 kBps 05s
# pkg add os-sensei-1.6.beta1.txz
Please be noted, although this has been thoroughly tested and reached beta stage, it's still not meant for production use. Use carefully.
Did a clean upgrade from 20.1.9 to 20.7 agian, uninstalled sensei and installed it from the packet.
Also did a fresh clean configuration of 1.6 beta.
tun device does work and looks like performance of 100Mbit/s isn't a issue here. (Can't test faster at time as this is the speed limit of this uplink).
As expected this doesn't make any difference of the performance issue reported in combination of opnsense 20.7, sensei & vmx interface on my server. Still arount 100-130 Mbit/s. (If sensei is stopped wirespeed around 950 Mbit/s) is possible.
One thing I want to mention is that there is no "Web Control" in this 1.6beta?
On my installation "Web Control" doesn't show categories at all. I just can select between "Permissive" / "Moderate Control" ... but if I select "Moderate Control" for e.g. I can't save this. It is just hanging on the load bar.
Will test further.
-
Hi @mb,
When I try to apply the 1.6 package via SSH I get the following:
the most recent version of os-sensei-1.5.2_1 is already installed
Just to confirm, this is the latest Sensei version, do we also need the latest kernel version?
What versions do you recommend we test with?
-
Yep, for 1.6 you need the netmap test kernel and you need to be on 20.7. You can try force installing:
pkg add -f os-sensei-1.6.beta1.txz
-
OK am running latest Kernel:
12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6 39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
And am running Sensei 1.6 Beta1
I can't see any interfaces in Sensei to apply to. Not even vtnet0 LAN which it is currently running on.
Should I hack in some bypasses https://forum.opnsense.org/index.php?topic=9521.msg84199#msg84199
-
Hi @bunchofreeds,
Sorry, filter was still there. Can you try again:
pkg add -f https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/os-sensei-1.6.beta1.txz
-
@mb,
I now can see the used 'LAN (vtnet0)' interface, but not PPPoE. Only it's parent interface 'Unassigned (vtnet1)'
-
Currently have the new kernel installed.
OPNsense 20.7.1-amd64
FreeBSD 12.1-RELEASE-p8-HBSD
OpenSSL 1.1.1g 21 Apr 2020
FreeBSD OPNsense.localdomain 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6 39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
And it seems my surricata is working fine and has started up monitoring PPPoE on my WAN.
2020-08-26T18:27:25 suricata[13692] [100200] <Notice> -- all 4 packet processing threads, 4 management threads initialized, engine started.
2020-08-26T18:27:25 suricata[13692] [101008] <Notice> -- opened netmap:pppoe4/T from pppoe4: 0x3e58c16d300
2020-08-26T18:27:25 suricata[13692] [101008] <Notice> -- opened netmap:pppoe4^ from pppoe4^: 0x3e58c16d000
2020-08-26T18:27:25 suricata[13692] [101000] <Notice> -- opened netmap:pppoe4^ from pppoe4^: 0x3e58b442300
2020-08-26T18:27:25 suricata[13692] [101000] <Notice> -- opened netmap:pppoe4/R from pppoe4: 0x3e58b442000
2020-08-26T18:27:25 suricata[13692] [100999] <Notice> -- opened netmap:vtnet0/T from vtnet0: 0x3e58abd4300
2020-08-26T18:27:25 suricata[13692] [100999] <Notice> -- opened netmap:vtnet0^ from vtnet0^: 0x3e58abd4000
2020-08-26T18:27:25 suricata[13692] [100992] <Notice> -- opened netmap:vtnet0^ from vtnet0^: 0x3e587ebc300
2020-08-26T18:27:25 suricata[13692] [100992] <Notice> -- opened netmap:vtnet0/R from vtnet0: 0x3e587ebc000
2020-08-26T18:26:27 suricata[9486] [100971] <Notice> -- This is Suricata version 5.0.3 RELEASE running in SYSTEM mode
If I understand correctly this kernel should fix vtnet-instability with the observed random crashes. Is the code below this line an example of the kind of crash which should now be fixed? (log below is from 20.7.1 with standard kernel and edited for a better reading experience)
2020-08-25T10:47:53 kernel 273.390513 [ 320] generic_netmap_register Emulated adapter for ovpnc1 activated
2020-08-25T10:47:53 kernel 273.390098 [1130] generic_netmap_attach Emulated adapter for ovpnc1 created (prev was NULL)
2020-08-25T10:47:53 kernel ovpnc1: permanently promiscuous mode enabled
2020-08-25T10:47:53 kernel 273.385399 [1035] generic_netmap_dtor Emulated netmap adapter for ovpnc1 destroyed
2020-08-25T10:47:53 kernel 273.385329 [1130] generic_netmap_attach Emulated adapter for ovpnc1 created (prev was NULL)
2020-08-25T10:47:53 kernel 273.360774 [ 83] vtnet_free_used 14 sgs dequeued from RX-0 (netmap=1)
2020-08-25T10:47:53 kernel 273.337532 [ 83] vtnet_free_used 15 sgs dequeued from RX-0 (netmap=1)
2020-08-25T10:47:53 kernel 273.313455 [ 83] vtnet_free_used 1 sgs dequeued from TX-0 (netmap=0)
2020-08-25T10:46:54 kernel ---<<BOOT>>---
2020-08-25T10:46:54 syslogd kernel boot file is /boot/kernel/kernel
2020-08-25T10:44:44 syslogd exiting on signal 15
2020-08-25T10:44:42 kernel 082.685532 [ 83] vtnet_free_used 23 sgs dequeued from RX-0 (netmap=1)
2020-08-25T10:44:42 kernel 082.656184 [ 83] vtnet_free_used 127 sgs dequeued from RX-0 (netmap=1)
2020-08-25T10:44:42 kernel 082.656155 [ 83] vtnet_free_used 1 sgs dequeued from TX-0 (netmap=1)
2020-08-25T10:44:42 kernel 082.656113 [1035] generic_netmap_dtor Emulated netmap adapter for ovpnc1 destroyed
2020-08-25T10:44:42 kernel 082.655669 [ 295] generic_netmap_unregister Emulated adapter for ovpnc1 deactivated
2020-08-25T10:44:42 kernel
2020-08-26T17:21:09 kernel 269.933029 [1035] generic_netmap_dtor Emulated netmap adapter for pppoe4 destroyed
2020-08-26T17:21:09 kernel 269.932647 [ 295] generic_netmap_unregister Emulated adapter for pppoe4 deactivated
2020-08-26T17:21:09 kernel 269.745860 [ 320] generic_netmap_register Emulated adapter for pppoe4 activated
2020-08-26T17:21:09 kernel 269.745712 [1130] generic_netmap_attach Emulated adapter for pppoe4 created (prev was NULL)
Update:
It seems Suricata doesn't receive packets from the PPPoE interface. Just changed a setting and this is the output with 0 packets for my PPPoE interface:
2020-08-26T18:50:06 suricata[13692] [100200] <Notice> -- Stats for 'pppoe4^': pkts: 0, drop: 0 (nan%), invalid chksum: 0
2020-08-26T18:50:06 suricata[13692] [100200] <Notice> -- Stats for 'pppoe4': pkts: 0, drop: 0 (nan%), invalid chksum: 0
2020-08-26T18:50:06 suricata[13692] [100200] <Notice> -- Stats for 'vtnet0^': pkts: 82103, drop: 0 (0.00%), invalid chksum: 0
2020-08-26T18:50:06 suricata[13692] [100200] <Notice> -- Stats for 'vtnet0': pkts: 74062, drop: 0 (0.00%), invalid chksum: 0
-
Hi @bunchofreeds, Sensei is not meant to be run on WAN. You can test a vpn interface for the tun support.
Speaking of PPPoE and Suricata, we'll revisit this after the first test kernel. Patience :)
Initial goal is to have a "stable" netmap kernel which works flawlessly for the existing drivers.
-
If I understand correctly this kernel should fix vtnet-instability with the observed random crashes. Is the code below this line an example of the kind of crash which should now be fixed? (log below is from 20.7.1 with standard kernel and edited for a better reading experience)
Hi @heresjody, if the firewall crashes and reboots after the messages, yes that is the crash this kernel is fixing.
It seems Suricata doesn't receive packets from the PPPoE interface. Just changed a setting and this is the output with 0 packets for my PPPoE interface:
Since we don't have a pppoe environment we cannot test this on our end, however I'll reach out to @bunchofreeds and yourself and ask to run a test binary. It'll better tell us if netmap is passing packets or not. Theoretically pppoe should work like the openvpn tun interface.
-
@mb,
Thanks for the updates.
Let me know when you have a PPPoE solution that needs testing.
-
@mb: Awesome, thanks for the great work. Will be standing by to test your binary.
-
I'm a bit late to the party but I've been experiencing crashes (shown in screenshot attached) every day or 2 on my firewall where it locks up and stops passing traffic. Between following this thread and github I first installed the sensei 1.6 beta, then per a gentleman on github did the following:
# opnsense-update -kr 20.7.1-netmap4
# opnsense-shell reboot
Firewall is up but now I'm having a weird issue where exceptions in sensei are not working? For example I have the "Ads" category blocked, but in order to access a site I use I have to whitelist a particular URL which falls into the category. Having the category blocked but a whitelisted URL in the same category worked fine before I did the above, now it is not.
I've tried restarting, stopping and starting the sensei engine but it seems like whitelists aren't working anymore if the category is blocked, where as it did before. Did I screw something up with the kernel update/sensei beta process or is this just a new bug?
-
I'm a bit late to the party but I've been experiencing crashes (shown in screenshot attached) every day or 2 on my firewall where it locks up and stops passing traffic. Between following this thread and github I first installed the sensei 1.6 beta, then per a gentleman on github did the following:
# opnsense-update -kr 20.7.1-netmap4
# opnsense-shell reboot
Firewall is up but now I'm having a weird issue where exceptions in sensei are not working? For example I have the "Ads" category blocked, but in order to access a site I use I have to whitelist a particular URL which falls into the category. Having the category blocked but a whitelisted URL in the same category worked fine before I did the above, now it is not.
I've tried restarting, stopping and starting the sensei engine but it seems like whitelists aren't working anymore if the category is blocked, where as it did before. Did I screw something up with the kernel update/sensei beta process or is this just a new bug?
Looks like the kernel recommended by the gentleman on github is different from the test kernel mentioned here. I've just installed the one from this thread : fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0822-1.tar.gz
Next I'll try adding the sensei 1.6 beta package from the CLI and will post back with results.
-
Hi @loganx1121,
I doubt kernel is the source for your problem, since exceptions are handled in the sensei packet engine.
Do try 1.6 beta1 and if it does not work out shoot a PR.
Can you point me to the github URL about 20.7.1-netmap4 kernel?
-
Hi @loganx1121,
I doubt kernel is the source for your problem, since exceptions are handled in the sensei packet engine.
Do try 1.6 beta1 and if it does not work out shoot a PR.
Can you point me to the github URL about 20.7.1-netmap4 kernel?
Sure it's here:
https://github.com/opnsense/core/issues/4305
So I uninstalled sensei completely via the GUI and now I'm trying to add the beta but I'm getting the following:
root@fw:~ # fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/os-sensei-1.6.beta1.txz
os-sensei-1.6.beta1.txz 25 MB 11 MBps 02s
root@fw:~ # pkg add os-sensei-1.6.beta1.txz
Installing os-sensei-1.6.beta1...
pkg: Missing dependency 'os-sensei-updater'
Failed to install the following 1 package(s): os-sensei-1.6.beta1.txz
And here is the kernel version:
root@fw:~ # uname -a
FreeBSD fw 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6 39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
Any help is appreciated
-
Hi logan, do not completely remove 1.5. just pkg add 1.6; otherwise it'll require dependencies.
-
Ok I'll reinstall sensei from the gui and then add the pkg from cli and report back. Thanks
-
Ok I'll reinstall sensei from the gui and then add the pkg from cli and report back. Thanks
Now it's failing on installing the logstash database. See below:
***ERROR: Elasticsearch service request returned error: Failed to parse mapping [conn]: No handler for type [array] declared on field [tags].***
***ERROR*** CODE:4***
I'm going to uninstall again, then reinstall, then go through the initial setup process, and then upgrade to the beta version and see if that works. Will report back.
-
Ok I'll reinstall sensei from the gui and then add the pkg from cli and report back. Thanks
Now it's failing on installing the logstash database. See below:
***ERROR: Elasticsearch service request returned error: Failed to parse mapping [conn]: No handler for type [array] declared on field [tags].***
***ERROR*** CODE:4***
I'm going to uninstall again, then reinstall, then go through the initial setup process, and then upgrade to the beta version and see if that works. Will report back.
So I uninstalled sensei, reinstalled, went through initial configuration, restored my backup sensei config, then upgraded to the beta version, and it's still basically broken. Still can't add whitelisted urls that belong to blocked categories.
An interesting thing...if I go to firmware and check for updates, I see the following:
Package Name Current Version New Version Required Action
kernel 20.7.1-netmap4 20.7.1 upgrade
Even though root@fw:~ # uname -a
FreeBSD Asgard-Wall.bifrost.local 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6 39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
Anyone have any suggestions?
-
I uninstalled sensei again. Reinstalled from gui, then upgraded to the beta version, blocked a category, then tried to whitelist a URL within that category, still the same thing. This time I didn't restore my sensei config to rule that out as a possible issue. Screenshot attached.
-
- Ok so, uninstalled sensei...again. Ran the firmware update since it struck me as weird that the firmware from github was showing AFTER I had followed the instructions to install the test kernel from this thread...after the reboot I was running:
root@fw:~ # uname -a
FreeBSD fw 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #2 505cf134d9b(stable/20.7)-dirty: Mon Aug 10 12:14:34 CEST 2020 root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
- Installed sensei again.
- Did NOT upload my sensei config backup
- Went through initial configuration.
- Activated my license file
- Blocked a category, tried to access a URL within that category and could not ( as expected)
- Whitelisted that URL and then was able to access it
So the above is normal behavior as I am used to it...
- ssh'd to firewall
- cd /boot- fetch https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/kernel-12.1-0822-1.tar.gz- mv kernel kernel.stock.save- tar zxf kernel-12.1-0822-1.tar.gz
Rebooted from cli. Output of kernel version after reboot:
root@fw:~ # uname -a
FreeBSD fw 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #6 39e30dc05(master)-dirty: Sat Aug 22 09:35:48 PDT 2020 root@sunnyvalley12.localdomain:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
- Confirmed previous sensei exception for the URL is still working
- Checked firmware updates again for funsies...said no updates
- From cli of firewall ran pkg add -f https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/os-sensei-1.6.beta1.txz- Went to sensei status page and stopped the packet engine, then started
- Tried to access the same URL again and I can't
So as soon as I install 1.6 beta, I can't access whitelisted URL's if they match a blocked category.
-
Tested with latest kernel at two sites' firewalls and the beta Sensei at one site.
- Lockups and issues with missing interfaces are resolved with kernel/both.
- Some speed problems at both sites (new kernel, but with and without the beta Sensei), where Sensei is shaving a LOT of the bandwidth (way more than it did previously). 1000mbit gets reduced to 100mbit at both sites for downloads...uploads appear mostly fine. I'm working with support on figuring this one out.
- On the beta Sensei, the Web Controls page is blank. Nothing there at all. App Controls is fine. I haven't opened a ticket on this yet. I'm more worried about the bandwidth issue.
-
The bandwidth issue is because the netmap and the kernel. The main issue is the way too paranoid "hardened" bsd project. I use freebsd on other systems too, this whole project is starting to get annoying and its more like pain in the ass. Much more sideeffect than benefit. I really dont like this where the freebsd developement is going.
-
The bandwidth issue is because the netmap and the kernel. The main issue is the way too paranoid "hardened" bsd project. I use freebsd on other systems too, this whole project is starting to get annoying and its more like pain in the ass. Much more sideeffect than benefit. I really dont like this where the freebsd developement is going.
Noone is forced to use netmap. On fast hardware you get perfect peflrformance, also with 20.7.
And the relation to HBSD compared to FreeBSD is just a guess ..
-
Huh, netmap is not hardened / slowed down. It's just general netmap performance implications. You can compare with FreeBSD 12.1 if you want on the same hardware, just set up a test lab?
Cheers,
Franco
-
https://forum.opnsense.org/index.php?topic=18754.msg86216#new
There is enough performance. Of course devs can not Test every hardware but I think with 20.7.4 theres a solution
-
The bandwidth issue is because the netmap and the kernel. The main issue is the way too paranoid "hardened" bsd project. I use freebsd on other systems too, this whole project is starting to get annoying and its more like pain in the ass. Much more sideeffect than benefit. I really dont like this where the freebsd developement is going.
Noone is forced to use netmap. On fast hardware you get perfect peflrformance, also with 20.7.
And the relation to HBSD compared to FreeBSD is just a guess ..
What would you consider fast hardware? I have an i7-2600 with 16GB of ram running OPNsense and I'm seeing my speeds almost cut in half when Sensei is enabled - from 900+ Mb/sec down to about 550+ Mb/sec.
This is quite the drop from 20.1 where I was able to max out my connection with both Suricata and Sensei enabled.
-
The main problem in this thread is ppl mixing their issues. Sensei iscthird party plugin, core team wont do any testing here. I'm quite sure there is space for improving the kernel, but maybe for 20.7.4 or .5.
OPNsense has fixed releases and sensei stuff may be not ready enough. But on the other side the community is too low for broad testingw. So what is the best way? Wait ages for testers and stick with old OS or go for a bit risk (which can also fail)?
P.S. I'm not core, just guessing
-
Hi, I'm from Sensei team.
For a month now, we've been doing a lot of testing on quite many deployments, hardware etc. I would like to share our findings.
There is a bunch of variables: OS, firewall hardware, ethernet adapter, ethernet adapter compatibility with netmap, test tool, test hardware, test device connectivity (wi-fi, wired), ISP being used etc.
Results can greatly vary in each particular test setting where one or more variable can affect the total throughput.
That being said, for our tests, I can say that; HardenedBSD did not have any significant effect. We found a %1-%2 difference between a FreeBSD 12/Stable kernel and stock OPNsense 20.7 kernel. This difference might also be a testing error.
Real problem is that iflib(4), the new network interface subsystem in FreeBSD, received a code refactoring. When I look at stable/12 commits, I can still - from time to time - see fixes for major issues.
The other thing is; this refactoring also severely affected netmap system. It was mostly incompatible with the new iflib code.
Good news is, we've come a long way in this short period of time. Most serious issues have been handled. I guess OPNsense team will be delivering these fixes with the upcoming releases.
Resources for open source projects can be constrained, so we're helping OPNsense team to create a netmap-iflib-stable kernel. We have sponsored another round of work on netmap side for new drivers and these bug-fixes. OPNsense team has been very cooperative and hard-working in trying to incorporate suggested commits. OPNsense version of 12.1 is likely to be more stable than the upstream.
The latest test kernel looks very promising. iflib work, although it creates a bit of headache now, has great implications for the future, which we'll all enjoy in the mid and long term.
Our initial focus is fixing the reliability problems. Next is performance. Latter is also a bit related to the former.
I'll keep you posted about the developments.
-
Thanks Murat, I'm quite sure with a .4 or .5 most of the issues are gone :)
But this can't work without the community and their responses.
-
@mb
Did you saw my message here: https://forum.opnsense.org/index.php?topic=17363.msg85741#msg85741
Exept of the performance stuff there is also an issue at all with "Web Controls" that are not configureable at all.
It's empty and can't be used. I do not have installed a license at this node so it is a "free" version in this case.
-
Hi @scream,
That was a a bug in 1.6.beta1, which got fixed. Stay tuned for the next beta.
-
New 1.6beta3:
(You need to have 1.5.2 or 1.6beta installed)
pkg add -f https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/os-sensei-1.6.beta3.txz
-
New 1.6beta3:
(You need to have 1.5.2 or 1.6beta installed)
pkg add -f https://updates.sunnyvalley.io/opnsense/updates/netmap-kernel/os-sensei-1.6.beta3.txz
Updated today from 1.6beta1 to 1.6beta3 without any issues.
Now "Web Controls" working fine and it also looks like blocking is working normally on my installation.
So after first 30 minutes I didn't see any issues exept the performance issue already reported with this test kernel and vmx interfaces.
Edit:
Performance:
vmx: 120-130 Mbit/s -> wirespeed is 1gbit/s.
ovpns: 175-213 Mbit/s -> may this is a limit of openvpn encryption.
-
Just adding info from my discussions with support to what Scream was saying. Might save a few people some steps in testing.
- We have 2 sites with latest kernel and 1.6b3, both with 1gbit down and varying upstream. One is VMX and one is igb/ix interfaces (either)
- With the new kernel and 1.6b3, all crashes are gone. It seems to run smoothly...but speed's not ideal.
- Site 1 (ix/igb): With sensei on, we get 20% up/downstream speed. Turn it off, we get 100%. With just bypass, we see about 50%.
- Site 2 (vmx): With sensei on, we get 20% downstream. Upstream is bad at the isp, so not testable. If we turn off Sensei, we get 95%.
- Both sites show about 1% cpu and 10% memory utilization. Doesn't appear to be a load problem.
Will add to the data as I learn more.
-
Here is a FreeBSD upstream discussion if someone is interested:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652
-
Here is a FreeBSD upstream discussion if someone is interested:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652
Thx. Don't know if this really applies to vmx too.
-
Here is a FreeBSD upstream discussion if someone is interested:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652
I had a troubleshooting session with a sensei engineer yesterday.
Created a test FW and two hosts on my esx to test performance between two servers passing opnsense fw.
He confirmed that the bug linked by mimugmail above seems to be the issue with vmx interfaces as well.
The test results on my testsetup looks at follows:
without sensei running at all: 856-935 Mbit/s
with sensei running in bypass mode: 225-405 Mbit/s
with sensei running in normal mode: 205.409 Mbit/s
-
Any chance of a post for a netmap kernel for 20.7.2 which includes the cyber crash fix?
@mb: Still willing and able to test PPPoE interface with netmap kernel fixes with Suricata and any test binary you may have. Thanks again for all your hard work.
-
@heresjody, our pleasure.
We'll post another test kernel based on 20.7.2 early next week. This will -possibly- have additional support for:
interfaces. We'll get back to pppoe once the first official test kernel gets shipped.
-
Any news on the 1.6 release, 1.5.2_1 with opnsense 20.7.2 mit mlx4en as lan (+several attached vlans to it) is not working. as soon as sensei is enabled all vlan traffic stops, firewall live log shows them as denied, untagged lan is still working.
sensei is running in routed mode (l3), dmesg shows that its using emulated netmap driver.
interface settings: disabled hw crc, tso, lro and hw vlan filtering.
no surricata etc running, as soon as sensei engine is stopped vlans are working again.
i reported this severals days ago via bugreport but ticket is still open.
edit if wrong subforum/thread please move.
-
Isnt it mlx5en? I did some testing with this driver and Connect X3 and it was working (without hardware offloading)
-
its a connectx3 with mlx4en driver loaded. would you mind to show enabled options? ifconfig -m mlx4enX
did you use the beta or stable sensei engine? was it using default netmap or the emulated driver?
thanks in advance
edit2:
mlxen0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
options=8c00a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
capabilities=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether f4:52:14:7a:9b:a0
inet6 fe80::f652:14ff:fe7a:9ba0%mlxen0 prefixlen 64 scopeid 0x7
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
supported media:
media autoselect
media 40Gbase-CR4 mediaopt full-duplex
media 10Gbase-CX4 mediaopt full-duplex
media 10Gbase-SR mediaopt full-duplex
media 1000baseT mediaopt full-duplex
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
also tested it with surricata in ips mode (https://docs.google.com/spreadsheets/u/0/d/1RVj8K3XOzWi-Bkjq6hUxWudu7Cxd8FFTqjLiBMzZWEM/htmlview#gid=0)
same result like active sensei, no vlan traffic/all denied.
ifconfig mlxen0 -vlanhwtso has no effect, still enabled afterwards.
mlx4_core0@pci0:2:0:0: class=0x020000 card=0x005515b3 chip=0x100315b3 rev=0x00 hdr=0x00
vendor = 'Mellanox Technologies'
device = 'MT27500 Family [ConnectX-3]'
class = network
subclass = ethernet
dev.mlx4_core.0.%parent: pci2
dev.mlx4_core.0.%pnpinfo: vendor=0x15b3 device=0x1003 subvendor=0x15b3 subdevice=0x0055 class=0x020000
dev.mlx4_core.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PCI0.PEG2.PEGP
dev.mlx4_core.0.%driver: mlx4_core
dev.mlx4_core.0.%desc: Mellanox driver (3.5.1)
dev.mlx4_core.%parent:
edit3: maybe related?
https://redmine.pfsense.org/issues/10836
-
After upgrading to 20.7.3 you can update to the latest netmap testing kernel with:
opnsense-update -kr 20.7.3-netmap
-
Hi @mr.x.y
Can you send the ifconfig mlxen0 command before and after you run below command:
ifconfig mlxen0 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
-
Can we continue this thread from here: https://forum.opnsense.org/index.php?topic=19175.new#new . This is more up to date I guess.
-
now I have managed to install to 20.7.3-netmap kernel.
after that I logged in on the web gui and did a "check for updates" - and this one shows me that there is an update for the kernel --> current version 20.7.3-netmap vs new version 20.7.3
is it possible that this is because of the different date/time when performing an "uname -a"?
the stock 20.7.3 kernel show something with "Mon Sep 21 16:xx:xx" while the 20.7.3-netmap kernel shows "Mon Sep 21 13:50:27"
the last time I tried to use sensei with the new netmap test kernel - it did not work out good - maybe because I performed an update after installing the -netmap kernel?!?
-
This is correct. Opnsense notices a different kernel than the 20.7.3 kernel so it sees it as an “update”.
-
thanks... so even if I lock the kernel in the packages tab and perform a check for updates, it still insists on updating the kernel.
besides that, does it decide to update the kernel based on "I am different than stock" or "I am older than the stock"?
-
Different than stock.
-
For those who
- are using Suricata/Sensei on VLANs on em(4)
- experiencing vmx crash
- experiencing vtnet crash
- want to have Suricata on PPPoE / OpenVPN interfaces
I have an (unofficial) test kernel ready. Please PM me if you'd like to give it a try.
PS: vmx patch fixes the kernel crash, but has an outstanding issue. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248494
Hi there is it still possible to get/test suricata with PPPoE WAN?
-
Hi @allebone,
tun(4) support is now in 20.7.4. It seems it's not working as expected for PPPoE. We're on it.
-
Hi @allebone,
tun(4) support is now in 20.7.4. It seems it's not working as expected for PPPoE. We're on it.
Thank you :)
@bunchofreeds
-
Thanks @allebone and @mb
Just had to ask in the right place :)
-
Hello @mb,
are there still issues with netmap on lagg interfaces with vlans on current 20.7.4 kernel? I'm using Sensei on a lagg interface with vlan child interfaces on igb nics. According to the spreadsheet this is a "PASS". However, from time to time my firewall becomes completely unreachable from the inside. This also happens sometimes when opening Sensei configuration pages. Often the situation stabilizes itself after a few minutes but sometimes I need to restart the whole firewall or unplug lagg members until it becomes reachable again. If in working state the issue can be observed by unplugging lagg member interfaces and starting to generate traffic while running a ping. The firewall becomes unreachable after a few seconds. If I disable Sensei this behaviour can't be observed - everything seems to work as expected from an lacp configuration.
Thanks!
-
I configured Sensei to use the physical interfaces (igb) in the lagg now. Seems to work a lot better so far! Unplugging the cables doesn't cause any unavailability anymore.
-
There is a new test kernel available, mostly for ix/ixl performance issues in iflib/netmap:
# opnsense-update -kr 20.7.4-next
# opnsense-shell reboot
You can see the relevant batch of new commits on the master branch[2].
Cheers,
Franco
[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652
[2] https://github.com/opnsense/src/commits/master
-
New kernel (20.7.4-next) helps ix 10G cards but some very strange results, seems like each thread has a cap but does parallel process much better.
20.7.4
IDS off (CPU shows around 5%)
Send
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 3.41 GBytes 5.85 Gbits/sec 0
Receive (-R)
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.14 sec 4.42 GBytes 3.74 Gbits/sec 0
IDS On - Hyperscan (CPU 40 - 50%)
Send
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 742 MBytes 1.25 Gbits/sec 0
Receive (-R)
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.14 sec 455 MBytes 742 Mbits/sec 0
20.7.4-next
IDS off (CPU shows around 30%)
Send
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.17 GBytes 4.44 Gbits/sec 0
Receive (-R)
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.13 sec 2.45 GBytes 2.08 Gbits/sec 0
IDS On - Hyperscan (CPU 60 - 80%)
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 16.1 GBytes 2.30 Gbits/sec
Receive (-R)
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.13 sec 2.43 GBytes 2.06 Gbits/sec 0
But running iperf3 in parallel mode to simulate real multiple streams, it really picks up performance. In previous kernel sum of parallel was about the same as 1 stream.
20.7.4
IDS On (CPU shows around 80%)
Send + Parallel 10 (-P 10)
[ ID] Interval Transfer Bitrate Retr
[SUM] 0.00-10.13 sec 7.79 GBytes 6.61 Gbits/sec
Receive (-R) + Parallel 10 (-P 10)
Almost same results for Receive and only about 10% less than with IDS Off.
Overall results with this kernel on an ix 10G interface
1) Double speed increase with netmap enabled
2) Multiple connections increase total throughput
3) New kernel with ix interface - CPU spikes way up both with and without netmap. Note, if network throughput is under 1g, then CPU stays around 5%, only when going over 1g does the CPU start to take a hit.
-
Thanks for taking the time to crunch some numbers here! :)
The patch is an "iflib" patch which means it also changes non-netmap behaviour and the changes in bulk handling for ix(l) certainly require further tweaking as per the comments in FreeBSDs bug tracker, but this is good enough to merge and then wait for more patches on the subject to land in the tree.
I asked for additional results by Sunny Valley, particularly on re(4) and vmxnet3(4). And either Murat or me will follow up here in a bit.
Cheers,
Franco
-
Not going to post all the detail stats but the 20.7.6 updated kernel performs slightly less than the 20.7.4-next kernel on the ix 10G cards. I am getting about 1.5gbs.
Please let me know when an updated kernel is available, but I know you are awaiting the BSD code to drop.
-
Expected. 20.7.6 does not include ix(l) specific patching from 20.7.4-next.
Cheers,
Franco
-
Hi Franco, does 20.7.4-next work with the 20.7.6 release? If so I can install it? But as per the other thread (no traffic monitoring), it's both 20.7.4-next and 20.7.6. Traffic works fine with 20.7.4 base kernel.
One item I have noticed since 20.7, ix(i) and vmxnet take a lot of cpu in heavy loads as compared to 20.1. But most of that is all the Harden BSD 12 code and not opnsense.
-
Kernels in the major version category always work interchangeably. But to be sure the latest code is tested I'm going to create a 20.7.6-next variant. Maybe after 21.1 is out we should work on a -devel type kernel track to avoid confusion and always upgrade to the new kernel if it is available.
Cheers,
Franco
-
20.7.6-next is now available both in kernel (-k) and base set (-b).
Cheers,
Franco
-
Is there an update on pppoe support at all?
-
Not that I know of, no.
Cheers,
Franco
-
I'm having some issues with LAGG on my LAN connection. It will just crash my LAN once I finish the install/wizard. I have searched this thread and a user has mentioned that instead of using the LAGG as the interface they used the 2 ports. In my case this would be ix0/1 connecting to my cisco switch.
I have 2 questions about doing this:
1, Doing what the user has done, would it be more taxing on the box and would it even work for protecting the network?
2, Do we have some kind of ETA as to when the next netmap will support LAGG interfaces?
For the moment I re-installed my opnsense and restored from a backup from before I installed Sensei since I was not even able to access my box from the LAN. I will be waiting before installing this again.
Thank you,
Jonathan
-
@franco
I am no longer able to downgrade to a fixed kernel on 2.7.8, I just upgraded from 2.7.5 to 2.7.8 and now getting this error message when trying to get the fixed kernel in place.
root@cerberus:~ # opnsense-update -kr 20.7.6-next
Fetching kernel-20.7.6-next-amd64.txz: .. failed, no signature found
root@cerberus:~ # opnsense-update -kr 20.7.4-next
Fetching kernel-20.7.4-next-amd64.txz: .. failed, no signature found
Thanks,
Tim
There is a new test kernel available, mostly for ix/ixl performance issues in iflib/netmap:
# opnsense-update -kr 20.7.4-next
# opnsense-shell reboot
You can see the relevant batch of new commits on the master branch[2].
Cheers,
Franco
[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652
[2] https://github.com/opnsense/src/commits/master
-
The next kernels were temporary kernels that largely resemble 21.1 kernels. There is no reason to keep them around.
Cheers,
Franco
-
Can I upgrade the kernel on 20.7 to 21.1 to get the netmap and intel driver fixes without upgrading the OS to 21.1?
-
I will need to force an upgrade to 21.1 it seems to get the fixed kernel.
root@cerberus:~ # opnsense-update -zkr 21.1
Fetching kernel-21.1-amd64.txz: ... failed, no signature found
root@cerberus:~ # opnsense-update -kr 21.1
Fetching kernel-21.1-amd64.txz: ............ failed, signature invalid
-
Any other version than 20.7.8_4 does not have the signature fingerprint for 21.1.. I'm not sure why you are not on this version. ;)
There is the "-i" parameter to do insecure upgrades, but it's nicer to not use it and use the latest 20.7 release with:
# opnsense-update -kr 21.1
# opnsense-shell reboot
Cheers,
Franco
-
I am running the business version of opnsense, I recently did the upgrade from 20.7.5 to 20.7.8 and allowed the kernel to get upgraded, after the upgrade my setup was running at a capped speed with IDS/IPS enabled. Having the -next kernel branch available would have saved me another upgrade to 21.1 tonight.
Any other version than 20.7.8_4 does not have the signature fingerprint for 21.1.. I'm not sure why you are not on this version. ;)
There is the "-i" parameter to do insecure upgrades, but it's nicer to not use it and use the latest 20.7 release with:
# opnsense-update -kr 21.1
# opnsense-shell reboot
Cheers,
Franco