Sensei on OPNsense - Application based filtering

[mb]

Hi @rene, looks like they ended up in the spam box. I have them right now. Thanks.

We'll be incorporating the suggested change with the next upcoming release (1.0.3)

[sol]

quick questions:

I cannot see a feature to resolve local hostnames in reports.
"show hostnames" does not show me names, just ips.
In Reporting / Insights opnsense will show names when using reverse lookup.
Do I miss a setting for this? Or is this not implented yet?
All local users have static ips with

Furthermore is there any way to show in a simple report how long a local ip has used the internet each day; e.g. a chart / graphic ip online from 2pm till 4 pm on Monday, online 5pm till 8pm on a Sunday or something

Cheers

[mb]

Hi @sol,

I cannot see a feature to resolve local hostnames in reports.
...
All local users have static ips with

Sensei does an in-flight enrichment of ip addresses with hostnames when it sees a related DNS transaction. Or, in the case of local nodes, Sensei also keeps track of MDNS messages for this purpose.

If the IP addresses are not resolved to hostnames, my first guess would be that you're running a local DNS server and most of the DNS messages are transported without Sensei in the scenes.

We also do not do an in-flight explicit DNS call for IP address resolution because of performance reasons.

What we can do is during reports viewing, we could try to resolve the IP address, when you have your mouse on one of them in the charts or grid reports. Actually this is what we do for remote addresses currently, we can do the same for local addresses if we see that it's not resolved beforehand.

Would that work?

Furthermore is there any way to show in a simple report how long a local ip has used the internet each day; e.g. a chart / graphic ip online from 2pm till 4 pm on Monday, online 5pm till 8pm on a Sunday or something

Not yet. In the roadmap  ;)

[sol]

Thank you mb.
I use unbound.
But only 1 local ip shows the hostname - even when I do not hover over it. See attachment.



Looking forward to the update on "online time". Will it be included in the free version?

[Wyrm]

I have installed opnsense 19.7.1 and installed sensei by guide on web.
In installation in SSH was all ok and success. In web gui all the settings were ok and after finishing and refreshing it says in status the service is not running. I correctly selected interfaces and all the settings.
When I click on start of service it says it does not have selected any interfaces, but they were selected in configuration!
HW is quad-core Xeon and 8GB RAM. It is VMWARE ESXI 6.7 virtual, but it should work.
I have also upgraded to actual production version which is 19.7.3

I have another installation where is opnsense 19.1 and it is running well.

Could you help me what is wrong ?

There is status screenshot included

Thanks very much

[mb]

Hi @sol,

This is most probably since Sensei was able to spot a dns transaction and get a hint for that IP. We'll introduce lookup of local IP's in the coming release (1.0.3).

We haven't yet thought about the edition of "online time" reporting.

As for @Wyrm's issue, it turned out that two python dependencies did not get installed although they are configured as the plugin's dependencies and the packages are available in the OPNsense LibreSSL package repository.

We couln't reproduce this in our lab.

Are there any other LibreSSL users  experiencing the same problem?

[BeNe]

Hi!
I have an error in my Logfile - every minute.
The strange thing is -> Sensei is complete disabled - but there are still jobs running ?!

There is also one with an Error:

Code Select Expand


Aug 29 12:46:00 configd.py: [5413e5ea-0d25-4052-8b5f-8d2a1f09b02b] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 484, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1.
Aug 29 12:46:00 configd.py: [5413e5ea-0d25-4052-8b5f-8d2a1f09b02b] captive login logout enrich
Aug 29 12:46:00 configd.py: [c12694fb-94c0-434c-8723-fefad2299514] check sensei engine health
Aug 29 12:46:00 configd.py: [c0c97d1e-9572-4363-9944-503805f19016] Runing periodical scripts
Aug 29 12:45:27 configd.py: [b1408ad6-4305-45ba-99aa-89785b7e1d38] view license
Aug 29 12:45:06 configd.py: [656dcab2-ba0a-4284-8bda-4eb63b4379e3] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 484, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1.
Aug 29 12:45:05 configd.py: [656dcab2-ba0a-4284-8bda-4eb63b4379e3] captive login logout enrich
Aug 29 12:45:00 configd.py: [6826b4d8-a469-4409-a06f-f9e2bae21679] check sensei engine health
Aug 29 12:45:00 configd.py: [0128a6ba-9005-4456-831c-8d5da47a1362] Runing periodical scripts
Aug 29 12:45:00 configd.py: [d9b4c8b8-6ffa-4a65-bbfc-1586848bc494] check sensei engine health
Aug 29 12:44:51 configd.py: [dfb2ad02-35ea-407e-839d-2c789acbd715] control services
Aug 29 12:44:29 configd.py: [a752df4d-1f04-4295-9e52-3aba5ddd37ea] check sensei updates
Aug 29 12:44:29 configd.py: [edbf53e3-085a-40ad-ab35-be0bcbccf271] view elasticsearch disk size
Aug 29 12:44:29 configd.py: [66a74d51-8631-4897-b52f-82e6d6cfebc6] control services
Aug 29 12:44:29 configd.py: [a76246b9-cbc1-40ac-816c-1cb8a6ffc2d8] check sensei ui version
Aug 29 12:44:29 configd.py: [2977d7e6-1d94-483f-9df6-3454b38f623c] check sensei db last modified
Aug 29 12:44:29 configd.py: [05bccd05-3e71-45fa-bb7f-79c365d8b60c] check sensei db version
Aug 29 12:44:29 configd.py: [275abcbd-a41b-4a55-aa04-b855946124fe] check sensei db last modified
Aug 29 12:44:29 configd.py: [cb42810a-74a8-4b3c-a5b3-30a06fbfbec4] check sensei db version
Aug 29 12:44:29 configd.py: [c636a48c-393a-4fcc-9ec8-821475effd62] check sensei last modified
Aug 29 12:44:29 configd.py: [6606bf25-295f-49d9-974c-3c45551f7d03] check sensei version
Aug 29 12:44:29 configd.py: [f66b94cc-138d-4a33-9d61-f0623205cd8f] control services
Aug 29 12:44:26 configd.py: [ebaf16ea-7086-4663-9e93-41268042a8a8] view elasticsearch disk size
Aug 29 12:44:26 configd.py: [b6248966-ac6d-4c33-ae11-86f3ef503415] control services
Aug 29 12:44:26 configd.py: [9b585355-19fd-4cfb-85a1-6a216f5ed7a1] check sensei ui version
Aug 29 12:44:26 configd.py: [d9b79260-5dfb-4b8f-b3e0-c69fe24d91ff] check sensei db last modified
Aug 29 12:44:26 configd.py: [bd339ddb-6073-407f-a17e-8318214e5b21] check sensei db version
Aug 29 12:44:26 configd.py: [77e95c98-9e7a-4186-8793-740dd19a654a] check sensei db last modified
Aug 29 12:44:26 configd.py: [9e789111-39b2-41b9-b85c-d4b00a42e771] check sensei db version
Aug 29 12:44:26 configd.py: [eaa3f74c-bb21-41a1-a7ed-678bbe16124c] check sensei last modified
Aug 29 12:44:26 configd.py: [4d463eb5-95d6-4437-a9c2-02326b8efdec] check sensei version
Aug 29 12:44:26 configd.py: [edc50189-fd8f-4e08-ad76-bb2843227fc3] control services
Aug 29 12:44:24 configd.py: [63f5b4df-30a0-4678-a0f2-a9e577bba2ed] check sensei updates
Aug 29 12:44:23 configd.py: [83b1e0cc-8cd6-42a0-a08f-d8ba551a4814] check hardware
Aug 29 12:44:22 configd.py: [061a0e97-d2ef-4859-885d-d80f82fb9b39] view license
Aug 29 12:44:00 configd.py: [af175a5c-bee8-4eab-93c2-d80969cbc6ff] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 484, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1.
Aug 29 12:44:00 configd.py: [af175a5c-bee8-4eab-93c2-d80969cbc6ff] captive login logout enrich
Aug 29 12:44:00 configd.py: [c043869a-d6ec-4a5e-9ed0-939262d08cce] check sensei engine health
Aug 29 12:44:00 configd.py: [e408fbac-3585-451f-97d6-0c8f02978f23] Runing periodical scripts
Aug 29 12:43:54 configd.py: [eede6a57-4704-4642-9e90-4337e9e4526e] request pfctl byte/packet counters
Aug 29 12:43:49 configd.py: [2baa7185-8ae9-4127-ab7c-9886ef7d10c8] request pfctl byte/packet counters
Aug 29 12:43:43 configd.py: [54f33596-62e2-43ec-89bd-3e1e809db62c] request pfctl byte/packet counters
Aug 29 12:43:00 configd.py: [f92788bb-fd0c-4177-a4f1-ad1f6568d204] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 484, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1.
Aug 29 12:43:00 configd.py: [f92788bb-fd0c-4177-a4f1-ad1f6568d204] captive login logout enrich
Aug 29 12:43:00 configd.py: [5c7c03be-071f-4914-b050-7895ce71974a] check sensei engine health
Aug 29 12:43:00 configd.py: [894347ec-50c4-4de6-85a3-3ef60b32c32b] Runing periodical scripts
Aug 29 12:42:00 configd.py: [735dad9a-a836-4f62-a8db-aaac917ea1bb] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 484, in execute stdout=output_stream, stderr=error_stream) File "/usr/local/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/Sensei/userenrich.py' returned non-zero exit status 1.
Aug 29 12:42:00 configd.py: [735dad9a-a836-4f62-a8db-aaac917ea1bb] captive login logout enrich
Aug 29 12:42:00 configd.py: [e83fd212-4ac4-4da3-9347-a964882163b7] check sensei engine health
Aug 29 12:42:00 configd.py: [9b10878e-3fe5-4acf-9424-2c11e29a533e] Runing periodical scripts

Searched in the Forum, but threre was not hit with userenrich.py. Does anyone else have the same errors ?

My Versions:
Engine Version:   1.0.2
App DB Version:   1.0.3
Rules DB Version:   1.0.3

Versions   OPNsense 19.7.2-amd64
FreeBSD 11.2-RELEASE-p12-HBSD
LibreSSL 2.9.2

Thanks!

[mb]

Hi @BeNe,

Batch jobs like userencricher (health check, updates check) continue to run in the background if you have Sensei installed. Stopping the packet engine just stops packet processing. Elasticsearch and background bookkeeping jobs will continue to run.

The duty of the Userenricher is to feed captive portal user/group information to Sensei so that it can map the ip addresses to users/groups.

In your case, you do not have Captive Portal enabled and this triggered this error (indeed a test code which tests this case),

Fixed as of now and for 1.0.3. Many thanks for reporting this.

[BeNe]

Thanks for you quick reply and the fix in Version 1.0.3
The Status e-Mail is also sent out if Sensei is disabled (packet engine and elasticsearch)

[mb]

Dear Sensei users,

We are aware of an issue affecting LibreSSL users. A few package dependencies, which are important for the operation of the plugin, do not get installed. This results in initial configuration being not written into configuration files.

As a workaround, for now, we advise that you install the dependencies manually:

Code Select Expand

pkg install py27-dnspython
pkg install py27-Jinja2
pkg install py27-sqlite3
pkg install os-sensei-updater


We'll issue the fix with 1.0.3.

[donald24]

Hi,

I have a problem after having upgraded to 19.7.3.

My configuration is still fairly out of the box, my LAN-side is using two separate VLANs next to its untagged main-traffic. I got notification, that my telephone is dead, my VOIP-vlan was not letting packets to the inside. I checked the VOIP-VLAN and no traffic was going to the internet. LAN was okay. I rebooted the firewall and afterwards I could not reach the firewall even from LAN-area anymore.
So I needed to hook the machine to a monitor and ran the uninstall steps, I have found in this thread:

Code Select Expand

service eastpect onestop
service elasticsearch onestop
pkg delete elasticsearch5
pkg delete os-sensei
rm -rf /var/db/elasticsearch/nodes/*

Though I remember that one pkg wasnt found, might be another name, but afterwards I had immediate access and running internet to all interfaces.

Is there still something missing for uninstallation? The configuration files is also having a lot of sensei parts in it, would I have to reinstall sensei, to run its uninstallation from the GUI, or is there even a manual way?

Thanks!

[mb]

Hi Donald,

Here are the manual steps to be able to remove Sensei from the system:

Code Select Expand

# service eastpect onestop
# pkg remove elasticsearch5
# pkg autoremove -y
# rm -rf /usr/local/sensei/
# rm -rf /var/db/elasticsearch/nodes/


On the other hand, I'm very much curious about what went wrong there. I'll be reaching out to you to see if we can have a look at your system together.

[tusc]

I also experienced the same situation as donald24 under 19.7.3. I lost complete access to the firewall and the Internet after running through the wizard. I had to stop service and uninstall the packages to reinstate connectivity.

I only have 4GB of ram on my OPNsense server so assumed I'm running into something related to that.

[mb]

Hi @tusc,

Your case looks more like you have a netmap-incompatible ethernet device. Let's have a look at your system together.

[tusc]

Really? I'm using a quad port Intel GigE card so wasn't aware this was netmap-incompatible:

Code Select Expand


root@OPNsense:~ # dmesg |egrep igb
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> mem 0xfe880000-0xfe8fffff,0xfe90c000-0xfe90ffff irq 27 at device 0.0 on pci1
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: xx:xx:xx:xx:xx:xx:xx
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: netmap queues/slots: TX 4/2048, RX 4/2048