Open Source DPI Engine - Netify Agent

[pbaldwin]

Hi OPNsense Community,

We're a Canadian company that has developed two bits of technology that can help with managing networks at the edge of the Internet.

• An open source deep packet inspection (DPI) engine that is ready to be integrated with the OPNsense firewall, QoS engines and other tidbits.
• A cloud-based network intelligence and visibility solution.  The solution provides full transparency into what’s happening on your network (screenshots).

These are really two different topics, so we have separated the announcements into different forum posts.

This one is about the first topic - the open source DPI engine.

Open Source DPI Engine

We merged the best parts of the open source l7-filter & nDPI projects, and then added our own innovations to create a new deep packet inspection (DPI) engine: netifyd.   The engine is able to extract useful metadata from a network conversation:

• Application (e.g. Netflix)
• Protocol (e.g. BitTorrent)
• SSL cipher, SNI, certificate common name
• Hostnames
• Agent strings
• DHCP fingerprints
• Network and packet information
• and more

The software only analyzes the first few packets in a network conversation and then publishes the metadata to a JSON-encoded data stream.  We’re not interested in the payload, just the metadata.  For example, the screenshot below is part of a stream of information coming from an Avast client on a Windows 7 desktop making an HTTPS/TLS 1.2 (version: 0x0303) connection to an Avast server (wildcard SSL certificate: *.avast.com”):



If you are curious, you can see another example of the data stream in our developer documentation - https://www.netify.ai/developer/netify-dpi-engine/netifyd.

So, What’s Next?

Okay, that’s nice… a stream of network information.  So, what’s next?

Third party tools in OPNsense can connect to this data stream and do all sorts of different things - firewalling, QoS, reporting, etc.  For example, in pfSense (gasp!), a netifyd-based app was created to block applications (e.g. Facebook) and protocols (e.g. BitTorrent). The same feature could be developed for OPNsense of course.  Or, the DPI data could be hooked into the QoS engine (even better).  In fact, we have another open source package called Netify Firewall Agent that provides the groundwork for making this type of feature.

Please let us know if you are interested in further development.  We’re happy to kickstart the process if someone is interested in running with it!

Kicking the Tires

If you want to start playing around with the network data flows, you can install netifyd on OPNsense today -- follow the Netify on OPNsense installation documentation. Note: you can skip the “netifyd --enable-sink” step -- that’s for enabling the cloud-based Netify product (see other forum thread) and not needed to run the DPI engine.  And ditto for skipping provisioning -- it’s not needed.

Once netifyd has been started, you can connect to the data stream on the Unix socket: /var/run/netifyd/netifyd.sock.  For direct command line interaction hackers, you can view the data stream in your console.  First, install the command line jq (JSON processor) tool in OPNsense 19.7.x or 20.1.x with:

Code: [Select]

pkg add http://pkg.freebsd.org/freebsd:11:x86:64/release_3/All/oniguruma-6.9.0.txz
pkg add http://pkg.freebsd.org/freebsd:11:x86:64/release_3/All/jq-1.6.txz

You can watch the JSON payloads go by on your terminal using netcat and jq:

Code: [Select]

nc -U /var/run/netifyd/netifyd.sock | jq .

It should look something like the screenshot above.

The Future

That’s it for now.  We’ll post updates for the netifyd engine from time to time.

Please only post responses to the open source DPI engine here.  Questions/comments about the cloud-based Netify solution should be posted in the other forum thread.

[athurdent]

Netify FWA for pfSense looks really cool, are you planning on porting this to OPNsense?
I‘m not comfortable with sending insight into my user’s traffic to the cloud, but a local version like FWA for application visibility would be terrific!

[Antaris]

I like to see and try another NGFW complex solution, especially with integrated QOS, so - go on

[pbaldwin]

Yes, if there's a good level of interest, we would port the Netify FWA app to OPNsense.  The first thing we're doing is porting the Netify Console tool (screenshot) to Python.   

[Antaris]

Most of us that runs OPNsense and have such needs using Sensei from SunnyValley Networks now. I'm not sure they can work simultaneously...

[athurdent]

Yes, if there's a good level of interest, we would port the Netify FWA app to OPNsense.  The first thing we're doing is porting the Netify Console tool (screenshot) to Python.

Really looking forward to this, did you find time to port it, yet?

[pbaldwin]

Not yet... sorry!

[athurdent]

Not yet... sorry!

Still interested in this :-)

Have you considered offering a „home“ subscription for Netify, for the typical family home admin that just wants some insight into what their kids are doing?

[Anael]

I'm too interrested to know about the question above 

[Georges]

Just check this : https://www.netify.ai/pricing

[RedeyeAce]

I sent them an email a couple of times, and they never responded.

It's baked into ClearOs and literally the very first thing in the menu. My concerns were that the cost for a home user at that time was $480 per year.

It seems to be $660 for the same thing now, neither of which is appealing to a home user. Given the inspection engine is free, I would of paid a small fee to host it my end. I don't know many home users that would pay $660 to provide their data and get fancy pictures from their website.

Shame as it looked good, tis also a shame that they don't answer people's questions.

[athurdent]

Just use Sensei, it's awesome!

[raje0000]

Hello everyone,

I'm encountering an issue while attempting to install a package on my FreeBSD 13.2-RELEASE-p3 system. Despite my system being FreeBSD 13.2, when I try to install a package using the following.

I receive the following error message:
Failed to install the following 1 package(s): http://download.netify.ai/netify/freebsd/12.1/stable/netifyd-3.03_1.txz
pkg: wrong architecture: FreeBSD:12:amd64 instead of FreeBSD:13:amd64

It seems that the package is built for FreeBSD 12.1 and not compatible with my FreeBSD 13.2 system. Could someone please advise me on how to install or run this package on FreeBSD 13.2? Additionally, when attempting to execute any command from this package, I encounter further error messages.

Any guidance on resolving this issue would be greatly appreciated.

[Patrick M. Hausen]

Try

Code: [Select]

IGNORE_OSVERSION=yes pkg add ...or

Code: [Select]

setenv IGNORE_OSVERSION yes; pkg add ...depending on which shell you use.

This wil get it installed but possibly it won't run. FreeBSD 12 is long EOL. That company must provide an updated package.

[athurdent]

Looks like they abandoned everything but Debian/Ubuntu.
Just use Zenarmor, it's awesome and the basic version is free.