New Plugin: NMAP Scanner

[MasterXBKC]

See the screenshots, its up and working, just waiting for it to get pulled into the repos.

[fabian]

You should move it to Interfaces -> Diagnostics and use this icon: https://fontawesome.com/icons/search?style=solid (for version 4 it is fa and not fas in the class)

NMAP has also XML output, which may be used to display it integrated in the GUI (requires parsing in the backend).

[fabian]

Very likely not going to be merged as it looks now.

[MasterXBKC]

So what your really saying is, im wasting my time trying to be helpfull, and that the hours ive spent putting both of these plugins togeather, would have been better and more productively spent drinking?

AND, for the record, i do not know how to set up the plugin using the non-static page stuff.   If i did i would have, so instead i modeled after and already existing static paged plugin, thats already in there.

[MasterXBKC]

here is a thought, since im apparently a moron here in how to setup proper, non static plugins, if someone wants to be so gracefull and helpfull, and either show me how, or make the needed changes to the nmap plugin i wrote and send me the modified version back so i can disect it and compare the differences and see how to do it for myself, then i might be able to be more usefull.

As for the pfmonitor plugin, its simple, you dont have a shell command function in the web ui, thus i cant print easy instructions for my users to install my plugin for monitoring their systems, without making them break down into ssh and such, when for pfsense installs it takes literally 5 seconds.   its a large handicap for people using my platform if they rightfully decide to use opnsense instead of pfsense.    It makes their and my job a lot harder, and if they could simple hit a plugin sign under plugins and be done, it would make so much of their jobs a lot easier.

I have 500+ firewalls from clients reporting to my platform, and adding more each month, so what your asking me, is to ask them, to ssh manually into each one, and using the shell install it manually.    which means i spend more time doing tech support for the ones who cannot figure it out themselves, and they get more frustrated with opnsense for this issue, and despite the JIM issue, because of this, they see a clear advantage in pfsense in that it takes so much less time and effort to get my plugin loaded.

I am one guy, i DO NOT have any staff under my pfmonitor project, no one.   I do all the tech support myself over email and phone, and right now 90% of the calls are, i cant get my opnsense unit to report in.   meaning they couldnt get the plugin loaded themselves.   Thankfully the operations of the platform have been built to be 90% autonomous and self maintaining, else i would have no free time at all.

Now im sorry for the frustration expressed in these posts, but here it is in my prospective, ive been trying to get my plugin in with you guys for months, Franco even provided generous help and a skeleton for the checkin plugin and everything, awsome on him for that.    I have been staying up late at night getting it all working and tested.   and working on it in lieu of time i usually use for sleep.   And this morning come to find all of it rejected....

[MasterXBKC]

and sorry for the early morning animosity, i havent got any coffee yet.

[MasterXBKC]

Very likely not going to be merged as it looks now.

Fabian i read thru yours comments on the code in github, and heres some thoughts of my own:
For your claim of my static colors breaking themes, i only colored a few individual words from results output for clarity, the pages retain the original theming from the plugin i borrowed code from.

For translations, i only speak english so i cant help you there.

I didnt see a reason to color on a host being down, since no usable information is actually output.

You seem to like your escapeshellarg, but if you actually test my preg replace regex code, its setup so you can use dns names, IPv4, IPv6, as well as hyphens, and slashes for use in CIDRs.   and to dump any other characters that do not belong.   Think of it like a character white list.

Then the section of CASE statements, you said were all the same make them dry....
There not all the same, they have minor differences in their layout and output vars.  Similar is not identical.

The no smart devices part, was a snipped i apparently missed trimming off, or else copied in without realizing, and it just needs AXEd.

REQUEST vs POST and GET, i didnt want $_COOKIE data getting lumped in too, i code from a standpoint of letting in as little information as possible, for security.   if i dont need data coming in all three holes, why even open all three holes???(GET, POST, COOKIE)

I took the stupidly basic smart plugin, and used it to make an nmap plugin, if you didnt gather that already.

As for the pfmonitor code being obfuscated, i have sent its original source to both franco and adsch, and i can even provide the obfuscator i used so the code can be verified.    HELL, if its such a big issue ill just upload the un-obfuscated version if it will save the headaches im trying to save by having that plugin available for one click install for my clients, which would save hours and hours for me playing tech support for users who are not familiar with ssh and linux/unix trying to install my plugin manually.

The pfmonitor plugin does send information to my server, cpu model, cpu usage, ram usage, load, and other stuff like this, it is for remote monitoring.   And i DO NOT want to have people openning their web interfaces to the outside world needlessly exposing them to brute forcing, when this plugin can just post it to my server, which does not need any open ports, or access to the opnsense api, its safer.

[MasterXBKC]

i fixed all the pertinent issues in the nmap plugin, not counting its staticness, and re-uploaded.

[fabian]

Here is the documentation via an example how to create a MVC plugin:
https://docs.opnsense.org/development/examples/helloworld.html

Plugins similar to yours are the iperf plugin (please note that a process management service may be too much for an nmap plugin) and the arp-scan plugin.

Put the HTML into the view (volt file) and the worker code into the API controller. Please note that highlighting must be done via JS in the browser since the API should not return HTML (either parsed and therefore structured data - the preferred way - or as an alternative the raw output as a single string). Colors must NOT be set by yourself but by the theme using Bootstrap classes (for example success for green and danger for red in the default theme).

[elektroinside]

Uuu, very nice, nmap! Can't wait to see this integrated :)
Thank you for your efforts and patience!

[franco]

So what your really saying is, im wasting my time trying to be helpfull, and that the hours ive spent putting both of these plugins togeather, would have been better and more productively spent drinking?

No, we have standards and best practices we need to adhere to. They have been established and refined over the past three years and now we are trying to stick to them while managing a growing community. They can change again, but for now it boils down to:

1. We want APIs and static pages don't offer them (regarding NMAP)
2. We want open source and obfuscated code is not going into official plugins shipped to our users (regarding pfMonitor)

AND, for the record, i do not know how to set up the plugin using the non-static page stuff.   If i did i would have, so instead i modeled after and already existing static paged plugin, thats already in there.

It's not your fault. It's our fault for not documenting everything and quickly explaining everyone about this complex ecosystem that we have inherited and extended. Now we're simply communicating what you did not know in order to bring you on the same level to be able to work out solutions together.

I'm more than happy to reiterate what I said over the years and hopefully it has been consistent. Time will change those established policies, but not against the wishes of a community we built upon the values of openness in code visibility, seamless tool chains and embracing future-proof technical design.

My greatest fear is that shortcuts will not lead to long-term solutions and put unnecessary pressure on the people that write OPNsense, which in turn makes the community less happy than it could be.


Cheers,
Franco

[MasterXBKC]

If you will agree for my pfmonitor plugin to pull, i will happily go in right now, and re-publish the code, with zero obfuscation.

Is this acceptable?  It is the same code i sent you before, but with the update function removed, as the package manager now handles that, and with 2-3 lines fixed for compatibility.  other than that its 100% the same skeleton you sent me before.

And while im at it, do i need to re-fork, so i can do a pull request just of pfmonitor, without the nmap, remember im not a git expert.

Also, im not sure what use the api will have for nmap functionality....

Separate question here:
Also, for the api itself, are you expecting everyone who wants to use it, to expose their web interfaces to the open internet?

[franco]

If you will agree for my pfmonitor plugin to pull, i will happily go in right now, and re-publish the code, with zero obfuscation.

One step in the right direction, but reviewing and merging it will help mostly yourself. I'm being reluctant and you know this. But I have helped as time permits and will help again when time at my day job is not taking most of my day.

Is this acceptable?  It is the same code i sent you before, but with the update function removed, as the package manager now handles that, and with 2-3 lines fixed for compatibility.  other than that its 100% the same skeleton you sent me before.

No, because it raises the review and beneficiary question mentioned above. I don't want to merge something we don't have a policy for. It's not a "no" for a merge, it's a "no, we really don't know" and from the core team we don't appreciate being pushed into this position instead of naturally easing into it by asking the right questions and discussing it with the community. If the community is silent, that then only means a decision must be made in the best interest, not necessarily in the voiced interest.

And while im at it, do i need to re-fork, so i can do a pull request just of pfmonitor, without the nmap, remember im not a git expert.

You can create separate branches for both plugins. Since they reside in separate directories, that won't be a problem.

Also, im not sure what use the api will have for nmap functionality....

Why are you not sure? Can you explain?

Separate question here:
Also, for the api itself, are you expecting everyone who wants to use it, to expose their web interfaces to the open internet?

No, it's for local scripting and UI flexibility. You don't like the current way the UI does things? Build a better UI on top of the API as a plugin, or combine them.... scan via nmap, do reverse DNS lookups, write firewall rules.. all in one plugin... eventually. Possibilities are endless, practially only limited by imagination.  ;)


Cheers,
Franco

[MasterXBKC]

If you will agree for my pfmonitor plugin to pull, i will happily go in right now, and re-publish the code, with zero obfuscation.

One step in the right direction, but reviewing and merging it will help mostly yourself. I'm being reluctant and you know this. But I have helped as time permits and will help again when time at my day job is not taking most of my day.

Yes it will help me largely, but i have 78 users on my platform, whom it would also benefit immensely, as a number of them, like myself, are very fed up with pfsense support price hikes, hardware price hikes, and their elimination of their 2220 cheaper end units, etc, im sure other reasons as well.   They want to move over to opnsense with a lot of their units, but do not want to give up the remote management and monitoring, and the reporting that my pfmonitor platform offers.    As of now the process to install my plugin for it is cumbersome, and requires ssh, etc.    This would ease their migration path greatly, by reducing the amount of time it takes to get the plugin loaded.   I myself work for a MSP and we have already been re-loading our pfSense devices with opnsense as we get onsite to them at their various sites.   That company which i work for would also be among those benefitted, including the other staff of said company.

If not for getting the plugin loaded into the repos, at the least a upload/browse button to more easily load custom plugins would also be acceptable.

Is this acceptable?  It is the same code i sent you before, but with the update function removed, as the package manager now handles that, and with 2-3 lines fixed for compatibility.  other than that its 100% the same skeleton you sent me before.

No, because it raises the review and beneficiary question mentioned above. I don't want to merge something we don't have a policy for. It's not a "no" for a merge, it's a "no, we really don't know" and from the core team we don't appreciate being pushed into this position instead of naturally easing into it by asking the right questions and discussing it with the community. If the community is silent, that then only means a decision must be made in the best interest, not necessarily in the voiced interest.

I can completely understand that you need policies for things, i am an open book, either with helping for the policy, providing input for it, whatever you need, just ask.

And while im at it, do i need to re-fork, so i can do a pull request just of pfmonitor, without the nmap, remember im not a git expert.

You can create separate branches for both plugins. Since they reside in separate directories, that won't be a problem.

Thank you, this explains how i screwed up the PR.

Also, im not sure what use the api will have for nmap functionality....

Why are you not sure? Can you explain?

i meant that the nmap plugin would not have much use in api form, expressing opinion thats all.

Separate question here:
Also, for the api itself, are you expecting everyone who wants to use it, to expose their web interfaces to the open internet?

No, it's for local scripting and UI flexibility. You don't like the current way the UI does things? Build a better UI on top of the API as a plugin, or combine them.... scan via nmap, do reverse DNS lookups, write firewall rules.. all in one plugin... eventually. Possibilities are endless, practially only limited by imagination.  ;)

I should hope not.   Would be a large security surface to worry about.

I am not trying to be difficult about this, i am very fond of this project, but please understand my frustration in this, that getting 1 little plugin added to a list, would help so much, and reduce the number of constant emails i get from people who do not know how to get SSH enabled, and how to SFTP the files to the opnsense device, and execute proper commands to get them loaded.

I fully apologize for the impressions from my above posts, but please guys, whatever i need to do to get this done, either with the plugin getting published, or even just an easy upload button to load a custom plugin, either one would solve the problem at least in short term.

[MasterXBKC]

i have re-published the pfmonitor plugin without the nmap, im trying to figure out how to do a seperate upload for the nmap still, it keeps trying to lump them in togeather.