Messages

1

Development and Code Review / Plugin Writes IP to Config File - IP Change?

« on: Today at 01:23:04 am »

I've made a load of improvements to a plugin but have hit a blocker. The plugin in question has to write the IP address for a service to listen on, into a config file. I can use the templates structure for the plugins, e.g.

Code: [Select]

{%     if helpers.exists('interfaces.'+int+'.ipaddr') %}
{%       set interface_ip = helpers.getNodeByTag('interfaces.'+int+'.ipaddr') %}
{%       if '.' in interface_ip %}
--- do stuff here ---
{%       endif %}
{%     endif %}

However that gets the IP address out of the config file. If an interface is set to DHCP, or SLAAC (for the ipaddrv6 property) there is no IP address in that field. How do we cope with that in a plugin, is there an event handler for an IP address change that can update config files for running services?

2

24.7 Production Series / Re: opnsense-version vs net-snmpd vs librenms

« on: November 08, 2024, 05:02:07 pm »

Actually, I've had a bit of a think about this. What I will do (if it's OK with you) is fork the snmp plugin and update that with what I've done here instead. That way we aren't building very specific solutions for very specific products into the body of the main OS - modifying system components for the sake of one 3rd party product. Instead I will roll what I've written here into the snmp plugin.

3

24.7 Production Series / Re: opnsense-version vs net-snmpd vs librenms

« on: November 08, 2024, 04:42:34 pm »

That would be awesome!

Minimal mucking about:

Code: [Select]

./opnsense-version -o
FreeBSD|SMP|amd64|OPNsense|24.7.8|vmware|
That should mean you could add something into your snmpd.conf template like:

Code: [Select]

{% if OPNsense.netsnmp.general.enableobservium == '1' %}
extend .1.3.6.1.4.1.2021.7890.1 distro /usr/local/sbin/opnsense-version -o
extend .1.3.6.1.4.1.2021.7890.2 hardware /bin/kenv smbios.planar.product
extend .1.3.6.1.4.1.2021.7890.3 vendor /bin/kenv smbios.planar.maker
extend .1.3.6.1.4.1.2021.7890.4 serial /bin/kenv smbios.planar.serial
{% endif %}
And then not have to bundle the distro.sh from Observium. It's actually better than distro.sh which will never detect if you are running in a hypervisor - despite having logic to check if it is running in FreeBSD, it then tries to call the detectvirt binary which is a systemd tool that doesn't exist here.  My virtualisation detection is rudimentary though - it doesn't detect containers and I can only actually test it on vmware and hyper-v.

I will put this in a PR and see what Franco et al think 

4

24.7 Production Series / Re: opnsense-version vs net-snmpd vs librenms

« on: November 08, 2024, 03:11:42 pm »

Just from a quick view, the format Observium is expecting for OS info is:

${OS}|${KERNEL}|${ARCH}|${DISTRO}|${VERSION}|${VIRT}|${CONT}

We could easily modify the built-in opnsense-version script to add another parameter (say, -o) to output Observium format. You would then not need distro.sh which contains a large amount of irrelevant info (e.g. checking to see if we are running on Solaris). Then you haven't got to maintain Observium's distro.sh script inside an OPNsense plugin, bundle in the Observium license, etc etc.

5

24.7 Production Series / Re: opnsense-version vs net-snmpd vs librenms

« on: November 08, 2024, 03:01:27 pm »

I understand now. That revision is far too new to be on my production systems I'm afraid - most are business edition! I will investigate it though because I would be keen to do this in such a way that would work any any NMS rather than specific 3rd party scripts for 1 product - e.g. some (most?) of the data from distro.sh is already provided by the net-snmp plugin in this other OID which is referenced by other 3rd party tools. And as mentioned I also have a bunch of work on the os-net-snmp and os-lldpd plugins in flight 

6

24.7 Production Series / Re: opnsense-version vs net-snmpd vs librenms

« on: November 08, 2024, 02:52:07 pm »

Ah, no sorry I didn't see that bit. I've not installed that plugin because I'm not using Observium, however it is interesting they have chosen to do it that way, as the info about architecture, release, and OS name (e.g. the stuff I posted about above) is exposed by net-snmpd by ticking the 'Display Version in OID' tickbox without needing any additional plugins.

What else does that plugin do? It can't just be that! Off to investigate... shouldn't need to install a plugin to make an nms work

7

24.7 Production Series / Re: opnsense-version vs net-snmpd vs librenms

« on: November 08, 2024, 02:45:49 pm »

The net-snmpd plugin has an option to expose the version of OPNsense under a specific OID. This is a tickbox in the GUI provided by this plugin:



This adds the following line to the snmpd config file:

Code: [Select]

extend    version   /usr/local/sbin/opnsense-version

This has the effect of putting whatever the output is of that command, into that OID.

Observium uses .1.3.6.1.4.1.2021.7890.1 which is what you get when you enable Observium support in the current version of the plugin. Just for reference - I have no idea why there are (at least) two different OIDs. Also Observium does not use opnsense-version but relies on its own "distro" script for all Unix platforms.

What does it report in that OID? I don't get anything on my OPNsense querying that:

Code: [Select]

root@x-y-z:~ # snmpwalk -v3 -u xxx -a sha -x aes -A xxx -X xxx -l authPriv 127.0.0.1 .1.3.6.1.4.1.2021.7890.1
UCD-SNMP-MIB::ucdavis.7890.1 = No Such Object available on this agent at this OID


8

24.7 Production Series / Re: 24.10 BE upgrade CRL errors

« on: November 08, 2024, 02:18:16 pm »

Is from updates, not a fresh install. Literally upgraded to 24.10_7 last night and it happened immediately.

I had HTTP_PROXY and HTTPS_PROXY set in the configd template and neither the gui nor update-crl-fetch.py worked. Adding http_proxy and https_proxy in the configd template made update-crl-fetch.py work, but the gui still not. Removing the file from /tmp made no difference - it would just be recreated. I ticked the Auto add CRLs tickbox and went to sleep, and when I woke up it was fixed.

I think that when I looked at the contents of the file, it just contained:

Code: [Select]

# [i] fetch certificate for https://opnsense-update.deciso.com

9

24.7 Production Series / Re: opnsense-version vs net-snmpd vs librenms

« on: November 08, 2024, 02:05:14 pm »

Happy to, I have a few more for net-snmpd / lldpd waiting in the wings too! The question I guess is, do any changes that have been made in the last 5 years now rely on the new default output of the command?

10

24.7 Production Series / opnsense-version vs net-snmpd vs librenms

« on: November 08, 2024, 02:01:57 pm »

The net-snmpd plugin has an option to expose the version of OPNsense under a specific OID. This is a tickbox in the GUI provided by this plugin:



This adds the following line to the snmpd config file:

Code: [Select]

extend    version   /usr/local/sbin/opnsense-version

This has the effect of putting whatever the output is of that command, into that OID. LibreNMS, a 3rd party network monitoring system, is aware of OPNsense. Libre, and I would assume other NMS-type products, look in this OID to figure out the OS and version, using this regex:

Code: [Select]

(?<version>[\d._]+) \((?<hardware>[^)]+)\)
However at some point the output of the opnsense-version command has changed, and it no longer outputs the arch by default. This means the string in the OID no longer matches the regex the NMS is using to identify the OS version.

Old output:

Code: [Select]

root@x-y-z:~ # ./opnsense-version
OPNsense 24.10_7 (amd64/OpenSSL)

New output:

Code: [Select]

root@x-y-z:~ # /usr/local/sbin/opnsense-version
OPNsense 24.10_7

Current opnsense-version with parameters to replicate old behaviour:

Code: [Select]

root@x-y-z:~ # /usr/local/sbin/opnsense-version -NvA
OPNsense 24.10_7 amd64

Believe it or not, this was caused by a commit from 2019! Commit id c1f839e in relation to issue #4500 changed the default output of the command, by introducing the DEFAULTS variable and setting this to include product_name and product_version.

There are a number of ways this could be addressed:

• Modify the default output of the opnsense-version command so it behaves as it did in the past
• Modify the template for the snmpd.conf file so that it adds -NvA as a parameter to opnsense-version, restoring the original functionality
• Add more tickboxes to the net-snmp plugin to further customise what information is presented in that OID
• Submit a PR to LibreNMS to change their regular expression

What do we collectively think is the correct way to approach this? Fundamentally, a change was introduced in OPNsense which inadvertently changed the way a plugin worked. Is the correct fix to modify the plugin, modify OPNsense to restore the original intended behaviour, or accept it is how it is and fix the 3rd party tool? I'll do the PR, would just like some guidance as to the correct way to fix it.

11

24.7 Production Series / Re: 24.10 BE upgrade CRL errors

« on: November 08, 2024, 01:35:57 pm »

Unfortunately in fixing it, I don't have that evidence from when it was broken anymore. I can't roll the environment back as it is semi-live... I'll see if I can recreate it in test.

12

24.7 Production Series / Re: 24.10 BE upgrade CRL errors

« on: November 08, 2024, 10:19:38 am »

No, trying again from the GUI just makes it recreate that file again. Each time I rm, the check from the gui brings it back.

However as suggested by another user in this thread, ticking System -> Trust -> Settings -> Autofetch CRLs (which shouldn't have an apostrophe in it ) and leaving it for a few hours has done the job. I suspect then from this that the fetch command that is run when you click 'Check for updates' is somehow running in some different environment/context which isn't inheriting the environment variables from configd, where the scheduled check (and pkg, when running 'Check for updates') is.

13

24.7 Production Series / Re: 24.10 BE upgrade CRL errors

« on: November 07, 2024, 09:54:03 pm »

Just upgraded some BE to 24.10_7 and am having this same issue, however I suspect there is a different root cause: my 2x BE devices are behind a proxy. They both have proxy environment variables set as per documentation here:

https://docs.opnsense.org/development/backend/configd.html#extending-the-environment

However this does not appear to be passing down the python requests library as it is clearly trying to connect directly.

Code: [Select]

root@x-y-z:~ # /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com
# [i] fetch certificate for https://opnsense-update.deciso.com
[!!] Chain fetch failed for https://opnsense-update.deciso.com (HTTPSConnectionPool(host='opnsense-update.deciso.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1ee0c0bfa110>: Failed to establish a new connection: [Errno 65] No route to host')))

And from the gui:

Code: [Select]

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10_7 at Thu Nov  7 20:41:56 GMT 2024
Fetching subscription information, please wait... Could not load CRL file /tmp/libfetch_crl.24110720
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/subscription: Authentication error
Fetching changelog information, please wait... Could not load CRL file /tmp/libfetch_crl.24110720
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/sets/changelog.txz: Authentication error
Updating OPNsense repository catalogue...
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/meta.txz: Authentication error
repository OPNsense has no meta file, using default settings
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.pkg: Authentication error
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
Could not load CRL file /tmp/libfetch_crl.24110720
pkg: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/latest/packagesite.txz: Authentication error
Unable to update repository OPNsense
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
Edit: Seems python-requests wants the environment variables in lower case. Added those and now the cmdline script runs without errors. The updates web gui still doesn't, though.

14

24.7 Production Series / Re: Plugin Development Questions & Some Bugs?

« on: August 14, 2024, 11:37:42 pm »

Ah, I am an idiot. The answer to question 1 is obvious - put it in /usr/local/opnsense/scripts

15

24.7 Production Series / Plugin Development Questions & Some Bugs?

« on: August 14, 2024, 09:53:36 pm »

Hi,

I am porting the scan agent for phpIPAM into OPNSense so you can perform subnet scans directly from the firewall/gateway/DHCP server/whatever role OPNSense has in your network. The agent itself is a threaded (using prctl) CLI PHP application from https://github.com/phpipam/phpipam-agent

I have written a plugin to perform GUI configuration of the agent and add scanning tasks as cron jobs in the web gui which all works fine and as expected. However there are some packaging questions I have:

• The phpipam-agent needs to reside somewhere. It is essentially a php script so there is no FreeBSD port of it for example. In my dev setup I have put it in /usr/local/share/phpipam-agent - where would be the appropriate place for this to live on the filesystem in production, and how would I package it into my plugin for it to live somewhere outside of the /usr/local/opnsense tree? I suspect I am going about this part incorrectly. Will I need to package the agent and the OPNsense plugin separately and somehow make both available?
• The phpipam-agent requires quite a few PHP packages that are not in the OPNsense ports collection but are in the FreeBSD ports collection, namely php82-gmp php82-pdo_mysql php82-iconv php82-posix. I have seen that one can request these ports be made available for OPNsense by opening a git issue - is that still the right approach?

Then, some possible bugs! I think there may be a couple of mistakes in the documentation regarding building. Several times in the build documentation the package vim-console is referenced, but this does not appear to be installable anymore. 

I believe there is a line of code in the example for the settingscontroller jquery which is incorrect. By this I mean the code shown in the documentation at https://docs.opnsense.org/development/examples/helloworld.html - the file in the github repo is correct. There is a line:

Code: [Select]

$result["validations"]["general.".$msg->getField()] = $msg->getMessage();Which I think should be:

Code: [Select]

$result["validations"]["helloworld.".$msg->getField()] = $msg->getMessage();
The original code results in an object being returned called general.general.fieldname when I think it should be helloworld.general.fieldname?

Additionally it does not appear possible to have an entry in the menu.xml file for a plugin where the tag name starts with php

This did not work in my testing:

Code: [Select]

<menu>
    <Services>
        <PhpIpamAgent visibleName="phpIPAM Agent" cssClass="fa fa-list-ol fa-fw">
            <Settings url="/ui/phpipamagent" />
        </PhpIpamAgent>
    </Services>
</menu>

This does work:

Code: [Select]

<menu>
    <Services>
        <ipamAgent visibleName="phpIPAM Agent" cssClass="fa fa-list-ol fa-fw">
            <Settings url="/ui/phpipamagent" />
        </ipamAgent>
    </Services>
</menu>

Thanks!