OPNsense Forum

Archive => 16.1 Legacy Series => Topic started by: ooboyle on June 06, 2016, 10:10:52 pm

Title: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 06, 2016, 10:10:52 pm
Hello,

I'm running the following ESX 5.5 VM:

OPNsense 16.1.14-amd64   
FreeBSD 10.2-RELEASE-p17   
OpenSSL 1.0.2h 3 May 2016

I am using OMD/Check_MK to monitor OPNsense but I'm getting incorrect memory data back from SNMP.

As you can see from the attached graph named Check_MK.jpg (which I've confirmed is correctly showing data it extracts from the firewall via SNMP v1 and v2c using both snmpwalk and snmpbulkwalk), the system is reporting that is only has .8GB of installed RAM when in fact, it has 2GB of 'installed' RAM. The .8GB amount actually correlates to the amount of 'used' RAM which can be seen via the OPNsense dashboard attachment named OPNsense.jpg which also happens to correctly report the amount of installed RAM.

The Check_MK guys have confirmed that the faulty numbers are coming from the OPNsense system. Here's what we see with an snmpwalk and some backup information about the check being used:

.1.3.6.1.2.1.25.2.3.1.3.1 = "Real Memory Metrics"
.1.3.6.1.2.1.25.2.3.1.4.1 = 4096
.1.3.6.1.2.1.25.2.3.1.5.1 = 204196
.1.3.6.1.2.1.25.2.3.1.6.1 = 201840
 
Description can be found in ~/share/check_mk/checks/hr_mem:
                                    3, # hrStorageDescr
                                    4, # hrStorageAllocationUnits
                                    5, # hrStorageSize
                                    6, # hrStorageUsed
 
The device says (via SNMP), it's mem sitze is 204196*4096 bytes and 201840*4096 are in use. Calculate these values gives us the values Check_MK shows which means the device's SNMP values appear to be faulty.

Has anyone had this issue before? Where is OPNsense getting its 'installed' and 'used' RAM figures from?

Oliver
Title: Re: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 08, 2016, 05:58:07 pm
This problem persists.

I'm also not receiving SNMP data for the em0 interface...

Anyone?
Title: Re: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 13, 2016, 05:26:21 pm
Is this not an OPNsense-related issue? Should I be contacting someone else?
Title: Re: SNMP providing faulty data to monitoring system
Post by: phoenix on June 13, 2016, 06:08:56 pm
Is this not an OPNsense-related issue? Should I be contacting someone else?
Have you considered upgrading to the most recent OPNsense release? I'm currently on 16.1.16 and I also use OMD/check_mk and LibreNMS to monitor the machines on my LAN and they all agree on the amount of RAM in use that's shown on the OPNsense lobby.
Title: Re: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 13, 2016, 06:48:58 pm
Hi Bill,

The upgrade doesn't appear to make a difference.

What version of cmk are you running? Is your OPNsense install physical or virtual? If virtual, what hypervisor are you running on?

Oliver
Title: Re: SNMP providing faulty data to monitoring system
Post by: phoenix on June 13, 2016, 07:49:53 pm
I've been running real hardware and virtual OPNsense on and off (at various times) for the past couple of weeks. The virtualized one is on an ESXi 6 server and that was my only platform for OPNsense for the past year, I've only recently been trying some real hardware and on both of those platform the memory usage figures have always been consistent. The current check_mk agent version is 1.2.6p16 and all my VMs have that from their respective repositories - most of the VMs are CentOS7 and my workstations are both openSUSE.
Title: Re: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 13, 2016, 10:01:14 pm
Hmm...

I'm running 1.2.6p9 but have a 1.2.8p4 that I can test with as well.

Did you install the cmk agent on the OPNsense devices or are you using just SNMP?

Oliver
Title: Re: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 14, 2016, 10:58:37 pm
Tested on cmk 1.2.8p4 and same result.

Bill, I'd love to know if you installed the cmk agent or if you're just using SNMP.
Title: Re: SNMP providing faulty data to monitoring system
Post by: phoenix on June 15, 2016, 07:34:27 am
Tested on cmk 1.2.8p4 and same result.

Bill, I'd love to know if you installed the cmk agent or if you're just using SNMP.
Sorry for the late reply, I seem to have missed your previous post. I have both the check_mk agent and SNMP being polled. I'll test that later today with only the SNMP poller.

[edit]Just for clarification, I also use LibreNMS tp monitor my network and that only polls SNMP and that does show the correct amount of RAM in use. My test later today will be for OMD to just use SNMP to see what the results are.
Title: Re: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 16, 2016, 03:42:39 pm
Thanks, Bill. If it's not too much to ask, re-try SNMP-only after completely uninstalling all monitoring agents.

Thanks,
Oliver
Title: Re: SNMP providing faulty data to monitoring system
Post by: franco on June 16, 2016, 08:52:47 pm
I've taken a quick peek: RAM size seems to be off by an order of magnitude, all other values look even funkier for RAM, I can't correctly correlate them.

SWAP itself is ok, disk size too.

No sure where to fix or report at this point.
Title: Re: SNMP providing faulty data to monitoring system
Post by: franco on June 16, 2016, 09:20:39 pm
Seems to be going on for quite a while... https://forums.freebsd.org/threads/41768/
Title: Re: SNMP providing faulty data to monitoring system
Post by: franco on June 16, 2016, 09:40:42 pm
If anyone wants to look at the code, I think this is it... https://github.com/freebsd/freebsd/blob/af3e10e5a78d3a/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c#L246
Title: Re: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 21, 2016, 03:53:40 pm
Thanks for digging into this Franco. Any chance we can get this added to a bug list and patched?
Title: Re: SNMP providing faulty data to monitoring system
Post by: franco on June 21, 2016, 08:42:30 pm
I will give this one more shot, otherwise it's best to take this to FreeBSD.

But with the actual fix it would make it easier to get the fix in. ;)
Title: Re: SNMP providing faulty data to monitoring system
Post by: ooboyle on June 22, 2016, 07:05:42 pm
Thanks, Franco.

Oliver