SNMP providing faulty data to monitoring system

[ooboyle]

Hello,

I'm running the following ESX 5.5 VM:

OPNsense 16.1.14-amd64   
FreeBSD 10.2-RELEASE-p17   
OpenSSL 1.0.2h 3 May 2016

I am using OMD/Check_MK to monitor OPNsense but I'm getting incorrect memory data back from SNMP.

As you can see from the attached graph named Check_MK.jpg (which I've confirmed is correctly showing data it extracts from the firewall via SNMP v1 and v2c using both snmpwalk and snmpbulkwalk), the system is reporting that is only has .8GB of installed RAM when in fact, it has 2GB of 'installed' RAM. The .8GB amount actually correlates to the amount of 'used' RAM which can be seen via the OPNsense dashboard attachment named OPNsense.jpg which also happens to correctly report the amount of installed RAM.

The Check_MK guys have confirmed that the faulty numbers are coming from the OPNsense system. Here's what we see with an snmpwalk and some backup information about the check being used:

.1.3.6.1.2.1.25.2.3.1.3.1 = "Real Memory Metrics"
.1.3.6.1.2.1.25.2.3.1.4.1 = 4096
.1.3.6.1.2.1.25.2.3.1.5.1 = 204196
.1.3.6.1.2.1.25.2.3.1.6.1 = 201840
 
Description can be found in ~/share/check_mk/checks/hr_mem:
                                    3, # hrStorageDescr
                                    4, # hrStorageAllocationUnits
                                    5, # hrStorageSize
                                    6, # hrStorageUsed
 
The device says (via SNMP), it's mem sitze is 204196*4096 bytes and 201840*4096 are in use. Calculate these values gives us the values Check_MK shows which means the device's SNMP values appear to be faulty.

Has anyone had this issue before? Where is OPNsense getting its 'installed' and 'used' RAM figures from?

Oliver

[ooboyle]

This problem persists.

I'm also not receiving SNMP data for the em0 interface...

Anyone?

[ooboyle]

Is this not an OPNsense-related issue? Should I be contacting someone else?

[phoenix]

Is this not an OPNsense-related issue? Should I be contacting someone else?

Have you considered upgrading to the most recent OPNsense release? I'm currently on 16.1.16 and I also use OMD/check_mk and LibreNMS to monitor the machines on my LAN and they all agree on the amount of RAM in use that's shown on the OPNsense lobby.

[ooboyle]

Hi Bill,

The upgrade doesn't appear to make a difference.

What version of cmk are you running? Is your OPNsense install physical or virtual? If virtual, what hypervisor are you running on?

Oliver

[phoenix]

I've been running real hardware and virtual OPNsense on and off (at various times) for the past couple of weeks. The virtualized one is on an ESXi 6 server and that was my only platform for OPNsense for the past year, I've only recently been trying some real hardware and on both of those platform the memory usage figures have always been consistent. The current check_mk agent version is 1.2.6p16 and all my VMs have that from their respective repositories - most of the VMs are CentOS7 and my workstations are both openSUSE.

[ooboyle]

Hmm...

I'm running 1.2.6p9 but have a 1.2.8p4 that I can test with as well.

Did you install the cmk agent on the OPNsense devices or are you using just SNMP?

Oliver

[ooboyle]

Tested on cmk 1.2.8p4 and same result.

Bill, I'd love to know if you installed the cmk agent or if you're just using SNMP.

[phoenix]

Tested on cmk 1.2.8p4 and same result.

Bill, I'd love to know if you installed the cmk agent or if you're just using SNMP.

Sorry for the late reply, I seem to have missed your previous post. I have both the check_mk agent and SNMP being polled. I'll test that later today with only the SNMP poller.

[edit]Just for clarification, I also use LibreNMS tp monitor my network and that only polls SNMP and that does show the correct amount of RAM in use. My test later today will be for OMD to just use SNMP to see what the results are.

[ooboyle]

Thanks, Bill. If it's not too much to ask, re-try SNMP-only after completely uninstalling all monitoring agents.

Thanks,
Oliver

[franco]

I've taken a quick peek: RAM size seems to be off by an order of magnitude, all other values look even funkier for RAM, I can't correctly correlate them.

SWAP itself is ok, disk size too.

No sure where to fix or report at this point.

[franco]

Seems to be going on for quite a while... https://forums.freebsd.org/threads/41768/

[franco]

If anyone wants to look at the code, I think this is it... https://github.com/freebsd/freebsd/blob/af3e10e5a78d3a/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c#L246

[ooboyle]

Thanks for digging into this Franco. Any chance we can get this added to a bug list and patched?

[franco]

I will give this one more shot, otherwise it's best to take this to FreeBSD.

But with the actual fix it would make it easier to get the fix in.