OPNsense Forum

English Forums => Tutorials and FAQs => Topic started by: faunsen on June 07, 2017, 06:08:00 pm

Title: Monit Mini Howto
Post by: faunsen on June 07, 2017, 06:08:00 pm

Monit is a software that can check your filesystems, disks, processes, system and many more.
It runs on the firewall host and sends messages or executes actions on various events.
With the os-monit plugin you can configure the Monit daemon.

Installation
To use Monit install the os-monit plugin first. It installs the monit package as a dependency.
After the installation has finished reload the GUI and navigate to Services->Monit->Settings.

Configuration
The first step is to check if the plugin installer has imported your System->Notification settings correctly. Then have a look at the other tabs. To help you get acquainted the installer has added some standard entries.

The order to configure a monitoring is to create Service Tests first, then Services to check and in the end Alerts.

Let's start with the Service Test Settings. A test contains a condition and an action. It can be assigned to one or more services. The Monit documentation (https://mmonit.com/monit/documentation/monit.html#SERVICE-TESTS) shows possible tests. You can simply adopt it by omitting the IF and THEN statements.

The next step is to configure service checks (https://mmonit.com/monit/documentation/monit.html#Service-checks). Depending on the service type we need to set a path or start/stop scripts etc. and assign tests we have defined before. You can assign the same tests to different service checks.

At the Alert Settings tab you can configure to whom sent alerts to on which events and to whom not.
Furthermore you can format the mail text (https://mmonit.com/monit/documentation/monit.html#ALERT-MESSAGES). E.g. Subject: $SERVICE failed on $HOST at $DATE

NOTE: For cluster setups you can synchronize the configuration to the other node if you check Monit System Monitoring
      under System->High Availabilty->Settings. Then go to Firewall->Diagnostics->Filter Reload and click on Force Config Sync.

Status
After starting Moint you can view detailed status information on the  Services->Monit->Status page.

System 'firewall.example.com'
  status                       Running
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  load average                 [0.59] [0.41] [0.35]
  cpu                          0.2%us 0.9%sy
  memory usage                 2.6 GB [8.1%]
  swap usage                   0 B [0.0%]
  uptime                       20d 4h 25m
  boot time                    Thu, 18 May 2017 13:27:11
  data collected               Wed, 07 Jun 2017 17:51:23

Filesystem 'RootFs'
  status                       Does not exist
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  data collected               Wed, 07 Jun 2017 17:51:23

Program 'IPMI'
  status                       Status ok
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  last exit value              0
  last output                  System Health: OK, Power Meter: 174 Watts, Ambient Temp: 19 degrees C, Fans: 39 percent
  data collected               Wed, 07 Jun 2017 17:51:23

Examples

FTP Proxy
Check FTP proxy and restart it. If it doesn't run after 5 restarts don't check it again.
Tests
Name: RestartLimit5
Condition: 5 restarts within 5 cycles
Action: Unmonitor
Name: FTPProxy8021
Condition: failed host 127.0.0.1 port 8021 type tcp
Action: Restart

Service
Name: FTPProxy8021
Type: Process
PID File: /var/run/osftpproxy.127_0_0_1_8021.pid
Start: /usr/local/sbin/configctl ftpproxy start 127_0_0_1_8021
Stop: /usr/local/sbin/configctl ftpproxy start 127_0_0_1_8021
Tests: FTPProxy8021, RestartLimit5


HPE ProLiant DL380 G7 via IPMI
Check HPE ProLiant DL380 G7 hardware via IPMI.
Install ipmitool, load the ipmi kernel driver and create a check script.

Code: [Select]

pkg install ipmitool
kldload ipmi
echo 'ipmi_load="YES"' >>/boot/loader.conf.local
create a script /usr/local/bin/CheckIPMI.sh

Code: [Select]

#!/bin/csh

set Status = 0
set FanCount = 0
set FanSum = 0
foreach Line (`/usr/local/bin/ipmitool sdr list | sed 's/ /\\t/g'`)
   set SensorData = `echo $Line | sed 's/\\t/ /g'`
   set SensorName = `echo $SensorData | awk -F\| '{print $1}'`
   set SensorValue = `echo $SensorData | awk -F\| '{print $2}'`
   set SensorStatus = `echo $SensorData | awk -F\| '{print $3}'`
   if ( "$SensorStatus" != "ok" && "$SensorValue" != "disabled" ) then
      echo "$SensorData"
      set Status = 1
   endif
   if ( "$SensorName" =~ "Power Supply*" ) then
      if ( "$SensorValue" == "0 Watts" ) then
         echo "$SensorData"
         set Status = 1
      endif
   endif
   if ( "$SensorName" == "Power Meter" ) set PowerMeter = "$SensorValue"
   if ( "$SensorName" == "Temp 1" ) set AmbientTemp = "$SensorValue"
   if ( "$SensorName" =~ "Fan *" ) then
     @ FanCount = ( $FanCount + 1 )
     set FanValue = `echo $SensorValue | awk '{print $1}' | awk -F. '{print $1}'`
     @ FanSum = ( $FanSum + $FanValue )
   endif
end
if ( $Status == 0 ) then
   @ FanSpeed = ( $FanSum / $FanCount )
   echo "System Health: OK, Power Meter: $PowerMeter, Ambient Temp: $AmbientTemp, Fans: $FanSpeed percent"
endif

exit $Status
Make it executable.

Code: [Select]

chmod +x /usr/local/bin/CheckIPMI.sh
Tests
Name: ExecStatus
Condition: status notequal 0
Action: Alert

Service
Name: IPMI
Type: Custom
Path: /usr/local/bin/CheckIPMI.sh
Tests: ExecStatus

Title: Re: Monit Mini Howto
Post by: Stephan on October 02, 2017, 11:36:20 am

Example for FILE CONTENT
https://mmonit.com/monit/documentation/monit.html#FILE-CONTENT-TEST (https://mmonit.com/monit/documentation/monit.html#FILE-CONTENT-TEST)

Example for scanning C-ICAP log for errors connecting to clamd

Test-condition: 

Code: [Select]

content = "(Registry 'virus_scan::engines' does not exist)|(clamd_connect: Can not connect to clamd server)" for 2 cycles

Service:

Code: [Select]

Service-Type: FILE

Code: [Select]

Path: /var/log/c-icap/server.log
In this case I used a script to start CLAMD and to restart C-ICAP afterwards:

Code: [Select]

#!/bin/sh

/usr/local/etc/rc.d/clamav-clamd start
sleep 5
/usr/local/etc/rc.d/c-icap restart

The Start and Stopp script entries for the service require a parameter like start, stop, restart - one can append it even though it's not used

Title: Re: Monit Mini Howto
Post by: dcol on March 09, 2018, 04:28:09 pm

How to add CPU temperature test to Monit.

You must have OPNsense 18.1.5 or higher installed using Monit 1.6

Add this script to /usr/local/bin And call it CPUTemp.sh for this example

Code: [Select]

#!/bin/csh

set MaxCPUTemp = $1
set NumCPUs = `sysctl -n kern.smp.cpus`
set CurrentCPU = 0

while ( $CurrentCPU < $NumCPUs )
   set CPUTemp = `sysctl dev.cpu.$CurrentCPU.temperature | awk '{print $2}' | awk -F. '{print $1}'`
   echo "CPU $CurrentCPU temp: $CPUTemp"
   if ( $CPUTemp >= $MaxCPUTemp ) then
      exit $CPUTemp
   endif
   @ CurrentCPU = $CurrentCPU + 1
end
exit $CPUTemp

Set Execute rights to this file
# chmod 755 /usr/local/bin/CPUTemp.sh

Now lets create the test in Monit

Service Test Settings - Create New
Name: Temps
Condition: status >= 60 #Make this number the temperature that triggers the alert
Action: Alert

Service Settings - Create New
Enable service checks - checked
Name: CPUtemp
Type: Custom
Path: /usr/local/bin/CPUTemp.sh '60' #This number sets the max temp ('quotes needed)
Start: <leave blank>
Stop: <leave blank>
Tests: Temps
Description: Check CPU Temperature

You can easily change the max temp by changing the number 60 to any other number.
Make sure you change it in both. The name I used 'Temps' can be whatever you want, just use the same name in the Service Settings Tests field.

That's it. Now, as per this example, all CPUs will be checked and will send a notification alert if temp exceeds 60, as per this example. You can check the status to make sure it is working. Test will show 'Initializing' Until the Start Delay is complete.

I find this test very useful because it can warn on fan failure or excessive CPU power draw.
Enjoy!

Title: Re: Monit Mini Howto
Post by: jenmonk on March 26, 2018, 04:32:24 pm

Thanks a lot.

Title: Re: Monit Mini Howto
Post by: docb on June 09, 2018, 10:50:33 am

Thanks a lot for the nice How-To. And thanks to an other idea by fausen I wrote a script to monitor active DHCP Leases. If someone has a use for it - have fun.http:// Yes, I am sure you could do that smoother in code, but I am an absolute Noob - it at least works ;-)

Code: [Select]

#!/bin/csh
set FILE = "/var/dhcpd/var/db/dhcpd.leases"
set LeaseCount = `grep -c "binding state active" $FILE`
echo "Active LeaseCount: $LeaseCount"
if ($LeaseCount > 0) then
 set LINE = ( `grep -n "binding state active" $FILE | cut -d: -f1` )
 foreach L ($LINE)
  echo "##################"
 # set EL = `expr $L + 8`
  set CL = `expr $L - 5`
  while ( $CL < `expr $L + 8` )
   if ( `sed -n "$CL p" $FILE | cut -d ' ' -f1` == "lease" ) then
    echo IP: `sed -n "$CL p" $FILE | cut -d ' ' -f2`
   endif
   if ( `sed -n "$CL p" $FILE | cut -d ' ' -f3` == "starts" ) then
    echo Start: `sed -n "$CL p" $FILE | cut -d ' ' -f5,6`
   endif
   if ( `sed -n "$CL p" $FILE | cut -d ' ' -f3` == "ends" ) then
    echo Ende: `sed -n "$CL p" $FILE | cut -d ' ' -f5,6`
   endif
   if ( `sed -n "$CL p" $FILE | cut -d ' ' -f3` == "hardware" ) then
    echo MAC: `sed -n "$CL p" $FILE | cut -d ' ' -f5`
   endif
   if ( `sed -n "$CL p" $FILE | cut -d ' ' -f3` == "client-hostname" ) then
    echo Name: `sed -n "$CL p" $FILE | cut -d ' ' -f4-`
   endif
   if ( `sed -n "$CL p" $FILE | cut -d ' ' -f3` == "set" ) then
    echo Device: `sed -n "$CL p" $FILE | cut -d ' ' -f6-`
   endif
   if ( `sed -n "$CL p" $FILE | cut -d ' ' -f1` == "}" && $CL > $L ) then
    set CL = `expr $CL + 8`
   endif
   @ CL++
  end
 end
 exit 1
else
 exit 0
endif

Enjoy

Title: Re: Monit Mini Howto
Post by: Blisk on July 22, 2019, 02:09:53 pm

Is it possible to check a file with monit.
I like to check if file exist in folder but can not specify name because it is also date and time related, so it can be only partial name.
But I tested that and doesn't work. Any idea how to do that?

  check file imagine_file with path /imagine/sync/MY_file*.7z
   if timestamp > 7 day then alert

Title: Re: Monit Mini Howto
Post by: faunsen on July 22, 2019, 04:02:32 pm

You could try to use a variable as descibed in the Message Format (https://mmonit.com/monit/documentation/monit.html#Message-format).
If that doesn't work you can use a script to check the existence of the file.

Title: Re: Monit Mini Howto
Post by: dcol on October 01, 2019, 12:30:39 am

The CPUtemp test I have laid out in a previous post no longer works.

Status never gets past 'Initializing'

If anyone knows how to fix, please share.

Thanks

Title: Re: Monit Mini Howto
Post by: faunsen on October 01, 2019, 08:55:43 am

Can you please start a new topic for that?
This is a how-to and not meant for resolving problems.

Thanks  :)