OPNsense Forum

English Forums => Tutorials and FAQs => Topic started by: bartjsmit on January 20, 2016, 02:02:16 pm

Title: Script to send emails for updates
Post by: bartjsmit on January 20, 2016, 02:02:16 pm
This script follows on from the example in the 'Using the API Howto' wiki entry. https://wiki.opnsense.org/index.php/Howto_use_the_API

Replace the redacted api_key and api_secret strings with your own and set the URL of your OPNsense instance as well as your name/email address.

It requires an MTA (set to localhost in the script) and can be run from cron. Tested with Python 2.7.5-34 on CentOS 7.
Title: Re: Script to send emails for updates
Post by: franco on January 20, 2016, 04:35:11 pm
Hi there,

This is really cool, thanks for sharing. I'm making this sticky in the general forum. :)


Cheers,
Franco
Title: Re: Script to send emails for updates
Post by: philamonster on January 28, 2016, 07:30:18 pm
Thanks for this. Confirmed on Debian wheezy. Got an update email today for 16.1.
Title: Re: Script to send emails for updates
Post by: franco on January 28, 2016, 08:42:08 pm
"Dude, where's my 16.1?"
Title: Re: Script to send emails for updates
Post by: bartjsmit on March 20, 2016, 02:51:21 pm
As some of you will have noticed, there is an issue with the script versus the Python request library, or rather with my (lack of) understanding of the latter.

Whereas before all JSON elements were returned as (unicode) list, some are now dict variables, causing the script to bomb out on a type error.

Attached is version 1.1 of fw_update.py, that tests for the returned type. As before, you can un-redact it and run it from cron.

Tested with Python 2.7.5-34 and python-requests-2.6.0-1 on CentOS 7

Enjoy!

Bart...
Title: Re: Script to send emails for updates
Post by: philamonster on March 23, 2016, 11:20:08 pm
Thanks for update. Working on Debian 7.9 w/python 2.7.3.
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on March 25, 2016, 12:16:07 am
The script was working for me until I just updated python on my Gentoo system.  Now I get this when I run it.  Any ideas would be greatly appreciated.

(163) -->  python2.7 fw_update1.1.py
/usr/lib64/python2.7/site-packages/urllib3/connectionpool.py:790: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "fw_update1.1.py", line 28, in <module>
    response = json.loads(r.text)
  File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Title: Re: Script to send emails for updates
Post by: bartjsmit on March 25, 2016, 10:28:22 am
The unverified HTTPS points to a x509 certificate chain issue, rather than Python.

Make sure you have a trusted certificate bound to the OPNsense webConfigurator under Certificates and that its root CA as well as any intermediate CA's in the trust chain are imported under Authorities. Both menus are under System:Trust in the web interface. Don't forget to restart the webConfigurator afterwards.

You can either create an internal CA for this and set your Gentoo to trust it,  or you can get a publicly trusted certificate for your firewall. I rent a cheap wildcard certificate from startssl.com for my home network.

If you just want stuff to work without worrying about certs, you can of course set verify=False in the request.get line, but Python will lecture you (quite rightly) about your poor OpSec  8)

Bart...
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on March 25, 2016, 04:34:49 pm
I have verify set to FALSE in the script -> "r = requests.get(url,verify=False"

It's the other errors that I don't know what to do with.

Traceback (most recent call last):
  File "fw_update1.1.py", line 28, in <module>
    response = json.loads(r.text)
  File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I'm using Python 2.7.11 on Gentoo.

Thanks.
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on March 25, 2016, 05:30:30 pm
Upon further investigation I'm running python 2.7.11-r2 which is listed as unstable in the Gentoo dev-lang/python package.  Apparently it's nearly impossible to move back the stable branch of python due to reverse dependencies.

So maybe 2.7.11-r2 is the issue and it would work with 2.7.10-r1?
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on April 29, 2016, 05:45:05 pm
I'm using this script on another deployment of OPNsense in an environment where I only have an OpenBSD server to use for the SMTP service of this script and I'm not able to send the script's email using that server.  I get this error:

Traceback (most recent call last):
  File "fw_update-1.1.py", line 58, in <module>
    s.sendmail(sender,recipients,message)
  File "/usr/local/lib/python2.7/smtplib.py", line 746, in sendmail
    raise SMTPDataError(code, resp)
smtplib.SMTPDataError: (550, '5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant')

Is anyone able to use an OpenBSD mail server to deliver this script without this error by chance?

This is what it looks like from the OpenBSD smtp server side:

<<< [MSG] <br>Click <a href="https://192.168.2.1/ui/core/firmware/">here</a> to fetch them.<br>
debug: 0x18f9f6123000: end of message, msgflags=0x8000
smtp: 0x18f9f6123000: >>> 550 5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant
smtp-in: Failed command on session 48cc59eadf55d2f9: "data" => 550 5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant
smtp: 0x18f9f6123000: STATE_BODY -> STATE_HELO
smtp: 0x18f9f6123000: <<< rset
smtp: 0x18f9f6123000: >>> 250 2.0.0: Reset state
smtp-in: Received disconnect from session 48cc59eadf55d2f9
debug: smtp: 0x18f9f6123000: deleting session: disconnecte

Thanks.
Title: Re: Script to send emails for updates
Post by: bartjsmit on April 29, 2016, 09:15:06 pm
Most likely cause for a 5.7.1 error is that your FreeBSD MTA denies relay from your firewall.

Can you get a message to the recipient manually through a telnet session on port 25?

Bart...
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on April 30, 2016, 12:44:58 am
I'm able to send a test email using this python script from the OPNsense server via the same OpenBSD mail server that is returning the "not RFC 2822 compliant" message for the fw_update-1.1.py script.

#!/usr/bin/python
import smtplib
sender = 'myname@mydomain'
receivers = ['myname@mydomain]
message = """From: From OPNsense <myname@mydomain>
To: To Person <myname@mydomain>
Subject: SMTP e-mail test
This is a test e-mail message.
"""
try:
   smtpObj = smtplib.SMTP('192.168.2.2')
   smtpObj.sendmail(sender, receivers, message)
   print "Successfully sent email"
   except SMTPException:
   print "Error: unable to send email"


Thanks.
Title: Re: Script to send emails for updates
Post by: franco on May 01, 2016, 07:42:17 pm
Well, what is the IMF output of the script? I guess the error is pretty clear, just need to figure out what is wrong with the message that should be dispatched.
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on May 01, 2016, 08:15:40 pm
Sorry for the stupid question, but what is IMF?  I'm just coming up with "International Monetary Fund" but I'm sure that's not what you mean.  ;)
Title: Re: Script to send emails for updates
Post by: bartjsmit on May 01, 2016, 08:54:02 pm
RFC 2822 deals with message content. Can you try and edit the script to say 'text/plain' instead of 'text/html' or even remark out these two lines completely:

#message += 'MIME-Version: 1.0\n'
#message += 'Content-type: text/html\n'

Bart...
Title: Re: Script to send emails for updates
Post by: franco on May 02, 2016, 08:00:48 am
IMF is Internet Message Format, a.k.a. RFC 2822 :)
Title: Re: Script to send emails for updates
Post by: franco on May 02, 2016, 08:05:54 am
I don't think the content headers matter, and I don't recall those being used by Email (MIME-Version and Content-type). Instead, maybe adhering to "\r\n" line breaks will help...

(And the last line dealing with reboot should probably also receive a newline.)
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on May 02, 2016, 11:53:39 pm
I tried using 'text/plain' instead of 'text/html', replacing all \n entries with \r\n and also tried remarking out both of these lines:
#message += 'MIME-Version: 1.0\n'
#message += 'Content-type: text/html\n'

but the script still comes back with "smtplib.SMTPDataError: (550, '5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant')"
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on May 05, 2016, 03:07:38 pm
This might be helpful.  On the network where I'm able to use the script without the RFC 2822 compliant (it relays through a Linux Postfix system, not an OpenBSD SMTP server), I noticed this in my Amavis quarantine this morning even though the email from the script was delivered:

X-Amavis-Alert: BAD HEADER SECTION, Missing required header field: "Date"

Could a missing Date header field be what's upsetting the SMTP daemon on the OpenBSD SMTP server who refuses to accept the email from the script due to the RFC 2822 compliant?

Just a thought.

Thanks,
ObecalpEffect.
Title: Re: Script to send emails for updates
Post by: franco on May 05, 2016, 03:23:29 pm
The RFC states "The only required header fields are the origination date field and the originator address field(s).  All other header fields are syntactically optional.". You may be right. :)
Title: Re: Script to send emails for updates
Post by: bartjsmit on May 06, 2016, 08:44:15 am
I'll have a dig through the smtplib docs to see if there is an option to add the timestamp to the email headers.

Bart...
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on May 06, 2016, 08:21:44 pm
I wish I could help, but I couldn't program my way out of a wet paper bag.   :-\

That said though, I was playing with getting a properly formatted date string with python while trying to get the script working and I think this would suffice and possibly be RFC 2822 compliant.  I hope it's helpful.  I'll try to poke through the smtplib docs too and see if I can find a way to add the date header.  Hopefully it is the date header that is causing OpenBSD's SMTP daemon to complain about RFC 2822 compliancy and we're not chasing our tails here.

#!/usr/local/bin/python
# import libraries
from email.Utils import formatdate
print formatdate(localtime = 6)

Title: Re: Script to send emails for updates
Post by: ObecalpEffect on May 06, 2016, 08:33:47 pm
Hmm, not sure if I'm dreaming this or not, but I think I fixed it with the addition of these three lines in the script:

from email.Utils import formatdate
and
dateheader = 'formatdate(localtime = 6)\r\n'
and
message += '\r\ndateheader\r\n

So the entire top section of the script now looks like this (and it actually worked!  :)   )

#!/usr/local/bin/python
# import libraries
import json
import requests
import smtplib
from email.Utils import formatdate
api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
api_secret = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
host       = 'localhost'
recipients = 'me@there.edu'
rcpt_name  = 'First Last'
dateheader = 'formatdate(localtime = 6)\r\n'
url      = 'https://' + host + '/api/core/firmware/status'
sender   = 'emailrelay@there.edu'
message  = 'From: emailrelay@there.edu\r\n'
message += 'To: me@there.edu>\r\n'
message += 'MIME-Version: 1.0\r\n'
message += 'Content-type: text/html\r\n'
message += 'Subject: Updates Available for OPNsense\r\n'
message += '\r\ndateheader\r\n'
# request and process data

Title: Re: Script to send emails for updates
Post by: bartjsmit on May 07, 2016, 12:33:34 am
Thanks for that - looks to me like a pretty sturdy paper bag ;-)

I'll do some regression testing with postfix and attach a new version to this thread.

Bart...
Title: Re: Script to send emails for updates
Post by: fabian on May 07, 2016, 10:35:53 am
You could also use a public git repository
Title: Re: Script to send emails for updates
Post by: bartjsmit on May 07, 2016, 04:07:28 pm
Also published on https://github.com/bartsmit/opnsense-update-email

ObecalpEffect, I've credited you with your forum handle.

Bart...
Title: Re: Script to send emails for updates
Post by: ObecalpEffect on May 26, 2016, 05:22:32 pm
Hi Bart,

Sorry to be a pain, but I noticed that the latest version of the script still isn't working with SMTP on my OpenBSD server.  SMTP on OpenBSD still complains "Message is not RFC 2822 compliant".

I was however able to get it working by replacing this line:
message += formatdate(localtime=True) + '\r\n'

With this line:
message += 'Date: = formatdate(localtime=True)'

*Please also note that it doesn't work when the line is appended with + '\r\n'

Thanks,
Obecalp.
Title: Re: Script to send emails for updates
Post by: bartjsmit on May 26, 2016, 10:29:33 pm
Thanks Obecalp,

I will do some regression testing and post an updated script.

Bart...