OPNsense Forum
English Forums => Tutorials and FAQs => Topic started by: bartjsmit on January 20, 2016, 02:02:16 pm
-
This script follows on from the example in the 'Using the API Howto' wiki entry. https://wiki.opnsense.org/index.php/Howto_use_the_API
Replace the redacted api_key and api_secret strings with your own and set the URL of your OPNsense instance as well as your name/email address.
It requires an MTA (set to localhost in the script) and can be run from cron. Tested with Python 2.7.5-34 on CentOS 7.
-
Hi there,
This is really cool, thanks for sharing. I'm making this sticky in the general forum. :)
Cheers,
Franco
-
Thanks for this. Confirmed on Debian wheezy. Got an update email today for 16.1.
-
"Dude, where's my 16.1?"
-
As some of you will have noticed, there is an issue with the script versus the Python request library, or rather with my (lack of) understanding of the latter.
Whereas before all JSON elements were returned as (unicode) list, some are now dict variables, causing the script to bomb out on a type error.
Attached is version 1.1 of fw_update.py, that tests for the returned type. As before, you can un-redact it and run it from cron.
Tested with Python 2.7.5-34 and python-requests-2.6.0-1 on CentOS 7
Enjoy!
Bart...
-
Thanks for update. Working on Debian 7.9 w/python 2.7.3.
-
The script was working for me until I just updated python on my Gentoo system. Now I get this when I run it. Any ideas would be greatly appreciated.
(163) --> python2.7 fw_update1.1.py
/usr/lib64/python2.7/site-packages/urllib3/connectionpool.py:790: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
Traceback (most recent call last):
File "fw_update1.1.py", line 28, in <module>
response = json.loads(r.text)
File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
-
The unverified HTTPS points to a x509 certificate chain issue, rather than Python.
Make sure you have a trusted certificate bound to the OPNsense webConfigurator under Certificates and that its root CA as well as any intermediate CA's in the trust chain are imported under Authorities. Both menus are under System:Trust in the web interface. Don't forget to restart the webConfigurator afterwards.
You can either create an internal CA for this and set your Gentoo to trust it, or you can get a publicly trusted certificate for your firewall. I rent a cheap wildcard certificate from startssl.com for my home network.
If you just want stuff to work without worrying about certs, you can of course set verify=False in the request.get line, but Python will lecture you (quite rightly) about your poor OpSec 8)
Bart...
-
I have verify set to FALSE in the script -> "r = requests.get(url,verify=False"
It's the other errors that I don't know what to do with.
Traceback (most recent call last):
File "fw_update1.1.py", line 28, in <module>
response = json.loads(r.text)
File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
I'm using Python 2.7.11 on Gentoo.
Thanks.
-
Upon further investigation I'm running python 2.7.11-r2 which is listed as unstable in the Gentoo dev-lang/python package. Apparently it's nearly impossible to move back the stable branch of python due to reverse dependencies.
So maybe 2.7.11-r2 is the issue and it would work with 2.7.10-r1?
-
I'm using this script on another deployment of OPNsense in an environment where I only have an OpenBSD server to use for the SMTP service of this script and I'm not able to send the script's email using that server. I get this error:
Traceback (most recent call last):
File "fw_update-1.1.py", line 58, in <module>
s.sendmail(sender,recipients,message)
File "/usr/local/lib/python2.7/smtplib.py", line 746, in sendmail
raise SMTPDataError(code, resp)
smtplib.SMTPDataError: (550, '5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant')
Is anyone able to use an OpenBSD mail server to deliver this script without this error by chance?
This is what it looks like from the OpenBSD smtp server side:
<<< [MSG] <br>Click <a href="https://192.168.2.1/ui/core/firmware/">here</a> to fetch them.<br>
debug: 0x18f9f6123000: end of message, msgflags=0x8000
smtp: 0x18f9f6123000: >>> 550 5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant
smtp-in: Failed command on session 48cc59eadf55d2f9: "data" => 550 5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant
smtp: 0x18f9f6123000: STATE_BODY -> STATE_HELO
smtp: 0x18f9f6123000: <<< rset
smtp: 0x18f9f6123000: >>> 250 2.0.0: Reset state
smtp-in: Received disconnect from session 48cc59eadf55d2f9
debug: smtp: 0x18f9f6123000: deleting session: disconnecte
Thanks.
-
Most likely cause for a 5.7.1 error is that your FreeBSD MTA denies relay from your firewall.
Can you get a message to the recipient manually through a telnet session on port 25?
Bart...
-
I'm able to send a test email using this python script from the OPNsense server via the same OpenBSD mail server that is returning the "not RFC 2822 compliant" message for the fw_update-1.1.py script.
#!/usr/bin/python
import smtplib
sender = 'myname@mydomain'
receivers = ['myname@mydomain]
message = """From: From OPNsense <myname@mydomain>
To: To Person <myname@mydomain>
Subject: SMTP e-mail test
This is a test e-mail message.
"""
try:
smtpObj = smtplib.SMTP('192.168.2.2')
smtpObj.sendmail(sender, receivers, message)
print "Successfully sent email"
except SMTPException:
print "Error: unable to send email"
Thanks.
-
Well, what is the IMF output of the script? I guess the error is pretty clear, just need to figure out what is wrong with the message that should be dispatched.
-
Sorry for the stupid question, but what is IMF? I'm just coming up with "International Monetary Fund" but I'm sure that's not what you mean. ;)
-
RFC 2822 deals with message content. Can you try and edit the script to say 'text/plain' instead of 'text/html' or even remark out these two lines completely:
#message += 'MIME-Version: 1.0\n'
#message += 'Content-type: text/html\n'
Bart...
-
IMF is Internet Message Format, a.k.a. RFC 2822 :)
-
I don't think the content headers matter, and I don't recall those being used by Email (MIME-Version and Content-type). Instead, maybe adhering to "\r\n" line breaks will help...
(And the last line dealing with reboot should probably also receive a newline.)
-
I tried using 'text/plain' instead of 'text/html', replacing all \n entries with \r\n and also tried remarking out both of these lines:
#message += 'MIME-Version: 1.0\n'
#message += 'Content-type: text/html\n'
but the script still comes back with "smtplib.SMTPDataError: (550, '5.7.1 Delivery not authorized, message refused: Message is not RFC 2822 compliant')"
-
This might be helpful. On the network where I'm able to use the script without the RFC 2822 compliant (it relays through a Linux Postfix system, not an OpenBSD SMTP server), I noticed this in my Amavis quarantine this morning even though the email from the script was delivered:
X-Amavis-Alert: BAD HEADER SECTION, Missing required header field: "Date"
Could a missing Date header field be what's upsetting the SMTP daemon on the OpenBSD SMTP server who refuses to accept the email from the script due to the RFC 2822 compliant?
Just a thought.
Thanks,
ObecalpEffect.
-
The RFC states "The only required header fields are the origination date field and the originator address field(s). All other header fields are syntactically optional.". You may be right. :)
-
I'll have a dig through the smtplib docs to see if there is an option to add the timestamp to the email headers.
Bart...
-
I wish I could help, but I couldn't program my way out of a wet paper bag. :-\
That said though, I was playing with getting a properly formatted date string with python while trying to get the script working and I think this would suffice and possibly be RFC 2822 compliant. I hope it's helpful. I'll try to poke through the smtplib docs too and see if I can find a way to add the date header. Hopefully it is the date header that is causing OpenBSD's SMTP daemon to complain about RFC 2822 compliancy and we're not chasing our tails here.
#!/usr/local/bin/python
# import libraries
from email.Utils import formatdate
print formatdate(localtime = 6)
-
Hmm, not sure if I'm dreaming this or not, but I think I fixed it with the addition of these three lines in the script:
from email.Utils import formatdate
and
dateheader = 'formatdate(localtime = 6)\r\n'
and
message += '\r\ndateheader\r\n
So the entire top section of the script now looks like this (and it actually worked! :) )
#!/usr/local/bin/python
# import libraries
import json
import requests
import smtplib
from email.Utils import formatdate
api_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
api_secret = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
host = 'localhost'
recipients = 'me@there.edu'
rcpt_name = 'First Last'
dateheader = 'formatdate(localtime = 6)\r\n'
url = 'https://' + host + '/api/core/firmware/status'
sender = 'emailrelay@there.edu'
message = 'From: emailrelay@there.edu\r\n'
message += 'To: me@there.edu>\r\n'
message += 'MIME-Version: 1.0\r\n'
message += 'Content-type: text/html\r\n'
message += 'Subject: Updates Available for OPNsense\r\n'
message += '\r\ndateheader\r\n'
# request and process data
-
Thanks for that - looks to me like a pretty sturdy paper bag ;-)
I'll do some regression testing with postfix and attach a new version to this thread.
Bart...
-
You could also use a public git repository
-
Also published on https://github.com/bartsmit/opnsense-update-email
ObecalpEffect, I've credited you with your forum handle.
Bart...
-
Hi Bart,
Sorry to be a pain, but I noticed that the latest version of the script still isn't working with SMTP on my OpenBSD server. SMTP on OpenBSD still complains "Message is not RFC 2822 compliant".
I was however able to get it working by replacing this line:
message += formatdate(localtime=True) + '\r\n'
With this line:
message += 'Date: = formatdate(localtime=True)'
*Please also note that it doesn't work when the line is appended with + '\r\n'
Thanks,
Obecalp.
-
Thanks Obecalp,
I will do some regression testing and post an updated script.
Bart...
-
The JSON response from OPNsense looks a bit different now. I have updated the script accordingly and made it python3 compatible.
-
Thanks for your contribution!
Bart...
-
I used the ChangeDetection.io (http://) Docker container i am running to monitor if my OPNsense has updates available. Configure a watch like this:
General > URL: https://hostnameofrouter/api/core/firmware/status
Request > Request method: POST
Request > Request headers: Authorization: Basic PUTBASE64HERE
Filters & Triggers > CSS/JSONPath/JQ/XPath Filters > json:$.status
You can create the PUTBASE64HERE with some PowerShell:
$key="xxxxxxxxx"
$secret="xxxxxxxxxx"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes("$($key):$($secret)"))
Write-Host $base64AuthInfo
As my OPNsense is currently updated, i haven't been able to check if it works completely right. But the JSON is displayed in ChangeDetection, so changes will be picked up if the 'status' field changes.
Or if you want to do it completely in PowerShell you could create a script yourself and use this as a start:
<#
This script retrieves the update status of OPNsense, to monitor if the system needs an update.
#>
$key="xxxxxx"
$secret="xxxxxx"
$hostname = 'puthostnamehere'
<#
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
#>
$user = $key
$pass = $secret
$pair = "$($user):$($pass)"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($pair))
$basicAuthValue = "Basic $encodedCreds"
$Headers = @{
Authorization = $basicAuthValue
}
$url = "https://$hostname/api/core/firmware/status"
Invoke-RestMethod -Uri $url -Method Post -Headers $Headers
You could then acces the JSON endpoints like so:
$status = $apiResponse.status
$statusMsg = $apiResponse.status_msg
$newPackages = $apiResponse.new_packages
$upgradePackages = $apiResponse.upgrade_packages
$reinstallPackages = $apiResponse.reinstall_packages
$needsReboot = $apiResponse.needs_reboot
I commented out the TrustAllCertsPolicy stuff as it wasn't needed (i use a self-signed certificate on OPNsense which is also present on my computer).
-
Unfortunately the above didn't seem to work, i wasn't notified of the most recent update. I think the POST-request and such works, but ChangeDetection.io is not playing nice with it.
I now configured changedetection to look at the URL https://forum.opnsense.org/index.php?board=11.0&action=.xml with the filter //recent-post[1]/subject to only return the first topic in the list. This seems to work better.
Edit: The above XPath also returns the 'Re:" topics. I changed my XPath to this (thanks ChatGPT :) so that i only get notified about new releases ):
//recent-post[not(contains(subject, 'Re:'))]/subject