OPNsense Forum
English Forums => General Discussion => Topic started by: orsomannaro on March 03, 2017, 02:56:26 pm
-
I'm migrating from PfSense to OPNsense.
To backup the configuration settings I'm following this guide:
https://wikit.firewall-services.com/doku.php/tuto/sauvegardes/sauvegarde_pfsense_2
but the script for OPNsense doesn't work:
#!/bin/bash -e
OUT='/var/backups/opnsense'
TMP=$(mktemp -d)
URL='https://opnsense.domain.tld'
LOGIN='backupusr'
PASS='p@ssw0rd'
# Submit the login form with the previous values, and save a new CSRF token
/usr/bin/wget -q -O /dev/null --keep-session-cookies --save-cookies $TMP/cookies.txt --no-check-certificate \
--post-data "login=Login&usernamefld=$LOGIN&passwordfld=$PASS" $URL/diag_backup.php
# Save only the config
/usr/bin/wget -q --keep-session-cookies --load-cookies $TMP/cookies.txt --save-cookies $TMP/cookies.txt --no-check-certificate \
--post-data "download=Download%20Configuration&donotbackuprrd=yes" $URL/diag_backup.php -O $OUT/config-pfsense.xml \
rm -f $TMP/*.txt
rmdir $TMP
Somebody can help me to adjust the script code in the right way?
Thanks for your help.
-
We don't have bash, replace:
#!/bin/bash -e
with
#!/bin/sh -e
It looks like there is nothing bash-specific in there, but I didn't verify.
Cheers,
Franco
-
Hi Franco.
Thanks for your reply, but change /bin/bash with /bin/sh doesn't solve the problem.
I suppose that the problem resides in the wget instruction parameters.
-
it could be your credentials as they need to be URL encoded.
-
We don't have an error message. It's very hard to debug this way. :)
-
can you remove the -q which is quet so you get some output?
-
The scripts seem to need to be updated with the new csrf stuff:
felix@plex: backups$ grep magic diag_backup.php
felix@plex: backups$ grep csrf *
diag_backup.php: <form class="clearfix" id="iform" name="iform" method="post" autocomplete="off" action="/diag_backup.php"><input type="hidden" id="__opnsense_csrf" name="UVZLYVFKSUlPbEdKb3RRL3ZkRjhOQT09" value="VlR2SUppNzFBYUJyR1hGUVVSQXErZz09"\/>
felix@plex: backups$
I've played a bit, but haven't hacked together the right values. Will try a bit later.
-
you need to grep for this line using something like
grep "__opnsense_csrf"
and then use
cut -d'"' -fnumberto get the key and the value
-
Hmm. A little closer, but I still get a 403 as I tried to modify some of the code:
wget -O- --keep-session-cookies --save-cookies cookies.txt --no-check-certificate http://192.168.86.1/diag_backup.php | grep "__opnsense_csrf" | sed 's/.*value="\(.*\)".*/\1/' > csrf.txt
wget -O- --keep-session-cookies --load-cookies cookies.txt --save-cookies cookies.txt --no-check-certificate --post-data "login=login&usernamefld=root&passwordfld=MYPASSWORD&__opnsense_csrf=$(cat csrf.txt)" http://192.168.86.1/diag_backup.php | grep "__opnsense_csrf" | sed 's/.*value="\(.*\)".*/\1/' > csrf2.txt
Error:
--2017-03-03 19:39:50-- http://192.168.86.1/diag_backup.php
Connecting to 192.168.86.1:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-03-03 19:39:50 ERROR 403: Forbidden.
I'm getting a token from the first wget:
cat csrf.txt
TWIxOFpIdFdUZ1RGZjBON0NXSERoUT09\
-
I made my own script to help you out. You can download it here:
https://github.com/fabianfrz/scripts/blob/master/OPNsense/backup_over_http.rb
It is a quick and dirty solution but it seems to work.
-
Thank you all for your help!
> can you remove the -q
The error is: 403 Forbidden
> I made my own script to help you out
Thank you very much Fabian!
I installed ruby. Ubuntu server 16.04 repository version is: ruby 2.3.1p112 (2016-04-26).
Runnin your script I have this error:
/usr/local/bin/Backuppc_OPNsense.rb:42:in `<main>': undefined method `scan' for nil:NilClass (NoMethodError)
In OPNsense I see the connection logs with pass status.
I'm searching this error on internet ...
-
this means you do not have the string "__opnsense_csrf" in the body. Do you use 17.1.2?
For debugging, can you add those three lines:
puts d.code
puts d.body
exit 0
after d is assigned (line 40)?
-
I changed the URL to just http for me and that worked fine for me.
I get the backup file.
I'm running 17.1.2.
-
this means you do not have the string "__opnsense_csrf" in the body. Do you use 17.1.2?
I'm running OPNsense 16.7
I've updated your script to use 4443 port:
indexpage = URI("https://#{SERVER_IP}:4443/index.php")
backuppage = URI("https://#{SERVER_IP}:4443/diag_backup.php")
For debugging, can you add those three lines:
puts d.code
puts d.body
exit 0
after d is assigned (line 40)?
This is th output:
200
<!doctype html>
<!--[if IE 8 ]><html lang="en" class="ie ie8 lte9 lte8 no-js"><![endif]-->
<!--[if IE 9 ]><html lang="en" class="ie ie9 lte9 no-js"><![endif]-->
<!--[if (gt IE 9)|!(IE)]><!--><html lang="en" class="no-js"><!--<![endif]-->
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="robots" content="index, follow, noodp, noydir" />
<meta name="keywords" content="" />
<meta name="description" content="" />
<meta name="copyright" content="" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" />
<title>Login</title>
<link href="/ui/themes/opnsense/build/css/main.css" rel="stylesheet">
<link href="/ui/themes/opnsense/build/images/favicon.png" rel="shortcut icon">
<!--[if lt IE 9]><script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.2/html5shiv.min.js"></script><![endif]-->
<script type="text/javascript">if (top != self) {top.location.href = self.location.href;}</script><script type="text/javascript">var csrfMagicToken = "sid:7a7f16c6317e0e693af8c8d09a4244ea2f82319e,1488785494;ip:a066ee2908007256ef908b4091d91f35f963ab4c,1488785494";var csrfMagicName = "__csrf_magic";</script><script src="/csrf/csrf-magic.js" type="text/javascript"></script></head>
<body class="page-login">
<div class="container">
<main class="login-modal-container">
<header class="login-modal-head" style="height:55px;">
<div class="navbar-brand">
<img src="/ui/themes/opnsense/build/images/default-logo.png" height="30" alt="logo"/>
</div>
</header>
<div class="login-modal-content">
<div id="inputerrors" class="text-danger"> </div><br />
<form class="clearfix" id="iform" name="iform" method="post" autocomplete="off" action="/index.php"><input type='hidden' name='__csrf_magic' value="sid:7a7f16c6317e0e693af8c8d09a4244ea2f82319e,1488785494;ip:a066ee2908007256ef908b4091d91f35f963ab4c,1488785494" />
<div class="form-group">
<label for="usernamefld">Username:</label>
<input id="usernamefld" type="text" name="usernamefld" class="form-control user" tabindex="1" autofocus="autofocus" autocapitalize="off" autocorrect="off" />
</div>
<div class="form-group">
<label for="passwordfld">Password:</label>
<input id="passwordfld" type="password" name="passwordfld" class="form-control pwd" tabindex="2" />
</div>
<button type="submit" name="login" value="1" class="btn btn-primary pull-right">Login</button>
</form>
</div>
</main>
<div class="login-foot text-center">
<a target="_blank" href="https://opnsense.org" class="redlnk">OPNsense</a> (c) 2014-2016 <a href="https://www.deciso.com/" class="tblnk">Deciso B.V.</a>
</div>
</div>
<script type="text/javascript">CsrfMagic.end();</script></body>
</html>
-
Thanks for your response, it was not required anymore to add this code to debug. You should always mention any changes you did which may break some scripts like a changed port.
-
Do you use 17.1.2?
Updating OPSsense to version 17.1.2 the script works fine.
Thank you so much Fabian!
-
does someone have a sh script for this?
-
As stated previously, that script should to do fine without bash. Can always check with "sh -n script.sh".
-
hmm it doesnt work for me...
I have modified it a little:
#!/bin/sh
TMP=$(mktemp -d)
URL=$1
LOGIN='mybackupuser'
PASS='mybackuppass'
# Submit the login form with the previous values, and save a new CSRF token
/usr/bin/wget -q -O /dev/null --keep-session-cookies --save-cookies $TMP/cookies.txt --no-check-certificate \
--post-data "login=Login&usernamefld=$LOGIN&passwordfld=$PASS" $URL/diag_backup.php
# Save only the config
/usr/bin/wget -q --keep-session-cookies --load-cookies $TMP/cookies.txt --save-cookies $TMP/cookies.txt --no-check-certificate \
--post-data "download=Download%20Configuration&donotbackuprrd=yes" $URL/diag_backup.php -O /srv/backup/config-$1-`date +%Y%m%d%H%M%S`.xml \
rm -f $TMP/*.txt
rmdir $TMP
gave the mybackupusr the permission to 'see' the backup site
and got this in /srv/backup/config-myhost-20180202094316.xml:
<!doctype html>
<!--[if IE 8 ]><html lang="en" class="ie ie8 lte9 lte8 no-js"><![endif]-->
<!--[if IE 9 ]><html lang="en" class="ie ie9 lte9 no-js"><![endif]-->
<!--[if (gt IE 9)|!(IE)]><!--><html lang="en" class="no-js"><!--<![endif]-->
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="robots" content="noindex, nofollow, noodp, noydir" />
<meta name="keywords" content="" />
<meta name="description" content="" />
<meta name="copyright" content="" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" />
<title>Login</title>
<link href="/ui/themes/opnsense/build/css/main.css" rel="stylesheet">
<link href="/ui/themes/opnsense/build/images/favicon.png" rel="shortcut icon">
<script type="text/javascript" src="/ui/js/jquery-3.2.1.min.js"></script>
<script type="text/javascript" src="/ui/js/jquery-migrate-3.0.1.min.js"></script>
<!--[if lt IE 9]><script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.2/html5shiv.min.js"></script><![endif]-->
<script type="text/javascript">
$( document ).ready(function() {
$.ajaxSetup({
'beforeSend': function(xhr) {
xhr.setRequestHeader("X-CSRFToken", "SEtpaEhPdXN6OWlQMmphdHdxNitadz09" );
}
});
});
</script>
</head>
<body class="page-login">
<div class="container">
<main class="login-modal-container">
<header class="login-modal-head" style="height:55px;">
<div class="navbar-brand">
<img src="/ui/themes/opnsense/build/images/default-logo.png" height="30" alt="logo"/>
</div>
</header>
<div class="login-modal-content">
<div id="inputerrors" class="text-danger"> </div><br />
<form class="clearfix" id="iform" name="iform" method="post" autocomplete="off" action="/diag_backup.php"><input type="hidden" name="RFlMUHFrV3p1M1RqNzhEcFdINFZLdz09" value="SEtpaEhPdXN6OWlQMmphdHdxNitadz09" />
<div class="form-group">
<label for="usernamefld">Username:</label>
<input id="usernamefld" type="text" name="usernamefld" class="form-control user" tabindex="1" autofocus="autofocus" autocapitalize="off" autocorrect="off" />
</div>
<div class="form-group">
<label for="passwordfld">Password:</label>
<input id="passwordfld" type="password" name="passwordfld" class="form-control pwd" tabindex="2" />
</div>
<button type="submit" name="login" value="1" class="btn btn-primary pull-right">Login</button>
</form>
</div>
</main>
<div class="login-foot text-center">
<a target="_blank" href="https://opnsense.org/" class="redlnk">OPNsense</a> (c) 2014-2018 <a href="https://www.deciso.com/" class="tblnk">Deciso B.V.</a>
</div>
</div>
</body>
</html>
Greetz
-
This means you are not authenticated.
-
Hi Folks, this script is it working ?
Im testint it with 18.7 series withou success.
best regards
Carlos
-
Just tested it - yes it works. Are you missing a dependency or the credentials?
-
Am I right if this doesn't work at all on 19.x ?
If it should work please add a working example.
Thanks!
-
> Am I right if this doesn't work at all on 19.x ?
No, it is ok.
-
> Am I right if this doesn't work at all on 19.x ?
No, it is ok.
Can you please post a working, tested, example as reference ? I think it clears up the topic as well. Thanks!
-
install os-api-backup plugin and see https://github.com/opnsense/plugins/pull/895#issuecomment-458158323
-
install os-api-backup plugin and see https://github.com/opnsense/plugins/pull/895#issuecomment-458158323
That is not the bashscript I requested for. I want to do something else with bash as well, like exporting certificates but want to have this working first.
I'm looking forward to it :)
-
This topic is about config.xml backups, sorry.
-
This topic is about config.xml backups, sorry.
Which I try to get with a working bash script you say which is working referring to a script you are not wiling to (re)post ?
From that I can see what I do else but this is my base, so please share what you say you have working or is working.
Thanks again!
-
I'm only trying to help and I feel you're not accepting it. I'm sorry I cannot communicate this any better.
-
I'm only trying to help and I feel you're not accepting it. I'm sorry I cannot communicate this any better.
Sorry for that, we might have some misunderstanding here! I have some other way I'm working on where I actually need this for so for now it's not needed anymore.
Thanks!
-
@Fabian
I recently switched to opnsense coming from pfsense.
I have about 12 pfsense in the field and now I added an opnsense.
To back-up the configs of these routers I use an hourly cronjob that fetches the latest config and if it is (about) the same it will throw it away. This way I end up with a list of configs through the years that only reflect the changes.
I have written that script in bash, but only the code surrounding the fetching of the config.
I've seen your example in Ruby, but this language is so unfamiliar to me that I can't use it to create a bash counterpart of it.
I am hoping the reverse is not true and that you are familiar enough with bash and are able to tell me how to convert my code to "opnsense".
Here's the code I use for pfsense which is working to this day:
if ! wget -t1 --timeout=10 -qO- --keep-session-cookies --save-cookies /tmp/${IDENTIFIER}_cookies.txt ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > /tmp/${IP}-csrf.txt ; then
echo "Error fetching cookie" >&2
exit 1
else
[ ${HEADLESS} ] || echo "Got session cookie" >&2
fi
if ! wget -t1 --timeout=10 -qO- --keep-session-cookies --load-cookies /tmp/${IDENTIFIER}_cookies.txt --save-cookies /tmp/${IDENTIFIER}_cookies.txt --post-data "login=Login&usernamefld=${USER}&passwordfld=${PASS}&__csrf_magic=`cat /tmp/${IP}-csrf.txt`" ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > /tmp/${IP}-csrf2.txt ; then
echo "Error pushing the session cookie" >&2
exit 1
else
[ ${HEADLESS} ] || echo "Pushed cookie" >&2
fi
if ! wget -t1 --timeout=30 -qO ${FNAME} --keep-session-cookies --load-cookies /tmp/${IDENTIFIER}_cookies.txt --post-data "download=download&donotbackuprrd=yes&__csrf_magic=$(head -n 1 /tmp/${IP}-csrf2.txt)" ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php ; then
echo "Error fetching ${FNAME}" >&2
rm -f "${FNAME}"
exit 1
else
[ ${HEADLESS} ] || echo "Fetched ${FNAME}" >&2
fi
-
How about a cron job along the line
rsync -av --update --partial --append --log-file=$HOME/.rsyncd.log <source> <destination>
?
https://www.freebsd.org/cgi/man.cgi?query=rsync (https://www.freebsd.org/cgi/man.cgi?query=rsync)
-
How about a cron job along the line
rsync -av --update --partial --append --log-file=$HOME/.rsyncd.log <source> <destination>
?
https://www.freebsd.org/cgi/man.cgi?query=rsync (https://www.freebsd.org/cgi/man.cgi?query=rsync)
Thanks, but preferably not. I prefer to pull the config which allows me to have all the code and configuration centralized on 1 server. If I let the router push the config it needs to be configured on both the server as the client.
This means troubleshooting only has to be done on 1 end.. not on both ends.
Furthermore it does much more.
It compares the downloaded config with the latest one and deletes it if it is the same in important parts.
This can't be done elegantly if you push the firmware.
This concept also works for very simple routers.
I pull configs from different routers for over 10 years and I prefer it that way.
It is possible to tweak my current procedure to enable it to work as the one for pfsense does.
I seek help to do that. No alternative ways to do a back-up.
-
eehm, rsync can be initiated from either side. Just saying. Apparently you never used it, otherwise you wouldn't mess around with wget... ;-)
-
Thanks for your valuable input...
Can someone give me the correct sequence of fetches that need to be done to get the config through the https-interface?
-
wget --http-user=mdcO...DsfO --http-passwd=1l0kZaw..d3C -O tmp_config.xml --no-check-certificate --auth-no-challenge https://firewall.test/api/backup/backup/downloadplease do not use "--no-check-certificate" - this call works on a test machine. This uses the os-backup-api plugin. You can get the API credentials from the user configuration as a file download. key is the user, secret the password.
-
Thanks...
That URL works.... ...in Chrome when I have previously logged in.
It doesn't when I logout.
With wget it gives me "Username/Password Authentication Failed."
So I can't directly use that URL without logging in first.
It seems I'm not providing the credentials.
I am using --no-check-certificate
You use it in your example, but write I shouldn't use it.
I take it as an advice to install a certificate. For now I need to to use it as it's not installed yet....
-
Thanks...
That URL works.... ...in Chrome when I have previously logged in.
It doesn't when I logout.
With wget it gives me "Username/Password Authentication Failed."
So I can't directly use that URL without logging in first.
It seems I'm not providing the credentials.
Please read my command carefully:
You provide the credentials using basic auth:
--http-user=mdcO...DsfO --http-passwd=1l0kZaw..d3C
This are not your standard credentials. This are the API credentials you get in your user configuration page.
I am using --no-check-certificate
You use it in your example, but write I shouldn't use it.
I take it as an advice to install a certificate. For now I need to to use it as it's not installed yet....
I copied the command 1:1 which I used on my development VM. On a real appliance, you should have a trusted certificate (even if it is your self singned CA which has been configured on your clients).
-
Thanks...
I was suspecting this as I could not imagine this to be not working at all.
I've been looking for some hints somewhere but could not find it.
Now you mention that there's something like an API-key, I googled that and found this.
I assume I will have no problem using it after using the API-credentials.
I was not aware of the existence of such a thing as API-credentials in OpnSense.
https://docs.opnsense.org/development/how-tos/api.html
Is there a way to create a user which is restricted to downloading a back-up?
-
Yes, you can create a user, create an API key for that user and assign the privilege "Backup API" to it. Then the user can log in on the web interface to change his password but nothing else and he can download the config XML as well.
-
I was able to assign a privilege to a group and made the user "apibackup" part of the group.
The interface is a bit counter-intuitive.
One needs to first create the group and only after editing it is possible to assign it privileges.
If you don't know beforehand it can be done there, it is hard to find out things without a manual.
I knew it had to be possible, so I first created the group.
Only when modifying the group, the "assign privileges" gets unlocked.
I don't like interfaces that "unlock" features when needed. Now that I know this is how it's done with opnsense I will be better prepared in the future...
I need to clean up my script a bit and will post it later on.
But many thanks...
I now have my backup-config working just like the one for my pfsense boxes.
I only need to install the api-backup, add a user and a group "backup"
Then I need to assign the privilege "GUI apibackup" to the group backup and assign an API key/secret to the user.