OPNsense Forum

English Forums => General Discussion => Topic started by: orsomannaro on March 03, 2017, 02:56:26 pm

Title: [SOLVED] Backup OPSsense configuration with wget
Post by: orsomannaro on March 03, 2017, 02:56:26 pm
I'm migrating from PfSense to OPNsense.

To backup the configuration settings I'm following this guide:

https://wikit.firewall-services.com/doku.php/tuto/sauvegardes/sauvegarde_pfsense_2

but the script for OPNsense doesn't work:

Code: [Select]
#!/bin/bash -e
 
OUT='/var/backups/opnsense'
TMP=$(mktemp -d)
URL='https://opnsense.domain.tld'
LOGIN='backupusr'
PASS='p@ssw0rd'
 
# Submit the login form with the previous values, and save a new CSRF token
/usr/bin/wget -q -O /dev/null --keep-session-cookies --save-cookies $TMP/cookies.txt --no-check-certificate  \
   --post-data "login=Login&usernamefld=$LOGIN&passwordfld=$PASS" $URL/diag_backup.php
 
# Save only the config
/usr/bin/wget -q --keep-session-cookies --load-cookies $TMP/cookies.txt --save-cookies $TMP/cookies.txt --no-check-certificate \
    --post-data "download=Download%20Configuration&donotbackuprrd=yes" $URL/diag_backup.php -O $OUT/config-pfsense.xml \
 
rm -f $TMP/*.txt
rmdir $TMP

Somebody can help me to adjust the script code in the right way?


Thanks for your help.
Title: Re: Backup OPSsense configuration with wget
Post by: franco on March 03, 2017, 04:13:40 pm
We don't have bash, replace:

#!/bin/bash -e

with

#!/bin/sh -e

It looks like there is nothing bash-specific in there, but I didn't verify.


Cheers,
Franco
Title: Re: Backup OPSsense configuration with wget
Post by: orsomannaro on March 03, 2017, 05:47:14 pm
Hi Franco.

Thanks for your reply, but change /bin/bash with /bin/sh doesn't solve the problem.

I suppose that the problem resides in the wget instruction parameters.

Title: Re: Backup OPSsense configuration with wget
Post by: fabian on March 03, 2017, 06:35:34 pm
it could be your credentials as they need to be URL encoded.
Title: Re: Backup OPSsense configuration with wget
Post by: franco on March 03, 2017, 07:31:35 pm
We don't have an error message. It's very hard to debug this way. :)
Title: Re: Backup OPSsense configuration with wget
Post by: fabian on March 03, 2017, 07:53:01 pm
can you remove the -q which is quet so you get some output?
Title: Re: Backup OPSsense configuration with wget
Post by: Animosity022 on March 03, 2017, 09:35:02 pm
The scripts seem to need to be updated with the new csrf stuff:

Code: [Select]
felix@plex: backups$ grep magic diag_backup.php
felix@plex: backups$ grep csrf *
diag_backup.php:            <form class="clearfix" id="iform" name="iform" method="post" autocomplete="off" action="/diag_backup.php"><input type="hidden" id="__opnsense_csrf" name="UVZLYVFKSUlPbEdKb3RRL3ZkRjhOQT09" value="VlR2SUppNzFBYUJyR1hGUVVSQXErZz09"\/>
felix@plex: backups$

I've played a bit, but haven't hacked together the right values. Will try a bit later.
Title: Re: Backup OPSsense configuration with wget
Post by: fabian on March 03, 2017, 09:51:51 pm
you need to grep for this line using something like
Code: [Select]
grep "__opnsense_csrf"
and then use
Code: [Select]
cut -d'"' -fnumberto get the key and the value
Title: Re: Backup OPSsense configuration with wget
Post by: Animosity022 on March 04, 2017, 01:44:44 am
Hmm. A little closer, but I still get a 403 as I tried to modify some of the code:

Code: [Select]
wget -O- --keep-session-cookies --save-cookies cookies.txt --no-check-certificate http://192.168.86.1/diag_backup.php  | grep "__opnsense_csrf" | sed 's/.*value="\(.*\)".*/\1/' > csrf.txt

wget -O- --keep-session-cookies --load-cookies cookies.txt --save-cookies cookies.txt --no-check-certificate --post-data "login=login&usernamefld=root&passwordfld=MYPASSWORD&__opnsense_csrf=$(cat csrf.txt)" http://192.168.86.1/diag_backup.php  | grep "__opnsense_csrf" | sed 's/.*value="\(.*\)".*/\1/' > csrf2.txt

Error:

Code: [Select]
--2017-03-03 19:39:50--  http://192.168.86.1/diag_backup.php
Connecting to 192.168.86.1:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-03-03 19:39:50 ERROR 403: Forbidden.

I'm getting a token from the first wget:
Code: [Select]
cat csrf.txt
TWIxOFpIdFdUZ1RGZjBON0NXSERoUT09\
Title: Re: Backup OPSsense configuration with wget
Post by: fabian on March 04, 2017, 09:00:55 am
I made my own script to help you out. You can download it here:
https://github.com/fabianfrz/scripts/blob/master/OPNsense/backup_over_http.rb

It is a quick and dirty solution but it seems to work.
Title: Re: Backup OPSsense configuration with wget
Post by: orsomannaro on March 04, 2017, 11:27:03 am
Thank you all for your help!


> can you remove the -q

The error is: 403 Forbidden


> I made my own script to help you out

Thank you very much Fabian!

I installed ruby. Ubuntu server 16.04 repository version is: ruby 2.3.1p112 (2016-04-26).

Runnin your script I have this error:

/usr/local/bin/Backuppc_OPNsense.rb:42:in `<main>': undefined method `scan' for nil:NilClass (NoMethodError)

In OPNsense I see the connection logs with pass status.

I'm searching this error on internet ...
Title: Re: Backup OPSsense configuration with wget
Post by: fabian on March 04, 2017, 12:42:20 pm
this means you do not have the string "__opnsense_csrf" in the body. Do you use 17.1.2?
For debugging, can you add those three lines:
Code: [Select]
puts d.code
puts d.body
exit 0

after d is assigned (line 40)?
Title: Re: Backup OPSsense configuration with wget
Post by: Animosity022 on March 04, 2017, 01:48:37 pm
I changed the URL to just http for me and that worked fine for me.

I get the backup file.

I'm running 17.1.2.
Title: Re: Backup OPSsense configuration with wget
Post by: orsomannaro on March 06, 2017, 08:37:14 am
this means you do not have the string "__opnsense_csrf" in the body. Do you use 17.1.2?

I'm running OPNsense 16.7

I've updated your script to use 4443 port:
Code: [Select]
indexpage = URI("https://#{SERVER_IP}:4443/index.php")
backuppage = URI("https://#{SERVER_IP}:4443/diag_backup.php")


Quote
For debugging, can you add those three lines:
Code: [Select]
puts d.code
puts d.body
exit 0
after d is assigned (line 40)?

This is th output:

Code: [Select]
200
<!doctype html>
<!--[if IE 8 ]><html lang="en" class="ie ie8 lte9 lte8 no-js"><![endif]-->
<!--[if IE 9 ]><html lang="en" class="ie ie9 lte9 no-js"><![endif]-->
<!--[if (gt IE 9)|!(IE)]><!--><html lang="en" class="no-js"><!--<![endif]-->
  <head>

    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">

    <meta name="robots" content="index, follow, noodp, noydir" />
    <meta name="keywords" content="" />
    <meta name="description" content="" />
    <meta name="copyright" content="" />
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" />

    <title>Login</title>

    <link href="/ui/themes/opnsense/build/css/main.css" rel="stylesheet">
    <link href="/ui/themes/opnsense/build/images/favicon.png" rel="shortcut icon">

    <!--[if lt IE 9]><script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.2/html5shiv.min.js"></script><![endif]-->

  <script type="text/javascript">if (top != self) {top.location.href = self.location.href;}</script><script type="text/javascript">var csrfMagicToken = "sid:7a7f16c6317e0e693af8c8d09a4244ea2f82319e,1488785494;ip:a066ee2908007256ef908b4091d91f35f963ab4c,1488785494";var csrfMagicName = "__csrf_magic";</script><script src="/csrf/csrf-magic.js" type="text/javascript"></script></head>
  <body class="page-login">

  <div class="container">
   

    <main class="login-modal-container">
      <header class="login-modal-head" style="height:55px;">
        <div class="navbar-brand">
          <img src="/ui/themes/opnsense/build/images/default-logo.png" height="30" alt="logo"/>
        </div>
      </header>

      <div class="login-modal-content">
        <div id="inputerrors" class="text-danger">&nbsp;</div><br />

            <form class="clearfix" id="iform" name="iform" method="post" autocomplete="off" action="/index.php"><input type='hidden' name='__csrf_magic' value="sid:7a7f16c6317e0e693af8c8d09a4244ea2f82319e,1488785494;ip:a066ee2908007256ef908b4091d91f35f963ab4c,1488785494" />

        <div class="form-group">
          <label for="usernamefld">Username:</label>
          <input id="usernamefld" type="text" name="usernamefld" class="form-control user" tabindex="1" autofocus="autofocus" autocapitalize="off" autocorrect="off" />
        </div>

        <div class="form-group">
          <label for="passwordfld">Password:</label>
          <input id="passwordfld" type="password" name="passwordfld" class="form-control pwd" tabindex="2" />
        </div>

        <button type="submit" name="login" value="1" class="btn btn-primary pull-right">Login</button>

      </form>

     
          </div>

      </main>
      <div class="login-foot text-center">
        <a target="_blank" href="https://opnsense.org" class="redlnk">OPNsense</a> (c) 2014-2016        <a href="https://www.deciso.com/" class="tblnk">Deciso B.V.</a>
      </div>

    </div>

    <script type="text/javascript">CsrfMagic.end();</script></body>
  </html>
Title: Re: Backup OPSsense configuration with wget
Post by: fabian on March 06, 2017, 03:15:56 pm
Thanks for your response, it was not required anymore to add this code to debug. You should always mention any changes you did which may break some scripts like a changed port.
Title: Re: Backup OPSsense configuration with wget
Post by: orsomannaro on March 09, 2017, 09:21:20 am
Do you use 17.1.2?

Updating OPSsense to version 17.1.2 the script works fine.

Thank you so much Fabian!
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: Perun on February 02, 2018, 09:19:08 am
does someone have a sh script for this?
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: franco on February 02, 2018, 09:25:06 am
As stated previously, that script should to do fine without bash. Can always check with "sh -n script.sh".
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: Perun on February 02, 2018, 09:47:46 am
hmm it doesnt work for me...

I have modified it a little:
Code: [Select]
#!/bin/sh
 
TMP=$(mktemp -d)
URL=$1
LOGIN='mybackupuser'
PASS='mybackuppass'
 
# Submit the login form with the previous values, and save a new CSRF token
/usr/bin/wget -q -O /dev/null --keep-session-cookies --save-cookies $TMP/cookies.txt --no-check-certificate  \
   --post-data "login=Login&usernamefld=$LOGIN&passwordfld=$PASS" $URL/diag_backup.php
 
# Save only the config
/usr/bin/wget -q --keep-session-cookies --load-cookies $TMP/cookies.txt --save-cookies $TMP/cookies.txt --no-check-certificate \
    --post-data "download=Download%20Configuration&donotbackuprrd=yes" $URL/diag_backup.php -O /srv/backup/config-$1-`date +%Y%m%d%H%M%S`.xml \
 
rm -f $TMP/*.txt
rmdir $TMP

gave the mybackupusr the permission to 'see' the backup site

and got this in /srv/backup/config-myhost-20180202094316.xml:

Code: [Select]
<!doctype html>
<!--[if IE 8 ]><html lang="en" class="ie ie8 lte9 lte8 no-js"><![endif]-->
<!--[if IE 9 ]><html lang="en" class="ie ie9 lte9 no-js"><![endif]-->
<!--[if (gt IE 9)|!(IE)]><!--><html lang="en" class="no-js"><!--<![endif]-->
  <head>

    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">

    <meta name="robots" content="noindex, nofollow, noodp, noydir" />
    <meta name="keywords" content="" />
    <meta name="description" content="" />
    <meta name="copyright" content="" />
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" />

    <title>Login</title>

    <link href="/ui/themes/opnsense/build/css/main.css" rel="stylesheet">
    <link href="/ui/themes/opnsense/build/images/favicon.png" rel="shortcut icon">

    <script type="text/javascript" src="/ui/js/jquery-3.2.1.min.js"></script>
    <script type="text/javascript" src="/ui/js/jquery-migrate-3.0.1.min.js"></script>


    <!--[if lt IE 9]><script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.2/html5shiv.min.js"></script><![endif]-->

 
            <script type="text/javascript">
              $( document ).ready(function() {
                  $.ajaxSetup({
                  'beforeSend': function(xhr) {
                      xhr.setRequestHeader("X-CSRFToken", "SEtpaEhPdXN6OWlQMmphdHdxNitadz09" );
                  }
                });
              });
            </script>
            </head>
  <body class="page-login">

  <div class="container">
   
    <main class="login-modal-container">
      <header class="login-modal-head" style="height:55px;">
        <div class="navbar-brand">
          <img src="/ui/themes/opnsense/build/images/default-logo.png" height="30" alt="logo"/>
        </div>
      </header>

      <div class="login-modal-content">
        <div id="inputerrors" class="text-danger">&nbsp;</div><br />

            <form class="clearfix" id="iform" name="iform" method="post" autocomplete="off" action="/diag_backup.php"><input type="hidden" name="RFlMUHFrV3p1M1RqNzhEcFdINFZLdz09" value="SEtpaEhPdXN6OWlQMmphdHdxNitadz09" />

        <div class="form-group">
          <label for="usernamefld">Username:</label>
          <input id="usernamefld" type="text" name="usernamefld" class="form-control user" tabindex="1" autofocus="autofocus" autocapitalize="off" autocorrect="off" />
        </div>

        <div class="form-group">
          <label for="passwordfld">Password:</label>
          <input id="passwordfld" type="password" name="passwordfld" class="form-control pwd" tabindex="2" />
        </div>

        <button type="submit" name="login" value="1" class="btn btn-primary pull-right">Login</button>

      </form>

     
          </div>

      </main>
      <div class="login-foot text-center">
        <a target="_blank" href="https://opnsense.org/" class="redlnk">OPNsense</a> (c) 2014-2018        <a href="https://www.deciso.com/" class="tblnk">Deciso B.V.</a>
      </div>

    </div>

    </body>
  </html>

Greetz
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: fabian on February 02, 2018, 08:14:03 pm
This means you are not authenticated.
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: ccesario on September 17, 2018, 02:31:38 pm
Hi Folks, this script is it working ?

Im testint it with 18.7 series withou success.

best regards

Carlos
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: fabian on September 17, 2018, 05:40:23 pm
Just tested it - yes it works. Are you missing a dependency or the credentials?
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: gislaved on February 10, 2019, 05:25:12 pm
Am I right if this doesn't work at all on 19.x ?

If it should work please add a working example.

Thanks!
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: franco on February 10, 2019, 05:30:30 pm
> Am I right if this doesn't work at all on 19.x ?

No, it is ok.
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: gislaved on February 10, 2019, 05:45:44 pm
> Am I right if this doesn't work at all on 19.x ?

No, it is ok.

Can you please post a working, tested, example as reference ? I think it clears up the topic as well. Thanks!
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: franco on February 10, 2019, 05:58:34 pm
install os-api-backup plugin and see https://github.com/opnsense/plugins/pull/895#issuecomment-458158323
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: gislaved on February 10, 2019, 06:01:13 pm
install os-api-backup plugin and see https://github.com/opnsense/plugins/pull/895#issuecomment-458158323

That is not the bashscript I requested for. I want to do something else with bash as well, like exporting certificates but want to have this working first.

I'm looking forward to it :)
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: franco on February 10, 2019, 06:02:35 pm
This topic is about config.xml backups, sorry.
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: gislaved on February 10, 2019, 06:30:17 pm
This topic is about config.xml backups, sorry.

Which I try to get with a working bash script you say which is working referring to a script you are not wiling to (re)post ?

From that I can see what I do else but this is my base, so please share what you say you have working or is working.

Thanks again!
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: franco on February 11, 2019, 08:48:48 am
I'm only trying to help and I feel you're not accepting it. I'm sorry I cannot communicate this any better.
Title: Re: [SOLVED] Backup OPSsense configuration with wget
Post by: gislaved on February 11, 2019, 02:31:56 pm
I'm only trying to help and I feel you're not accepting it. I'm sorry I cannot communicate this any better.

Sorry for that, we might have some misunderstanding here! I have some other way I'm working on where I actually need this for so for now it's not needed anymore.

Thanks!