[SOLVED] Backup OPSsense configuration with wget

[gislaved]

I'm only trying to help and I feel you're not accepting it. I'm sorry I cannot communicate this any better.

Sorry for that, we might have some misunderstanding here! I have some other way I'm working on where I actually need this for so for now it's not needed anymore.

Thanks!

[frater]

@Fabian

I recently switched to opnsense coming from pfsense.
I have about 12 pfsense in the field and now I added an opnsense.

To back-up the configs of these routers I use an hourly cronjob that fetches the latest config and if it is (about) the same it will throw it away. This way I end up with a list of configs through the years that only reflect the changes.

I have written that script in bash, but only the code surrounding the fetching of the config.
I've seen your example in Ruby, but this language is so unfamiliar to me that I can't use it to create a bash counterpart of it.
I am hoping the reverse is not true and that you are familiar enough with bash and are able to tell me how to convert my code to "opnsense".

Here's the code I use for pfsense which is working to this day:

Code: [Select]

  if ! wget  -t1 --timeout=10 -qO- --keep-session-cookies --save-cookies /tmp/${IDENTIFIER}_cookies.txt ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > /tmp/${IP}-csrf.txt ; then
    echo "Error fetching cookie" >&2
    exit 1
  else
    [ ${HEADLESS} ] || echo "Got session cookie" >&2
  fi

  if ! wget -t1 --timeout=10 -qO- --keep-session-cookies --load-cookies /tmp/${IDENTIFIER}_cookies.txt --save-cookies /tmp/${IDENTIFIER}_cookies.txt --post-data "login=Login&usernamefld=${USER}&passwordfld=${PASS}&__csrf_magic=`cat /tmp/${IP}-csrf.txt`" ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php  | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > /tmp/${IP}-csrf2.txt ; then
    echo "Error pushing the session cookie" >&2
    exit 1
  else
    [ ${HEADLESS} ] || echo "Pushed cookie" >&2
  fi

  if ! wget -t1 --timeout=30 -qO ${FNAME} --keep-session-cookies --load-cookies /tmp/${IDENTIFIER}_cookies.txt --post-data "download=download&donotbackuprrd=yes&__csrf_magic=$(head -n 1 /tmp/${IP}-csrf2.txt)" ${WGETOPT} ${PROTO}://${IP}:${PORT}/diag_backup.php ; then
    echo "Error fetching ${FNAME}" >&2
    rm -f "${FNAME}"
    exit 1
  else
    [ ${HEADLESS} ] || echo "Fetched ${FNAME}" >&2
  fi


[chemlud]

How about a cron job along the line

Code: [Select]

rsync -av --update --partial --append --log-file=$HOME/.rsyncd.log <source> <destination>
?

https://www.freebsd.org/cgi/man.cgi?query=rsync

[frater]

How about a cron job along the line

Code: [Select]

rsync -av --update --partial --append --log-file=$HOME/.rsyncd.log <source> <destination>
?

https://www.freebsd.org/cgi/man.cgi?query=rsync

Thanks, but preferably not. I prefer to pull the config which allows me to have all the code and configuration centralized on 1 server. If I let the router push the config it needs to be configured on both the server as the client.
This means troubleshooting only has to be done on 1 end.. not on both ends.

Furthermore it does much more.
It compares the downloaded config with the latest one and deletes it if it is the same in important parts.
This can't be done elegantly if you push the firmware.
This concept also works for very simple routers.

I pull configs from different routers for over 10 years and I prefer it that way.
It is possible to tweak my current procedure to enable it to work as the one for pfsense does.
I seek help to do that. No alternative ways to do a back-up.

[chemlud]

eehm, rsync can be initiated from either side. Just saying. Apparently you never used it, otherwise you wouldn't mess around with wget... ;-)

[frater]

Thanks for your valuable input...

Can someone give me the correct sequence of fetches that need to be done to get the config through the https-interface?

[fabian]

Code: [Select]

wget --http-user=mdcO...DsfO --http-passwd=1l0kZaw..d3C -O tmp_config.xml --no-check-certificate --auth-no-challenge https://firewall.test/api/backup/backup/downloadplease do not use "--no-check-certificate" - this call works on a test machine. This uses the os-backup-api plugin. You can get the API credentials from the user configuration as a file download. key is the user, secret the password.

[frater]

Thanks...

That URL works....   ...in Chrome when I have previously logged in.
It doesn't when I logout.

With wget it gives me "Username/Password Authentication Failed."

So I can't directly use that URL without logging in first.
It seems I'm not providing the credentials.

I am using --no-check-certificate
You use it in your example, but write I shouldn't use it.
I take it as an advice to install a certificate. For now I need to to use it as it's not installed yet....

[fabian]

Thanks...

That URL works....   ...in Chrome when I have previously logged in.
It doesn't when I logout.

With wget it gives me "Username/Password Authentication Failed."

So I can't directly use that URL without logging in first.
It seems I'm not providing the credentials.

Please read my command carefully:

You provide the credentials using basic auth:

Code: [Select]

--http-user=mdcO...DsfO --http-passwd=1l0kZaw..d3C

This are not your standard credentials. This are the API credentials you get in your user configuration page.

I am using --no-check-certificate
You use it in your example, but write I shouldn't use it.
I take it as an advice to install a certificate. For now I need to to use it as it's not installed yet....

I copied the command 1:1 which I used on my development VM. On a real appliance, you should have a trusted certificate (even if it is your self singned CA which has been configured on your clients).

[frater]

Thanks...
I was suspecting this as I could not imagine this to be not working at all.
I've been looking for some hints somewhere but could not find it.

Now you mention that there's something like an API-key, I googled that and found this.
I assume I will have no problem using it after using the API-credentials.
I was not aware of the existence of such a thing as API-credentials in OpnSense.

https://docs.opnsense.org/development/how-tos/api.html

Is there a way to create a user which is restricted to downloading a back-up?

[fabian]

Yes, you can create a user, create an API key for that user and assign the privilege "Backup API" to it. Then the user can log in on the web interface to change his password but nothing else and he can download the config XML as well.

[frater]

I was able to assign a privilege to a group and made the user "apibackup" part of the group.
The interface is a bit counter-intuitive.
One needs to first create the group and only after editing it is possible to assign it privileges.

If you don't know beforehand it can be done there, it is hard to find out things without a manual.
I knew it had to be possible, so I first created the group.
Only when modifying the group, the "assign privileges" gets unlocked.

I don't like interfaces that "unlock" features when needed. Now that I know this is how it's done with opnsense I will be better prepared in the future...

I need to clean up my script a bit and will post it later on.

But many thanks...
I now have my backup-config working just like the one for my pfsense boxes.
I only need to install the api-backup, add a user and a group "backup"
Then I need to assign the privilege "GUI    apibackup" to the group backup and assign an API key/secret to the user.