Messages

[/root/rogers/git/conf_backup]

I have gone through the following steps now.

1) Rebuilt my git repo from my gui backups and put it at /root/rogers/git/conf_backup
2) Added symbolic link as:

Code Select Expand

cd /conf/backup
ln -s ~/rogers/git/conf_backup git

A couple of quick test changes are now showing up in git, so I'll stick with this for now.

I do prefer the git over the xml files as it keeps the full history in much less space.

I'll probably look for a good place to programmatically add the link back upon reboot so it's always present since I expect I'll always reboot after a restore.

[rm /conf/backup/*.xml]

Maybe even rm /conf/backup/*.xml would be a safer implementation of the flush operation ?

[os-git-backup]

Thanks for all the feedback!

Yes, it was os-git-backup that I have been using, but only with a local repo in /conf/backup/git, which unfortunately I had not cloned elsewhere.

It sounds like the symlink idea is worth testing. I can at least build a new repo using the backups I have made through the GUI, since those are stored on a different machine and give me a reasonable amount of history on their own.

I guess it also makes sense for me to add a nightly cron job to do an additive rsync of /conf/backup to another machine, since it sounds like I cannot uncheck this option and make that setting stick.

I do feel that the "Flush (full) local configuration history" option should be off by default, though. I would not normally expect a restore operation to destroy the history that would allow me to undo that restore.

I could also envisage a workflow where you roll back to an older configuration temporarily and then want to roll forward again. Flushing the history makes that impossible unless you have another copy elsewhere.

[System → Configuration → Backups → Restore]

Hi,

I recently did a System → Configuration → Backups → Restore configuration and quickly discovered that I lost everything under /conf/backup.

I guess I should have been a bit more cautious. After looking into it, I see that I had "Flush (full) local configuration history" selected, which I assume explains what happened.

From what I have found, I assume the deleted files are not recoverable through OPNsense itself, particularly since I am using UFS rather than ZFS. I do still have the configuration backups that I downloaded through the interface and stored externally, so I could probably reconstruct at least some of the Git history from those, or simply accept the loss.

Is there any practical way to recover either the old Git repository or the deleted XML configuration files on UFS?

I was also wondering whether the restore behavior could be made a little less aggressive. For example, could "Flush (full) local configuration history" be unchecked by default?

Finally, would moving the Git repository outside /conf/backup and replacing it with a symlink make it safer from this particular operation? For example:

Code Select Expand

mkdir -p /backup/git
ln -s /backup/git /conf/backup/git
Would the flush operation follow the symlink and delete the target contents, or would it only remove the symlink itself?

Thanks!

Cool, thanks for the explanation/reassurance!

Thanks Franco! I am no longer stuck on 18.7. Running that looked like it was bumping me to 19.1.3 and that is what my Dashboard reports too after completion, and my change log. It seemed that there were still some upgrades available though which were for 19.1.2. That did not seem to make sense, but I have applied everything and now it is running happily with dashboard reporting 19.1.3 and audit reporting 19.1.2 and no updates available. Hopefully I didn't break anything and versions will smooth out at the release following 19.1.3

I have gone through this upgrade process quite a few times now and the kernel parts seem fine, but opnsense never makes it to new new version. I tried doing an update on the console instead and I see that the issue may well be related to a package I have installed.

https://www.sunnyvalley.io/sensei

I clicked on the unlock to perform the upgrade from 18.7.10 to 19.1 and got the following message.

***GOT REQUEST TO UPGRADE: maj***
Fetching packages-19.1-OpenSSL-amd64.tar: .................................. done
Fetching base-19.1-amd64.txz: ............ done
Fetching kernel-19.1-amd64.txz: .............................................. failed, no signature found
***DONE***

At this point it seemed like there was no way to upgrade and so searching the forum I found:

• https://forum.opnsense.org/index.php?topic=11199.0

Rather than remove the files suggested in the final post there, I mv'ed them out of the way and things looked promising. I had an upgrade button again and so I proceeded with the upgrade. All looked good, a few reboots, etc and the system was back up and functional.

Checking the versions now, though, things do not look right. The lobby dashboard now shows

Versions   OPNsense 18.7.10_4-amd64
FreeBSD    11.2-RELEASE-p8-HBSD
OpenSSL    1.0.2q 20 Nov 2018

And my upgrade screen shows what it did previously (That I am on 19.7.10) but only very briefly. After s second or two, It changes its mind and instead offers me the following upgrade options.

There are 2 updates available, total download size is 98.7MiB. This update requires a reboot.

Package Name   Current Version   New Version   Required Action
base           19.1              18.7.10       upgrade
kernel         19.1              18.7.10       upgrade

Given that things seem to be working fine but the state is looks confused, I am not sure what the safest path forward would be.

Great, thanks for clarifying!

I am currently running 18.7.10 and now see vulnerabilities being reported. To upgrade I have to unlock the next release. I am wanting to remain on stable releases and avoid development releases or release candidates. Initially, when 19.1 was the only choice I thought it was a release candidate and and that the unlock would be removed when it became stable, however it looks like I mis-understood as there are now multiple releases queued up (19.1, 19.1.1, 19.1.2, 19.1.3) and the forums label it as production.

Looking at https://opnsense.org/about/road-map/, I see:

• NEXT RELEASE 19.1 - Januari 2019
• LATEST RELEASE 18.7 - July 31, 2018

So I feel 18.7 is latest prod.

From the "book link" by 19.1.3 on the UI, I read "recommend running 18.7 until this is taken care of" so again I think 18.7 is current prod.

The forum however labels 18.7 as legacy and 19.1 as production.

So, will the unlock go away when 19.1+ is as stable as it will get or will it always require a conscious click?
Since 18.7 is EOL, I guess vulnerabilities will never get fixed, so is the 19.1 path going to lead to better security with a little upgrade risk?

I'b de grateful for any tips of fixing or troubleshooting the most granular Insight graphs.

Insight worked fine for a while, but broke for me a while back after an upgrade through the UI. I am now running 18.7.8 and had hoped my problem would go away with subsequent updates, but it has persisted. I have read numerous articles, but everything I have seen that may help looked like it would loose my current history. I am hoping to be able to fix the granular graphs and preserve my history.

My Insight works partially as follows

Interface totals (bits/sec) - Last 2 hours - No Data Available
Interface totals (bits/sec) - Last 8 hours - No Data Available
Interface totals (bits/sec) - Last 24 hours - No Data Available
Interface totals (bits/sec) - All the rest seem to work fine.

Top usage ports / sources (bytes) - Last 2 hours - No Data Available
Top usage ports / sources (bytes) - Last 8 hours - No Data Available
Top usage ports / sources (bytes) - All the rest seem fine.

I have rebooted the system.
I have reinstalled the package flowd.
I ran Reporting --> Settings --> Repair Netflow Data
I checked /var/log/flowd.log and see current data using flowd-reader flowd.log

The flow logs seem to be updating fine and rotating.

root@OPNsense:/var/log # ls -lh flowd.log*
-rw-------  1 root  wheel   1.4M Nov 23 09:34 flowd.log
-rw-------  1 root  wheel    11M Nov 23 08:17 flowd.log.000001
-rw-------  1 root  wheel    11M Nov 22 23:09 flowd.log.000002
-rw-------  1 root  wheel    11M Nov 22 18:45 flowd.log.000003
-rw-------  1 root  wheel    11M Nov 22 08:45 flowd.log.000004
-rw-------  1 root  wheel    11M Nov 21 22:08 flowd.log.000005
-rw-------  1 root  wheel    11M Nov 21 13:38 flowd.log.000006
-rw-------  1 root  wheel    11M Nov 20 22:34 flowd.log.000007
-rw-------  1 root  wheel    11M Nov 20 11:28 flowd.log.000008
-rw-------  1 root  wheel    11M Nov 19 22:27 flowd.log.000009
-rw-------  1 root  wheel    11M Nov 19 07:06 flowd.log.000010

I checked unique flowd type messages in system.log (One issue starting flowd_aggregate a while ago)

root@OPNsense:/var/log # grep flowd system.log | cut -d' ' -f 5- | sort -u
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/metadata.sqlite
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/metadata.sqlite [done]
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/src_addr_000300.sqlite
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/src_addr_000300.sqlite [done]
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/src_addr_003600.sqlite
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/src_addr_003600.sqlite [done]
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/src_addr_086400.sqlite
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/src_addr_details_086400.sqlite
OPNsense flowd_aggregate.py: sqlite3 repair /var/netflow/src_addr_details_086400.sqlite [done]
OPNsense flowd_aggregate.py: start watching flowd
OPNsense flowd_aggregate.py: startup, check database.
OPNsense flowd_aggregate.py: vacuum /var/netflow/dst_port_000300.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/dst_port_003600.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/dst_port_086400.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/interface_000030.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/interface_000300.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/interface_003600.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/interface_086400.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/src_addr_000300.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/src_addr_003600.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/src_addr_086400.sqlite
OPNsense flowd_aggregate.py: vacuum /var/netflow/src_addr_details_086400.sqlite
OPNsense flowd_aggregate.py: vacuum done
OPNsense pkg-static: flowd reinstalled: 0.9.1_3 -> 0.9.1_3
OPNsense root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate
flowd_aggregate.py: start watching flowd
flowd_aggregate.py: startup, check database.
flowd_aggregate.py: vacuum /var/netflow/dst_port_000300.sqlite
flowd_aggregate.py: vacuum /var/netflow/dst_port_003600.sqlite
flowd_aggregate.py: vacuum /var/netflow/dst_port_086400.sqlite
flowd_aggregate.py: vacuum /var/netflow/interface_000030.sqlite
flowd_aggregate.py: vacuum /var/netflow/interface_000300.sqlite
flowd_aggregate.py: vacuum /var/netflow/interface_003600.sqlite
flowd_aggregate.py: vacuum /var/netflow/interface_086400.sqlite
flowd_aggregate.py: vacuum /var/netflow/src_addr_000300.sqlite
flowd_aggregate.py: vacuum /var/netflow/src_addr_003600.sqlite
flowd_aggregate.py: vacuum /var/netflow/src_addr_086400.sqlite
flowd_aggregate.py: vacuum /var/netflow/src_addr_details_086400.sqlite
flowd_aggregate.py: vacuum done
root@OPNsense:/var/log #
root@OPNsense:/var/log # grep 'failed to start flowd_aggregate' system.log
Nov  4 09:01:15 OPNsense root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate
Nov  4 09:01:24 OPNsense root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate
Nov  4 09:01:31 OPNsense root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate
Nov  4 09:06:46 OPNsense root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate
Nov  4 09:15:22 OPNsense root: /usr/local/etc/rc.d/flowd_aggregate: WARNING: failed to start flowd_aggregate

I took a shot at making sure databases were valid.

# pwd                                                                                                                                                                             
/var/netflow
# for sqlite in `ls *sqlite`
do
echo $sqlite; sqlite3 ${sqlite} 'pragma integrity_check;' '.exit'
done
dst_port_000300.sqlite
ok
dst_port_003600.sqlite
ok
dst_port_086400.sqlite
ok
interface_000030.sqlite
ok
interface_000300.sqlite
ok
interface_003600.sqlite
ok
interface_086400.sqlite
ok
metadata.sqlite
ok
src_addr_000300.sqlite
ok
src_addr_003600.sqlite
ok
src_addr_086400.sqlite
ok
src_addr_details_086400.sqlite
ok


I did not do I ran Reporting --> Settings --> Reset Netflow Data
I have been avoiding this as I did not want to loose what it has collected.