insight stopt working

[syd]

Hello all,

After a reboot, insight stopt working. I get "No Data Available"
I'm also sending netflow data to prtg that is still working fine.
I have try-ed to populate the data in the netflow tab again.
No messages in the syslog, any ideas where to look ?

Syd

[syd]

Just upgraded to 16.1.17 problem is still there.
Anybody ??

[franco]

Are you using a nano image and/or periodic backups?

[syd]

Hello Franco,

Running:
OPNsense 16.1.17-amd64
FreeBSD 10.2-RELEASE-p18
OpenSSL 1.0.2h 3 May 2016

On esxi 5.5 started with config of monowall in early time of opnsense and updated along the way.
No scheduled backups

[franco]

Hi Syd,

Ok, there's an unconditional backup task at reboot time, too. Just making sure this one is affected too.

What happens is that the database gets copied during an inconsistent state and can't be recovered after boot.

I had an install where this undid itself after a while, but that's obviously not the right fix.

Here's a dump of my current files, I'm curious to know how yours looks:

Code: [Select]

# ls -lah /var/netflow/
total 103036
drwxr-x---   2 root  wheel   768B Jun 16 20:24 .
drwxr-xr-x  16 root  wheel   960B Jun 16 03:01 ..
-rw-r-----   1 root  wheel    76K Jun 16 20:24 dst_port_000300.sqlite
-rw-r-----   1 root  wheel   144K Jun 16 20:24 dst_port_003600.sqlite
-rw-r-----   1 root  wheel   9.4M Jun 16 20:24 dst_port_086400.sqlite
-rw-r-----   1 root  wheel   1.8M Jun 16 20:24 interface_000030.sqlite
-rw-r-----   1 root  wheel   1.3M Jun 16 20:24 interface_000300.sqlite
-rw-r-----   1 root  wheel   416K Jun 16 20:24 interface_003600.sqlite
-rw-r-----   1 root  wheel    40K Jun 16 20:24 interface_086400.sqlite
-rw-r-----   1 root  wheel   8.0K Jun 16 20:24 metadata.sqlite
-rw-r-----   1 root  wheel   284K Jun 16 20:24 src_addr_000300.sqlite
-rw-r-----   1 root  wheel   936K Jun 16 20:24 src_addr_003600.sqlite
-rw-r-----   1 root  wheel    16M Jun 16 20:24 src_addr_086400.sqlite
-rw-r-----   1 root  wheel    70M Jun 16 20:24 src_addr_details_086400.sqlite

Cheers,
Franco

[syd]

Hey franco,

First thnx for the help,

total 72172
drwxr-x---   2 root  wheel   512B Jun 12 20:43 .
drwxr-xr-x  29 root  wheel   512B Jun  1 06:43 ..
-rw-r-----   1 root  wheel   196K Jun  5 22:06 dst_port_000300.sqlite
-rw-r-----   1 root  wheel   432K Jun  5 22:06 dst_port_003600.sqlite
-rw-r-----   1 root  wheel   748K Jun  5 22:06 dst_port_086400.sqlite
-rw-r-----   1 root  wheel   2.1M Jun  5 22:13 interface_000030.sqlite
-rw-r-----   1 root  wheel   1.5M Jun  5 22:06 interface_000300.sqlite
-rw-r-----   1 root  wheel   252K Jun  5 22:06 interface_003600.sqlite
-rw-r-----   1 root  wheel    28K Jun  5 22:06 interface_086400.sqlite
-rw-r-----   1 root  wheel   8.0K Jun  5 22:06 metadata.sqlite
-rw-r-----   1 root  wheel   616K Jun  5 22:06 src_addr_000300.sqlite
-rw-r-----   1 root  wheel   2.9M Jun  5 22:06 src_addr_003600.sqlite
-rw-r-----   1 root  wheel   9.2M Jun  5 22:06 src_addr_086400.sqlite
-rw-r-----   1 root  wheel    52M Jun  5 22:06 src_addr_details_086400.sqlite

is my /var/netflow

[onazari]

hello;
I have the same problem too

[syd]

Hi,

The files that where in /var/netflow didn't change from the time the problem started, i turned netflow of removed all the files and turned it back on. It recreated the files again, but it still doesn't work.

This is the new dir:

Code: [Select]

total 140
drwxr-x---   2 root  wheel   1.0K Jun 18 10:25 .
drwxr-xr-x  29 root  wheel   512B Jun  1 06:43 ..
-rw-r-----   1 root  wheel    12K Jun 18 10:11 dst_port_000300.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 dst_port_003600.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 dst_port_086400.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 interface_000030.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 interface_000300.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 interface_003600.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 interface_086400.sqlite
-rw-r-----   1 root  wheel     0B Jun 18 10:11 metadata.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 src_addr_000300.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 src_addr_003600.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 src_addr_086400.sqlite
-rw-r-----   1 root  wheel    12K Jun 18 10:11 src_addr_details_086400.sqlite

Any clues ?

[franco]

Hi syd,

Your database files and metadata was definitely trashed. There is no data to be found. Can you give the following output too on your mounted filesystems?

# df -h

There is also netflow cache file in /var/log that may be corrupted too. The reporting database pulls the standard netflow data from there... Maybe deleting this file if seemingly damaged would be a better starting point.

# ls -lah /var/log/flowd.*

So far I haven't been able to reproduce the problem anymore.

The frontend needs a while time to collect initial data, but it should really jump back to life after 10 minutes.


Thanks,
Franco

[syd]

Hey Franco,

Code: [Select]

root@mono:~ #  ls -lah /var/log/flowd.*
-rw-------  1 root  wheel   2.8G Jun 21 22:32 /var/log/flowd.log
-rw-------  1 root  wheel    11M Jun  5 19:58 /var/log/flowd.log.000001
-rw-------  1 root  wheel    11M Jun  5 16:28 /var/log/flowd.log.000002
-rw-------  1 root  wheel    11M Jun  5 12:51 /var/log/flowd.log.000003
-rw-------  1 root  wheel    11M Jun  5 10:17 /var/log/flowd.log.000004
-rw-------  1 root  wheel    11M Jun  5 06:15 /var/log/flowd.log.000005
-rw-------  1 root  wheel    11M Jun  5 01:15 /var/log/flowd.log.000006
-rw-------  1 root  wheel    11M Jun  4 21:41 /var/log/flowd.log.000007
-rw-------  1 root  wheel    11M Jun  4 18:02 /var/log/flowd.log.000008
-rw-------  1 root  wheel    11M Jun  4 14:38 /var/log/flowd.log.000009
-rw-------  1 root  wheel    11M Jun  4 11:58 /var/log/flowd.log.000010
root@mono:~ # df -h
Filesystem     Size    Used   Avail Capacity  Mounted on"+

/dev/da0s1a     31G    4.3G     24G    15%    /
devfs          1.0K    1.0K      0B   100%    /dev
devfs          1.0K    1.0K      0B   100%    /var/dhcpd/dev+

Will get rid of -rw-------  1 root  wheel   2.8G Jun 21 22:32 /var/log/flowd.log
and see if it recreats it.

Grtz

Syd

[syd]

Quick update,

It works again

-rw-------   1 root   wheel   3.6M Jun 22 13:51 flowd.log

Is recreated and updating.
@ Franco If you want i could send you the faulty flowd.log, it is quit big 2.7 gig.

Thanx for pointing me in the right dir.

Grtz

Syd

[franco]

Hi Syd,

If we could pull that trade off it would be nice. This is a huge file and the likely cause of the graph not regenerating. Thanks for your help so far.

Do you have a server where you could push this file to grab privately?


Cheers,
Franco