Messages

1726

21.7 Legacy Series / Re: 21.7.3. - high CPU and MEM usage

« on: September 30, 2021, 11:18:49 pm »

probably my post is too verbose but is on topic.
High cpu usage post upgrade to 21.7.3 and I'm trying to point to what might be a problem. Or rather seeking confirmation if is normal to have flowd_aggregate.py running if there is no netflow enabled.
Sorry if is not clear.

1727

21.7 Legacy Series / Re: 21.7.3. - high CPU and MEM usage

« on: September 30, 2021, 07:57:29 pm »

I'm sorry for the "me too" post that mixes issues but it might warrant it.
I upgraded today from 21.7.2 to 21.7.3  and rebooted the firewall.
Yesterday I had disabled Netflow by leaving " Listening interfaces" and " WAN interfaces" as "Nothing selected".
Once all services had restarted post-boot, I gave it a few minutes to settle down and checked memory and cpu consumption. The cpu was high for a while.

htop gave me one process to check that looked like interesting.
/usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.
This had a WCPU value of 98%, constant for some 5 minutes.
A couple of ploints: an fstat showed the open file that the script was using was about 15 MB in size.
A quick look at the script I read: that it will rotate files at 10 MB and it's main method will check and repair it's sqlite db and log 'start watching flowd' to syslog.
It was there:
2021-09-30T17:58:32   /flowd_aggregate.py[54199]   vacuum done   
2021-09-30T17:58:31   /flowd_aggregate.py[54199]   vacuum interface_086400.sqlite   
2021-09-30T17:58:31   /flowd_aggregate.py[54199]   vacuum interface_003600.sqlite   
2021-09-30T17:58:31   /flowd_aggregate.py[54199]   vacuum interface_000300.sqlite   
2021-09-30T17:58:31   /flowd_aggregate.py[54199]   vacuum interface_000030.sqlite   
2021-09-30T17:58:15   /flowd_aggregate.py[54199]   vacuum dst_port_086400.sqlite   
2021-09-30T17:58:15   /flowd_aggregate.py[54199]   vacuum dst_port_003600.sqlite   
2021-09-30T17:58:15   /flowd_aggregate.py[54199]   vacuum dst_port_000300.sqlite   
2021-09-30T17:58:07   /flowd_aggregate.py[54199]   vacuum src_addr_086400.sqlite   
2021-09-30T17:58:07   /flowd_aggregate.py[54199]   vacuum src_addr_003600.sqlite   
2021-09-30T17:58:07   /flowd_aggregate.py[54199]   vacuum src_addr_000300.sqlite   
2021-09-30T17:57:48   dhclient[74977]   Creating resolv.conf   
2021-09-30T17:57:39   /flowd_aggregate.py[54199]   vacuum src_addr_details_086400.sqlite   
2021-09-30T17:57:08   /flowd_aggregate.py[54199]   start watching flowd   
2021-09-30T17:56:24   opnsense[53768]   plugins_configure newwanip (execute task : webgui_configure_do(,wan))   
2021-09-30T17:56:24   opnsense[53768]   plugins_configure newwanip (execute task : vxlan_configure_interface())   
2021-09-30T17:56:15   opnsense[53768]   plugins_configure newwanip (execute task : unbound_configure_do(,wan))   
2021-09-30T17:56:15   opnsense[53768]   plugins_configure newwanip (execute task : openssh_configure_do(,wan))   
2021-09-30T17:56:15   opnsense[53768]   plugins_configure newwanip (execute task : opendns_configure_do())   
2021-09-30T17:56:14   opnsense[53768]   plugins_configure newwanip (execute task : ntpd_configure_do())   
2021-09-30T17:56:13   opnsense[53768]   /usr/local/etc/rc.newwanip: Curl error occurred: Resolving timed out after 15001 milliseconds   
2021-09-30T17:56:12   /flowd_aggregate.py[54199]   startup, check database.

I thought it took longer than the 2 mins the logs show but maybe I my recollection is incorrect.

What I see now is that that's the "hoovering" routine that spikes the cpu. At first I thought new python version, needs to maybe convert the data in files or db and was needed as a one-off.
However the script runs on a loop that seems outside the in-script "vacuum_interval = (60*60* # 8 hour vacuum cycle"

Question: is it normal that it runs even when Netflow is disabled?

I've not sighup'ed it in case is needed and this is how is meant to be.

1728

General Discussion / Re: NOOB question

« on: September 30, 2021, 02:18:04 pm »

What I've found is that it helps to have the wan and lan interfaces plugged in when installing so that the routines set a lot of it up. Then for DNS it will also be defaulting to normal values. The same goes for DHCP and NAT.
But just in case just go over the basics in the manual and there should be very little to change, and some of it will be for preference.
https://docs.opnsense.org/manual/install.html#initial-configuration

1729

General Discussion / Re: LAN lose internet every ~14d

« on: September 30, 2021, 01:45:02 pm »

I would look at services consuming resources over time. Log to shell after a few days and see what is consuming ram for example.
Unbound can have it's log verbosity increased from the UI but as always, it needs more attention for the storage used.
Top is in base and htop can be added from mimugmail's repo.

1730

21.1 Legacy Series / Re: how to limit stepson's usage of internet? new to opsense.

« on: September 28, 2021, 04:01:05 pm »

It would be useful to know what you tried and how are things setup at the moment in your network.

1731

21.7 Legacy Series / Re: Can't install updates

« on: September 28, 2021, 03:23:33 pm »

You're on the right track I think, DNS problem.
What do you have set in System > Settings > General > DNS Servers
and is " Do not use the local DNS service as a nameserver for this system " ticked?

1732

General Discussion / Re: OPN as a PXE boot server

« on: September 26, 2021, 02:21:57 am »

thanks for the pointer Fabian.
I'm happy to report I had success. I had to read documentation from syslinux, the other sense firewall distribution and digging around opn system files, plus a packet capture.
I'll write it up sometime for anyone trying to do this.

1733

General Discussion / Re: OPN as a PXE boot server

« on: September 25, 2021, 09:54:56 pm »

Well this is interesting. tftpd seems to be working. I went to basics and simply connected with a machine in the LAN via tftp. I was able to download the files requested without problem.
So that part seems fine at the moment. Now I need to figure out why clients seem to be unable to get the pxe booting options offered by tftpd on OPN. I suspect I need to read some RFCs.
I shall post when I know more.

1734

General Discussion / Re: OPN as a PXE boot server

« on: September 25, 2021, 12:44:18 am »

I just got back from a few days away. I'll resume this.
I can't figure out what the plugin is meant to do/not do and how so I can plug the gaps. That's all I need at the moment.

1735

General Discussion / Re: OPN as a PXE boot server

« on: September 21, 2021, 08:04:09 pm »

Anyone with pointers?

1736

General Discussion / Re: unbound just died during the night, how to use monit

« on: September 21, 2021, 06:32:25 pm »

ok if the service dies on you then of course to you it is unstable. That is fair enough.
I am trying to say many of use don't see that behaviour. Mine never stops unless I cause it to.
The problem is one only you can diagnose, a forum can sign-post where to look but every setup is different.

I'm not an expert in Unbound, I used to use DNSMasq but I've set it up and works fine.
What I would do is approach it like every problematic service: find out where it logs, if is possible to increase the verbosity, try to reproduce the problem and check logs.

1737

General Discussion / Re: unbound just died during the night, how to use monit

« on: September 19, 2021, 10:38:25 pm »

Unbound is stable. If I may say, of course monitoring his important you'd be better off solving the problem with Unbund rather than knowing when it has failed.

1738

General Discussion / OPN as a PXE boot server

« on: September 17, 2021, 12:17:37 am »

Hi.
I'm trying to setup OPN to serve PXE booting options to install OSs from the network.
Same question as https://forum.opnsense.org/index.php?topic=23128.0 but it's an old post.
Once working I shall create the relevant documentation.

I've installed the os-tftp plugin and created the directory /usr/local/tftp and put in there my pxe files extracted from the syslinux-6.03.pkg i.e. pxelinux.0  and .c32 files

I've created my /usr/local/tftp/pxelinux.cfg/default file with my entries there pointing to a webserver I have in my LAN and started testing.

My tests are failing with:
"PXE-T01:File not found"

What I've found is that the plugin starts the tftp server and I can get the files with a tftp client and a "get" verb.
Also that it doesn't seem to use inetd as the service control mechanism and rc.d instead. All good.
I've also as a test included the OPN LAN address in the Services > DHCP4 > TFTP and the absolute path & NBP file.

Is the plugin still in development and not finished? I'd like some pointers to troubleshoot.

1739

21.7 Legacy Series / Re: Mutli-wan with dual-lan does not work

« on: September 16, 2021, 12:05:09 pm »

that's exactly what I was going to suggest to check/follow.
I don't have a multi-wan setup at the moment to check things. I'm sure someone will peep with suggestions.

1740

21.7 Legacy Series / Re: Mutli-wan with dual-lan does not work

« on: September 15, 2021, 09:54:29 pm »

I would start by checking your configured gateway(s).
https://docs.opnsense.org/manual/gateways.html