Messages

1

21.1 Production Series / Re: High CPU usage with flowd_aggregate.py ... IPv6 is disabled ... any ideas?

« on: June 24, 2021, 11:59:11 pm »

It's a bit buried. I took a screenshot showing where you can check it and set the profile. You could also try the other profiles but I find HiAdaptive works pretty well. It won't hurt to try them all and see if this helps.

2

21.1 Production Series / Re: High CPU usage with flowd_aggregate.py ... IPv6 is disabled ... any ideas?

« on: June 24, 2021, 11:49:25 pm »

I don't personally use the Prometheus plugin so I'm unsure if that requires netflow to be enabled.

I'd also check that PowerD is enabled and set to HiAdaptive so that the CPU is scaling to its turbo clock as needed when an intensive task kicks off. Other than that I'm not sure what else to suggest. I see the CPU spikes on mine too but it's just a single thread and it's useful data so I don't turn off netflow. I haven't seen a situation where the task is running for a long time as a result of high bandwidth usage, and this is on a 500/500 connection that gets used pretty heavily. On a system with 4 dedicated cores I kind of consider it a non-issue.

If you just need bandwidth totals vnStat is a great plugin and more lightweight to run than using netflow.

3

21.1 Production Series / Re: Unbound DNS Locking Up

« on: June 24, 2021, 06:44:06 pm »

Just to confirm, are you saying that the Unbound service is stopping/crashing? Do you see any errors in the log file?

I've been using a custom config forwarding DoT to Quad9 for years as soon as it was supported by Unbound back in 2018. This has been very stable and the Unbound service itself has never shown any issues. There may be some clues in the log if it's a DNS provider problem.

4

21.1 Production Series / Re: High CPU usage with flowd_aggregate.py ... IPv6 is disabled ... any ideas?

« on: June 24, 2021, 05:35:13 pm »

If you have netflow enabled this is just the result of the schedule job aggregating new stats for you. It doesn't have anything to do with IPv6, it's just aggregating any traffic that the router is passing, including all IPv4.

You can stop this by turning off netflow.

I run a J3455 platform and I've only seen the aggregator use a single CPU thread and it only spikes briefly, usually less than 15 seconds. If you're seeing sustained CPU usage I would suspect something is slowing the ability of the aggregate job to complete its task, perhaps a bottleneck on storage? I'm using a 120GB Sata SSD.

5

Hardware and Performance / Re: APU2D4 very low throughput 1Gbit

« on: June 04, 2021, 11:09:15 pm »

I think the next thing to try just to rule out some weird inconsistency would be to attempt the same tests on the latest pfsense 2.5 and report back? If you're seeing the same limited throughput on the same platform that Teklager benchmarked then there has to be some other piece of the puzzle missing here. Maybe firmware or some other oddity?

6

Hardware and Performance / Re: APU2D4 very low throughput 1Gbit

« on: June 02, 2021, 06:07:15 pm »

While I do not consider myself an expert I do think Teklager actually left a hint. They specifically say no tuning is needed on pfSense 2.5 (which is FreeBSD 12.x based). What this really means is that anything 12.x based, to include OPNsense as well, will respond in a similar fashion.

Teklager also goes on to show single thread transfer tests with lower performance values when using pfSense 2.5 compared to pfSense 2.4 (and the FreeBSD 11.x tweaks).

Miroco posted the link with Teklager hinting at this.

https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/

"Gigabit config for pfSense 2.5.0. No tweaks are required! Don't follow any of the information listed below for pfSense 2.4.5."

At this point I would try these 4 things and report back. It's also important to make sure that the iperf tests you run are pushing traffic through the firewall (have the client on LAN, and another server on WAN). Don't just host iperf on one of the firewall interfaces.

In your tuneables set the following:

Code: [Select]

hw.ibrs_disable: 1 (just disable this to test throughput, there are security implications)
vm.pmap.pti: 0 (just disable this to test throughput, there are security implications)
dev.igb.0.eee_control: 0 (disable Energy Efficient Ethernet, do this for all IGB interfaces present on the device)
dev.igb.0.fc: 0 (disable Flow Control, do this for all IGB interfaces present on the device)
Set those tuneables and reboot. Then re-run the throughput tests and see if there is an improvement. All traffic shaping and the Netflow Insight plugin on OPNsense should also be disabled during these tests.

7

21.1 Production Series / Re: Installer crashes on amd x2 250

« on: June 02, 2021, 02:47:09 am »

I would agree with you, this seems to be a long standing issue. I suggested starting on the 19.x series because that was based on FreeBSD 11.x. However based on the links you provided, it appears a lot of people also see this issue on the 11.x installs too. Sorry I don't know of a reliable source of mirrors that provide older versions.

EDIT: On 2nd thought, would it be possible to just install OPNsense on a different system and swap the drive over to the older Athlon x2 system? That should get around the install bug, yes?

8

21.1 Production Series / Re: Installer crashes on amd x2 250

« on: June 02, 2021, 12:09:54 am »

I would start with the earliest available version on the official mirrors, which is OPNsense 19.7.

https://mirror.wdc1.us.leaseweb.net/opnsense/releases/19.7/

Start there and see if the install completes, then upgrade to the latest. I would recommend against using the i386 install. Download and attempt to use the AMD64 version first, it will have a much longer lifespan. i386 is no longer supported after OPNsense 20.1.

9

Hardware and Performance / Re: APU2D4 very low throughput 1Gbit

« on: May 31, 2021, 05:12:24 pm »

Again want to say, I don't own one of these devices but I think a lot of the configs posted here will not work with later versions of OPNsense (20.7 and 21.1). Both OPNsense 20.7+ and pfSense 2.5+ use FreeBSD 12.x for their base. FreeBSD 12.x uses iflib for NIC queues and no longer contains many of the old tunables what we would have used in FreeBSD 11.x.

Because of this, most of the configs being posted here will not have any impact.

There are still some tunables that you can set on the igb NIC driver, primarily disabling flow control and disabling EEE. These are the "new" tunables needed in the FreeBSD 12.x series:

Code: [Select]

dev.igb.X.fc (X is the interface number)
dev.igb.X.eee_control (X is the interface number)Setting both of these to 0 should disable the feature.

If you wish to check which options are available for the igb NICs, you can run the following at an SSH console

Code: [Select]

sysctl -a | grep igb
You will notice that if you run this command, there are now many different configurable settings that do not match any of the previously used configs that we relied on in FreeBSD 11.x.

10

21.1 Production Series / Re: OPNsense frequent crashes

« on: May 28, 2021, 03:16:03 pm »

It may be UEFI only. Try GPT volumes and UEFI booting and see if you can get the diagnostics to fire up?

11

21.1 Production Series / Re: OPNsense frequent crashes

« on: May 27, 2021, 06:38:23 pm »

No worries, let us know how you get on with the diagnostics.

Something else to add, I find that with these integrated SoC based X86 systems they tend to have components that aren't needed at all on a firewall. While you're in there checking the BIOS and potentially updating it, I'd also go through and disable all the onboard devices that you don't need. The sound card, LPT ports, serial ports (if unused), etc. can easily be disabled if you're just using VGA or HDMI output to a monitor.

Also, if the BIOS has a power management setting, make sure that it is set to "performance" or if possible, disable it and just let the system run at maximum speed.

12

Hardware and Performance / Re: Move to new appliance - config import

« on: May 27, 2021, 05:49:20 pm »

I'd just make a backup of the backup so that you always have the original config, just in case an edit goes wrong.

I've never tried to import or edit a pfSense config on to an OPNsense install. It may work but, who knows? At this point the projects are divergent enough I wouldn't trust it and if it was me, I would personally start over from scratch. At the very least, I would try it on a VM first in lab, snapshot the VM and then import the hacked config file to see what happens.

13

Hardware and Performance / Re: Move to new appliance - config import

« on: May 27, 2021, 05:25:31 pm »

I've sort of done this when moving between VMs on different platforms (VMware to HyperV) or when converting a VM to a physical platform.

I'll caveat this with my configs not using IDS/IPS/Sensei. I'm mainly just using a traffic shaper, DHCP, Unbound DNS, NTP server, basic stuff that is supported without add-ons.

For example, if I'm going from a HyperV based install (hnX NICs) to a VMware based installed (VMX NICs), I just edit the config file, find the 'hnX' adapters, and change these to 'vmxX', where X is the adapter number. The only thing you should double check is the order of the adapters in the new appliance that you are moving to. If you get this order correct, everything will come up perfectly after a config restore and reboot. It's been a very simple process for me and I've used this method dozens of times now.

14

21.1 Production Series / Re: OPNsense frequent crashes

« on: May 27, 2021, 06:24:25 am »

Have any diagnostics been performed on the hardware yet? If not, start with the basics like updating to the latest BIOS firmware, then do a full memtest (this will take a long time) and run some CPU stress tests. The memtest and CPU stress tests can be done with a bootable linux ISO such as Ubuntu or Fedora.

Other things to check would be the power supply and the CPU temperature. Both can be a common source of instability.

15

Hardware and Performance / Re: OPNSENSE and RealTek-NIC

« on: May 27, 2021, 12:50:52 am »

Hmm, you didn't mention before that you were using Sensei? That's a major factor that would have been good to know. 

You need to re-baseline and just run the performance tests with Sensei disabled and no other IPS/IDS packages running. If you have a traffic shaper configured, turn that off too.

Then re-run a download test with your newest (igb) network adapter and report back the results. I would suggest either using two iperf clients to push traffic through the firewall, or use a single client on the LAN side to download several linux ISOs via torrent. This should max out your connection and give you a good idea of what the max throughput will be.

While doing the above tests, also watch the output of this command at the SSH console: top -aSCHIP
Screenshot the CPU usage of that console when the throughput tests are running.