Messages

I have now also set my powerd to maximum and restarted. Can confirm that the problem does not occur.

Code Select Expand


Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
-       -       523     55.64   940     998     536
0       0       431     46.42   927     999     194
1       1       486     52.01   935     998     123
2       2       619     65.20   950     998     92
3       3       557     58.92   945     998     127


This would mean that the problem occurs in connection with powerd.

Short-term recommendation for maximum performance would be to switch off powerd or set it to profile Maximum. However, with the effect that unnecessarily much energy is consumed.

I have a ticket open with the BIOS developers and will provide the information there. Because the goal should be that we can use the power dynamically without burning energy senselessly.

That is interesting! Thanks.

[@hushcoden:]

Thanks! Interesting!

@hushcoden: Your CPU seems to run normal - up to 1GHz. How long has it been running, when you run turbostat?

@pmhausen: Your CPU seems to be stuck at 600 MHz. As are mine after some time after a reboot.

The question is: What are the differences between the setups?

• Are all your NIC ports connected (under some load) and thus producing IRQs?
• What are your custom tunables?

What I can say: I have tried multiple BIOS versions without any difference. And even disabling Core Power Boost in BIOS didn't help. CPU is still stuck a 600MHz after one or two hours.

Maybe we will find the root cause.

OPNsense is such a useful software that I was happy to donate something via paypal. Thanks to all involved!

In my context, editing alias works normally.

[I would be interested to know how the situation looks like with your APU. If you want to participate, you can post your observations here. In the following I describe how you can determine the maximum busy clock.]

The APU2/3/4 devices are predestined for use as routers under OPNsense: Inexpensive reliable hardware with open source BIOS. Accordingly, it is often used by users here in the forum.

The performance of the used SOC AMD GX412TC is certainly much lower compared to i3 or i5 systems. However, I can say from experience that they are more than sufficient in many cases and more performance is simply not necessary. And that's why I prefer such a cost-effective and power-saving device whenever possible.

However, questions have been piling up here and in other forums over the past few months as to why APU-based firewall performance dips and doesn't seem to be enough for some, while it's fast enough for others in a similar context.

Of course, this always depends on the individual case: What services are enabled on the firewall? Does WAN have to be connected via PPPoE? How large is the MTU? Are the correct tunables set? Etc.

Meanwhile I am not sure if there is not one more aspect: I have two APU2 in use (APU2C4 and APU2E4). I noticed that they are really responsive and powerful after a reboot. After some time (several hours), they become noticeably sluggish and the CPU utilization suddenly seems higher.

I then noticed that after a reboot, the APU can scale up the frequency to the nominal maximum, which is 1GHZ. After a while, however, the maximum achievable frequency seems to be reduced to 600MHz. This then remains the same until a reboot. After the reboot, the 1GHz can be reached again for a certain time. Until suddenly the 600MHz limit applies again.

I would be interested to know how the situation looks like with your APU. If you want to participate, you can post your observations here. In the following I describe how you can determine the maximum busy clock.


You need console or SSH access to your OPNsense:

All the measurements are done on the console or in an SSH shell on the OPNsense. If you do not have access to the console, you can set up SSH access as follows:

• System: Settings: Administration
• Secure Shell Server: Enable Secure Shell
• Root Login: Permit root user login
• Authentication Method: Permit password login
• Now you should be able to access with ssh YOUR_USER@YOUR_FIREWALL_IP
• If you do not access with root, you may have to become super user: su

Note: After taking the measurements, the access can be deactivated again for security reasons.


You need to install and use the tool turbostat:

The measurements are done with the tool turbostat, which can be installed as follows:

Code Select Expand

pkg add http://pkg0.isc.freebsd.org/FreeBSD:12:amd64/latest/All/turbostat-4.17_2.txz
rehash

Before using turbostat you have to load the kernel module cpuctl once before doing measurements:

Code Select Expand

kldload cpuctl


A measurement series is started as follows:

Code Select Expand

turbostat --interval 3

After that the tool prints the CPU statistics every 3 seconds.

After a reboot everything runs normally and the output shows that the Bzy_MHz is near 1GHz:

Code Select Expand


root@router:~ # turbostat --interval 3
turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
turbostat: /dev/cpuctl0 missing, kldload cpuctl: No such file or directory
root@router:~ # kldload cpuctl
root@router:~ # turbostat --interval 3
turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): AuthenticAMD 13 CPUID levels; family:model:stepping 0xf:30:1 (15:48:1)
CPUID(1): SSE3 MONITOR - - - TSC MSR - -
CPUID(6): APERF, No-TURBO, No-DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, No-EPB
CPUID(7): No-SGX
NSFOD /sys/devices/system/cpu/cpu1/cpufreq/scaling_driver
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
-       -       355     38.76   915     998     170
0       0       247     28.12   880     998     32
1       1       222     25.44   871     998     69
2       2       414     44.66   928     998     43
3       3       536     56.82   943     998     26
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
-       -       313     34.39   910     998     334
0       0       410     44.19   928     998     184
1       1       336     36.68   915     998     80
2       2       236     26.66   885     998     49
3       3       269     30.02   898     998     21
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
-       -       267     29.60   904     998     247
0       0       520     54.88   947     998     68
1       1       289     31.59   914     998     82
2       2       127     15.56   813     998     42
3       3       135     16.36   825     998     55
^C


When the OPNsense has been running for a few hours, the output shows that the Bzy_MHz is below 600MHz (even under maximum load):

Code Select Expand


root@router:~ # turbostat --interval 3
turbostat version 17.06.23 - Len Brown <lenb@kernel.org>
CPUID(0): AuthenticAMD 13 CPUID levels; family:model:stepping 0xf:30:1 (15:48:1)
CPUID(1): SSE3 MONITOR - - - TSC MSR - -
CPUID(6): APERF, No-TURBO, No-DTS, No-PTM, No-HWP, No-HWPnotify, No-HWPwindow, No-HWPepp, No-HWPpkg, No-EPB
CPUID(7): No-SGX
NSFOD /sys/devices/system/cpu/cpu1/cpufreq/scaling_driver
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
-       -       52      8.61    599     998     162
0       0       78      13.01   599     998     37
1       1       37      6.21    599     998     69
2       2       44      7.40    599     998     35
3       3       47      7.82    599     998     21
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
-       -       57      9.56    598     997     208
0       0       87      14.48   598     996     51
1       1       45      7.53    598     996     79
2       2       55      9.15    599     998     62
3       3       42      7.09    599     998     16
Core    CPU     Avg_MHz Busy%   Bzy_MHz TSC_MHz IRQ
-       -       59      9.81    599     999     297
0       0       94      15.67   600     1000    67
1       1       45      7.56    600     1000    104
2       2       60      9.97    599     998     68
3       3       36      6.01    599     998     58
^C


When you report your observations, it would be interesting to know which BIOS version you have installed (can be conveniently viewed in the Hardware Information widget) and whether you have the Core Performance Boost feature set to enabled or disabled in the BIOS (in newer BIOSes, the default value is enabled).

On the APU2E4, the update ran completely automatically and smooth in less than 10 minutes. Many thanks to all who contributed to this!

[headless systems]

The PC Engines APU devices are a cost-effective, stable and reliable hardware for a firewall based on OPNsense. Accordingly, I have two of them in use.

The APUs are so-called headless systems. I.e. they have no monitor output but are initially connected via the serial port. This works fine and once OPNsense is up and running, console access is usually no longer necessary, since OPNsense can be managed completely via the web UI (and occasionally ssh for very special needs).

In very rare cases, however, it is necessary to access the console, e.g. to change the BIOS settings or if a manual intervention is necessary after an unsuccessful update (but I have never had to do this).

Of course, in such a rare case you can go with the notebook close to the firewall and temporarily access the console with a Serial2USB cable. I have done this in the past maybe once a year at most and otherwise tried to prevent it, also because it was a bit tedious.

Recently I saw on the web that you can easily create a telnet access to a serial port with the tool ser2net. Usually you can find instructions for a Raspberry Pi. But this works for all such single board computers. I had an old Orange Pi PC lying around, which I converted to my Serial2Network device:

• install armbian
• install ser2net as an autostarting service
• configure ser2net
• access it with telnet [IP] [portnumber

The Orange Pi PC is now sitting in our tech basement on top of the APU. It takes power from the USB port on the APU. And the Serial2USB cable from PC Engines connects the console of the APU to the USB port of the Orange Pi PC.

Here I came across the possibility and it is described: https://www.jpaul.me/2019/01/how-to-build-a-raspberry-pi-serial-console-server-with-ser2net/

If it is some kind of incompatibility between the firewall and the central switch, interposing a small switch might help for a test. This might solve the problem by "normalizing" the communication. It would be worth a try.

Bug report is filed: https://github.com/opnsense/core/issues/4648

Thank you!

Thank you franco! I will do that later today.

OPNsense 20.7.8-amd64

Today I noticed, that the "Firewall Logs" Widget showed many entries. But the firewall log itself (e.g. plain view) only contains two entries.

When opening the dashboard page, initially there are only two entries. But every second these two entries are added again and again until the maximum of entries in the widget is reached.

Is this a known problem?

I may not have understood the problem.

But in my case WAN1 and WAN2 are running with their own shaper and a group gateway WAN_Group_Failover over them. I.e. the rules of the shapers must work on WAN1 and WAN2 level.

Works in my context.

It depends.  ;)

It depends on how many and how complex rules you have at firewall and IDS level. I would expect that the small Protectli has enough power to run your load. Maybe Sensei could be too much. Give it a try.