Web and Ssh extremely slow (apu2)

[huzergackl]

On my box the web ui and even ssh takes extremely long to load. The main page usually opens after one minute after login (without showing the cpu load and stuff - this takes another 30 seconds). Also switching to the page with the firewall rules at least takes 30 Seconds. On logging in over ssh it takes about 45 seconds until the main menu gets visible. It's no fun to work with and to me it looks like something is horribly wrong with the box.

I just have one hand full of simple firewall rules but use quite a few services like dhcp, haproxy, suricata (constantly at about 3% cpu), igmp proxy and unbound dns but the system has very less traffic and the overall performance of the traffic is totally ok and i am able to use the maximum speed of my internet connection without any problems. Hardware offloading is off due to using suricata.

Here the ususal load if i do nothing over the web ui:

Code: [Select]

last pid: 94959;  load averages:  0.05,  0.08,  0.40                                                                                      up 0+00:59:22  11:32:52
36 processes:  1 running, 35 sleeping
CPU:  0.7% user,  0.0% nice,  0.0% system,  0.6% interrupt, 98.7% idle
Mem: 669M Active, 141M Inact, 805M Wired, 130M Buf, 2288M Free
Switching to the firewall rules page (cpu stays that high until the page is loaded):

Code: [Select]

last pid: 34001;  load averages:  0.78,  0.33,  0.44                                                                                      up 0+01:02:13  11:35:43
36 processes:  2 running, 34 sleeping
CPU: 25.6% user,  0.0% nice,  0.6% system,  0.6% interrupt, 73.2% idle
Mem: 694M Active, 366M Inact, 807M Wired, 131M Buf, 2037M Free
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
90839 root          1  98    0   115M    99M CPU2     2   2:18 100.14% php-cgi
 9158 root          7  20    0  2761M   338M nanslp   3   4:56   3.56% suricata
Gstat shows almost no io while opening the firewall rule page (that took around 45 seconds this time):

Code: [Select]

dT: 1.003s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      2      0      0    0.0      2     64    0.6    0.1| ada0
    0      0      0      0    0.0      0      0    0.0    0.0| ada0p1
    0      0      0      0    0.0      0      0    0.0    0.0| ada0p2
    0      2      0      0    0.0      2     64    0.6    0.1| ada0p3
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/bf8f817a-46e6-11eb-bbd7-000db943b3f0
    0      0      0      0    0.0      0      0    0.0    0.0| msdosfs/EFISYS
    0      0      0      0    0.0      0      0    0.0    0.0| gpt/bootfs
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/c00913a5-46e6-11eb-bbd7-000db943b3f0
    0      2      0      0    0.0      2     64    0.6    0.1| gpt/rootfs

To me it looks like there is something horribly wrong and i don't quite get why the web ui or ssh is so extremely slow. The small box does not have a lot of processing power but i would have expected quite a bit more with its 4 core 1ghz cpu. (I also found the form post related to issues with Firefox and tested with chrome. It has nothing to do with the browser in my case)

Did somebody have similar issues? What else can i do to troubleshoot this problem?
Thanks in advance.

[huzergackl]

Still no clue whats going on with the web ui. Did some Packet captures to see if there are any issues within the network but everything works just fine and i don't have an idea how to further troubleshoot the cpu consumption of the php process.

Here the performance test of the lan interface (over which i access the box). It looks fine:

Code: [Select]

iperf3 -c 10.0.10.1 -u -t 60 -i 10 -b 1000M
Connecting to host 10.0.10.1, port 5201
[  4] local 10.0.10.10 port 51720 connected to 10.0.10.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec  144413
[  4]  10.00-20.00  sec  1.11 GBytes   951 Mbits/sec  145051
[  4]  20.00-30.00  sec  1.11 GBytes   951 Mbits/sec  145053
[  4]  30.00-40.00  sec  1.11 GBytes   951 Mbits/sec  145051
[  4]  40.00-50.00  sec  1.11 GBytes   951 Mbits/sec  145052
[  4]  50.00-60.00  sec  1.11 GBytes   951 Mbits/sec  145052
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-60.00  sec  6.64 GBytes   950 Mbits/sec  0.034 ms  1975/869671 (0.23%)
[  4] Sent 869671 datagrams

iperf Done.


[almodovaris]

I have an APU2. Tweaking it is difficult, and overdoing it is a problem.

[huzergackl]

Holy shit, i finally found out whats the issue. It is the suricata ids.

If you had it enabled once with a lot of rules you are lost. Disabling it is not enough and does not improve the situation.

The rules are added to the config and the file size increases drastically (from 58 kb to 2.3 MB in my case). Even if you disable all rules and suricata in the ui they will stay inside the configuration file. Note that the ui says that there are no rules in this case.

What helped for me is to backup the configuration, manually remove the rules from the ids section and restore it afterwards.