Slow WebGUI even when downloading packet capture

[voluhar]

On my HA setup I have very strange situation where I can not resolve slow WebGUI problem.
DNS resolving works normally, updates work normally but when I navigate from one to another page it takes like 10s.
Even if I click to download .cap file after finish of packet capture it takes 11.25s before it starts (packet capture file is 24Byte in size)
I was already looking into iostat and top but nothing special that will help me with resolution.

My version:
OPNsense 19.7.3-amd64
FreeBSD 11.2-RELEASE-p14-HBSD
OpenSSL 1.0.2s 28 May 2019

A will update as soon as possible but this is probably not problem because we have problem with Slow interface since we get this server in production.
I checked disks and health is OK.
Average load is 2.00 and CPU is Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz (8 cores)

Does anybody have any suggestion where can I start debugging it?

Best regards, Robi

[Goldorak92]

Hi,

I have the same issue. Login page takes 20s to load, some pages after login take less than a second and some other 10s.

I tried a fresh install on a vm, first config works fine (less than a second per page), importing the prod's backup, reboot, and them think, more than 10s on some pages...

Top command shows that php process hang at 105% of a core, but not one every pages.

My config: Opnsense 19.7.4 one a Qotom i5, 8gb ram and ssd.
Services/plugins : dhcpv4, network time, HA proxy, IPS (activated and even desativated).
Webgui: 8443
Ha front: 443

I planned to reinstall and re config, but if the issue could be resolved in another way... I will be happy with helping debugging.

G.

[Goldorak92]

As an update, even the ssh connection hangs and take more than 30s...

Here is an example of php-cgi processes hanging...

Code Select Expand


last pid: 87712;  load averages:  1.29,  0.54,  0.34                                             up 9+00:22:44  19:15:01
52 processes:  4 running, 48 sleeping
CPU: 57.7% user,  0.0% nice, 11.0% system,  0.2% interrupt, 31.2% idle
Mem: 46M Active, 1660M Inact, 713M Wired, 413M Buf, 5433M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
62467 root             1  95    0    98M 90848K CPU0    0   5:35 102.63% php-cgi
57577 root             1  95    0 83804K 74840K CPU3    3   1:42  99.84% php-cgi
33949 root             1  52    0 36752K 32428K piperd  2   0:00  22.92% python3.7
10737 root             1  21    0  1033M  2408K wait    3   0:00   0.31% flock
98488 root             1  20    0  1033M  2928K piperd  2   0:00   0.11% cron
13850 root             3  20    0 24356K 11356K kqread  0   1:39   0.04% syslog-ng
85038 root             1  20    0  1034M  3640K CPU2    2   0:00   0.04% top
98241 root             1  20    0  1033M  2888K nanslp  1   0:05   0.03% cron
81608 root             1  20    0  1033M  2928K bpf     2   1:12   0.02% filterlog
78097 root             1  20    0  1033M  2940K select  2   0:40   0.01% syslogd
  407 root             1  20    0 30900K 27560K select  2 302:46   0.01% python3.7
92159 root             5  52    0  6908K  2348K uwait   3   1:02   0.01% dpinger
45669 root             1  20    0  1039M  6776K select  3   0:33   0.01% ntpd
53758 root             1  20    0  1038M  7704K select  1   0:00   0.00% sshd
42916 root             1  20    0 15096K  9820K kqread  2   0:20   0.00% lighttpd
49523 dhcpd            1  20    0 16584K 12184K select  2   0:14   0.00% dhcpd
46587 www              1  20    0 19040K 14108K kqread  2   0:27   0.00% haproxy
52650 root             1  20    0 10356K  5892K kqread  0   0:15   0.00% lighttpd
35681 root             1  22    0 90012K 81960K accept  2   5:23   0.00% php-cgi
98634 root             1  23    0 85852K 77596K accept  3   5:08   0.00% php-cgi
96612 root             1  21    0 73564K 64344K accept  0   2:27   0.00% php-cgi
29030 root             1  22    0 77660K 68300K accept  2   1:53   0.00% php-cgi


I made a test, the php-cgi hang also when I try a new ssh connection...

Any suggestion guys ?

G.

[voluhar]

In my case it is same behaviour php-cgi is taking whole core(s).

I hardly find some pages that loads normaly and these are:
Services/Monit/Settings
Services/Monit/Status
Services/Unbound DNS/Statistics
Services/Web Proxy/Administration
Services/Zabbix
Power/Reboot
Power/Poweroff

I updated system to latest version from first post and results are still the same.
Current version
OPNsense 19.7.5_5-amd64
FreeBSD 11.2-RELEASE-p14-HBSD
OpenSSL 1.0.2t 10 Sep 2019

Same behaviour I have on master and slave box.

[jeremiah]

Ad made a plugin that may help, install the 'os-cache' plugin and see if that changes anything.

[voluhar]

Hi, I installed mentioned plugin and it did not make any difference for me. I installed it on both boxes and web interface have same responsiveness.

It is not mentioned but should I reboot it ?

Best regards, Robi

[jeremiah]

Hi Robi, If it isn't too much trouble, it might be good to reboot the boxes and see if the issue is resolved.

I had a sluggish experience initially as well, I just ended up reinstalling the software as I hadn't really configured anything at that point yet. In your case, I would try a reboot on both the master/slave boxes to see if that resolves your issues.

[Goldorak92]

Hi,

For infos, I tried to change the front's listen address of Ha Proxy from 0.0.0.0:port to Fqdn:port, apply, reboot... No changes, the Web Ui still hanging...

G.

[voluhar]

Today I rebooted and problem is still unsolved.

Problem is still present even on update to version:
OPNsense 19.7.6-amd64
FreeBSD 11.2-RELEASE-p14-HBSD
OpenSSL 1.0.2t 10 Sep 2019

[DanDanger]

What Web-Browser are you using?
I had similar issues. Acessing the WebGUI via Firefox took ~20s for each page to load. Switching to Chrome solved those issues and the WebGUI is fully responsive...

[Goldorak92]

Hi,
The Web brower can't be the cause regarding Cpu's load.
I tried another Web brower (chrome), and same issue, more than 15s for the page's load and cpu increase as before.

G.