[Solved] Web interface stops working

[else]

Hi,

after reboot, the web interface stops working for me and instead I instead get a 503. This seems to be happening only after upgrading to 22.1. A reboot fixes it.

Lighttpd logs show:

<27>1 2022-02-14T23:17:49+01:00 router.example.com lighttpd 50624 - [meta sequenceId="60"] (gw_backend.c.285) establishing connection failed: socket: unix:/tmp/php-fastcgi.socket-1: Connection refused
<27>1 2022-02-14T23:17:49+01:00 router.example.com lighttpd 50624 - [meta sequenceId="61"] (gw_backend.c.285) establishing connection failed: socket: unix:/tmp/php-fastcgi.socket-0: Connection refused
<27>1 2022-02-14T23:17:49+01:00 router.example.com lighttpd 50624 - [meta sequenceId="62"] (gw_backend.c.995) all handlers for /index.php? on .php are down.

php-fpm seems to be terminating:

[07-Feb-2022 19:55:10] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful
[07-Feb-2022 19:55:11] NOTICE: fpm is running, pid 14183
[07-Feb-2022 19:55:11] NOTICE: ready to handle connections
[07-Feb-2022 20:05:26] NOTICE: Terminating ...
[07-Feb-2022 20:05:26] NOTICE: exiting, bye-bye!

I tried starting it again but it does not seem to help:

root@router:/var/log # ps ax | grep fpm
81734  -  Ss       0:00.03 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
82120  -  I        0:00.00 php-fpm: pool www (php-fpm)
82340  -  I        0:00.00 php-fpm: pool www (php-fpm)

Restarting the web gui does not seem to help either:

root@router:/var/log # /usr/local/etc/rc.restart_webgui
Starting web GUI...failed.
Generating RRD graphs...done.

This also does not work:

root@router:/var/log # configctl webgui restart
OK

How can I debug it further?

[franco]

Nothing from core uses php-fpm. You may be looking at your own modifications.


Cheers,
Franco

[else]

Good to know. I have not done any modification besides installing some plugins (IIRC os-*).

What could be causing the 503 then?

[franco]

os-nginx? might be interference from there.


Cheers,
Franco

[fabian]

The nginx plugin starts two php-fpm instances. One as root and one as a restricted user. They can be used to host small PHP utilities or the web interface.

However it is like that since the first version.

Maybe lighttpd is now using php-fpm as well.

For now I would recommend to stop the php-fpm service and afterwards try to restart the lighttpd service. It does not matter if you don't use the hosting feature or the advanced authentication.

[else]

It *looks* like I found the issue.

I am using the os-git-backup-1.0_1 package and am backing up my configuration to Github. Now the token expired, and that seems to have caused a PHP exception.

I was able to kill the lighttpd process and restart using /usr/local/etc/rc.start_webgui.

I would love to provide the stacktrace but I used the integrated reporter to report the PHP exception and now it's gone. Where does it actually end up? 

I will refresh the token now and see if that helps.

[else]

No, that did not seem to do the trick. Today again I got a 503. Latest lighttpd logs show this:

<27>1 2022-02-16T18:53:33+01:00 router.example.com lighttpd 58562 - [meta sequenceId="1"] (server.c.2057) server stopped by UID = 0 PID = 23715
<27>1 2022-02-16T18:53:34+01:00 router.example.com lighttpd 28460 - [meta sequenceId="2"] (server.c.1551) server started (lighttpd/1.4.63)
<27>1 2022-02-16T18:53:34+01:00 router.example.com lighttpd 28460 - [meta sequenceId="3"] (gw_backend.c.380) child signalled: 1
<27>1 2022-02-16T18:53:34+01:00 router.example.com lighttpd 28460 - [meta sequenceId="4"] (gw_backend.c.380) child signalled: 1
<27>1 2022-02-16T19:42:20+01:00 router.example.com lighttpd 28460 - [meta sequenceId="1"] (gw_backend.c.285) establishing connection failed: socket: unix:/tmp/php-fastcgi.socket-1: Connection refused
<27>1 2022-02-16T19:42:20+01:00 router.example.com lighttpd 28460 - [meta sequenceId="2"] (gw_backend.c.285) establishing connection failed: socket: unix:/tmp/php-fastcgi.socket-0: Connection refused
<27>1 2022-02-16T19:42:20+01:00 router.example.com lighttpd 28460 - [meta sequenceId="3"] (gw_backend.c.995) all handlers for /index.php? on .php are down.
<30>1 2022-02-16T19:42:20+01:00 router.example.com lighttpd 28460 - [meta sequenceId="4"] 10.42.23.67 router.example.com - [16/Feb/2022:19:42:20 +0100] "GET / HTTP/2.0" 503 361 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"
<30>1 2022-02-16T19:42:21+01:00 router.example.com lighttpd 28460 - [meta sequenceId="5"] 10.42.23.67 router.example.com - [16/Feb/2022:19:42:21 +0100] "GET / HTTP/2.0" 503 361 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0"
<27>1 2022-02-16T19:42:23+01:00 router.example.com lighttpd 28460 - [meta sequenceId="6"] (gw_backend.c.362) gw-server re-enabled: unix:/tmp/php-fastcgi.socket-1  0 /tmp/php-fastcgi.socket
<27>1 2022-02-16T19:42:23+01:00 router.example.com lighttpd 28460 - [meta sequenceId="7"] (gw_backend.c.362) gw-server re-enabled: unix:/tmp/php-fastcgi.socket-0  0 /tmp/php-fastcgi.socket
<30>1 2022-02-16T19:42:52+01:00 router.example.com lighttpd 28460 - [meta sequenceId="8"] 10.42.23.67 router.example.com - [16/Feb/2022:19:42:52 +0100] "PRI * HTTP/2.0" 100 902 "-" "-"

Installed os-* packages:

root@router:/var/log/lighttpd # pkg info | grep os-
os-acme-client-3.8             ACME Client
os-ddclient-1.0                Dynamic DNS client
os-dyndns-1.27_2               Dynamic DNS Support
os-frr-1.26                    The FRRouting Protocol Suite
os-git-backup-1.0_1            Track config changes using git
os-mdns-repeater-1.0_1         Proxy multicast DNS between networks
os-netdata-1.1                 Real-time performance monitoring
os-node_exporter-1.1           Prometheus exporter for machine metrics
os-tayga-1.1_2                 Tayga NAT64
os-vnstat-1.3                  vnStat is a console-based network traffic monitor
os-wireguard-1.10              WireGuard VPN service

Any suggestion on where to look regarding the 503?

[fabian]

Maybe your PHP interpreter is defect.

You may check out this files - these are the config files used by the nginx plugin. When you can start the php-fpm interpreter, then it is a good sign. If it fails, try to reinstall PHP.

https://github.com/opnsense/plugins/tree/master/www/nginx/src/opnsense/service/templates/OPNsense/Nginx

[else]

Upgrading to 22.1.1 seems to have done the trick.