WebGUI unaccessible for 3-4 minutes --> when modyfing "Interfaces"

[AES777GCM]

Okay - so in your case - the browser has an impact and everything else worked fine for you. That's weird but okay.

As I mentioned in the thread, my both Systems! I gave a try [with 24.7.x] behave the same: After Applying any switch action in INTERFACE SECTION you can see that save is done, but instead of "coming back after doing his things" the WebGui Server hangs up for 3-4 minutes.

In this state of lighttpd task being "locked" you can't even reset this task. I have two choices then: Wait for 3-4 minutes; somehow the task is then being killed automatically and comes back or killing the "stalled" lighttpd manually.

The Browser could not be an impact for me in this time.

[franco]

"locked" means your lighttpd is in the process of already being reconfigured. No idea why it's taking so long. You could add

log_msg("my message here");

commands to https://github.com/opnsense/core/blob/master/src/etc/inc/plugins.inc.d/webgui.inc#L137-L174

My guess is lighttpd doesn't stop or start and hangs the PHP script which keeps the lock over the lifetime of the PHP script. I will not speculate over where it may hang. You'll find it if you want to and can report back.


Cheers,
Franco

[AES777GCM]

Hi Franco,

I have no clue how to insert this code line cause I don't see any /src directory when using shell.
I help myself right now by killing lighttpd and reloading all services again.

For other who may have same issue and needs a step by step manual:

1. Open a ssh session
2. Insert "su" as first line and authorize with ROOT Password
3. Change to menu item SHELL

Code: [Select]

0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 8

root@obelix:/home/udo #
Now find out how many lighttpd tasks your system has

Code: [Select]

root@obelix:/home/udo # ps aux | grep lighttpd
root    17036   0.0  0.1  22784  10068  -  S    18:05     0:00.35 /usr/local/sbin/lighttpd -f /usr/local/etc/lighttpd_w
root    31424   0.0  0.0  14448   4084  -  S    18:04     0:00.02 /usr/local/sbin/lighttpd -f /var/etc/lighttpd-acme-ch
root    48669   0.0  0.0  12716   2396  0  S+   18:18     0:00.00 grep lighttpd
root@obelix:/home/udo #
In this example I got 2 running services. When System stalls, it is is NOT the acme service, so have a look at the PID of the other one --> 17036

So - when system stalls after "Applying", just kill this "locked" task and everything is fine again.

Code: [Select]

root@obelix:/home/udo # kill 17036

Th variant for lazy Guys is more like a HAMMER Method, but same effective.
I know via PS-AUX that I have 2 lighttpd tasks. So what, if we kill all lighttpd tasks and reload all services again?

Code: [Select]

root@obelix:/home/udo # ps aux | grep lighttpd
root    95866   2.8  0.1  20220   9564  -  S    18:24     0:00.02 /usr/local/sbin/lighttpd -f /usr/local/etc/lighttpd_w
root    26470   1.3  0.0  14448   4088  -  S    18:24     0:00.00 /usr/local/sbin/lighttpd -f /var/etc/lighttpd-acme-ch
root    12748   0.0  0.0  12716   2392  0  S+   18:24     0:00.00 grep lighttpd
root@obelix:/home/udo # pkill lighttpd -f
root@obelix:/home/udo # pkill lighttpd -f
root@obelix:/home/udo # configctl service reload all
OK
root@obelix:/home/udo #
Sorry, but I am not a professional coder and getting the things work "somehow" is my intention.

But nevertheless - thanks tho all Folks working directly and indirectly on codebase and keeps IT a little safer for all of us.

cheers,
Udo

[Red Squirrel]

I've been experiencing weird issues, particularly when setting up interfaces. I thought my whole setup was being bricked as I was not waiting long enough. I wonder if this bug is what I'm experiencing. I made a separate thread already but just thought I'd post to say I may also be experiencing this. Next time it happens I will give it time to see if it fixes itself. I will have a continuous ping, then do a change to interfaces, then lose everything and the ping also fails. Can't access web interface from any interfaces.

At one point I also did a reload service within the local console, and during that process I got one ping in then it failed again. So that did something, sort of.

I had one issue where one interface was handing out the wrong DHCP range, but then it fixed itself overnight. So it seems to me there may be some issues where things don't happen instantly when you make changes and there's something happening in the background.

I don't want to hijack this thread though and I already made my own, but starting to wonder if it's related and just thought I'd mention it.

[gstrauss]

This thread reminds me of race conditions in the PHP and shell scripts that I pointed out in https://github.com/opnsense/core/issues/6351 (make sure to show comments which might be hidden by github due to the long conversation thread)

I am a professional programmer (lighttpd developer) and made suggestions how to eliminate the race conditions I identified, and some -- but not all -- changes were implemented.  It might be useful if some one else with scripting and process management experience would review that code again and provide feedback to Franco, as my posts in https://github.com/opnsense/core/issues/6351 were not enough.

[franco]

Glenn, with all due respect your preachy behaviour is betting a bit old on the subject like lighttpd start race or OCSP.

I did address your concerns in the meantime and more regarding it but now it's gotten worse and your attitude needs a shift now that a proper lock is in place. We're still running into lockups... not that I am overly surprised to hear, but I also cannot reproduce. Can you?


Cheers,
Franco

[franco]

The locking code. To be frank I cannot see anything other than lighttpd locking up as witnessed by the user having to kill it.

https://github.com/opnsense/core/commit/4182f1993

[AES777GCM]

Thx for Glenn and Franco talking about the problem. Only by accepting a problem hope to fix it can be assumed.

@Franco:: If you wanna reproduce it - I'd like to send you my Fujitsu (you can keep it afterwards) so you can see I'm not telling a fairy tale. The fact I got this issue on 2 pieces of hardware (and being not alone in the world) tells one simple truth: There must be a problem somewhere in Code.

If Glenn is able to help - pls let him help.

[franco]

Ok, I'll send you a PM.


Thanks,
Franco