Hello fellow opnsensers,
my box used to hum right away running 24.1. After upgrading to 24.7 the webgui is unable to start at boot. To "fix" this issue a login via ssh is needed. Restarting the webgui via 'configctl webgui restart' make the webgui accessable again.
There are hints in the boot.log:
2024-08-03T18:38:20 webgui_configure_do[299] failed.
2024-08-03T18:38:20 webgui_configure_do[299] Starting web GUI...
digging a bit deeper i found this in the general.log:
2024-08-03T18:38:20 Notice kernel <118>Starting web GUI...failed.
2024-08-03T18:38:20 Notice kernel <118>Configuring OpenSSH...done.
2024-08-03T18:38:20 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(1))
2024-08-03T18:38:20 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure dhcp (1)
2024-08-03T18:38:20 Notice opnsense /usr/local/etc/rc.bootup: ROUTING: setting inet default route to 192.168.2.1
2024-08-03T18:38:20 Notice opnsense /usr/local/etc/rc.bootup: ROUTING: configuring inet default gateway on wan
2024-08-03T18:38:20 Notice opnsense /usr/local/etc/rc.bootup: ROUTING: entering configure using defaults
2024-08-03T18:38:20 Error opnsense /usr/local/etc/rc.bootup: The command '/usr/local/bin/flock -ne /var/run/lighty-webConfigurator.pid /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf' returned exit code '255', the output was '2024-08-03 18:38:20: (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.76/src/network.c.604) bind() [fe80::20c:29ff:fe34:fc02]:4444: Can't assign requested address'
2024-08-03T18:38:20 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure early (execute task : webgui_configure_do(1))
2024-08-03T18:38:20 Notice opnsense /usr/local/etc/rc.bootup: plugins_configure early (execute task : unbound_cache_flush(1))
also the restart seems to find an 'old' instance:
2024-08-03T18:41:50 Error opnsense /usr/local/etc/rc.restart_webgui: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 97477'
bit i couldn't find anymore hints for reason the webgui is not starting. Is this a known bug or does anybody know where to look for debug info?
Cheers,
Cilahn
https://docs.opnsense.org/manual/settingsmenu.html#listen-interfaces
Hi Patrick,
thanks for pointing this out. I double checked the Interfache Settings. The WebGUI ist listening to port 4444 on Interface LAN and WLAN. Both are set to IPV4 only (IPV6 'NONE') and have static IPs.
If there where a conflict a configctl webgui restart should fail too, but it doesn't. In the log is an bind error belonging to an ipv6 address which make no sense, because IPV6 is set to none on all interfaces.
Cheers,
Cilahn
Change to "All (recommended)" and test, please.
i did, still needed a manual restart after reboot.
general.log is not happy:
2024-08-03T20:29:30 Error opnsense /usr/local/etc/rc.bootup: The command '/usr/local/bin/flock -ne /var/run/lighty-webConfigurator.pid /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf' returned exit code '255', the output was '2024-08-03 20:29:30: (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.76/src/network.c.604) bind() [fe80::20c:29ff:fe34:fc02]:4444: Can't assign requested address'
it is strange, kinda found a workaround ..all interfaces are configured with IPV6 Configuration "None". Anyway 'ifconfig show' still shows ipv6 addresses on all interfaces. vmx0 is the interface which is unable to bind the IPV6 listening port stated in the general.log.
Also tried Interface->Settings->Allow IPv6 'off', which kinda fixed it ...
before IPv6 off ....
vmx0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 00:0c:29:34:fc:02
inet6 fe80::20c:29ff:fe34:fc02%vmx0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
vmx1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 00:0c:29:34:fc:0c
inet 192.168.2.2 netmask 0xffffff00 broadcast 192.168.2.255
inet6 fe80::20c:29ff:fe34:fc0c%vmx1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
options=0
groups: enc
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pfsync0: flags=0 metric 0 mtu 1500
options=0
maxupd: 128 defer: off version: 1400
syncok: 1
groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
options=0
groups: pflog
vmx0_vlan100: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN (lan)
options=4000000<MEXTPG>
ether 00:0c:29:34:fc:02
inet 192.168.80.250 netmask 0xffffff00 broadcast 192.168.80.255
inet6 fe80::20c:29ff:fe34:fc02%vmx0_vlan100 prefixlen 64 scopeid 0x7
groups: vlan
vlan: 100 vlanproto: 802.1q vlanpcp: 0 parent interface: vmx0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vmx0_vlan300: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: DMZ (opt1)
options=4000000<MEXTPG>
ether 00:0c:29:34:fc:02
inet 172.23.23.1 netmask 0xffffff00 broadcast 172.23.23.255
inet6 fe80::20c:29ff:fe34:fc02%vmx0_vlan300 prefixlen 64 scopeid 0x8
groups: vlan
vlan: 300 vlanproto: 802.1q vlanpcp: 0 parent interface: vmx0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vmx0_vlan400: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WLAN (opt2)
options=4000000<MEXTPG>
ether 00:0c:29:34:fc:02
inet 10.10.10.1 netmask 0xffffff00 broadcast 10.10.10.255
inet6 fe80::20c:29ff:fe34:fc02%vmx0_vlan400 prefixlen 64 scopeid 0x9
groups: vlan
vlan: 400 vlanproto: 802.1q vlanpcp: 0 parent interface: vmx0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
.... reboot ...vmx0 is still IPv6 enabled, but the vlans are now IPv6 free and lighthttpd seems to bind without trouble after the reboot ...
vmx0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 00:0c:29:34:fc:02
inet6 fe80::20c:29ff:fe34:fc02%vmx0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
vmx0_vlan100: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN (lan)
options=4000000<MEXTPG>
ether 00:0c:29:34:fc:02
inet 192.168.80.250 netmask 0xffffff00 broadcast 192.168.80.255
groups: vlan
vlan: 100 vlanproto: 802.1q vlanpcp: 0 parent interface: vmx0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
sockstat -4ln
0 lighttpd 914 7 tcp4 127.0.0.1:4444 *:*
0 lighttpd 914 10 tcp4 192.168.80.250:4444 *:*
0 lighttpd 914 11 tcp4 10.10.10.1:4444 *:*
0 lighttpd 914 12 tcp4 127.0.0.1:80 *:*
0 lighttpd 914 15 tcp4 192.168.80.250:80 *:*
0 lighttpd 914 16 tcp4 10.10.10.1:80 *:*
0 sshd 95203 8 tcp4 *:22 *:*
sorry, but this feels like a bug in the config daemon.
Quote from: Cilahn on August 03, 2024, 08:57:51 PM
all interface are configures with IPV6 Configureation "None". Anyway ifconfig show still ipv6
addresses on all interface.
This is not how it works, not configuring IPv6 does not disable the link-local IPs - and never has.
Quote from: doktornotor on August 03, 2024, 09:17:33 PM
Quote from: Cilahn on August 03, 2024, 08:57:51 PM
all interface are configures with IPV6 Configureation "None". Anyway ifconfig show still ipv6
addresses on all interface.
This is not how it works, not configuring IPv6 does not disable the link-local IPs - and never has.
This is surely true, i'm not to deep into ipv6, but this shouldn't prevent the start of webgui anyway. Something broke here between 24.1 and 24.7
Thinking a bit about it, so why are there no link-local adresses on the vlan interfaces?
i'm did a deeper dig today and found some clues i not really can glue together.
First I saw that all VLans on on the vmx0 1q-Trunk Interfaces share the same ipv6 link-local addresses, but starting the webgui via configctl just binds theses addresses fine. Even when the shared link-local IP was mentioned in the lighty startup general.log error.
Could it be, that under ESXi the VLans are somewhat setup late and i observe a race condition?
BTW the VLan Trunk is setup correctly with ID 4095 and has being working for years now.
So to dig deeper i added an evil hack to /usr/local/etc/inc/plugins.inc.d/webgui.inc just to see how the setup is reacting to a little delay of 30 seconds. Guess what. It works after this fix.
/* start lighttpd */
/* .. some evil hack.. */
sleep(30);
if (!count($listeners) || mwexec('/usr/local/bin/flock -ne /var/run/lighty-webConfigurator.pid /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf')) {
service_log("failed.\n", $verbose);
} else {
service_log("done.\n", $verbose);
}
Any Ideas what could be the reason behind all this?
If the share the MAC address the share the link-local address. Nothing wrong with that. Link-local means exactly that. Unique per link.
Did you try changing the listen interfaces to "All"?
yes i did. Listen to all didn't help :(
Weird. Only thing I can say for now is that the link-local addresses are perfectly normal.
I think the link-local similarity is misleading because all interfaces are perfectly mapped when the webgui is delayed or manually started.
sockstat -ln | grep lighttpd
0 lighttpd 84760 7 tcp4 127.0.0.1:4444 *:*
0 lighttpd 84760 8 tcp6 ::1:4444 *:*
0 lighttpd 84760 9 tcp6 fe80::1%lo0:4444 *:*
0 lighttpd 84760 10 tcp4 192.168.80.250:4444 *:*
0 lighttpd 84760 11 tcp6 fe80::20c:29ff:fe25:92df%vmx0_vlan100:4444 *:*
0 lighttpd 84760 12 tcp4 10.10.10.1:4444 *:*
0 lighttpd 84760 13 tcp6 fe80::20c:29ff:fe25:92df%vmx0_vlan400:4444 *:*
0 lighttpd 84760 14 tcp4 127.0.0.1:80 *:*
0 lighttpd 84760 15 tcp6 ::1:80 *:*
0 lighttpd 84760 16 tcp6 fe80::1%lo0:80 *:*
0 lighttpd 84760 17 tcp4 192.168.80.250:80 *:*
0 lighttpd 84760 18 tcp6 fe80::20c:29ff:fe25:92df%vmx0_vlan100:80 *:*
0 lighttpd 84760 19 tcp4 10.10.10.1:80 *:*
0 lighttpd 84760 20 tcp6 fe80::20c:29ff:fe25:92df%vmx0_vlan400:80 *:*
some minor findings. The timing of the interface being initalized and bindung the lighttpd daemon to them is right on the same time stamp in the logs. May be this is a hint?
Timing
system/latest.log
<11>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot opnsense 278 - [meta sequenceId="321"] /usr/local/etc/rc.bootup: The command '/usr/local/bin/flock -ne /var/run/lighty-webConfigurator.pid /usr/local/sbin/
lighttpd -f /var/etc/lighty-webConfigurator.conf' returned exit code '255', the output was '2024-08-04 21:48:52: (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.76/src/network.c.604) bind() [fe80::20c:29ff:fe
25:92df]:4444: Can't assign requested address'
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot opnsense 278 - [meta sequenceId="322"] /usr/local/etc/rc.bootup: ROUTING: entering configure using defaults
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="304"] <6>vlan0: changing name to 'vmx0_vlan100'
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="305"] <6>vlan1: changing name to 'vmx0_vlan300'
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="306"] <6>vlan2: changing name to 'vmx0_vlan400'
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="307"] <118>done.
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="308"] <118>Configuring DMZ interface...done.
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="309"] <118>Configuring LAN interface...done.
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="310"] <118>Configuring WAN interface...
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="311"] <6>vmx1: link state changed to UP
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="312"] <118>done.
<13>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot kernel - - [meta sequenceId="313"] <118>Configuring WLAN interface...done.
audit/latest.log
<38>1 2024-08-04T21:48:52+02:00 firewall.mittelerde.mot configd.py 238 - [meta sequenceId="3"] action allowed interface.linkup.start for user root
Quote from: Cilahn on August 04, 2024, 10:06:27 PM
yes i did. Listen to all didn't help :(
Just to be sure because this looks like a waste of debugging time that has already been done over the years:
"Listen to all" here means DESELECT ALL interfaces, not selecting all interfaces.
Cheers,
Franco
Hi franco,
i reverted to 24.1 by now. So i can't retest it. Looking at my Screenshots taken during the debugging on the 24.7 i'm pretty sure all Interfaces where unticked and the "All (recommended)" was shown. Also asked by Patrick before and i could reproduce the problem under 24.7.
I will stop investigating further. I'm relatively new to opnsene and currently evaluating it against pfsense and i must admit when in the help box under the feature is written "Only accept connections from the selected interfaces. Leave empty to listen globally. Use with care" i assume it is not a good idea to listen on the global/WAN side. When this is a common problem it would be better to not enable this config item or at least post a clear Warning in the Help Text.
I really appreciate your and Patrick very quick support. Thank you for that. Sadly opnsense doesn't print the best picture here.
Kind Regards,
Cilahn
I might have the same issue after an upgrade to OPNsense 24.1.10_8 (and consecutive upgrade to 24.7 as well).
Previous version: 24.1.6
Issue appeared after upgrade to: 24.1.10_8
Issue: No access via web interface after boot (timeout).
Reloading the services via console (menu item 11) restores access via web interface until next reboot.
System: Log Files: General shows during boot:
Quote2024-08-05T14:38:13 Error opnsense /usr/local/etc/rc.bootup: The command '/usr/local/bin/flock -ne /var/run/lighty-webConfigurator.pid /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf' returned exit code '255', the output was '2024-08-05 14:38:13: (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.76/src/network.c.604) bind() [fe80::7802:45ff:fe40:c66d]:4433: Can't assign requested address'
I can confirm that setting the listening interface at
System: Settings: Administration to "All (recommended)" by deselecting all interfaces solves the issue. When the listening interface is set manual, even if all interfaces are selected, the above error message gets thrown and access to the web interface is not possible after reboot.
All interfaces are configured for static IPv4. IPv6 configuration type is set to "None".
Unchecking "Allow IPv6" at
Interfaces: Settings and two reboots in a row solves the issue as well.
Quote from: linore on August 05, 2024, 04:41:52 PM
I can confirm that setting the listening interface at System: Settings: Administration to "All (recommended)" by deselecting all interfaces solves the issue. When the listening interface is set manual, even if all interfaces are selected, the above error message gets thrown and access to the web interface is not possible after reboot.
"All (recommended)" and selecting each interface individually are fundamentally different things. There's a reason why
- the menu selection says "recommended"
- the help text says "use with care"
- the documentation explicitly states "Misconfigurations likely lead to a non accessible web interface"
"All (recommended)" is the only supported setting. Just don't mess with it unless you know exactly what you are doing.
I explained this again and again and again on this forum, so here's some links:
https://forum.opnsense.org/index.php?topic=33145.msg167914#msg167914
https://forum-opnsense-org.translate.goog/index.php?topic=40843.msg204479&_x_tr_sl=de&_x_tr_tl=en&_x_tr_hl=de&_x_tr_pto=wapp#msg204479
HTH,
Patrick
Quote from: Patrick M. Hausen on August 05, 2024, 05:03:19 PM
"All (recommended)" is the only supported setting. Just don't mess with it unless you know exactly what you are doing.
The setting should be hidden under "Advanced" at minimum. Also creating floating firewall rules (and disabling the anti-lockout if required on LAN) is a much better way of restricting access than messing with the listening interfaces.
Wherever you hide these settings and however many popups you create to warn people, they will find it and use it.
The thing is - this is only really useful when you need to run something on the web GUI port on some interface(s) - e.g. OpenVPN TCP/443 on WAN, or HAProxy, but still want to keep the OPNsense GUI on the default ports for whatever reason... Just way too visible as it is now.
The static PHP page with the admin settings never having had an advanced button is probably the largest offender. I can agree with that.
Cheers,
Franco
Quote from: doktornotor on August 05, 2024, 09:35:53 PM
The setting should be hidden under "Advanced" at minimum. Also creating floating firewall rules (and disabling the anti-lockout if required on LAN) is a much better way of restricting access than messing with the listening interfaces.
Absof*inglutely! Anti-lockout, reply-to, ... I disable all of this "magic" stuff. I hate implicit, "magic" configuration with a vengeance. Everything should be explicit. If you change your LAN rules in a way so you cannot connect to the UI - tough shit. And the default is "allow * *", anyway. So why this anti-lockout thingy?
I really appreciate the change that IPsec VPN connections do not automatically create "allow" rules for ESP and AH, anymore, for example.
But of course - admittedly - I also never managed arp, ND, DHCP ... manually. So this is an area where I think these automatic rules are OK.
It's a difficult decision - not complaining about the current state of OPNsense. Just the remark that I disable everything that performs "magic", i.e. implicit rather than explicit rules, on every system I manage.
Kind regards,
Patrick
Quote from: doktornotor on August 05, 2024, 09:57:39 PM
The thing is - this is only really useful when you need to run something on the web GUI port on some interface(s) - e.g. OpenVPN TCP/443 on WAN, or HAProxy, but still want to keep the OPNsense GUI on the default ports for whatever reason... Just way too visible as it is now.
Bind to 4443, use a NAT port forward rule on LAN, done. 8)
There is a GitHub issue about this topic: https://github.com/opnsense/core/issues/7649
Wow this boiled hot.
Sorry but you guys don't see that this is not the way to lure new users like me into the opnsense universe?
Scratiching my Head and starring into infinity ....
Not sure which thread you read. This was rather technical and productive.
Cheers,
Franco