Virtual IP Bind HAProxy Issue

[AryasDad]

Just updated to OPNsense 22.1.3-amd64 this morning and finally resolved an issue with HAProxy not starting.

HAProxy was working fine before the update, but after HAProxy would not start.  Config file reported no errors, but I could not find anything in log files as to why HAProxy was not starting.  So I finally connected via terminal and tried to start HAPRoxy manually and got an error that it could not bind to my internal Virtual IP.

SO I went to the VirtualIP settings in the GUI and confirmed that I had Allow service binding turned on.  As a test I re-saved and applied the settings.  After that HAProxy started up just fine.

I rebooted to test and the same thing happened.  I had to manually re-save the Virtual IP and apply the settings and the HAPRoxy could be started.

Is there some new setting somewhere that I need to activate to get this to work automatically like before?

[franco]

So who reverts the option? Can you check the configuration history?

In general, however, your issue is that something listens to the IP you want HAProxy to listen to. Disabling the service bind doesn't do anything for HAProxy plugin so it's another service, likely the web GUI configured to listen to said interface manually, which we don't recommend for exactly this reason.


Cheers,
Franco

[AryasDad]

So who reverts the option? Can you check the configuration history?

I checked config history and the only "change" to VIP settings is me re-saving the settings.  But maybe I didn't explain myself clearly. Nothing is changed and I don't actually change the settings, I just navigate to the settings and click Save.  After that HAproxy starts up with no issues.  It will not start until I do this step manually.

In general, however, your issue is that something listens to the IP you want HAProxy to listen to. Disabling the service bind doesn't do anything for HAProxy plugin so it's another service, likely the web GUI configured to listen to said interface manually, which we don't recommend for exactly this reason.

I moved from pfsense over a year ago to OPNsense.  I started using HAProxy in pfsense to redirect to various servers on my local LAN.  I ran into an issue where when I was on the local LAN, I could not reach the internal servers using the domains I setup for them that worked from outside my LAN.  I found a guide that instructed me to setup a combination of Virtual IP and firewall rules to get it working internally as it did externally.

When I moved to OPNsense, it was not a direct 1 to 1 transfer and it took me some time to get it all working but I eventually did.  I have been running OPNsense with this configuration for over a year with no issues until I did the recent update.

Are you saying that I should NOT have "Allow service binding" active on my Virtual IP in order for HAProxy to properly bind my listening frontends to?

[franco]

What was your last working version? Something on 21.7.x or 22.1.2?

I'm just saying that in order for HAProxy to fail some other service needs to bind first where you intend to bind it.

I'm guessing this is due to port 80/443 use which suggests that you moved the web GUI to listen to specific interfaces (system: settings: administration). If I'm wrong I apologise because there is not much info on VIP and interface setup here and it's easy to miss something obvious.


Cheers,
Franco

[Bunch]

I have the same issue as OP.
My last working version is 22.1.2.
My WEB UI is using 8443, and I have disabled http redirect.
My HAProxy is listening to port 80 and port 443 of VIP.

Whenever I restart opnsense. HAProxy cannot start as it cannot bind these two ports of the VIP.
(Probably another process already listening to the VIP, but I don't know what it is)
After I click edit for the VIP, save without any changes, apply changes.
I can start HAProxy without any issue. (But the same thing happen after a reboot)

VIP detail FYI:
Mode: IP Alias
Interface: Loopback
IP address type: single address (grey)
Address: 192.168.6.1/32
Allow service binding: true
Gateway: (blank)
VHID group: none
Advertising Frequency: base 1, Skew 0

Probably the same issue here

[franco]

I can see where confusion and supposedly buggy behaviour may come from.

The thing is "Allow service binding" needs to be turned off in order to work with HAProxy. Really, the label should say "Allow automatic service binding" since HAProxy takes the explicit virtual IP as an input, which is a manual configuration.

If the setting is turned on it seems now in 22.1.3 it is enforced more vigorously, which is certainly a good thing as it exposes these problematic configurations of overlapping VIP/port use might disable one service or another. Likely what happens when you save the configuration for the VIP is that the address is reset in the system which frees it for HAProxy to grab and the other service that had it bound will lose it.


Cheers,
Franco

[Bunch]

Thanks for your reply.

I have just turned off Allow service binding, then reboot the router.
But the issue still exists.

[Bunch]

An update for this issue.

I have just reboot opnsense again. And try to check which process is listening to 192.168.6.1:443 and 80

Code: [Select]

root@OPNsense:~ # sockstat -4 -l
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
unbound  unbound    83031 3  udp4   *:53                  *:*
unbound  unbound    83031 4  tcp4   *:53                  *:*
unbound  unbound    83031 5  udp4   *:53                  *:*
unbound  unbound    83031 6  tcp4   *:53                  *:*
unbound  unbound    83031 7  tcp4   127.0.0.1:953         *:*
dhcpd    dhcpd      16219 10 udp4   *:67                  *:*
root     ntpd       43089 21 udp4   *:123                 *:*
root     ntpd       43089 22 udp4   (my_WAN_IP):123     *:*
root     ntpd       43089 25 udp4   127.0.0.1:123         *:*
root     ntpd       43089 26 udp4   192.168.4.1:123       *:*
root     ntpd       43089 27 udp4   192.168.3.1:123       *:*
root     lighttpd   4810  4  tcp4   127.0.0.1:43580       *:*
root     lighttpd   51954 5  tcp4   *:8443                *:*
root     sshd       49729 4  tcp4   *:22                  *:*
Nothing is listening to it.
And I cannot ping 192.168.6.1 in SSH
(I also try to create another VIP 192.168.7.1 with same setting, it can ping normally)

Thus, the issue should be VIP not up after system reboot.
This error also found in general log, which is right after I save and apply the VIP setting
Probably not related to this issue, just hope it can give some hint

Code: [Select]

/firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.6.1' -alias' failed to execute

[franco]

> /firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.6.1' -alias' failed to execute

Log files with errors make me happy because they are so clinically precise.

https://github.com/opnsense/core/commit/9a618ba6

# opnsense-patch 9a618ba6


Cheers,
Franco

[franco]

Hmm, no feedback at all?


Cheers,
Franco

[AryasDad]

Hmm, no feedback at all?

Sorry, have not had time to come back and follow up on this issue.  I appreciate Bunch being more thorough with the troubleshooting than I have been.

Can you point me to a tutorial on how to apply patches to my install?  I have seen these posted from time to time, but have never learned how to apply them manually (which I assume is something I need to do).

[AryasDad]

I just checked my Logs and have the same error related to Virtual IP as Bunch reported.

/firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '10.17.0.1' -alias' failed to execute

So, if I can get a gentle nudge on how to apply the patch, I can verify if it resolves the issue.  Do I manually edit the parameters in the files or is there a script that executes the patch?

[AryasDad]

Can you point me to a tutorial on how to apply patches to my install?  I have seen these posted from time to time, but have never learned how to apply them manually (which I assume is something I need to do).

Sorry to spam the thread.  I just dug a bit deeper in your post and realized you already provided the way to apply the patch via https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-patch: opnsense-patch 9a618ba6

I will try to apply this week and report back.

[Bunch]

Hmm, no feedback at all?


Cheers,
Franco

Sorry, I was a bit busy these few days.
I have just tested the patch, it fixed the VIP issue and no more error related to VIP in log.
Thanks for your help.

[franco]

AryasDad: ah yes the infamous "opnsense-patch", sometimes even I'm surprised by the simplicity of it 

Bunch: Yay, good news. I'll add this fix to 22.1.4. It should refrain from breaking now for the foreseeable future the way it's implemented now.


Cheers,
Franco