HAProxy synchronization with Map Files failing to apply HAProxy settings

[laralar]

I have a setup of two OPNsense servers in HA mode. and also running os-haproxy plugin on them

I have a setup where I am using 4 map files for the configuration.

oot@opnsense1:/tmp/haproxy/mapfiles # ls -la
total 24
drwxr-x---  2 www   www   512 Jan 12 17:06 .
drwxr-x---  6 www   www   512 Jan 12 17:06 ..
-rw-------  1 root  www   681 Jan 12 17:06 5e162c72d8a779.05394596.txt
-rw-------  1 root  www  1709 Jan 12 17:06 5e164a6e713049.47297243.txt
-rw-------  1 root  www   180 Jan 12 17:06 5e165dc4c9c242.74140417.txt
-rw-------  1 root  www    41 Jan 12 17:06 5e165dd31a9bd4.08263470.txt


WHen testing the HA failover, shutting down the server, I was getting some NO SERV requests in the /var/log/haproxy.log file

clog -f /var/log/haproxy.log

root@opnsense2:/tmp/haproxy/mapfiles # cat /var/log/haproxy.log |grep NOSRV
Jan 12 17:05:57 opnsense2 haproxy[82574]: 10.20.10.1:44784 [12/Jan/2020:17:05:52.277] http-map http-map/<NOSRV> -1/-1/-1/-1/5001 400 211 - - CR-- 30/30/0/0/0 0/0 "<BADREQ>"
Jan 12 17:05:57 opnsense2 haproxy[82574]: 10.20.10.1:44734 [12/Jan/2020:17:05:52.277] http-map http-map/<NOSRV> -1/-1/-1/-1/5001 400 211 - - CR-- 29/29/0/0/0 0/0 "<BADREQ>"
Jan 12 17:05:57 opnsense2 haproxy[82574]: 10.20.10.1:44050 [12/Jan/2020:17:05:52.579] http-map http-map/<NOSRV> -1/-1/-1/-1/5001 400 211 - - CR-- 30/30/0/0/0 0/0 "<BADREQ>"
Jan 12 17:05:57 opnsense2 haproxy[82574]: 10.20.10.1:44088 [12/Jan/2020:17:05:52.579] http-map http-map/<NOSRV> -1/-1/-1/-1/5001 400 211 - - CR-- 29/29/0/0/0 0/0 "<BADREQ>"
Jan 12 17:05:57 opnsense2 haproxy[82574]: 10.20.10.1:44090 [12/Jan/2020:17:05:52.579] http-map http-map/<NOSRV> -1/-1/-1/-1/5001 400 211 - - CR-- 28/28/0/0/0 0/0 "<BADREQ>"
Jan 12 17:05:57 opnsense2 haproxy[82574]: 10.20.10.1:44048 [12/Jan/2020:17:05:52.579] http-map http-map/<NOSRV> -1/-1/-1/-1/5001 400 211 - - CR-- 27/27/0/0/0 0/0 "<BADREQ>"



While checking the Files on both servers after the Manual synchronization, they seemed similar, (the only difference was a missing \n (end of line) at the end of the copied file on the BACKUP server.  and I am not getting this NOSRV requests on the MASTER server.  why Is this?

I looked further into it and found out that even though the files seemed the same in the GUI, they were not the same on the BACKUP server:
failove


root@opnsense2:/tmp/haproxy/mapfiles # ls -la
total 24
drwxr-x---  2 www   www   512 Jan 12 12:57 .
drwxr-x---  6 www   www   512 Jan 12 12:57 ..
-rw-------  1 root  www   666 Jan 12 12:57 5e162c72d8a779.05394596.txt
-rw-------  1 root  www  1693 Jan 12 12:57 5e164a6e713049.47297243.txt
-rw-------  1 root  www   164 Jan 12 12:57 5e165dc4c9c242.74140417.txt
-rw-------  1 root  www    26 Jan 12 12:57 5e165dd31a9bd4.08263470

See the difference in sizes.  comparing the files showed the differences.

Now.. I went to the backup server and hit Test Sysntax/ Apply on the HAProxy plugin and voila!!  Files are now the same:

root@opnsense2:/tmp/haproxy/mapfiles # ls -la
total 24
drwxr-x---  2 www  www   512 Jan 12 12:57 .
drwxr-x---  6 www  www   512 Jan 12 12:57 ..
-rw-------  1 www  www   681 Jan 12 17:11 5e162c72d8a779.05394596.txt
-rw-------  1 www  www  1708 Jan 12 17:11 5e164a6e713049.47297243.txt
-rw-------  1 www  www   179 Jan 12 17:11 5e165dc4c9c242.74140417.txt
-rw-------  1 www  www    41 Jan 12 17:11 5e165dd31a9bd4.08263470.txt
root@opnsense2:/tmp/haproxy/mapfiles #


But Still I am getting the NOSRV requests even though the files are the same


Restarted haproxy plugin,, still getting the NOSRV.. Restarted the BACKUP server and I am still getting the NOSRV.


Any idea why this could be happening?

I have detailed log enabled,, Any idea how to know which request key is getting the NOSRV ?

I also checked the /usr/local/etc/haproxy.conf files in both servers and they are the same

Thanks

[cmdr.adama]

Sounds like it's a HA Proxy issue.

I haven't done anything with Map files but I'd check your configs for each server, backend, etc. to see if there's something wrong/missing

[laralar]

I've found this in the HAProxy documentation:

 >>> haproxy[18113]: 127.0.0.1:34548 [15/Oct/2003:15:18:55.798] px-http \
          px-http/<NOSRV> -1/-1/-1/-1/8490 -1 0 - - CR-- 2/2/2/0/0 0/0 ""

    => the client never completed its request and aborted itself ("C---") after
       8.5s, while the proxy was waiting for the request headers ("-R--").
       Nothing was sent to any server.


It seems they were lost requests by the client (maybe related to session that doesn't persist in the HA failover)

There seems to be a bug though in the HA Synchronization, if I don't do Apply on HAProxy, the Map Files don't get updated.