Today it finally hit me... classic example of PEBKAC.
During upgrade, I found error in my HA settings and while fixing and reviewing the setup, I added "Web GUI" to synced services by mistake. So every time I synced configuration, it tried to use same certificate as on first node and not finding it, it generated new one.
During upgrade, I found error in my HA settings and while fixing and reviewing the setup, I added "Web GUI" to synced services by mistake. So every time I synced configuration, it tried to use same certificate as on first node and not finding it, it generated new one.