High Availability Setup in vSphere 6.0

[bdelacour]

Hi !

I successfully installed OPNsense as a firewall and gateway between 3 VLANs. Everything works like a charm and I've also been able to setup some IPsec connections.

Now, I want to add some High Availability setting up a second OPNsense node. I read the tutorial on "How to setup CARP" and I have been able to setup connectivity between my two nodes.

The problem is at the end, when I test my virtual IPs. I can see them in my ARP table on all my virtual machines of VLANs but I can't ping them. The reason is : we must enable promiscuous mode on all DVS (my VLANs).

BUT, I can't enable it for two reasons :
* My provider blocks this setting because it allows packet sniffing and is not secure
* I don't want my CPU to be overstressed receiving packets from all VMs

How it should work attached to this post.

Do you know if another high availability setup exists ? For example, a tool like Keepalived (VRRP) would fix the virtual ip problem (not the xmlrpc sync but if I had to make a choice, I would choose working virtual ips).

Thank you !

[Supermule]

Run HA on Vsphere level....

[bdelacour]

Uuhh, yes ?

If the host containing my OPNsense VM is down, I don't want to wait vSphere HA to restart my VM on another host, I want a quick switch...

On my vSphere I have multiple ESX, I already had the problem where one of my ESX host isn't offline but cannot be accessed. So, when something like this happens, vSphere won't quickly detect my VM as down but all my network will be down... -> downtime
BUT, if I have another node, it won't be able to contact the master node and will be elected -> no downtime

[bdelacour]

UP

Somebody for a serious answer ? 

[bartjsmit]

Which licence do you have for your ESXi hosts? You may be able to run vSphere FT on an OPNsense VM.

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.avail.doc/GUID-7525F8DD-9B8F-4089-B020-BAA4AC6509D2.html

Bart...

[bdelacour]

Bart,

Thank you for your answer. I am able to enable vSphere FT on this VM but it doesn't fix all my issues :/
The advantage of CARP is : if I want to update one node, I can update the second, switch master to the second, update the first and switch master to the first.

Here, when I activate failover, I lose control on my VM. It only handles "failover" side.

Do you know if I can do the same with vSphere FT ?

If not, do you know another solution ?

Kind regards

[bartjsmit]

FT is a pure availability solution and requires a maintenance window to perform patching. Sorry.

Bart...

[SigurdM]

Hi,

if you haven't found another solution I suggest you ask your provider to reconsider his choices;

"* My provider blocks this setting because it allows packet sniffing and is not secure"

This is not really relevant for a router/firewall, it will see all traffic going in/out of the network anyway and it having promiscuous mode capability will not change much.

"* I don't want my CPU to be overstressed receiving packets from all VMs"

You set the options as override on a per vlan basis, so it will not get traffic from things outside the vlans you enable this on.

Also make a note of this KB, it might be you need Net.ReversePathFwdCheckPromisc = 1 on the VMware server:
https://kb.vmware.com/s/article/59235

My setup is fairly similar to yours, but I don't use distributed switches as my VMware-servers are standalone and used to utilize the hardware better and has no shared storage.  It works great, CARP failover with PFSYNC gives a few packets dropped when one of the nodes goes down.