Two OPNvpn boxes to access the same LAN at an emergency

[danbet]

I run an IBM server with VMware ESXi with a few virtual machines. I installed OPNsense in one of them and made sure that it was always started first. I have configured the ESXi management interface so that it is only accessible on the LAN because VMware cannot protect it enough to make it accessible via a public IP address. So I can only access the management interface if I am connected to my infrastructure via VPN to OPNsense. This also means that if the VM with OPNsense is not running, I cannot reach the management interface and cannot manage any VM.

Therefore, I wanted to build a second VM with OPNsense so that I could access the management interface via VPN with this other OPNsense in an emergency. However, that doesn't work.

OPNsense 1 (active firewall):
LAN 192.168.1.1
WAN 10.10.1.1
FW 22.7

OPNsense 2 (operation to access the ESXi management interface in an emergency):
LAN 192.168.1.7
WAN 10.10.1.7
FW 23.7

I'll try to briefly describe what doesn't work with OPNsense 2.

During the installation I am connected to the OPNsense 1 via OpenVPN. The installation takes place via the ESXi console. Afterwards I can use the IP address 192.168.1.7 to configure everything needed for a VPN in the OPNsense 2 using the web browser and create a rule for the WAN interface so that I can access it. But I'm not allowed to specify the WAN address yet, otherwise I'll lose the connection to 192.168.1.7. I don't understand that, but I learned that I  configure the IP address of the WAN interface at the end.

The VPN tunnel to OPNsense 2 (10.10.1.7) works at this point as does access to the LAN interface 192.168.1.7 as soon as the VPN is active. Unfortunately I cannot access the IP address 192.168.1.2 because that is where the ESXi management interface is running and I would like to achieve that as described above.

I can see on the OPNsense 2 in the live view log that traffic is going out to 192.168.1.2, but on the OPNsense 1 I don't see it coming in. 192.168.1.7 and 192.168.1.2 are in the same LAN and therefore the traffic does not go through the firewall at all.

What am I doing wrong or what am I not understanding?

[Monviech (Cedrik)]

I think you should look into building a HA setup with two OPNsense VMs.

https://docs.opnsense.org/manual/hacarp.html#high-availability
https://docs.opnsense.org/manual/how-tos/carp.html

[danbet]

Thanks, after reading through the CARP documentation I think I can configure this. Only in the section about DHCP did I stumble across the following:
Next thing to consider is there will be two servers active at the same time, which should know of each others pools.

I don't know how to do this with OPNsense. I have already done a failover DHCP configuration with the ISC DHCP server, but not with OPNsense.

[danbet]

Oh, I have found the entries to define the DHCP failover.

[danbet]

What IP address does OpenVPN have to point to in an HA setup? The CARP Virtual IP address or the interface IP address?

From what I understand, it should be the CARP Virtual IP address, because I don't know whether the master or the backup is running.

[Monviech (Cedrik)]

Yes all services that should work in HA should always use the CARP VIP so that they can failover.

[danbet]

That's what I thought, but I can't establish an OpenVPN connection to the CARP IP address, I get a TSL handshake error. However, I can establish an OpenVPN connection to both the public IP address of the master and the public IP address of the backup.

My two OPNsenses that are in the HA configuration are currently still running alongside the one that previously provided the firewall. However, I was careful not to use an IP address twice.

OPNsense active
LAN address: 10.127.63.1
WAN address: 217.27.99.242

OPNsense01 (master)
LAN address: 10.127.63.7
CARP address: 10.127.63.12
WAN address: 217.27.99.252
CARP address: 217.27.99.250

OPNsense active
LAN address: 10.127.63.8
CARP address: 10.127.63.12
WAN address: 217.27.99.253
CARP address: 217.27.99.250

I can't imagine that the two new firewalls (master and backup) will work just like that if I turn off the one that is active now. What do I need to consider before doing this?

[Monviech (Cedrik)]

I sadly can't help with OpenVPN, never used it before on the OPNsense in HA. I have running IPsec and Wireguard connections though that use multiple CARP VIPs just fine (especially IPsec). Maybe the listen interface of the OpenVPN is wrong, or there are firewall rules missing that allow the traffic to the CARP VIP. Since OpenVPN uses a certificate, there might be problems with the Hostname and the A-Record too? Just assumptions though.

Turning the old firewall off won't be seamless. You have to have a downtime window where you can turn the old firewall off, and give its IP addresses as CARP VIPs to the new firewalls. That way, all clients can reach the former IP Addresses on the new Firewalls and everything continues to work. And then you have to configure those CARP VIPs to provide all of the former services that were reachable on the old Firewall. You can have as many CARP VIPs as you want, they all just need to be in seperate VHID groups. So just add them additionally to your already existing ones.

Also make sure the CARP VIPs always have the same subnet as their parent interface. The CARP VIPs have to be /24 if the parent interface is also /24. Don't use /32 ones.

[danbet]

I am aware that there is a service outage.

Is it better to disable the IPsec configuration before starting the switch? I can reach every OPNsense via public IP address.

[danbet]

Turning the old firewall off won't be seamless. You have to have a downtime window where you can turn the old firewall off, and give its IP addresses as CARP VIPs to the new firewalls. That way, all clients can reach the former IP Addresses on the new Firewalls and everything continues to work. And then you have to configure those CARP VIPs to provide all of the former services that were reachable on the old Firewall. You can have as many CARP VIPs as you want, they all just need to be in seperate VHID groups. So just add them additionally to your already existing ones.

Also make sure the CARP VIPs always have the same subnet as their parent interface. The CARP VIPs have to be /24 if the parent interface is also /24. Don't use /32 ones.

Everything worked without any problems, but unfortunately no traffic is accepted at the VIPs. The VIP's are configured correctly as far as I can see. I've read through all of them three or four times and can't find a mistake.

These are not physical machines, but virtual ones under VMware ESXi.

So I now have two OPNsense boxes, I can switch manually by setting the IP addresses of all interfaces accordingly, but unfortunately I don't have an automatic failover.

[Monviech (Cedrik)]

I think you have to enable "promiscuous mode" and/or "mac address spoofing" on each port group or virtual switch with a VIP to get the VIPs working, since they use CARP MAC addresses. The vSwitch on ESXi will probably filter them out since it looks like the vNICs of the OPNsense VM has a spoofed MAC address if it sends with 00:00:5e:00:01:XX...

https://docs.opnsense.org/manual/firewall_vip.html
The virtual MAC address of a CARP interface is 00:00:5e:00:01:XX, where the last two digits will be populated by its vhid.
CARP uses IP protocol number 112 (0x70), to detect priority it will send out advertisements using 224.0.0.18 or FF02::12.

[danbet]

I have MAC address changes enabled and Forged transmits, but not Promiscuous mode. I observed traffic to 224.0.0.18.

So I'll activate promiscuous mode and test it again. Unfortunately I won't be able to try this out until the next maintenance window.