Opnsense High Availability Questions

[fearz]

Hello,

Just new to Opnsense HA and its getting me really interested but i'm having several issues & questions.

I have a primary Opnsense VM and its working proplery with Multi-WAN on a proxmox VM.

Now i wanted to achieve HA so i created a clone from the existing one (exact replica), and following this video to create CARP/Virtual IPs as it has the exact setup of mine:

https://youtu.be/I5n3QXOlxmw?si=Yi8GepDm2M11afeD

My setup is as follows:

Modem 1 (DHCP ON) LAN port -> OpnSense Primary
Modem 2 (DHCP ON) LAN port -> Opnsense Primary
Modem 1 (DHCP ON) Lan port 2 -> Opnsense Secondary

Opnsense Primary: 192.168.4.1
Opnsense Secondary: 192.168.4.2

All services are ON on both (DHCP, DNS, AdGuard, ZenArmor, Crowdsec, etc)

The 1st problem is that when I turn on the secondary VM, the internet stops working or starts to stutter, I think that maybe related to that I have DHCP service on for the same subnet (192.168.4.x) and as well the other VLANs.

Should that be turned off on the secondary VM and HA sync should be taking place, so the question comes to my mind, what if the primary VM fails / offline, if DHCP is off on the secondary VM, how will the clients take IPs?

The 2nd problem is that when the primary is down, the secondary does not have internet to clients, if i login to opnsense via SSH and ping 1.1.1.1 - it pings normally, but the clients can't ping, not sure what is wrong, it shows in Virtual IPs status that the secondary is MASTER for both LAN & WAN.

The video was created for Opnsense 24.7.x however i'm using 25.1 - there were slightly different settings found in 25.1 vs. whats in the video, i just played around a bit with them but not sure if what i did was correct.

I'm ready to provide whatever configuration you need.

Your assistance is very much appreciated.

Thank you.

[fearz]

Any support is appreciated.

[Patrick M. Hausen]

Then please do not refer to youtube videos but to the documentation. Most regulars have neither time nor motivation to watch stuff on YT looking for errors. If it doesn't work the way the video claims to, ask the author of the video.

Willing to help any time but not going to watch that.

First and foremost post screen shots of your settings - HA sync and the virtual CARP IPs, please.

[fearz]

On FW1:

I have pfsync interface set to 10.0.0.1
on Firewall rules:
pfsync interface -> pass any to any
LAN & WAN -> pass any to any CARP

I can ping interface 10.0.0.2 from FW1 and vice-versa

its just when i unplug FW1 LAN cable, i lose all connectivity, however in Virtual IPs - > status - it shows as MASTER on FW2 for both LAN & WAN..

[fearz]

FW 1 HA

[fearz]

It's worth to mention that both FWs are on different Proxmox nodes but in same subnet 192.168.4.x

I have MAC Filtering in Proxmox disabled for both VMs.

[fearz]

FW1 NAT outbound rule

[Patrick M. Hausen]

I'll look into the details later today.

[fearz]

Thanks Patrick,

Also, My interfaces names & identifiers are identical between both FWs.

[Patrick M. Hausen]

A couple of things I notice:

- Set the CARP VIPs to /32 instead of /24 (should not be cause of any problems, but "cleaner").
- Do you have a dedicated interface for pfsync, virtual or not? If yes, why NAT? And why not the default of directed multicast for pfsync?
- You have private networks on both sides, LAN and WAN, so the uplinks are Ethernet, right? Make sure to disable reply-to (Firewall > Settings > Advanced).
- Both firewalls have a plain Ethernet connection on all Interfaces with CARP? Not only to some modem but also to each other, possibly via the modem's builtin switch?

Kind regards,
Patrick

[fearz]

Hi Patrick,

- Set the CARP VIPs to /32 instead of /24 (should not be cause of any problems, but "cleaner"). <- Done on both FWs
- Do you have a dedicated interface for pfsync, virtual or not? If yes, why NAT? And why not the default of directed multicast for pfsync? Yes, dedicated interface via Proxmox, I'm not sure of the remaining questions, I just followed the video.
- You have private networks on both sides, LAN and WAN, so the uplinks are Ethernet, right? Make sure to disable reply-to (Firewall > Settings > Advanced). - < Done on both FWs
- Both firewalls have a plain Ethernet connection on all Interfaces with CARP? Not only to some modem but also to each other, possibly via the modem's builtin switch? I don't fully understnd the question but the modem (WEWAN) is connected to same box as FW1 via Proxmox bridge to Opnsense


I attached the interfaces overview, it would help..

Also let me confirm if the tests i'm doing is right, once I go to FW1 System HA, and resync everything to FW2..

I have ping -t set for:

192.168.4.1
192.168.4.2
192.168.4.14
10.0.0.2
10.0.0.1
8.8.8.8

I dissconnect the FW1 LAN cable & here are the results:

192.168.4.1 - fails
192.168.4.2 - still pinging
192.168.4.14 - still pinging
10.0.0.2 - fails
10.0.0.1 - fails
8.8.8.8 - fails

[fearz]

Patrick? Anyone?

[Patrick M. Hausen]

I don't quite understand the IP addresses on your WEWAN. For HA you need a static /29 at least. One IP address for each firewall, one CARP. Why do you have a single global address and RFC 1918 addresses on the same interface?

[fearz]

WEWAN is an interface configured with DHCP from modem, i have in proxmox 3 interfaces, 1 WAN, 1 LAN & 1 for HA/pfsync

[Patrick M. Hausen]

You need static addresses on all interfaces that should support CARP. I am not aware of HA supporting DHCP (as a client).