OPNsense Forum
English Forums => High availability => Topic started by: Dunuin on February 20, 2021, 02:22:47 pm
-
Hi,
I followed this (https://www.thomas-krenn.com/de/wiki/OPNsense_HA_Cluster_einrichten) tutorial what basically is doing the same like here (https://docs.opnsense.org/manual/how-tos/carp.html).
The primary OPNsense is running inside a VM on a Proxmox host. The secondary OPnsense is running inside a VM on a FreeNAS host. Both VMs are using 8 virtio NICs (all interfaces with identical interface names). 7 of them are tagged later on the host and are connected to a switch over a single 10G NIC. One is using a Gbit NIC for pfsync so both OPNsense VMs are directly connected using a dedicated patch cable.
I disabled "mac filtering" on the proxmox host so MAC spoofing should work for CARP and I think FreeNAS isn't preventing MAC spoofing out of the box. I wound no info on how to allow MAC spoofing only one half year old feature wish to disable MAC spoofing to increase the bhyve security.
Right now I only setup 4 Interfaces:
Firewall 1 (primary on Proxmox) is using:
DMZ 192.168.42.2
LAN 192.168.43.2
PFSYNC 192.168.4.2
WAN 192.168.0.2
Firewall 2 (secondary on FreeNAS) is using:
DMZ 192.168.42.3
LAN 192.168.43.3
PFSYNC 192.168.4.3
WAN 192.168.0.3
My ISPs router (Fritzbox) IP is 192.168.0.1.
Firewall 1s CARP dashboard plugin shows me this:
WAN@1 MASTER 192.168.0.4
LAN@3 MASTER 192.168.43.1
DMZ@5 MASTER 192.168.42.1
Firewall 2s CARP dashboard plugin shows me this:
WAN@1 MASTER 192.168.0.4
LAN@3 MASTER 192.168.43.1
DMZ@5 MASTER 192.168.42.1
Pfsync is working and I can sync configs from firewall 1 to firewall 2.
What looks strange to me:
1.) Dashboards of both Firewalls are showing "MASTER" at the same time. Shoudn't one be shown as SLAVE or something like that?
2.) If I look at my ISPs router I always see that two hosts with the same IP (192.168.0.4) but different MACs are connected. But there is always only 192.168.0.2 OR 192.168.0.3 connected and both are using the identical MAC. Even if both OPNsense VMs are running.
If I shutdown one VM 192.168.0.2 switches to 192.168.0.3 and if I'm starting the VM again and shutdown the other VM it switches back from 192.168.0.3 to 192.168.0.2.
I thought the idea was that firewall 1 is always connected with 192.168.0.2 and a unique MAC, firewall 2 always connected with 192.168.0.3 and a unique mac and that there should be only one host with 192.168.0.4 (the virtual IP) connected at the same time. And that 192.168.0.4 is pointing to the master whoever that might be. So both VMs should share the same IP 192.168.0.4 and MAC but only one of them at the time.
3.) If I ping google.de I get this:
Code:
--- google.de ping statistics ---
7 packets transmitted, 7 received, +2 duplicates, 0% packet loss, time 257ms
rtt min/avg/max/mdev = 5.099/5.221/5.384/0.103 ms
I never saw before that I recieve duplicates. I thought maybe both VMs are running in parallel as master and because of that I receive duplicate answers?
If I shutdown one of the two VMs, ping shows normal results without duplicates.
Do you know what could went wrong?
I already double checked my config and the tutorial but I don't see what I could have done different.
-
This is what my ISPs router is telling me whats on the WAN net:
Master-VM is on, Backup-VM is off:
FE:41:DC:03:E2:67 192.168.0.4
00:00:5E:00:01:01 192.168.0.2
Master-VM is off, Backup-VM is on:
00:A0:98:6F:54:71 192.168.0.4
00:00:5E:00:01:01 192.168.0.3
Master-VM is on, Backup-VM is on:
FE:41:DC:03:E2:67 192.168.0.4
00:A0:98:6F:54:71 192.168.0.4
00:00:5E:00:01:01 192.168.0.2 or 192.168.0.3 but never together
And these are the WAN interfaces:
(https://forum.opnsense.org/index.php?action=dlattach;topic=21631.0;attach=15471)
Is it possible that virtual IPs/CARP and static IPs are somehow switched?
-
It might be worth setting the group ID differently on each interface (as that will ensure the MAC addresses are unique). I had weird issues till I did that. Certainly the example configuration (https://docs.opnsense.org/manual/how-tos/carp.html) has different groups set per interface.
I also think the double-master you have is hurting you - are you certain you have different skews on the two hosts? Normally I think it's base 1 + skew 0 on the primary and base 1 + skew 100 on the secondary.
Having said that - I think I have other broken NAT issues on my 21.1 environment, which I'm still troubleshooting, so I could be way off.
-
I'm not familiar with Proxmox, but, you need to make sure that the virtual switch / virtual interface card allows MAC address spoofing whoch is normally not allowed.
CARP must be allowed to spoof MAC addresses, that's how CARP works.
https://forum.proxmox.com/threads/allow-mac-spoofing.84424/ (https://forum.proxmox.com/threads/allow-mac-spoofing.84424/)