Messages

1

High availability / Re: CARP with DHCP on WAN

« on: October 26, 2024, 12:06:03 am »

I am a Google Fiber subscriber. My environment is simple with an active/passive firewall -  a KVM VM with hardware passthrough of a quad port NIC, and physical hardware firewall with some intel NICs. I have a single WAN, and a single LAN interface running CARP.  The VPNs I use continue to function after failover. Stateful protocols such as ipsec or openvpn will drop and need to re-negotiate, but can reconnect immediately. Wireguard has no such issue.

Spali's github post is very useful: https://gist.github.com/spali/2da4f23e488219504b2ada12ac59a7dc

I have updated my personal script to the following, which is a mash of theirs and mine, which I posted in Reddit some time ago: https://www.reddit.com/r/opnsense/comments/runb4r/diy_ha_activepassive_for_home_internet/

Code: [Select]

#!/usr/local/bin/php
<?php
require_once("config.inc");
require_once("interfaces.inc");
require_once("util.inc");
$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';
if ($type != 'MASTER' && $type != 'BACKUP') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}
if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}
foreach($config['interfaces'] as $ifkey => $interface) {
    if ($ifkey=='opt3') {
        if ($type == 'MASTER') {
            log_msg("Carp Status is now Master!");
            log_msg("Enabling interface: $ifkey - {$interface['if']}");
            shell_exec("/sbin/ifconfig {$interface['if']} up");
            $config['interfaces'][$ifkey]['enable'] = '1';
            write_config("enable interface '$ifkey' due CARP event '$type'", false);
            interface_configure(false, $ifkey, false, false);
            sleep(1);
            log_msg("Restarting DHCPD");
            shell_exec('pluginctl -s dhcpd restart');
            sleep(1);
            log_msg("Issueing dhclient command to request a DHCP lease");
            shell_exec("dhclient {$interface['if']}");
        } else if ($type == 'BACKUP') {
            log_msg("Carp Status is now Backup!");
            log_msg("Disabling interface: $ifkey - {$interface['if']}");
            shell_exec("/sbin/ifconfig {$interface['if']} down");
            unset($config['interfaces'][$ifkey]['enable']);
            write_config("disable interface '$ifkey' due CARP event '$type'", false);
            interface_configure(false, $ifkey, false, false);
            log_msg("Stopping DHCPD");
            shell_exec('pluginctl -s dhcpd stop');
        }
    }
}
?>

(the forum is breaking the greater than and less than in the PHP brackets at the start and end, correct them yourself)

• This version will also manually "down" interfaces, as disabling them does not appear to fully "shut" the interface in my environment. This can cause mac flapping, and all of the issues related to that condition.
• My version also stops the DHCP Daemon, which ensures that I only have one DHCP server running on my LAN. I need the backup device to actually become "passive". Calling dhclient may not be necessary with the interface_configure call, but it's a holdover from when I previously only used shell_exec("/sbin/ifconfig {$interface['if']} down");   to up/down the interfaces, instead of enabling/disabling the interfaces.
• I use log_msg instead of log_error so that these events show up in the general system log as a "notice".

I do recommend creating a gateway with "Upstream Gateway" checked and a higher metric than the normal WAN gateway, as per spali's github comments to allow the backup to reach the internet via the LAN.

I also recommend disabling the "Backup" router's WAN interface - so that your secondary device will boot up with the WAN in disabled state, and the CARP script will re-enable the interface if CARP goes master. This prevents the devices from both booting up and each having active WAN interfaces.

Saw your reddit post and your most recent post on github. Thank you to you and Spali on figuring this out.
Im on 24.7.6, does the script no longer work or is the one you posted here working with .6?

I haven't set this up yet, but I have been looking into doing this for a while.

My setup:

• opnsense main: 192.168.29.1
• opnsense backup: 192.168.29.100
• pfsync/halink between the two: 10.0.0.1 and 10.0.0.2
• GPON ATT is on VLAN 842 (to bypass the need for the ATT Fiber gateway)

What should my CARP virtual IPs be for WAN and LAN?
Should I keep the backup a fully clean OPNsense state or add things like the VLAN 842 for the GPON or restore a proxmox backup so its all the same and just change the CARP settings and such??