Messages

Thanks for all your help. It has been a long learning journey for me:)

I just wanted to give an update. I ended up using gateway groups instead of frr. WAN on node1 as tier1, node2 (LAN) as tier2. In addition I added a 5g mobile usb router as tier3. If WAN fails, I still have internet routing through node 2, and if both WAN on node 1 and 2 is down, I still have net through the mobile router. If I shut down node 1, node 2 becomes master. So at the moment it seems to work as I hoped. I guess my system is more or less failsafe now without frr.

Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            81.xxx.xxx.1       UGS             re0
8.8.8.8            81.xxx.xxx.1       UGHS            re0
10.10.10.0/24      link#7             U               wg0
10.10.10.1         link#3             UHS             lo0
10.10.10.2         link#7             UHS             wg0
10.10.10.3         link#7             UHS             wg0
10.10.10.4         link#7             UHS             wg0
81.xxx.xxx.0/21    link#1             U               re0
81.xxx.xxx.1       link#1             UHS             re0
81.xxx.xxx.185     link#3             UHS             lo0
127.0.0.1          link#3             UH              lo0
192.168.50.0/24    link#2             U            vtnet0
192.168.50.1       link#3             UHS             lo0
192.168.50.2       link#3             UHS             lo0

Unplugged:
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
10.10.10.0/24      link#7             U               wg0
10.10.10.1         link#3             UHS             lo0
10.10.10.2         link#7             UHS             wg0
10.10.10.3         link#7             UHS             wg0
10.10.10.4         link#7             UHS             wg0
127.0.0.1          link#3             UH              lo0
192.168.50.0/24    link#2             U            vtnet0
192.168.50.1       link#3             UHS             lo0
192.168.50.2       link#3             UHS             lo0

I ran all configs and logs into AI. I dont know how accurate this is but:
"Your OSPF daemon is learning routes, but not injecting them into the FreeBSD kernel routing table.
That's why you don't see a second default route from the peer node."

Some more

Some of my configs

Running NIC in passthrough on both nodes. Interface go down as it should when I disconnect the cable. When running netsstat when disconnected, the WAN device dissappears from the routing table. It is present when wan is connected. I`ve read somewehre that some people experience a problem with default route persisting when upstream gateway is down.

Ahh, I see. I thought i mentioned that I run it in proxmox somewhere earlier. Thanks for clearing it up. I may try to run pci passthrough for WAN.

Oh sorry. :)
It does not seem to lead to the backup node 192.168.50.3 when disconnected.

Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            151.130.80.1       UGS          vtnet0
8.8.8.8            151.130.80.1       UGHS         vtnet0
10.10.10.0/24      link#7             U               wg0
10.10.10.1         link#3             UHS             lo0
10.10.10.2         link#7             UHS             wg0
10.10.10.3         link#7             UHS             wg0
10.10.10.4         link#7             UHS             wg0
127.0.0.1          link#3             UH              lo0
192.168.50.0/24    link#2             U            vtnet1
192.168.50.1       link#3             UHS             lo0
192.168.50.2       link#3             UHS             lo0
151.130.80.0/20    link#1             U            vtnet0
151.130.84.90      link#3             UHS             lo0

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::1                               link#3                        UHS             lo0
fe80::%lo0/64                     link#3                        U               lo0
fe80::1%lo0                       link#3                        UHS             lo0

I think so. Here are some more screenshots that might help. (Black theme is node1 192.168.50.2, white is node 2 192.168.50.3.)
Routing table:

Thanks!
I  have tried different configurations but for some reasopn I can`t get it to work. The nodes do communicate with each other, master state is "Full/DR" and backup is "Full/Backup", however OSPF does not respond if WAN gateway  is down. When unplugging, gateway is down within just a few seconds. These are my configurations (neighbors, prefix lists and route maps are empty). I also turned off BFD until I get OSPF working. I have no gateway groups. My CARP VIP LAN ip is 192.168.50.1. Master router: 192.168.50.2, backup is 192.168.50.3. I have tried both with carp failover/demote, but that neither worked. So my current configuration avoids CARP so node1 always is master.


My ISP gateway ip

Thank you, that was exactly what I was looking for. I have set up OSPF with BFD on both nodes, so the service is up and running and seems to communicate with each other, but I still loose internet when disconnecting WAN on the master node, so there is no rerouting to node 2 and I am trying to figure out what may be wrong. First of all, is it sufficient for me to use OSPF with BFD or do I need to setup BGP to achive this? This is only for a small homelab, so I am not trying to make this more complex than necessary and I understand BGP can be quite complex.

As I understand, HA CARP only detects when LAN interface is down.
Gateway and ip monitoring is for multi-wan, but I can`t set gateway groups with only one wan port on my node so multi wan is not an option for me.
My goal is to have continous internet connection. I have two different ISPs connected to each node which is in cluster. If my main ISP is disconnected, I want my other node/secondary ISP to automaticly take over.
Somebody mentioned dynamic routing. Is this the best way to achieve that with my 2 nic setup??

I am running opnsense on two proxmox nodes, both have 1 WAN and 1 LAN port each. WAN on both nodes are connected to a switch, which is connected to a fiber modem (bridge). I get two public ips from my ISP so each node has its own IP and both are online and have internet. It should be setup correctly. I also setup monitor IP 1.1.1.1 on wan, on both nodes. When I unplug, the WAN interface is still registered as up, while in gateway configuration WAN_DHCP is registered as down.
I have configured CARP VIP LAN 192.168.1.1 on both nodes. HA works and I am able to sync configurations. Master VHID 1 (freq. 1/0), backup VHID 1  (freq. 1/100).
The node does not seem to know that the WAN is unplugged, because the interfase is still up.

I`m starting to wonder if this has something to do with proxmox bridging NIC to vmbr0, and vmbr0 always think link status is connected even when disconnected.

I run OPNsense on two separate nodes, both receive their own public ip from my ISP. Enabled high availability and CARP. If I shut down the master node, the backup node immediately becomes the master and resumes the internet connection after just a few secs. However, if I unplug the ethernet cable (instead of powering down the master node), the node is still listed as the master node and the backup node still remain backup causing me to loose internet connection. How can I fix this?