Messages

Hello everyone,

I tested again with raw FreeBSD and I can't reproduce the issue there. So there must be something with the gateway code of OPNsense that causes the loose of the default gateway.
I configured two FreeBSD machines with 3 networks each. I did not actually use vtnet1, but added it anyways, if I want to test state sync later.

The following is set in rc.conf:

router1:
hostname="freebsd-router1"
keymap="de.kbd"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
#networking stuff
#Management-Interface/LAN
ifconfig_vtnet0="inet 192.168.122.100 netmask 255.255.255.0"
#PFSYNC
ifconfig_vtnet1="inet 192.168.1.1 netmask 255.255.255.0"
#WAN
ifconfig_vtnet2="inet 192.168.2.1 netmask 255.255.255.248"
ifconfig_vtnet2_alias0="inet vhid 1 pass testpass alias 192.168.3.1/29"

#routes
#defaultrouter="192.168.3.4"
static_routes="default"
route_default="-net 0.0.0.0 192.168.3.4"

router2:
hostname="freebsd-router2"
keymap="de.kbd"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
#networking stuff
#Management-Interface/LAN
ifconfig_vtnet0="inet 192.168.122.83 netmask 255.255.255.0"
#PFSYNC
ifconfig_vtnet1="inet 192.168.1.2 netmask 255.255.255.0"
#WAN
ifconfig_vtnet2="inet 192.168.2.2 netmask 255.255.255.248"
ifconfig_vtnet2_alias0="inet vhid 1 advskew 100 pass testpass alias 192.168.3.1/29"

#routes
#defaultrouter="192.168.3.4"
static_routes="default"
route_default="-net 0.0.0.0 192.168.3.4"

No matter how hard I try to break the default route, the problem does not exist in raw FreeBSD.
Similiar to the OPNsense-setup I disabled carp on the primary by deactivating the whole interface:
ifconfig vtnet2 down
after that i made sure that carp is indeed disabled:
output from ifconfig vtnet2:
vtnet2: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 52:54:00:7b:a5:0a
   inet 192.168.2.1 netmask 0xfffffff8 broadcast 192.168.2.7
   inet 192.168.3.1 netmask 0xfffffff8 broadcast 192.168.3.7 vhid 1
   carp: INIT vhid 1 advbase 1 advskew 0
   media: Ethernet 10Gbase-T <full-duplex>
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
As you can see carp is in INIT state.
After looking at the router2 it was indeed in master state, but the default route was still there on both systems:
Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.3.4        UGS      vtnet2
127.0.0.1          link#4             UH          lo0
192.168.1.0/24     link#2             U        vtnet1

I also tried spamming ifconfig vtnet2 vhid1 state backup to somehow provoke bad behaviour, but there was not success. :o
Then I tried setting the default route in different ways in rc.conf like adding the option defaultrouter, but this also works without an issue.
The OPNsense-Team will definitely have to look in their gateway-code to solve this issue.

Hi guys,

I can completly reproduce the problem. It was also present in pfSense and seamingly fixed at some point, but started occuring again there aswell. https://redmine.pfsense.org/issues/8465 I bet this is a upstream bug from FreeBSD. This is really simple to reproduce. Just set an upstream gateway not directly reachable by the interfaces of the firewalls, but instead only reachable by the carp ip subnet on these interfaces. As soon as you disable carp on the master or the slave the gateway is lost. It also does not matter if you set far gateway or not. The funniest thing is applying the gateway config again seems to solve the problem until a reboot. I also don't understand why this problem is not top priority. This is a serious issue that probably a lot of people never noticed, because they did not check the routing table and or did not test ha properly and just deployed it. :o Not everyone has 2 WAN IP addresses to waste for ha...  I think I will try to reproduce the problem on a raw FreeBSD install and try to reproduce it there to report it to the FreeBSD guys. I bet it will happen aswell.