Messages

I don't see that setting in the CARP config page (attached). I have FRR installed but disabled, so maybe it came from there?

OPNsense 21.1.7_1-amd64

I have two pairs of firewalls and they're both behaving in the same way and I'm not sure why. The primary has CARP interfaces with base:skew values of 1:0. Secondary is 1:100. Occasionally the primary shows CARP status of BACKUP and secondary shows MASTER. I see this in the log on the primary while in BACKUP status (newest on top):

Code Select Expand

2021-07-06T15:10:26 opnsense[54731] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.1.254 - LAN CARP (1@em1)" has resumed the state "BACKUP" for vhid 1
2021-07-06T15:10:26 kernel em1: deletion failed: 3
2021-07-06T15:10:26 kernel carp: 1@em1: MASTER -> BACKUP (more frequent advertisement received)
2021-07-06T15:09:51 opnsense[23015] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.2.0.1 - LDC01-TEST CARP (2@em0_vlan2)" has resumed the state "BACKUP" for vhid 2
2021-07-06T15:09:50 opnsense[76639] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.100.2.1 - LDC01-TDC01-L2 (3@em0_vlan910)" has resumed the state "BACKUP" for vhid 3
2021-07-06T15:09:50 kernel em0_vlan2: deletion failed: 3
2021-07-06T15:09:50 kernel carp: 2@em0_vlan2: MASTER -> BACKUP (more frequent advertisement received)
2021-07-06T15:09:50 kernel em0_vlan910: deletion failed: 3
2021-07-06T15:09:50 kernel carp: 3@em0_vlan910: MASTER -> BACKUP (more frequent advertisement received)

Following the page at https://docs.opnsense.org/development/backend/carp.html, I see the following on the primary (while in BACKUP status):

Code Select Expand

root@LDC01A:~ # sysctl net.inet.carp.demotion
net.inet.carp.demotion: 1048576
root@LDC01A:~ # sysctl net.inet.carp.ifdown_demotion_factor
net.inet.carp.ifdown_demotion_factor: 240
root@LDC01A:~ # sysctl net.inet.carp.senderr_demotion_factor
net.inet.carp.senderr_demotion_factor: 240
root@LDC01A:~ # sysctl net.inet.carp_demotion_factor
sysctl: unknown oid 'net.inet.carp_demotion_factor'
root@LDC01A:~ # sysctl net.pfsync.carp_demotion_factor
net.pfsync.carp_demotion_factor: 240


Not sure what to make of it, I do this:

Code Select Expand

root@LDC01A:~ # configctl interface update carp service_status
OK

Within seconds, the primary firewall changes its CARP status to MASTER and the secondary to BACKUP. I see this in the log on the primary:

Code Select Expand

2021-07-07T10:44:02 opnsense[99061] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.2.0.1 - LDC01-TEST CARP (2@em0_vlan2)" has resumed the state "MASTER" for vhid 2
2021-07-07T10:44:01 opnsense[54887] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.1.254 - LAN CARP (1@em1)" has resumed the state "MASTER" for vhid 1
2021-07-07T10:44:00 opnsense[21941] /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "10.100.2.1 - LDC01-TDC01-L2 (3@em0_vlan910)" has resumed the state "MASTER" for vhid 3
2021-07-07T10:43:59 kernel carp: 2@em0_vlan2: BACKUP -> MASTER (preempting a slower master)
2021-07-07T10:43:59 kernel carp: 1@em1: BACKUP -> MASTER (preempting a slower master)
2021-07-07T10:43:59 kernel carp: 3@em0_vlan910: BACKUP -> MASTER (preempting a slower master)
2021-07-07T10:43:59 kernel carp: demoted by -1048576 to 0 (sysctl)
2021-07-07T10:43:59 carp[40586] carp promoted by 1048576 due to service recovery

• Why did CARP status swap in the first place?
• Why does it not swap back until I manually run that code in the shell?

I have two pairs of OPNsense firewalls. All are running OPNsense 21.1.7_1 and both pairs are configured as CARP pairs with equivalent configuration, except the CARP master election appears inconsistent between the two pairs and I don't know why.

Pair 1:
router A CARP: 192.168.1.254/24 (vhid 1 , freq. 1 / 0)
router B CARP: 192.168.1.254/24 (vhid 1 , freq. 1 / 100)
CARP MASTER: router B

Pair 2:
router A CARP: 10.15.0.17/24 (vhid 1 , freq. 1 / 0)
router B CARP: 10.15.0.17/24 (vhid 1 , freq. 1 / 100)
CARP MASTER: router A

The election of CARP master is consistent. IE, if I reboot the CARP master in either pair, the backup router will promote to CARP master until the first router finishes booting, at which point the rebooted router resumes as master. Same if I disable CARP on the master then re-enable it.

The documentation indicates that the member with the lower Skew number is normally master. Why does my second pair follow this convention but the first pair does not?

I don't understand your recommendation. I have added nothing in the Networks tab. If I try to add an entry in the Networks tab there are references to the prefix lists I created, but I don't know the purpose of this dialogue.

Routes attached.

Configs sanitised and attached.

Green is md5 hash
Black is WAN IP or network of tdc01b
Red is WAN IP or network of ldc01b

Notice that the far side WAN network appears as a K route (static, added by me) and as an O route (added by OSPF on the Wireguard interface). I'd like to prevent it from being added as an O route so I could also not have to override it with the static route.

I have two OSPF peers, both running OPNsense 21.1.7 and FRR. There is a Wireguard tunnel between them running over the WAN interfaces and OSPF enabled on the Wireguard interface such that both firewalls are sharing connected and static routes over the tunnel. The problem is that with OSPF running, after some short interval Wireguard starts sending its UDP packets via the tunnel rather than the WAN. After sending a few packets over the tunnel using the WAN destination address, Wireguard on the peer starts using the tunnel endpoint address as its peer address, such that its trying to form a tunnel within the tunnel.

I can circumvent this undesired behaviour by adding a static route for the remote endpoint via the WAN gateway, but I'd prefer to keep the routing table as small as possible, and there are other routes I'd prefer not to share via OSPF, such as the PFSYNC network and the network of the wireguard tunnel, which end up being redundant in the routing table and would be better not distributed via OSPF.

I have read the filtering section in the FRR documentation, but it's too sparse and appears to be written for somebody that is already familiar with the software. I tried adding prefix lists for these networks on both routers in Routing : OSPF : Prefix Lists with action Deny, but this didn't prevent these routes from being distributed or used.

It appears I can't filter a received route with OSPF. ref: https://forum.opnsense.org/index.php?topic=22852.0. Is there a good way to prevent FRR from either distributing these routes or from using received routes? Am I just doing it wrong?

I changed the password and now it works. The old password had a colon (:) in it, maybe that was a problem for XMLRPC.

I'm getting this error while trying to perform an initial synchronisation between two OPNsense boxes. I have confirmed that

• target IP is correct (pfsync interface)
  
• user is root
  
• password is correct
  
• pass rule in place for TCP/443 on pfsync interface
  
• firewall log on target shows pass rule working
  
• tcpdump on target shows S>A>P>F packets between both hosts on pfsync interface
  
• another pair of OPNsense firewalls with apparently same config (but different passwords) syncing just fine

In System: Log Files: General on the target host after attempting to sync I see

2021-06-22T10:57:45   api[21532]   [2021-06-22T10:57:45-04:00][error] no active session, user not found   
2021-06-22T10:57:45   api[21532]   [2021-06-22T10:57:45-04:00][error] no active session, user not found

What am I missing?

Are 3 real IPs needed or can private IPs be used?

You must use 3 IPs that are valid on the subnet you are trying to communicate with. So if that's a public network, then you need 3 IP addresses in that subnet. If it's on a public network, then you need 3 IPs in that subnet.

That does seem unexpected. How did you verify which box is responding?

With carp both firewalls share a single IP address.

I think CARP is the obvious choice on the LAN sides for that reason. I'm less sure that I need it on the inter-router links where OSPF will be active regardless due to frequent changes on the LAN routes.

My main concern is whether there are gotchas with using CARP on one interface and OSPF on another.

You can also load balance and fail over by having two routes with the same metric (load balance) or a different one (fail over).

How well does this work when OSPF is balancing and CARP is in failover mode? For example, suppose A1 and B1 are both CARP master on their LAN and OSPF is balancing traffic between A and B sides. I expect to see all LAN A traffic coming from A1 and being forwarded to both B1 and B2 to be forwarded to LAN B. Return traffic from LAN B is picked up by B1 and forwarded to A1 and A2. Does pfsync manage this asymmetry in a way that is conducive to a good experience for network users?

And what happens if the OSPF peer goes down on the CARP master? Will OPNsense recognise that those upstream OSPF gateways are down and demote itself on the LAN CARP?

I am going to test some or all of these modes, but I thought I'd see if there was any wisdom floating in the community that could save me going down any blind alleys.

Our goal is to set up a VPN link between two sites with a pair of OPNsense firewalls at each site acting as VPN endpoints. We are leaning toward Wireguard as our VPN protocol, but that's not the focus of this post.

Diagram:

           /------OPNsense A1--------OPNsense B1------\
LANs A-----                      X                     ------LANs B
           \------OPNsense A2--------OPNsense B2------/

Goals:

• LANs A and B should have bilateral (non-NAT) IPv4 communication.
• Firewalls should share LAN routes between A and B side.
• Connectivity should be able to tolerate the failure of one firewall on each side with minimal convergence / failover period.

Questions:

• Our initial thought was to use CARP for failover and OSPF for route propagation. While I think CARP makes sense on the LAN interfaces, I am wondering if it makes sense to skip CARP on the A-B link. Can OSPF provide that redundancy / failover given that it will be active on the VPN interface anyway?
• Should I be looking at gateway groups for failover rather than CARP or OSPF for the A-B link? I'm not really sure what the overlap is between these protocols and I haven't found any documentation that really makes it clear to me what the interplay would be in a setup like this.

They're not blocking all upstream ICMP though, right? Can you just set your monitor IP to the second or third hop?

ESXi 6.5.0
OPNsense-21.1-OpenSSL-dvd-amd64.iso
2 CPU
2 GB RAM
8 GB disk

I booted from ISO in EFI mode. I designated vmx0 as WAN (connected with dhcp service), vmx1 as LAN (disconnected). Logged in as installer and accepted default console settings. On the next screen the installer appears to hang (console.png). If I hit the Enter, down-arrow or F10 keys on the keyboard, nothing happens for many minutes.

Then I connected the LAN interface and was able to ssh in as installer, but it still appears stuck (ssh.png).

I read the sections in the docs for Initial Installation and Configuration and Virtual & Cloud based Installation and I don't see that I did anything wrong. Any pointers?