CARP and IP Alias set up

[whitwye]

On pfSense, where multiple IPs on a WAN interface are to be controlled by CARP, first one of them is set to CARP, then for subsequent IPs when setting them to IP Alias the Interface drop-down menu includes not just the major interfaces, but also an entry for the CARP address, in the form

<ip> (vhid: n)

OPNsense does not offer that. What is the OPN sense method for tying a set of VIPs to CARP for an interface? If this is in the doc, I haven't found it.

[whitwye]

I'm sure there's more than one way to do this. Part of my question is about the sometimes ambiguous translation between GUI options and back-end implementation. On the face of it, I'm assuming that I shouldn't just try to define all the virtual IPs of each WAN as CARP, as that would result in a separate CARP signal for each IP -- far too noisy. But clever code translating between the GUI and the back-end might see that and translate to a single CARP signal. Or it could be that when IP Aliases are assigned to an interface which as a CARP signal, that those aliases are automatically bound to the CARP signal -- accomplishing what pfSense does explicitly through its GUI option to do just that. I'd like to avoid spending hours experimenting to work out something that no doubt others already know.

Someone here must know what the OPNsense way to handle this is. Obviously OPNsense is not just for home routers, where the ISP has handed out only one or several IPs. The doc at https://docs.opnsense.org/manual/how-tos/carp.html describes setting up one VIP via CARP, but doesn't speak to what to do when, like us, you have a /27 on each WAN. I'm assuming this is a documentation bug (or lack of a feature there at least), and not that OPNsense itself can't handle this. What is the right way, through the menus, to have OPNsense handle this?

[whitwye]

Hope someone can advise me soon on this.

Experimenting, I'm getting a contradictory result. On the first WAN interface, I've added two CARP IPs, both using the same VHID number. On the second WAN interface, I've added one CARP IP, using a second VHID number. But when I try to add a second CARP IP, OPNsense complains:

The following input errors were detected:

    VHID 12 is already in use on interface WAN2. Pick a unique number on this interface.

In the CARP widget in the lobby, it clearly shows the first two IPs as "WAN1@11 > MASTER <ip>", and the second as "WAN2@12 <ip>", where 11 and 12 are the CARP VHIDs. Note the second interface's IP isn't showing it as master either. What difference is allowing multiple CARP IPs with the same VHID on one WAN interface, and blocking it from being set up that way on the other?

Also, WAN2 is not sending out any CARP packets. It's receiving some with another VHID from another system on that interface. But it's not sending out any of its own on WAN2, not with VHID12 or anything else. Also, it has not add the one CARP IP to its interface.

[whitwye]

Looking with ifconfig, I see that interface igb1 (WAN1) has  promiscuous mode enabled, igb2 (WAN2) does not. Why would this be? Could it be related to the problems?

[mimugmail]

Promiscuous could be related to Suricata.

As far as I remember you have to create for each WAN IP a single group. Noisy, but I think there's no other way.

[whitwye]

Promiscuous could be related to Suricata.

As far as I remember you have to create for each WAN IP a single group. Noisy, but I think there's no other way.

Interesting. I haven't enabled Suricata, let alone pointed it at any particular interface.

One of my questions is why WAN1 here is allowing 2 IPs to be assigned with the same CARP VHID (broadcast ID), while OPNsense blocks me from setting up WAN2 that way. That's incoherent behavior -- either it should be blocked or allowed on both interfaces.

Also, since WAN2 isn't sending out any broadcasts on WAN2 at all, there's no way this can work without some sort of fix.

[mimugmail]

I can't reproduce it right now. It seems there's a bug not correctly validating that you use one group on 2 CARP IPs. Try to delete this, make 2 groups on WAN1 and then try WAN2

[whitwye]

I can't reproduce it right now. It seems there's a bug not correctly validating that you use one group on 2 CARP IPs. Try to delete this, make 2 groups on WAN1 and then try WAN2

By "make 2 groups" do you mean use 2 vhids on WAN1, each for several IPs?

My bigger problem is the CARP IP already assigned to WAN2 is not resulting in any CARP broadcast from it at all. I have the firewall rules explicitly allowing both originating and outgoing (which wasn't necessary on WAN1), but all that's seen on the WAN2 interface is an incoming signal from an unrelated system on that subnet, not any signal from itself.

Now, it's Tier2 while WAN1 is Tier1. We need outgoing here to use only WAN1 unless the line goes down. But we also need incoming traffic to work across both WANs, so that DNS, mail and other servers are always reachable on both. If the logic is "I'm not going to send CARP signals because this is Tier2," that's not good. Beyond that guess I haven't a clue why this shouldn't be working.

I've used CARP's Linux port, UCARP for years, without any trouble like this.