Topics

Franco suggests perhaps my gateway monitoring isn't set up correctly. So let's run through that here.

At System > Gateways > All

Name    Interface    Gateway    Monitor IP    Description    
GW_WAN    GLOBAL    207.239.<offuscated>    8.8.4.4    GlobalGW    
GW_WAN_2 (default)    COGENT    38.105.<obfuscated>    8.8.8.8    CogentGW

So far so good. But indeed double-checking on the "DIsable Gateway Monitoring" boxes shows them checked. [Note on interface conventions: 99 times out of 100 checkboxes are used to enable things, not disable them.] Is this it? Uncheck both boxes and apply.

Then I disable the COGENT interface. And ...

root@OPNsense:/tmp # route get 207.136.236.70
route: route has not been found

But it does know the special route to the check IP:

root@OPNsense:/tmp # route get 8.8.4.4
   route to: google-public-dns-b.google.com
destination: google-public-dns-b.google.com
    gateway: 207.239.<obfuscated>
        fib: 0
  interface: igb2
      flags: <UP,GATEWAY,HOST,DONE,STATIC>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

And can ping that. But it can't ping 8.8.8.8, or 207.136.236.70. And when I try to connect to one of the public IPs on the GLOBAL interface from outside, I can't.

Now I do nothing but enable the COGENT interface.

root@OPNsense:/tmp # route get 207.136.236.70
   route to: vt.electrainfo.com
destination: default
       mask: default
    gateway: g<obfuscated>1.atlas.cogentco.com
        fib: 0
  interface: igb1
      flags: <UP,GATEWAY,DONE,STATIC>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

And I can both ping IPs on the GLOBAL interface, and connect to services NATed behind them -- which were unavailable with the COGENT interface down.

So my bad on misreading the checkbox function. Yet, getting that right's not enough to make MultiWAN work. Is there documentation on what's supposed to be going on under the covers here, so I can check on where that might be going wrong?

Thanks again,
Whit

Given that OPNsense isn't using multiple routing tables (which is how Linux is typically configured for policy routing), but instead is using PF's route-to and reply-to options, where can I learn about what in theory should be happening with those as interface availability changes?

I'm intrigued by Franco's statement that there's an optimal pf rule set that will make the result robust, but puzzled on what that rule set should look like. As I've mentioned in other threads, so far I can't get OPNsense to handle WAN2 correctly when WAN1 is taken down. I'll be thankful for any suggestions of recipes that should work, or pointers to documentation that gives enough background to deduce what such recipes should look like.

Specifically, what rules applied to either the floating or WAN2 interface rule set would enable WAN2 to successfully return or originate traffic, regardless of WAN1's state? I see no evidence that OPNsense will originate traffic on WAN2 ever. But it at least returns traffic on WAN2 while WAN1 is up, yet fails to return traffic on WAN2 once WAN1 is down -- an odd and unexpected dependency. Taking WAN1 down removes the default route from the system; but apparently the power of pf route-to and reply-to rules should make success independent of that. Besides, the WAN1 default being their or not shouldn't on the face of it affect the success of WAN2 in replying on its IPs, since this is working with WAN1 up, where the reply doesn't take that default route anyway.

Separating this out from multiple issues discussed on https://forum.opnsense.org/index.php?topic=5765.0:

On WAN1, with a CARP IP set, the CARP VHID broadcasts are going out as expected.

On WAN2, with a CARP IP set, there are no outgoing CARP VHID broadcasts appearing on the interface except for some originating from an unrelated system on the same subnet.

The inspection is done with:

tcpdump -nvi igbX -c10 proto 112 -T carp

where "X" is 1 and 2 for the two interfaces. Nor are the missing CARP VHID broadcasts appearing on any other interface. And OPNsense is not adding the CARP IP in question to the second interface (as shown by ifconfig), although the first interface has the IP added.

Additionally, OPNsense has allowed 2 CARP IPs to be assigned to the first interface with the same VHID, while it rejects an attempt to assign a second CARP IP to the second interface with the same VHID as the first there, claiming this is a conflict. However since not even the first CARP IP on the second interface works at all yet, maybe solving that problem will also make this one go away.

We have a LAN using several subnets, some in 172.17.0.0/16, plus one in 192.168.1.0/24. Nagios runs from an IP in the latter. With a gateway group set as the gateway for the LAN firewall rule, as the Multi WAN doc says to do, the static route set up to send 192.168.1.0/24 traffic to the LAN gateway fails.

Should I take it this is just how it works, that static routing gets over ridden by any use of gateway groups, so that achieving the results that would normally be done through static routing instead requires special firewall rules? If so, are they then required on each interface, or will floating handle it?

In trying to work out how to best set up CARP to handle multiple IPs per interface, I'm looking for how to check the current status of CARP. By way of contrast and example, in pfSense there's a screen for that at Status > Carp (Failover). The equivalent of the Status menu is Services in OPNsense, but there's nothing for CARP there. I'm not saying pfSense is doing CARP right (in my brief experience it has deep problems in the implementation), just that that screen is useful (for example, in helping see the deep problems in its CARP implementation).

Is there any representation of the CARP status in OPNsense, short of using ifconfig in a shell?

On pfSense, where multiple IPs on a WAN interface are to be controlled by CARP, first one of them is set to CARP, then for subsequent IPs when setting them to IP Alias the Interface drop-down menu includes not just the major interfaces, but also an entry for the CARP address, in the form

<ip> (vhid: n)

OPNsense does not offer that. What is the OPN sense method for tying a set of VIPs to CARP for an interface? If this is in the doc, I haven't found it.

The doc refers to Policy Routing, without details on backend implementation. On Linux it's standard to use multiple routing tables for Multi WAN. FreeBSD supports that (https://lists.freebsd.org/pipermail/freebsd-arch/2007-December/007331.html, discussed at https://www.mmacleod.ca/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/). When I checked the pfSense kernel, it looked like that team left this feature out there. Is OPNsense using multiple tables for its Policy Routing? If so, what's the right way to view the multiple tables? It doesn't look like netstat has an option for that.

The goal here is to use Multi WAN in failover configuration for outgoing connections. That is, for connections originating on the LAN, use ISP1 unless it's down, then go to ISP2.

But for incoming connections, we need to handle them on whichever ISP they come in on. We haven't fully experimented with this yet, but already we see a problem. We have ports 443 and 22 enabled for connections from a remote admin IP to "This Firewall" as a Floating Rule. We reach the admin interface fine on port 443 on the IP assigned to ISP1; but on the IP assigned to ISP2 we got the just part of the initial signin screen, and then nothing. SSH works fine to ISP1's IP; to ISP2's IP it works for long enough to log in, but then freezes up within the first minute, in repeated trials.

Is the way the gateway is handled for a failover Multi WAN config incompatible with actively using the IPs of both WANs? We have various services behind the WANs we need to NAT from each of the WANs, and have available if that WAN can be reached, regardless of the failover state for outgoing connections. This part we haven't tested yet. The docs don't seem to talk about this aspect of Multi WAN setup at all. What are the implications?

As the main admin of this, located outside the office we're setting this up for, having the admin interface accessible on both WANs is important in itself. Does this require a load-balancing rather than failover setup to work?

This goes in the "most useless error message" sweepstakes. With two ISPs, for the second one I'm trying to add the necessary gateway. All I get is the error message:

Sorry, we could not create your IPv4 gateway at this time.

Now, it may be that the docs will straighten me out on the way to get this done. I just wanted to complain that the error message is nothing but frustrating, and totally without useful content.

At Interfaces: Other Types: LAGG, when I go to Parent interface, the drop down list is empty. Currently only 2 of the 6 NICs are assigned. Shouldn't the other 4 show up as choices here?

SInce we use Nagios extensively to track system health, I'm getting nrpe going. It's a fine thing to see so many packages available, especially after trying pfSense, which has cut itself off from standard FreeBSD packages! (Yes, pfSense has nrpe as part of it, but the interface page for it is crude.)

The pkg installation presents this message:

**********************************************************************

Enable NRPE in /etc/rc.conf with the following line:

   nrpe2_enable="YES"

A sample configuration is available in /usr/local/etc/nrpe.cfg.sample.
Copy to nrpe.cfg where required and edit to suit your needs.

**********************************************************************

What's the equivalent thing to do on OPNsense, which lacks the /etc/rc.conf file? Create /etc/rc.conf? Add that to /etc/defaults/rc.conf? It looks from this like maybe /etc/rc.conf is required:

root@OPNsense:/usr/local/etc/rc.d # more nrpe2
#!/bin/sh

# $FreeBSD$
#
# PROVIDE: nrpe2
# REQUIRE: LOGIN
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf to enable nrpe2:
# nrpe2_enable (bool):    Set to "NO" by default.
#                         Set it to "YES" to enable nrpe2.
# nrpe2_flags (str):      Not set by default.
# nrpe2_configfile (str): Set to "/usr/local/etc/nrpe.cfg" by default.
...

Is there a doc page somewhere that covers running available FreeBSD packages on OPNsense?

Searching before posting, I found this discussion: https://forum.opnsense.org/index.php?topic=5121.msg21235#msg21235

The context here is different. In a fresh OPNsense installation from the amd64 VGA USB version to a system in which OPNsense has been used to totally reformat the drive (in manual mode, but accepting each default), the resulting system boots, but with this complaint:

Aug  8 13:11:25 OPNsense kernel: Trying to mount root from ufs:/dev/ufs/OPNsense_Install [ro,noatime]...
Aug  8 13:11:25 OPNsense kernel: GEOM: da0: the secondary GPT header is not in the last LBA.
Aug  8 13:11:25 OPNsense kernel: GEOM: diskid/DISK-4C531001610425107534: the secondary GPT header is not in the last LBA.
Aug  8 13:11:25 OPNsense kernel: GEOM: diskid/DISK-4C531001610425107534: the secondary GPT header is not in the last LBA.
Aug  8 13:11:25 OPNsense kernel: GEOM: diskid/DISK-4C531001610425107534: the secondary GPT header is not in the last LBA.
Aug  8 13:11:25 OPNsense kernel: GEOM: diskid/DISK-4C531001610425107534: the secondary GPT header is not in the last LBA.

This looks like an inconsistency between the FreeBSD tools used to set up the drive, and where the FreeBSD kernel expects the secondary GPT header to be. Will this be a problem if trouble develops with the primary GPT header? Is there a utility which can adjust the secondary GPT header's location to make the kernel happy on next boot?

I did a second full installation on the system, just to be sure it wasn't a one-time problem. It showed up both times.

Bootblocks were successfully installed!

Segmentation fault (core dumped)
The installation was aborted.

Bad RAM??

| Your selected environment uses the    │
│ following console settings, shown in  │
│ parentheses. Select any that you wish │
│ to change.   

Nothing is shown in parentheses, at all.

Since the guided installation from USB to HD failed when running it from a shell in a iDRAC session, I'm now trying to use SSH. The manual says:

The GUI will listen on https://192.168.1.1/ for user "root" with password "opnsense". Using SSH, the "root" and "installer" users are available as well on IP 192.168.1.1. Note that these install medias are read-only, which means your current live configuration will be lost after reboot.

In this case I've installed the LAN interface at 172.17.10.3. The web version is available. SSH is not.  When I try to ssh to localhost there is a daemon there. And telneting to port 22 on localhost shows it's the standard OpenSSH. But from another system, that can reach the HTTPS interface for 172.17.10.3, I can't reach the SSH daemon.

How long should the phase, at 63% of "/usr/local/bin/cpdup -vvv -I -o /usr/local /mnt/usr/local" take? I ask because it's been stuck there for 20 minutes with just the spinning |/-\ running.

This is in trying to install from the "VGA" USB image.

While the end point will be a two-WAN config, while setting this up initially we have OPNsense just connected via LAN. I'm trying to reset the default gateway to use the LAN for now. If this were Linux I'd just do "ip ro del default; ip ro add default via a.b.c.d dev whatever." But here, from the console, when I try "route del default" I get "route: writing to routing socket: Address already in use \  del net default fib 0: gateway uses the same route".

Since I'd like to configure OPNsense before connecting directly to either WAN, including downloading updates, this has me stuck for now. What are the steps to get FreeBSD to let me set a new default route, via the LAN, for now? I've Googled, but the docs I find elsewhere imply that "route del default" will just work, and don't say what's required when it refuses to.

The current initial OPNsense configuration routine seems to assume that the whole setup is new, so the WAN interface can be immediately used. Our situation is one where we're getting set to replace existing firewalls, which will remain in charge of our WANs until we've go OPNsense fully configured, and ready to substitute for them. Is there a documented procedure somewhere for our type of situation? Thanks.