Messages

I am also getting this error.  Nothing in the logs except

Code Select Expand

2023-04-01T15:15:43 Warning haproxy Proxy webrouter stopped (cumulated conns: FE: 0, BE: 0).

I think it's been like this since the upgrade to 23.1.   I can't see anything wrong with the config, and indeed it was previously working and I have not changed it. I am loathe to post config here as it contains internal network configuration details I would rather keep private.   

Does anyone have any idea why my proxy would immediately stop on server startup in this way?  Thanks.

I have now made it go away, probably by disabling and re-enabling some things.

Sorry for the delayed reply - got busy with other stuff and this got put on the back burner.

The output is:

Code Select Expand

root@<host>:~ # /usr/local/etc/rc.filter_synchronize
send >>>
Host: 192.168.66.4
User-Agent: XML_RPC
Content-Type: text/xml
Content-Length: 117
Authorization: Basic cm9vdDpQaWJqSXBzSUxwVEFmNHlZOTZ4Uw==
<?xml version="1.0"?>
<methodCall>
<methodName>opnsense.firmware_version</methodName>
<params>
</params></methodCall>received >>>
error >>>
fetch error. remote host down?root@fenchurch:~ # send >>>
Missing name for redirect.
<methodName>opnsense.firmware_version</methodName>
<params>
</params></methodCall>received >>>
error >>>
fetch error. remote host down?

This did enable me to discover that I wasn't able to traceroute/ping the backup interface from the main interface.   I went through all my firewall rules to try to work out what was wrong, and the only difference I could find was that for entirely inexplicable reasons, some automatic outbound NAT rules were being generated for the backbone on the primary router (perhaps because the primary router is configured to route by the backbone if the primary internet goes down.  Anyway these happened after outbound NAT was manually disabled for the interface, and I checked that when disabling outbound rules the problem still existed.)

In any event having been through all that and disabled and re-enabled gateways I am now at a point where ping, ssh and http over the backbone are working again,  and so sync is back up and running.  Subsequent runs are not producing an error, and my sync menu is now back.   I do not know and probably never will know which traceroute over the backbone works on the secondary, but not the primary router.  Thanks for your help with this.  I do wish routers were a little less "black box" sometimes.

I am still seeing this error.  Is it still safe to attempt to apply the patch above to the latest OpnSense?   Has this made its way into the main tree?  Thanks.

Thanks for the suggestion.  Yes, there is.  This is a previously working config that appears to have stopped working at some point.  I did have to reinstall the primary server at one point and did so using the USB stick config transfer method.  No passwords have changed.  The problem is that the diagnostic message I'm getting is so non-specific as to leave me lost as to how to even investigate what's not working.

Still looking for some help on this.  Even being pointed at where I might find some useful diagnostic logs as to why the link is not operative would be a help.

As of relatively recently my HA setting has stopped working.  When I try to access "High Availability" -> "Status" I get an error message:

    The backup firewall is not accessible or not configured.

CARP is still working fine, which I why I hadn't noticed and can't say when this begun.  There are no firewall rules preventing traffic on the direct ethernet link between the two firewalls.   Can anyone suggest how I might investigate / fix this?

Thanks

Thanks, this sounds like a workable solution in principle, but in practice, would OpnSense overwrite a custom ddclient.conf?

You'd probably find it less necessary to go over the same things if you had read what I wrote a little more carefully, but you're right about one thing, which is that continuing this discussion further is pointless.

2. By the time 22.7 will be released, ddclient will surely support a lot more providers than current release, and if it doesn't, I don't think devs will remove dyndns from the repo leaving dyndns users without an official alternative.

Why would you think that?  It would not be the first time functionality present in a previous version of the firmware has been removed.  In fact, the devs have indicated in this thread that they will never support a "custom" method like os-dyndns does, so practically it seems that as a user of Mythic Beasts, a small yet highly clueful UK ISP it is unlikely to ever get support in os-ddclient, my options will be

• Learn enough about ddclient and the internals of opnsense to write and submit a patch myself, which may or may not get accepted.
• Move my dyndns config into scripts and cron jobs.
• Leave the old unmaintained plugin in place.

It's likely to be one of the latter option sI take, as I don't really have the time for option 1.  I'm am lucky enough to have sufficient knowledge that option 2 is actually viable for me, but I am sure there are many others who won't be.  I get where the devs are coming from - maintaining legacy code is a pain, but removing functionality is always going to be a pain point for existing users, and I would strongly prefer the option of offering ddclient as an alternative and leaving dyndns in on an "as is" basi, so option 3 is likely to win out by default.

I think there has also been an element of glib "this is not a problem" response that also fails to acknowledge that for many, it will in fact, be a problem.  To say that there are sound technical reasons for causing those problems for a minority users is a stance that may well  be valid (I'm not in a position to judge), but breezily trying to convince us that this is an unproblematic change when it may well cause me a problem and take at least half a day of my time to work around does not sit well with me, I am afraid.  Things may continue to work, and I hope they will, but there is of course no guarantee of that.

So the FreeBSD people tell me that there are in fact two different watchdogs on the emulated chip, and the one that Qemu emulates is not supported by BSD.  Apparently it would not be a particularly hard driver to write, but having looked at the kernel sources, I don't have the faintest idea what is going on or the desire to work it out.

The upshot is that external watchdog monitoring when running under qemu is not possible, and is likely to remain unsupported for the foreseeable future.

[pciconf -l -v]

I have been experiencing some problems with the router becoming unresponsive overnight due to some sort of leak which I have found impossible to diagnose.   As such I have been trying to enable FreeBSD's built in watchdog functionality to qemu to reboot my OpnSense VM if it becomes unresponsive.   Unfortunately I cannot get it to work at all.

I am running a couple of OpnSense (based on FreeBSD 12.1) routers in qemu virtual machines in Proxmox 6.4. 

Promox has its own non-libvert way of configuring VMs, but to add a watchdog device, one adds the following line to the VM config file:

Code Select Expand

    watchdog: model=i6300esb,action=reset

I can see that this is working and the virtual device is present in the VM because pciconf -l -v in the guest includes the following output:

Code Select Expand

    none0@pci0:0:4:0:       class=0x088000 card=0x11001af4 chip=0x25ab8086 rev=0x00 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = '6300ESB Watchdog Timer'
        class      = base peripheral

Unfortunately, although the ichwd driver supports the emulated chipset, it is not being detected, and there are is nothing in the dmesg logs about it.

I have enabled watchdogd by adding the line watchdog_enable=yes to /etc/rc.conf.  This is working, but it will default to using a software watchdog, and appears to produce no debugging information, so is not helpful.

I can also force the kernel to load the ichwd driver by adding ichwd_load="YES" to boot/loader.conf (actually in OpnSense this is done by adding it to System -> Settings -> Tuneables in the GUI, but the final effect is identical).  Output from 'kldstat` shows

Code Select Expand

    Id Refs Address                Size Name
    12    1 0xffffffff82959000     70c8 ichwd.ko


I am thus reasonably certain that the virtual device is present on the system and the correct device driver loaded, but am unable to coax any further debugging information out of qemu, ichwd, or watchdogd.  What should I try next?

OK, so it turns out that this is in fact totally possible using monit.

In my design I have used a primary and backup router, but there may be appropriate tweaks for other setups.  CARP setup is the same for both routers, save that the skew on the backup machine is set to 50 (any number between 0 and the number used for demotion will do.  I have set monit to run every 5 seconds.  This does not seem to have a noticeable performance impact on my router.

The whole process is managed by a couple of shell scripts.  You will need the "bash" package installed to use them.  (Sorry, I do not do csh.)  Doubtless these could be much tidier and more robust than they are, but perhaps someone can use them as a starting point for their own more sensible scripts.  The "uplink-status" script monitors a named gateway, and dpinger monitoring will have to be configured for that gateway.  It ignores packet loss and other statuses, and stores the last (fully up or fully down) gateway status in /usr/local/var/run.  It will emit a non-zero return code when the link changes from its previous status to the opposite one.

Code Select Expand

#!/usr/local/bin/bash
#
# uplink-status - check whether we have internet connectivity
#

UPLINK_NAME="MY_GW"
UPLINK_STATE_FILE=/usr/local/var/run/uplink-status

LINK_UP=2
LINK_DOWN=1

# Cleanup
quit() {
    EC=$1

    exit $EC
}

checkrc() {
    PRC=$1; if [ $PRC -ne 0 ]; then
       echo "$0: Exiting due to error ($PRC)" 1>&2
       exit 0
    fi
}

GATEWAY_STATUS_JSON=`pluginctl -r return_gateways_status`
UPLINK_PRESENT=`echo $GATEWAY_STATUS_JSON | grep -c $UPLINK_NAME`

if [ "$UPLINK_PRESENT" -eq 0 ]; then
    CUR_STATUS="down"

else

    # Check dpinger status of the uplink
    CUR_STATUS=`echo $GATEWAY_STATUS_JSON | python3 -c "import sys, json; print(json.load(sys.stdin)['dpinger']['${UPLINK_NAME}']['status'])"`
    checkrc $?
fi

if [ ! -f "$UPLINK_STATE_FILE" ]; then
   echo "up" > $UPLINK_STATE_FILE
   checkrc $?
fi

LAST_STATUS=`cat $UPLINK_STATE_FILE`
checkrc $?

if [ "$LAST_STATUS" = "up" ] && [ "$CUR_STATUS" = "down" ]; then
    echo "down" > $UPLINK_STATE_FILE
    echo "down"
    quit $LINK_DOWN
elif [ "$LAST_STATUS" = "down" ] && [ "$CUR_STATUS" = "none" ]; then
    echo "up" > $UPLINK_STATE_FILE
    echo "up"
    quit $LINK_UP
else
    if [ "$CUR_STATUS" = "none" ]; then CUR_STATUS="up"; fi
    echo "no change ($CUR_STATUS)"
    quit 0
fi


When monit receives a non-zero return code from the above script, it then calls "set-carp-demotion" which will decrease the interface's carp priority if the link has gone down, and increase it if it has come back up.

Code Select Expand


#!/usr/local/bin/bash
#
# set-carp-demotion - demote/promote a link dependent on interface status
# Called by monit on link status change
#

DEMOTION_VALUE=100
LINK_STATUS=${MONIT_PROGRAM_STATUS}

LINK_UP=2
LINK_DOWN=1

# Cleanup
quit() {
    EC=$1

    if [ $EC -ne 0 ]; then echo "$0: Exiting due to error ($EC)"; fi
    exit $EC
}

checkrc() {
   PRC=$1; if [ $PRC -ne 0 ]; then quit $PRC; fi
}

CUR_DEMOTION=`sysctl -n net.inet.carp.demotion`
checkrc $?

if [ "$LINK_STATUS" -eq "$LINK_UP" ]; then
    if [[ "$CUR_DEMOTION" -gt "0" ]]; then
        sysctl -q net.inet.carp.demotion=-$DEMOTION_VALUE
        checkrc $?
    fi

elif [ "$LINK_STATUS" = "$LINK_DOWN" ]; then
    if [[ "$CUR_DEMOTION" -lt "$DEMOTION_VALUE" ]]; then
        sysctl -q net.inet.carp.demotion=$DEMOTION_VALUE
        checkrc $?
    fi

else
    echo "Invalid link status '$LINK_STATUS'."
    quit 1
fi

quit 0


While this solution is quite specific to my own situation, I hope that by posting it here, it may help someone else who is trying to have finer grained control over their CARP interface.

That seems like a shame.  I'm currently considering firing up two OpnSense instances (in VMs) to act as pure WAN interfaces (WAN on one side, unfirewalled VPN on the other) so that I can configure my redundant firewall routers (also VMs) to talk to both interfaces via the unfirewalled VPN, but this seems like a very complex way to accomplish something that I think ought to be a lot simpler than this.

Hello, I have two routers connected to different uplinks in a redundant CARP setup, one configured as primary CARP, the other as backup.  I would like the primary machine to demote itself when when its uplink fails and to re-establish itself as primary when the link comes back up.  Is there a way to do this with OpnSense? Thanks.