Messages

@Gareth

> ROUTING: refusing to set inet6 gateway on addressless wan
> ROUTING: refusing to set inet6 gateway on addressless wan

It looks like the newly added log message didn't trigger on the reload here, not sure why or otherwise it would have said "wan(xxx)" to point to the device it's been using.

Looking further and hearing about PPPoE I think you don't have "Use IPv4 connectivity" set for your WAN IPv6 configuration? That's why it wants to use igb2 instead of pppoeX and here we can se igb2 does not have an address indeed.

If, however, the option is set then at least we know where to look further.


Cheers,
Franco

PS: Which WAN IPv6 mode are you using?

Hi Franco,

I ran the procedure a couple of times to make sure it wasn't me, but output was the same on both occasions, so just posted.

In terms of "Use IPv4 connectivity" it is set and my IPv6 mode is set to static because my ISP does not support PPPoEv6 or DHCPv6, few pictures attached showing further info.

I've kept the static IPv6 for the WAN interface and the gateways in the same /64 range so it's basically:

WAN Interface: face:face:face:face::254/64
WAN Gateway: face:face:face:face::50/64

As I recall, when I configured this a pretty long time back now, this was the only way that worked for me and I can see both before the upgrade to 23.1.7_3 and after that the face:face:face:face::254 gateway is being automatically selected, it's just that this gateway refuses to come up after the upgrade, but strangely dpinger shows it as green even though it's offline, I have attached a picture of this also.

The internal clients are given IPv6 addresses using a manually configured radvd since there is no interface to track here.

Thanks for the help

Gareth

There are two patches to help with diagnose..

https://github.com/opnsense/core/commit/8beb293c5
https://github.com/opnsense/core/commit/48855143b

This is on a clean 23.1.7, opnsense-revert used to make sure:

# opnsense-revert opnsense && opnsense-patch 48855143b 8beb293c5
# /usr/local/etc/rc.routing_configure
# opnsense-log | grep refusing

In the last log line there is a hint of the interface and device being used, e.g.:

> ROUTING: refusing to set inet gateway on addressless wan(igb1)

For the device is parenthesis run:

# pluginctl -D igb1

Depending on this output the log line is generated and the route refused. If data is there we might be looking at a timing issue, if not then it's something more fundamental.


Cheers,
Franco

Hi Franco,

Ok, next set of tests as requested above (slightly edited for privacy):

root@OPNSense:~ # opnsense-revert opnsense && opnsense-patch 48855143b 8beb29                                                                                                        3c5
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following packages will be fetched:

New packages to be FETCHED:
        opnsense: 23.1.7_3 (4 MiB: 100.00% of the 4 MiB to download)

Number of packages to be fetched: 1

The process will require 4 MiB more space.
4 MiB to be downloaded.
Fetching opnsense-23.1.7_3.pkg: 100%    4 MiB   4.4MB/s    00:01
opnsense-23.1.7_3: already unlocked
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
        opnsense-23.1.7_3

Number of packages to be reinstalled: 1
[1/1] Reinstalling opnsense-23.1.7_3...
[1/1] Extracting opnsense-23.1.7_3: 100%
Stopping configd...done
Resetting root shell
Updating /etc/shells
Unhooking from /etc/rc
Unhooking from /etc/rc.shutdown
Updating /etc/shells
Registering root shell
Hooking into /etc/rc
Hooking into /etc/rc.shutdown
Starting configd.
>>> Invoking update script 'refresh'
Writing firmware setting...done.
Writing trust files...done.
Configuring login behaviour...done.
Configuring system logging...done.
=====
Message from opnsense-23.1.7_3:

--
I'm no chicken
Fetched 48855143b via https://github.com/opnsense/core
Fetched 8beb293c5 via https://github.com/opnsense/core
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 48855143b0c5e2d3f70a29a841e80a45210d74e2 Mon Sep 17 00:00:00 2001
|From: Franco Fichtner <franco@opnsense.org>
|Date: Wed, 10 May 2023 14:37:38 +0200
|Subject: [PATCH] system: add 'if' to message in case of mismatch
|
|PR: https://forum.opnsense.org/index.php?topic=33864.0
|---
| src/etc/inc/system.inc | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/src/etc/inc/system.inc b/src/etc/inc/system.inc
|index 722900df88..7666a0e740 100644
|--- a/src/etc/inc/system.inc
|+++ b/src/etc/inc/system.inc
--------------------------
Patching file etc/inc/system.inc using Plan A...
Hunk #1 succeeded at 619 (offset -2 lines).
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 8beb293c53e3d14c5256cd648b3a834667595c2d Mon Sep 17 00:00:00 2001
|From: Franco Fichtner <franco@opnsense.org>
|Date: Mon, 15 May 2023 10:11:38 +0200
|Subject: [PATCH] pluginctl: add an ifconfig mode for easier debugging and
| later use
|
|PR: https://forum.opnsense.org/index.php?topic=33864.0
|---
| src/sbin/pluginctl | 7 +++++--
| 1 file changed, 5 insertions(+), 2 deletions(-)
|
|diff --git a/src/sbin/pluginctl b/src/sbin/pluginctl
|index afa7e674ce..eb531b8e97 100755
|--- a/src/sbin/pluginctl
|+++ b/src/sbin/pluginctl
--------------------------
Patching file sbin/pluginctl using Plan A...
Hunk #1 succeeded at 63.
Hunk #2 succeeded at 78.
done
All patches have been applied successfully.  Have a nice day.
root@OPNSense:~ # /usr/local/etc/rc.routing_configure
Setting up routes...done.
Setting up gateway monitors...done.
Configuring firewall.......done.
root@OPNSense:~ # opnsense-log | grep refusing
<11>1 2023-05-15T18:36:30+01:00 OPNSense.domain.local opnsense 301 - [meta sequenceId="12"] /usr/local/etc/rc.bootup: ROUTING: refusing to set inet6 gateway on addressless wan
<11>1 2023-05-15T18:36:44+01:00 OPNSense.domain.local opnsense 4898 - [meta sequenceId="43"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet6 gateway on addressless wan
root@OPNSense:~ # pluginctl -D igb2
{
    "igb2": {
        "flags": [
            "up",
            "broadcast",
            "running",
            "simplex",
            "multicast"
        ],
        "capabilities": [
            "rxcsum",
            "txcsum",
            "vlan_mtu",
            "vlan_hwtagging",
            "jumbo_mtu",
            "vlan_hwcsum",
            "tso4",
            "tso6",
            "lro",
            "wol_ucast",
            "wol_mcast",
            "wol_magic",
            "vlan_hwfilter",
            "vlan_hwtso",
            "netmap",
            "rxcsum_ipv6",
            "txcsum_ipv6",
            "nomap"
        ],
        "options": [
            "vlan_mtu",
            "jumbo_mtu",
            "nomap"
        ],
        "macaddr": "a0:36:9f:7d:55:7b",
        "ipv4": [],
        "ipv6": [],
        "supported_media": [
            "autoselect",
            "1000baseT",
            "1000baseT full-duplex",
            "100baseTX full-duplex",
            "100baseTX",
            "10baseT/UTP full-duplex",
            "10baseT/UTP"
        ],
        "mtu": "1500",
        "media": "1000baseT <full-duplex>",
        "media_raw": "Ethernet autoselect (1000baseT <full-duplex>)",
        "status": "active"
    }
}


Hopefully offers you something useful, apologies on the earlier mistake, which I've now corrected in this post. There doesn't seem to be any info attached to the igb2 interface above.

It did lead me to consider one thing which I thought I'd mention, in the grep for refusing, it shows the interface as wan but no hint. I know the wan interface is tied to igb2 on my virtual firewall (VMware Passthrough, Intel Card) as it is on my physical one.

But ultimately it's a PPPoE interface which is then linked to igb2, so wonder if it's not picking it up because of this?

In any case, hope the output is of some use, going to rollback again for now, let me know if there is anything further you would like me to do.

Thanks

Gareth

Let's look at this differently... I'm assuming the bad change is https://github.com/opnsense/core/commit/a8e9862b410073 and it may work again if it's reverted?

# opsense-patch a8e9862b410073

The commit does two things: it adds IP address family specific reload functionality, but it should not matter for when e.g. rc.configure_routing is called which is what WireGuard is doing.

The other thing is it tries to verify that the gateway selected for default gateway use does have a matching interface with at least one address in it (the equivalent of calling ifconfig to see if that has an address). The latter one is easy to try... I do think that at least one address must be present anyway, but perhaps if it's a tunnel device the address might not show up correctly?

Looking forward to verification that the patch is the issue...


Cheers,
Franco

Hi Franco,

So first things first, I've spent a few minutes running tests again and it does in fact look as if the wireguard vpn is coming up after I upgrade to 23.1.7_3, I have double checked and it does appear to be working as expected.

The failure is on the IPv6 gateway which refuses to come online and therefore stops all IPv6 traffic. I guess this would be consistent with the error in the log around refusing to apply an inet6 GW.

I tried to apply the patch you mentioned in the above post and got the below error output:

root@OPNSense:~ # opnsense-patch a8e9862b410073
Fetched a8e9862b410073 via https://github.com/opnsense/core
1 out of 2 hunks failed while patching etc/rc.syshook.d/monitor/10-dpinger
root@OPNSense:~ # opnsense-patch a8e9862b410073
Found local copy of a8e9862b410073, skipping fetch.
1 out of 2 hunks failed while patching etc/rc.syshook.d/monitor/10-dpinger

I tried it a couple of times just to be sure, but the half a patch doesn't resolve or make any impacts.

I'll move on to the next set of tests to try and feedback further

Thanks

Gareth

Hi Nic,

Just to make matters more confusing, i'm afraid in my case I'm not using any static routes no.

The far gateway in use in my case is dynamically generated by the fingerlessgloves script to connect to my PIA VPN, found here:

https://github.com/FingerlessGlov3s/OPNsensePIAWireguard

This script does however pull routes which I can see in the routing table, it doesn't do this immediately at boot though. instead using a Cron job to add them or refresh every 5 minutes, so you will often see the dpinger as red for a couple of minutes right after booting while it waits for the script to kick in. This I believe to be normal behaviour when using this script.

It does use/enable a dynamic far gateway though because this is what's needed to route the traffic to PIA over wireguard after established, so it's likely somewhat similar.

It's interesting to note though that after booting and before the script runs, its already knocked out the IPv6 routing on my main internet connection and is also showing the error around refusing to add a default gateway which was in Shuppie's example.

That does give me something to compare later to see if it also fails to add the normal route after upgrading to 23.1.7_3, but off to work now so will have to figure that out later.

Thanks

Gareth

Hi Franco,

Thanks for coming back on this.

Just to be double sure I wasn't going crazy, I've looked at my config.xml again and double checked. There are no duplicate gateways/names on different interfaces and the number and spec on the gateways matches exactly, unlike the example showed by Struppie where there were clearly 2x entries in the gateways section of the config file and only a single gateway visible in the GUI.

I could only find one section in the config.xml that contained gateway info, looking as below(Some IP's altered for privacy):

<gateways>
    <gateway_item>
      <interface>wan</interface>
      <gateway>dynamic</gateway>
      <name>WAN_PPPOE</name>
      <priority>254</priority>
      <weight>1</weight>
      <ipprotocol>inet</ipprotocol>
      <interval/>
      <descr>Interface WAN_PPPOE Gateway</descr>
      <monitor>8.8.8.8</monitor>
      <defaultgw>1</defaultgw>
    </gateway_item>
    <gateway_item>
      <interface>wan</interface>
      <gateway>face:face:face:face::254</gateway>
      <name>WAN_GW</name>
      <priority>254</priority>
      <weight>1</weight>
      <ipprotocol>inet6</ipprotocol>
      <interval/>
      <descr>Interface WAN_GW Gateway</descr>
      <monitor>2001:4860:4860::8844</monitor>
    </gateway_item>
    <gateway_item>
      <interface>opt8</interface>
      <gateway>dynamic</gateway>
      <name>WAN_PIAGW_IPv4</name>
      <priority>255</priority>
      <weight>1</weight>
      <ipprotocol>inet</ipprotocol>
      <interval/>
      <descr/>
      <fargw>1</fargw>
    </gateway_item>
  </gateways>

The above section also matches exactly the gateways that appear in the GUI.

It's entirely possible I'm missing something obvious, but could only find one section that contained the above info.

Thanks for your help.

Gareth

Hi Guys,

I'll weigh in here, since I seem to be having a similar problem as Nic with the exception that my IPv4 outbound continues to work but it affects my outbound VPN and IPv6 outbound connectivity.

I've been ripping my hair out a bit with this issue and in the end have set myself up with a virtual version of my usually physical firewall so I could perform quick changes and roll back without impacting my main prod too much. It definitely appears to be related to far gateways that dont have physical interfaces since after upgrading thats exactly what I experience.

The specific GW causing the trouble seems to be my WireGuard VPN which uses a far GW and doesn't have it's own physical interface, and as a side impact also seems to impact my IPv6 GW which is created by my PPPoE connection over IPv4, although this is not a far GW.

Problem first appeared for me after upgrading from 23.1 to 23.1.7_3 from which I had to rollback to get things working again properly.

When I run: opnsense-log | grep refusing

I see in the log last entry is:
2023-05-14T16:50:30+01:00 myopnsensename.local opnsense 47013 - [meta sequenceId="4"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet6 gateway on addressless wan

Might be worth knowing, my ISP doesn't offer DHCPv6 or PPPoEv6, so I've always had it set to a static IPv6 within my allocated /64 which has always worked correctly before and still does when I roll back.

I have checked my config.xml and as far as I can tell have the exact same number of GW (3) as are supposed to be starting at boot.

Now I'm running virtual, testing should be minimally impactfull and my IPv4 outbound continues to work, I just lose my IPv6 and VPN outbound connectivity, so happy to try and perform testing to assist if needed.

Thanks so much to everybody for all the hard work on OPNsense, it's really a great piece of work.

Gareth

Thank you Franco, you are a superstar, I'll get this patch applied  ;D

Hi All,

I've just updated my OPNSense from 22.7.5>6 this morning and I'm now seeing an error around HAProxy being unable to start due to a CRL problem because I use client certificate authentication.

All certs are being issued by a local CA on the OPNSense firewall

I've already tried removing and recreating the CRL then re-adding to the HAProxy frontend, none of which has made any difference. for now to get HAProxy to start correctly I've had to remove the CRL from the public facing frontend, but this is less than ideal.

If I try and re-add it I see the pictured error when doing a test syntax from the HAProxy GUI, so it's definitely related to the CRL somehow, but I cant figure out whats actually wrong.

Coupled to this, the crash reporter is now also regularly reporting the below error, even though HAProxy is functional albeit with no CRL for any revoked certs:

[14-Oct-2022 08:29:49 Europe/London] PHP Fatal error:  Uncaught Error: Call to undefined function crl_update() in /usr/local/opnsense/scripts/OPNsense/HAProxy/exportCerts.php:74
Stack trace:
#0 {main}
  thrown in /usr/local/opnsense/scripts/OPNsense/HAProxy/exportCerts.php on line 74
[14-Oct-2022 08:30:30 Europe/London] PHP Fatal error:  Uncaught Error: Call to undefined function crl_update() in /usr/local/opnsense/scripts/OPNsense/HAProxy/exportCerts.php:74
Stack trace:
#0 {main}
  thrown in /usr/local/opnsense/scripts/OPNsense/HAProxy/exportCerts.php on line 74
[14-Oct-2022 08:30:58 Europe/London] PHP Fatal error:  Uncaught Error: Call to undefined function crl_update() in /usr/local/opnsense/scripts/OPNsense/HAProxy/exportCerts.php:74
Stack trace:
#0 {main}
  thrown in /usr/local/opnsense/scripts/OPNsense/HAProxy/exportCerts.php on line 74

I know there was some changes to CRL handling, but I thought this was only supposed to impact OpenVPN which seems to be working fine.

Any help in resolving would be very much appreciated guys.

Many thanks

Gareth

Hi Franco,

No worries on that, just wanted to check if there was anything else I needed to do.

Thanks very much for the help guys :)

Gareth

Hi,

Is there a guide somewhere on how to apply the workaround, or has this already been resolved in the  21.7.3_3 update which I'm already running?

Thanks

Gareth

[Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 7]

Hi Guys,

Not sure if I'm an isolated case, but since updating to 20.7.6 I seem to be getting the same error every time when trying to forcefully renew the certificate chain as below:

Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 7

When looking at the URL it lists error 7 as:

CURLE_COULDNT_CONNECT (7)

Failed to connect() to host or proxy.

Turning up the logging level shows the below errors in the logs:

[Thu Dec 17 11:25:45 GMT 2020]    Giving up sending to CA server after 20 retries.
[Thu Dec 17 11:25:43 GMT 2020]    Could not get nonce, let's try again.
[Thu Dec 17 11:25:43 GMT 2020]    ret='7'
== Info: Immediate connect fail for    2606:4700:60:0:f53d:5624:85c7:3a2c: No route to host
== Info: Immediate connect fail for    172.65.32.248: Host is down
[Thu Dec 17 11:25:43 GMT 2020]    == Info: Trying 172.65.32.248:443...

So it looks as if the acme client is trying to connect to something thats offline or the url is incorrectly configured in the request, however when I manually ping the address: 172.65.32.248 from my private network behind the firewall it answers and if I access it with HTTPS it lets me know its a letsencrypt server running boulder.

I run two OPNsense firewalls, both set up similarly and using Hurricane electric DNS and the DNS01 challenge for renewal but for different domains and in different physical locations.

The other one I haven't updated from 20.7.5 yet and that still seems to be working without issues.

I dont use the HAproxy integration and just select the named cert in the haproxy frontend.

I have tried stripping all LetsEncrypt configuration, resetting the configuration uninstalling and reinstalling the LetsEncrypt plugin and reconfiguring from scratch and still receive the same error as above, changing the password for Hurricane Electric in case there is some new limitation introduced, but none of this seems to resolve at all.

My original cert from previous versions is still in place in the trust store and valid until the 21st Jan so I have a little time to resolve, but I'm really running out of ideas, so asking for help if you can provide any?

Thanks

Gareth

Didn't get much chance to do any testing, but what I can say for sure was that the firewall gateway monitoring could no longer reach/ping the PIA DNS servers at 209.222.18.218 after the upgrade was complete, but when running 19.1.10 this works, just double checked it after rolling back.

Hi Guys,

Just tried again by running the update to 19.7.3 and I'm afraid I'm still experiencing the exact same issue as before, rolled back for now again.

Thats great news,

Thanks for the response mimugmail  :D

I noticed this same issue occurring on my OPNsense instance after upgrading from 19.1.10 to 19.7.*

I have cloned the 19.1.10 version and spent significant time playing around and trying to work out what's going on, but nothing obvious is presenting itself as the cause. I basically rolled back for now, but did anybody figure out what is causing the issue?

Essentially traffic just stops flowing to the internet out of the PIA gateway altogether after the upgrade. rolling back to 19.1.10 solves the issue again for me.

My setup is pretty much identical to the one described by DanMC85.