OPNsense Forum
Archive => 23.1 Legacy Series => Topic started by: My_Network on May 05, 2023, 02:49:34 pm
-
Hi every one!,
I upgraded to the latest version on Opnsense *23.1.7_1* and iv’e not been able to come back online since. My firewall is between my modem and a Cisco router. It was working fine on 23.1.6 in conjunction with Zenarmor. But now the pingeer that tests the reachability of my « far gateway’s » (10.10.0.0/24 | 10.10.10.0/24 | 10.10.20.0/24 | 192.168.1.0/24 VIA static routes that point to 192.168.12.0 /24 via 192.168.12.1 (Cisco WAN interface), keeps bouncing around erratically from an online status to no ping at all but with the green online status. And they refuse to actualy reach the router like it was doing before. Has anybody experience this? I’m totaly in the dark here. Tried reinstalling from scrach but it does noting to help. It was all working fine with 21.1.6. In then mean time i reactivated my "Residential Gateway" and IP NAT OUSIDE on my Cisco router to bypass my Firewall temporarely... Im at a loss here.
Hoping for some Help :)
Thank you,
Nic
-
Hi,
Can you make sure to update to 23.1.7_2 and then apply the following on top?
https://github.com/opnsense/core/issues/6544#issuecomment-1535790249
Cheers,
Franco
-
Hi Franco,
Just did the upgrade to 23.1.7_2 with the proposed patch on github, and at first glaced it fixed the eractic pinger behaviour. I will get back to you later when i move back my network behing my firewall.
Thank you,
Nic
-
Thanks, in that case I'll weave that into the hotfix.
Cheers,
Franco
-
Hi Franco,
So as promised here were I stand with version 23.1.7_3. Attached you will find a screen capture of "single gateways" currently running with all updated packages but with a #opnsense-revert -r 23.1.6 in shell to force the firewall to use 23.1.6 instead of 23.1.7_3. VLAN'S 10-20-30-40 and CISCO_WAN_INT are all "FAR GATEWAYS" configured with a routes as shown in screen capture called "Routes". These networks all opperates on my cisco 891f running downstream via 192.168.12.0/24 (Router Wan interface). Has soon as I update to 23.1.7_1-2-3 and try to add a gateway or reboot the firewall it renderes my network to the ground.
Thank you,
Nic :o
-
Similar problems here. Did the update from 23.1.6 to 23.1.7_3 with no configuration change (details in attached screenshot). Before the update everything was working fine.
Scenario:
(Internet) ---- (public IP - Router - private IP Network A) ----- (private IP Network A - OPNSense - private IP Network B) ----- (private Network)
After the update I couldn't access from my private Network any public machine of the Internet.
No ping, no http etc.
ping to private IP Network A of Router worked, but ping to public IPs did not work.
Restored backup => Everything fine again.
Seems something got broken with the update...
-
Same here, since 23.1.7_x suddenly starting to have weird behaviors while nothing is changed on config.
With updates to 23.1.7_2 and 23.1.7_3 I hoped this was fixed but unfortunately not yet.
From my OpenVPN tunnels is one tunnel not responding (ping) on tunnel-subnet anymore (from both sides) but local LAN and remote LAN are still working.
Other tunnel is not responding on tunnel subnet and remote subnet.
Rebooted both ends but no luck.
Restored back to 23.1.6 all working fine again. :)
-
Ok, so to bring a little structure into this:
1. Are you using default gateway switching? If yes does it sort of work if you disable default gateway switching?
Because that was the only thing being switched over...
2. How do your routing tables look if it works on 23.1.6 vs. 23.1.7(_3) where it doesn't work. Usually "nothing works" is easy to spot in terms of entries in the routing table... I sounds a bit like hoping it just works vs. explicit configuration that is going on here.
Cheers,
Franco
-
Hi Franco,
No, default gateway switching is not enable since im not using the gateway for multiwan purpuse but for Far gateways. I will try to send you the info running on 23.1.7_3 without the usualy "Someone broke the internet again"..... situation.
Thank you,
Nicolas
-
Hi Franco,
1: No default gateway switching in use
2: I will try to do another update attempt and report on this
Thanks for taking care!
-
Hi Franco,
I did the update from 23.1.6 to 23.1.7_3 again.
Immediately after the update everything still runs fine.
Then I did a reboot and the issue was back, that no traffic to the public internet was working.
Your hint to look at the routing table was good.
The comparison of before and after the update+reboot shows, that the first entry of the routing table was missing!
The one with "destination" default etc - refer to attached screenshot (red rectangle).
Interestingly the status of the gateway was marked as "online".
After adding the default route as a static route as interims solution, everything is working fine again.
But I think the default route should be created automatically as in former times?
-
It's rather odd that default routes are missing and default gateway switching is not in use.
What is your WAN setup? IPv4 DHCP?
Cheers,
Franco
-
I configured a WAN interface (using vlan) with static IPv4 (192.168.30.254).
Additionally I configured the IPv4 upstream Gateway on this interface, which is the single Gateway I have.
(inner leg of a second router to the public internet)
Looks like this:
Scenario:
(Internet) ---- (public IP - Router - private IP Network A) ----- (private IP Network A - OPNSense - private IP Network B) ----- (private Network)
OPNSense: (private IP Network A) is the 192.168.30.254
Router: (private IP Network A) is the gateway IP 192.168.30.1
-
I have an idea... Can you grep for this:
# opnsense-log | grep refusing
Cheers,
Franco
-
seems to be a good idea ;)
Here's the output:
root@OPNsense:~ # opnsense-log | grep refusing
<11>1 2023-05-10T11:43:30+02:00 OPNsense.dimo.nil opnsense 285 - [meta sequenceId="8"] /usr/local/etc/rc.bootup: ROUTING: refusing to set inet gateway on addressless wan
<11>1 2023-05-10T11:43:35+02:00 OPNsense.dimo.nil opnsense 17719 - [meta sequenceId="32"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet gateway on addressless wan
-
The full log around the log message would be helpful. A static addressing with a WAN and no address seems a bit strange still but I'm sure we can find the loophole now.
Cheers,
Franco
-
many thanks for looking into this issue - much appreciated! :)
I attached 2 logs:
- opnsense-badcase.log: this is the full log after the update and reboot (23.1.7_3)
for comparison reasons (there you can see, that setting the default route is successful):
- opnsense-goodcase.log: this is the full log with the former version of OPNsense (23.1.6)
-
I've tried this twice and couldn't reproduce. There seems to be a more obvious issue WRT parsing ifconfig or possibly mismatching the interface name somewhere. I'm leaning towards the latter, perhaps a duplicated gateway entry issue we have had previously?
# grep -nr \<gateway_item /conf/config.xml
There should be as many entries as you can see from the GUI (which can be deleted). If there is one more try to look if there is an overlap...
Cheers,
Franco
-
hmm, the relevant part of the conf.xml looks like this:
<gateways>
<gateway_item>
<interface>opt8</interface>
<gateway>192.168.30.1</gateway>
<name>WAN_GW</name>
<priority>255</priority>
<weight>1</weight>
<ipprotocol>inet</ipprotocol>
<interval>1</interval>
<descr>Interface WAN Gateway</descr>
<monitor>8.8.8.8</monitor>
<defaultgw>1</defaultgw>
</gateway_item>
<gateway_item>
<descr>Interface WAN Gateway</descr>
<defaultgw>1</defaultgw>
<ipprotocol>inet</ipprotocol>
<interface>wan</interface>
<gateway>192.168.30.1</gateway>
<monitor_disable>1</monitor_disable>
<name>WAN_GW</name>
<interval>1</interval>
<weight>1</weight>
</gateway_item>
</gateways>
But the GUI shows only one, attachment OPNSence-system-gateways.png
Additionally I attached the interface configuration itself, where I configured the IPv4 Upstream Gateway, attachment OPNSence-interface-wan30.png
All this looked the same before the update and worked fine.
Any hint what to change regarding the interface/gateway configuration is much appreciated, if this is the reason why it doesn't work anymore or is somehow wrong....
-
exceeding max size of attachment, therefore attached with 2nd post
-
Ok the <name/> overlaps here which is the issue since <interface/> differs and that probably causes an empty lookup. Can you check the assignments page for internal identifier of your wan (either 'wan' or 'opt8')? And then just drop the wrong <gateway_item/> from config file.
Cheers,
Franco
-
I think it's now up to me to spend you a beer, or two ;)
After removing the second <gateway_item/> part and reboot OPNsense does now set the default route again etc.
Seems to run fine now!
I will monitor it a bit, but seems you find the root cause!
I think I wouldn't have found this without your help! Many thanks!
-
Okay so far so good. This was added to some other 23.1.x prior which tries to prevent gateway duplication:
https://github.com/opnsense/core/commit/4b03f1c88d
The code to prevent overlap is still not perfect. Let me try to find more loopholes. :)
Cheers,
Franco
-
I think the wizard did some silliness here... https://github.com/opnsense/core/commit/db69027dda5
Hard to reach this state but not impossible. Thanks for the report!
Cheers,
Franco
-
good findings and perfect support! Many thanks!
-
Hi Struppie,
Could you please add a before and after of your conf.xml for reference? Im not sure I completely follow the changes you had to make to get it working.
Thank you ;D
Nic
-
Hi Nic,
Let's first see if you have a similar issue:
# opnsense-log | grep refusing
Does this bring up something?
Cheers,
Franco
-
Could you please add a before and after of your conf.xml for reference? Im not sure I completely follow the changes you had to make to get it working.
I'll be happy to assist, but let's first check - as suggested by franco - if you are facing the same problem. Otherwise we may make it worse than better ;)
What does the following command spit out on your side?
# opnsense-log | grep refusing
-
Hi Guys,
So I finally had the chance to get a little downtime...23. 1.7_3 still breaks my Firewall. Unfortunately, the proposed command did not spit out a result. But I stood upon something wierd in my log that was not there before the upgrade to 23.1.7_3. Dowgraded to 23.1.6 since to get back up and running using opnsense-revert -r 23.1.6. Please see screen capture. You will see an error about the wrong gateway beeing chosen or something like that. The gateway for these vlan should be 192.168.12.1 and not 192.168.15.1. Why is it choosing the wrong one?
Thank you :o
Nic
-
Hi Nic,
Part of this looks odd. Let's assume it's also a side effect from a duplicated gateway entry. I wrote a patch to quickly diagnose this via https://github.com/opnsense/core/commit/c1784ad1a
# opnsense-patch c1784ad1a
Warnings should appear in the general log in that case.
I'd still suspect the configuration side has an issue on your end not directly related to 23.1.7.
Cheers,
Franco
-
Hi Franco,
Finally was abe to get something out of opnsense-log | grep refusing. Dont know if this give you any clues on what could me my issue? Oh and intresting fact, just after the upgrade to 23.1.7_3, the issue arise only if I reload the "routing" service out of any other services!
<11>1 2023-05-12T16:43:28-04:00 OPNsense.localdomain opnsense 98415 - [meta sequenceId="92"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet gateway on addressless lan
Other intresting general log entry:
<13>1 2023-05-12T16:31:59-04:00 OPNsense.localdomain opnsense 74858 - [meta sequenceId="124"] /usr/local/etc/rc.linkup: Chose to bind CISCO_WAN_INT on 192.168.15.1 since we could not find a proper match.
Look like 192.168.12.0/24 is not being consired at all!
+
If we look on the log of onw of Struppi awnsers! We see the same error but with the WAN interface:
- opnsense-badcase.log: this is the full log after the update and reboot (23.1.7_3)
<11>1 2023-05-10T12:33:04+02:00 OPNsense.dimo.nil opnsense 8359 - [meta sequenceId="32"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet gateway on addressless wan
Thank you,
Nic
-
Look in /conf/config.xml for <gateway_item/> with the same name... I think you have more than one which causes this issue. It looks to be the same as struppie's issue now.
Cheers,
Franco
-
Hi Franco,
So after further investigation, it seem's that i dont have the same issue. There is no duplicate gateways. But the issue seems to be with: /usr/local/etc # cat rc.routing_configure. In 23.1.7_3 this script changes and seem to interfere somehow with ''FAR GATEWAYS'' that dont have physical interface.
Im out of ideas here.
Nic
-
Hi Guys,
I'll weigh in here, since I seem to be having a similar problem as Nic with the exception that my IPv4 outbound continues to work but it affects my outbound VPN and IPv6 outbound connectivity.
I've been ripping my hair out a bit with this issue and in the end have set myself up with a virtual version of my usually physical firewall so I could perform quick changes and roll back without impacting my main prod too much. It definitely appears to be related to far gateways that dont have physical interfaces since after upgrading thats exactly what I experience.
The specific GW causing the trouble seems to be my WireGuard VPN which uses a far GW and doesn't have it's own physical interface, and as a side impact also seems to impact my IPv6 GW which is created by my PPPoE connection over IPv4, although this is not a far GW.
Problem first appeared for me after upgrading from 23.1 to 23.1.7_3 from which I had to rollback to get things working again properly.
When I run: opnsense-log | grep refusing
I see in the log last entry is:
2023-05-14T16:50:30+01:00 myopnsensename.local opnsense 47013 - [meta sequenceId="4"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet6 gateway on addressless wan
Might be worth knowing, my ISP doesn't offer DHCPv6 or PPPoEv6, so I've always had it set to a static IPv6 within my allocated /64 which has always worked correctly before and still does when I roll back.
I have checked my config.xml and as far as I can tell have the exact same number of GW (3) as are supposed to be starting at boot.
Now I'm running virtual, testing should be minimally impactfull and my IPv4 outbound continues to work, I just lose my IPv6 and VPN outbound connectivity, so happy to try and perform testing to assist if needed.
Thanks so much to everybody for all the hard work on OPNsense, it's really a great piece of work.
Gareth
-
I see in the log last entry is:
2023-05-14T16:50:30+01:00 myopnsensename.local opnsense 47013 - [meta sequenceId="4"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet6 gateway on addressless wan
Might be worth knowing, my ISP doesn't offer DHCPv6 or PPPoEv6, so I've always had it set to a static IPv6 within my allocated /64 which has always worked correctly before and still does when I roll back.
The code itself looks at ifconfig if there is *any* address set on the device, which should already be true for link-local alone. But again here seems to be the problem that the gateway section in the config.xml might have a duplicated gateway on different interfaces which breaks the setup now when before it didn't.
Cheers,
Franco
-
Hi Franco,
Thanks for coming back on this.
Just to be double sure I wasn't going crazy, I've looked at my config.xml again and double checked. There are no duplicate gateways/names on different interfaces and the number and spec on the gateways matches exactly, unlike the example showed by Struppie where there were clearly 2x entries in the gateways section of the config file and only a single gateway visible in the GUI.
I could only find one section in the config.xml that contained gateway info, looking as below(Some IP's altered for privacy):
<gateways>
<gateway_item>
<interface>wan</interface>
<gateway>dynamic</gateway>
<name>WAN_PPPOE</name>
<priority>254</priority>
<weight>1</weight>
<ipprotocol>inet</ipprotocol>
<interval/>
<descr>Interface WAN_PPPOE Gateway</descr>
<monitor>8.8.8.8</monitor>
<defaultgw>1</defaultgw>
</gateway_item>
<gateway_item>
<interface>wan</interface>
<gateway>face:face:face:face::254</gateway>
<name>WAN_GW</name>
<priority>254</priority>
<weight>1</weight>
<ipprotocol>inet6</ipprotocol>
<interval/>
<descr>Interface WAN_GW Gateway</descr>
<monitor>2001:4860:4860::8844</monitor>
</gateway_item>
<gateway_item>
<interface>opt8</interface>
<gateway>dynamic</gateway>
<name>WAN_PIAGW_IPv4</name>
<priority>255</priority>
<weight>1</weight>
<ipprotocol>inet</ipprotocol>
<interval/>
<descr/>
<fargw>1</fargw>
</gateway_item>
</gateways>
The above section also matches exactly the gateways that appear in the GUI.
It's entirely possible I'm missing something obvious, but could only find one section that contained the above info.
Thanks for your help.
Gareth
-
Good evening Gazd25,
Look like we have the exact same issue! In my config file there is no duplicate gateway's, iv'e looked and looked again. The issue is really regarding routing to ''far gateway's'' that's not working anymore in 23.1.7_3, but in 23.1.6 it works fine. Do you have static routes that points to that specific gateway? In my environnement I have 5 static routes that point to vlan's on my cisco router downstream with it's WAN interface. Hoping to get this sorted out. Would be more that happy to share more details if needed.
Regard :o
Nic
-
Hi Nic,
Just to make matters more confusing, i'm afraid in my case I'm not using any static routes no.
The far gateway in use in my case is dynamically generated by the fingerlessgloves script to connect to my PIA VPN, found here:
https://github.com/FingerlessGlov3s/OPNsensePIAWireguard
This script does however pull routes which I can see in the routing table, it doesn't do this immediately at boot though. instead using a Cron job to add them or refresh every 5 minutes, so you will often see the dpinger as red for a couple of minutes right after booting while it waits for the script to kick in. This I believe to be normal behaviour when using this script.
It does use/enable a dynamic far gateway though because this is what's needed to route the traffic to PIA over wireguard after established, so it's likely somewhat similar.
It's interesting to note though that after booting and before the script runs, its already knocked out the IPv6 routing on my main internet connection and is also showing the error around refusing to add a default gateway which was in Shuppie's example.
That does give me something to compare later to see if it also fails to add the normal route after upgrading to 23.1.7_3, but off to work now so will have to figure that out later.
Thanks
Gareth
-
Let's look at this differently... I'm assuming the bad change is https://github.com/opnsense/core/commit/a8e9862b410073 and it may work again if it's reverted?
# opsense-patch a8e9862b410073
The commit does two things: it adds IP address family specific reload functionality, but it should not matter for when e.g. rc.configure_routing is called which is what WireGuard is doing.
The other thing is it tries to verify that the gateway selected for default gateway use does have a matching interface with at least one address in it (the equivalent of calling ifconfig to see if that has an address). The latter one is easy to try... I do think that at least one address must be present anyway, but perhaps if it's a tunnel device the address might not show up correctly?
Looking forward to verification that the patch is the issue...
Cheers,
Franco
-
There are two patches to help with diagnose..
https://github.com/opnsense/core/commit/8beb293c5
https://github.com/opnsense/core/commit/48855143b
This is on a clean 23.1.7, opnsense-revert used to make sure:
# opnsense-revert opnsense && opnsense-patch 48855143b 8beb293c5
# /usr/local/etc/rc.routing_configure
# opnsense-log | grep refusing
In the last log line there is a hint of the interface and device being used, e.g.:
> ROUTING: refusing to set inet gateway on addressless wan(igb1)
For the device is parenthesis run:
# pluginctl -D igb1
Depending on this output the log line is generated and the route refused. If data is there we might be looking at a timing issue, if not then it's something more fundamental.
Cheers,
Franco
-
Let's look at this differently... I'm assuming the bad change is https://github.com/opnsense/core/commit/a8e9862b410073 and it may work again if it's reverted?
# opsense-patch a8e9862b410073
The commit does two things: it adds IP address family specific reload functionality, but it should not matter for when e.g. rc.configure_routing is called which is what WireGuard is doing.
The other thing is it tries to verify that the gateway selected for default gateway use does have a matching interface with at least one address in it (the equivalent of calling ifconfig to see if that has an address). The latter one is easy to try... I do think that at least one address must be present anyway, but perhaps if it's a tunnel device the address might not show up correctly?
Looking forward to verification that the patch is the issue...
Cheers,
Franco
Hi Franco,
So first things first, I've spent a few minutes running tests again and it does in fact look as if the wireguard vpn is coming up after I upgrade to 23.1.7_3, I have double checked and it does appear to be working as expected.
The failure is on the IPv6 gateway which refuses to come online and therefore stops all IPv6 traffic. I guess this would be consistent with the error in the log around refusing to apply an inet6 GW.
I tried to apply the patch you mentioned in the above post and got the below error output:
root@OPNSense:~ # opnsense-patch a8e9862b410073
Fetched a8e9862b410073 via https://github.com/opnsense/core
1 out of 2 hunks failed while patching etc/rc.syshook.d/monitor/10-dpinger
root@OPNSense:~ # opnsense-patch a8e9862b410073
Found local copy of a8e9862b410073, skipping fetch.
1 out of 2 hunks failed while patching etc/rc.syshook.d/monitor/10-dpinger
I tried it a couple of times just to be sure, but the half a patch doesn't resolve or make any impacts.
I'll move on to the next set of tests to try and feedback further
Thanks
Gareth
-
There are two patches to help with diagnose..
https://github.com/opnsense/core/commit/8beb293c5
https://github.com/opnsense/core/commit/48855143b
This is on a clean 23.1.7, opnsense-revert used to make sure:
# opnsense-revert opnsense && opnsense-patch 48855143b 8beb293c5
# /usr/local/etc/rc.routing_configure
# opnsense-log | grep refusing
In the last log line there is a hint of the interface and device being used, e.g.:
> ROUTING: refusing to set inet gateway on addressless wan(igb1)
For the device is parenthesis run:
# pluginctl -D igb1
Depending on this output the log line is generated and the route refused. If data is there we might be looking at a timing issue, if not then it's something more fundamental.
Cheers,
Franco
Hi Franco,
Ok, next set of tests as requested above (slightly edited for privacy):
root@OPNSense:~ # opnsense-revert opnsense && opnsense-patch 48855143b 8beb29 3c5
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following packages will be fetched:
New packages to be FETCHED:
opnsense: 23.1.7_3 (4 MiB: 100.00% of the 4 MiB to download)
Number of packages to be fetched: 1
The process will require 4 MiB more space.
4 MiB to be downloaded.
Fetching opnsense-23.1.7_3.pkg: 100% 4 MiB 4.4MB/s 00:01
opnsense-23.1.7_3: already unlocked
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):
Installed packages to be REINSTALLED:
opnsense-23.1.7_3
Number of packages to be reinstalled: 1
[1/1] Reinstalling opnsense-23.1.7_3...
[1/1] Extracting opnsense-23.1.7_3: 100%
Stopping configd...done
Resetting root shell
Updating /etc/shells
Unhooking from /etc/rc
Unhooking from /etc/rc.shutdown
Updating /etc/shells
Registering root shell
Hooking into /etc/rc
Hooking into /etc/rc.shutdown
Starting configd.
>>> Invoking update script 'refresh'
Writing firmware setting...done.
Writing trust files...done.
Configuring login behaviour...done.
Configuring system logging...done.
=====
Message from opnsense-23.1.7_3:
--
I'm no chicken
Fetched 48855143b via https://github.com/opnsense/core
Fetched 8beb293c5 via https://github.com/opnsense/core
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 48855143b0c5e2d3f70a29a841e80a45210d74e2 Mon Sep 17 00:00:00 2001
|From: Franco Fichtner <franco@opnsense.org>
|Date: Wed, 10 May 2023 14:37:38 +0200
|Subject: [PATCH] system: add 'if' to message in case of mismatch
|
|PR: https://forum.opnsense.org/index.php?topic=33864.0
|---
| src/etc/inc/system.inc | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/src/etc/inc/system.inc b/src/etc/inc/system.inc
|index 722900df88..7666a0e740 100644
|--- a/src/etc/inc/system.inc
|+++ b/src/etc/inc/system.inc
--------------------------
Patching file etc/inc/system.inc using Plan A...
Hunk #1 succeeded at 619 (offset -2 lines).
done
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 8beb293c53e3d14c5256cd648b3a834667595c2d Mon Sep 17 00:00:00 2001
|From: Franco Fichtner <franco@opnsense.org>
|Date: Mon, 15 May 2023 10:11:38 +0200
|Subject: [PATCH] pluginctl: add an ifconfig mode for easier debugging and
| later use
|
|PR: https://forum.opnsense.org/index.php?topic=33864.0
|---
| src/sbin/pluginctl | 7 +++++--
| 1 file changed, 5 insertions(+), 2 deletions(-)
|
|diff --git a/src/sbin/pluginctl b/src/sbin/pluginctl
|index afa7e674ce..eb531b8e97 100755
|--- a/src/sbin/pluginctl
|+++ b/src/sbin/pluginctl
--------------------------
Patching file sbin/pluginctl using Plan A...
Hunk #1 succeeded at 63.
Hunk #2 succeeded at 78.
done
All patches have been applied successfully. Have a nice day.
root@OPNSense:~ # /usr/local/etc/rc.routing_configure
Setting up routes...done.
Setting up gateway monitors...done.
Configuring firewall.......done.
root@OPNSense:~ # opnsense-log | grep refusing
<11>1 2023-05-15T18:36:30+01:00 OPNSense.domain.local opnsense 301 - [meta sequenceId="12"] /usr/local/etc/rc.bootup: ROUTING: refusing to set inet6 gateway on addressless wan
<11>1 2023-05-15T18:36:44+01:00 OPNSense.domain.local opnsense 4898 - [meta sequenceId="43"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet6 gateway on addressless wan
root@OPNSense:~ # pluginctl -D igb2
{
"igb2": {
"flags": [
"up",
"broadcast",
"running",
"simplex",
"multicast"
],
"capabilities": [
"rxcsum",
"txcsum",
"vlan_mtu",
"vlan_hwtagging",
"jumbo_mtu",
"vlan_hwcsum",
"tso4",
"tso6",
"lro",
"wol_ucast",
"wol_mcast",
"wol_magic",
"vlan_hwfilter",
"vlan_hwtso",
"netmap",
"rxcsum_ipv6",
"txcsum_ipv6",
"nomap"
],
"options": [
"vlan_mtu",
"jumbo_mtu",
"nomap"
],
"macaddr": "a0:36:9f:7d:55:7b",
"ipv4": [],
"ipv6": [],
"supported_media": [
"autoselect",
"1000baseT",
"1000baseT full-duplex",
"100baseTX full-duplex",
"100baseTX",
"10baseT/UTP full-duplex",
"10baseT/UTP"
],
"mtu": "1500",
"media": "1000baseT <full-duplex>",
"media_raw": "Ethernet autoselect (1000baseT <full-duplex>)",
"status": "active"
}
}
Hopefully offers you something useful, apologies on the earlier mistake, which I've now corrected in this post. There doesn't seem to be any info attached to the igb2 interface above.
It did lead me to consider one thing which I thought I'd mention, in the grep for refusing, it shows the interface as wan but no hint. I know the wan interface is tied to igb2 on my virtual firewall (VMware Passthrough, Intel Card) as it is on my physical one.
But ultimately it's a PPPoE interface which is then linked to igb2, so wonder if it's not picking it up because of this?
In any case, hope the output is of some use, going to rollback again for now, let me know if there is anything further you would like me to do.
Thanks
Gareth
-
Hi Franco,
I think I found what causing our issues with "FAR GATEWAYS". By removing this from src/etc/inc/filter.inc. I think you broke the ability to reach or find any other static route that point to where that gateway is. Would explain the issue that Gazd25 is having also.
}
$default_gw = $fw->getGateways()->getDefaultGW($down_gateways, $ipprotocol);
if ($default_gw !== null && !empty($default_gw['gateway'])) {
system_default_route($default_gw['gateway'], $default_gw['if'], isset($default_gw['fargw']));
}
And then this bit of code also prevents FAR GATEWAYS that are "DOWN" from being a canditate to be gateways.
foreach (['inet', 'inet6'] as $ipproto) {
/* determine default gateway without considering monitor status */
$gateway = $gateways->getDefaultGW([], $ipproto);
$logproto = $ipproto == 'inet' ? 'IPv4' : 'IPv6';
if ($gateway != null) {
log_msg("ROUTING: {$logproto} default gateway set to {$gateway['interface']}", LOG_INFO);
if ((empty($interface) || $interface == $gateway['interface']) && !empty($gateway['gateway'])) {
log_msg("ROUTING: setting {$logproto} default route to {$gateway['gateway']}");
system_default_route($gateway['gateway'], $gateway['interface'], isset($gateway['fargw']));
} else {
log_msg("ROUTING: skipping {$logproto} default route");
Thank you,
Nic
-
@Gareth
> ROUTING: refusing to set inet6 gateway on addressless wan
> ROUTING: refusing to set inet6 gateway on addressless wan
It looks like the newly added log message didn't trigger on the reload here, not sure why or otherwise it would have said "wan(xxx)" to point to the device it's been using.
Looking further and hearing about PPPoE I think you don't have "Use IPv4 connectivity" set for your WAN IPv6 configuration? That's why it wants to use igb2 instead of pppoeX and here we can se igb2 does not have an address indeed.
If, however, the option is set then at least we know where to look further.
Cheers,
Franco
PS: Which WAN IPv6 mode are you using?
-
@Nic
Far gateway behaviour was not changed at all. I don't see the immediate issue from your post.
Cheers,
Franco
-
@Gareth
> ROUTING: refusing to set inet6 gateway on addressless wan
> ROUTING: refusing to set inet6 gateway on addressless wan
It looks like the newly added log message didn't trigger on the reload here, not sure why or otherwise it would have said "wan(xxx)" to point to the device it's been using.
Looking further and hearing about PPPoE I think you don't have "Use IPv4 connectivity" set for your WAN IPv6 configuration? That's why it wants to use igb2 instead of pppoeX and here we can se igb2 does not have an address indeed.
If, however, the option is set then at least we know where to look further.
Cheers,
Franco
PS: Which WAN IPv6 mode are you using?
Hi Franco,
I ran the procedure a couple of times to make sure it wasn't me, but output was the same on both occasions, so just posted.
In terms of "Use IPv4 connectivity" it is set and my IPv6 mode is set to static because my ISP does not support PPPoEv6 or DHCPv6, few pictures attached showing further info.
I've kept the static IPv6 for the WAN interface and the gateways in the same /64 range so it's basically:
WAN Interface: face:face:face:face::254/64
WAN Gateway: face:face:face:face::50/64
As I recall, when I configured this a pretty long time back now, this was the only way that worked for me and I can see both before the upgrade to 23.1.7_3 and after that the face:face:face:face::254 gateway is being automatically selected, it's just that this gateway refuses to come up after the upgrade, but strangely dpinger shows it as green even though it's offline, I have attached a picture of this also.
The internal clients are given IPv6 addresses using a manually configured radvd since there is no interface to track here.
Thanks for the help
Gareth
-
@Gareth
Thanks for your help! At first glance I was unable to reproduce but it looks like PPPoE Use IPv4 connectivity with static setup is a bit of a loophole with regard to bringing up the IPv6 default route since the IPv4 is dynamically assigned to a spawned PPPoE interface and only then the IPv6 address is added.
The proposed patch is:
https://github.com/opnsense/core/commit/766f1f0c5a3
# opnsense-patch 766f1f0c5a3
Cheers,
Franco
-
Hi Franco,
So just applied the patch and I'm afraid it doesn't seem to alter the behaviour at all.
I applied the 766f1f0c5a3 patch to a newly updated 23.1.7_3 and it didn't seem to have any impact, rebooted a couple of times to rule out any issue with timing.
Next to make double sure, I reverted to my working 23.1 using a VMware snapshot, installed the 23.1.7_3 update and then added the 766f1f0c5a3 patch again and rebooted a few more times to check if the issue was timing, but behaviour remained consistent.
Whilst doing this, I did remember something that happens rarely but occassionally and all this rebooting has made the issue more obvious. Sometimes I would say maybe once out of every 5 boots, I will have to click the dpinger play button manually to start the IPv6 gateway. It had almost slipped my mind since it's rare I'll need to do it, but it does occasionally happen when running an update or rebooting for some reason and when I manually click the dpinger button the gateway had always previously started correctly.
Have a feeling that behaviour has been there for a while but would happen so rarely that it was simple just to click the button and not worry.
I wonder if some of the new code you have introduced has crystalised that issue in a way it hadn't on the previous version?
In any case, I also attempted the same thing to manually click the dpinger button, after the 23.1.7_3 update and the patch and it doesn't start the gateway I'm afraid so no IPv6 traffic.
I did also run the "opnsense-log | grep refusing" command against the updated and patched version and it shows the same error:
<11>1 2023-05-16T10:08:45+01:00 OPNSense.local opnsense 78232 - [meta sequenceId="8"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet6 gateway on addressless wan
Let me know what else I can do to help.
Thanks
Gareth
-
It's still missing the crucial bit of https://github.com/opnsense/core/commit/48855143b to know it's not reading the wrong interface information.
# opnsense-patch 48855143b
I'm sure 766f1f0c5a3 already helps with the edge case. But to sure let me state the goal of the session here: the default route for IPv6 is not set on reboot / or not at all (even when reapplying routes from the GUI).
Cheers,
Franco
-
Hi Franco,
So just applied the 48855143b patch on top of the 766f1f0c5a3.
Same as before, no change to the IPv6 gateway which remains offline.
Thanks
Gareth
-
It would be best if you could reboot and let me have the recent "ROUTING" output of that boot (can also share via mail franco AT opnsense DOT org or private message).
Cheers,
Franco
-
Hi Franco,
Just rebooted and taken a copy of the routing table from System > Routes > Status, which is hopefully what you are after and emailed it over to you.
Thanks
Gareth
-
Hi Gareth,
Just to clarify I'm looking for:
# opnsense-log | grep ROUTING
(Email not received yet -- if it's in there nevermind.)
Cheers,
Franco
-
Hi Franco,
Here is an output of the asked LOG with all proposed patch:
<13>1 2023-05-16T18:34:57-04:00 OPNsense.localdomain opnsense 63457 - [meta sequenceId="3"] /usr/local/sbin/pluginctl: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:34:57-04:00 OPNsense.localdomain opnsense 63457 - [meta sequenceId="4"] /usr/local/sbin/pluginctl: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:34:57-04:00 OPNsense.localdomain opnsense 63457 - [meta sequenceId="5"] /usr/local/sbin/pluginctl: ROUTING: keeping current inet default gateway 'xxxxx'
<13>1 2023-05-16T18:34:59-04:00 OPNsense.localdomain opnsense 89022 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:34:59-04:00 OPNsense.localdomain opnsense 89022 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:34:59-04:00 OPNsense.localdomain opnsense 89022 - [meta sequenceId="11"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:12-04:00 OPNsense.localdomain opnsense 93565 - [meta sequenceId="3"] /usr/local/sbin/pluginctl: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:12-04:00 OPNsense.localdomain opnsense 93565 - [meta sequenceId="4"] /usr/local/sbin/pluginctl: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:12-04:00 OPNsense.localdomain opnsense 93565 - [meta sequenceId="5"] /usr/local/sbin/pluginctl: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:14-04:00 OPNsense.localdomain opnsense 54979 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:14-04:00 OPNsense.localdomain opnsense 54979 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:14-04:00 OPNsense.localdomain opnsense 54979 - [meta sequenceId="11"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:50-04:00 OPNsense.localdomain opnsense 27513 - [meta sequenceId="12"] /usr/local/sbin/pluginctl: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:50-04:00 OPNsense.localdomain opnsense 27513 - [meta sequenceId="13"] /usr/local/sbin/pluginctl: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:50-04:00 OPNsense.localdomain opnsense 27513 - [meta sequenceId="14"] /usr/local/sbin/pluginctl: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:52-04:00 OPNsense.localdomain opnsense 48269 - [meta sequenceId="18"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:52-04:00 OPNsense.localdomain opnsense 48269 - [meta sequenceId="19"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:52-04:00 OPNsense.localdomain opnsense 48269 - [meta sequenceId="20"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:53-04:00 OPNsense.localdomain opnsense 68645 - [meta sequenceId="21"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:53-04:00 OPNsense.localdomain opnsense 68645 - [meta sequenceId="22"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:53-04:00 OPNsense.localdomain opnsense 68645 - [meta sequenceId="23"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:39:01-04:00 OPNsense.localdomain opnsense 56798 - [meta sequenceId="25"] /usr/local/sbin/pluginctl: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:39:01-04:00 OPNsense.localdomain opnsense 56798 - [meta sequenceId="26"] /usr/local/sbin/pluginctl: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:39:01-04:00 OPNsense.localdomain opnsense 56798 - [meta sequenceId="27"] /usr/local/sbin/pluginctl: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:39:03-04:00 OPNsense.localdomain opnsense 76629 - [meta sequenceId="31"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:39:03-04:00 OPNsense.localdomain opnsense 76629 - [meta sequenceId="32"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:39:03-04:00 OPNsense.localdomain opnsense 76629 - [meta sequenceId="33"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
My "FAR GATEWAY" is still not working on 27.1.7_3..
Log message:
2023-05-16T18:39:01-04:00
opnsense /usr/local/sbin/pluginctl: Chose to bind CISCO_WAN on 192.168.15.1 since we could not find a proper match.
Reverting back to 23.1.6
Nic
-
I'm really not sure what is wrong. The code didn't even change for dpinger between 23.1.6 and 23.1.7 and the routing code is not relevant here (only VIPs are):
2023-05-16T18:39:01-04:00
opnsense /usr/local/sbin/pluginctl: Chose to bind CISCO_WAN on 192.168.15.1 since we could not find a proper match.
If you want to reach a monitor 192.168.12.1 you just need a VIP on CISCO_WAN for 192.168.12.XX/24 and it should work... but that was always the case?
Cheers,
Franco
-
Hi Franco,
I kind of politely disagre that the new code in src/etc/inc/filter.inc and /usr/local/sbin/pluginct dont have any impact on routing desision because as soon as we upgrade to the new firmware it fails to reach 192.168.12.0/24 that is perfecly reachable on 23.1.6 via my static route and to make the matter worse we are seeing error messages that are only present in the new code. On another note, I never had to make Virtual IP's to make my setup work.
Regards,
Nic
-
We can disagree on this but if you insist without pointing to a line of code where the bug is you will have to trust me that this might be configuration-related.
Sometimes we fix consistency issues that will highlight misconfigurations as "bugs" on the user end.
I'm only here trying to help which is a bit of a time sink at the moment (lot of time spent, not much progress made).
Cheers,
Franco
-
Hi Franco,
Im sorry I was out of line. On a previous post you asked to show us our routing table before and after the upgrade. On 23.1.7_3 does not contain what 23.1.6 does. :o In 23.1.6 you can clearly see that 192.168.12.0/24 is the gateway for my network.
Nic
-
Hi All,
I realise I've gone a bit quiet since Franco has been helping me get to the bottom of the problem I was experiencing so wanted to provide an update.
After a couple more patches and troubleshooting steps supplied by Franco, we were able to identify that the problem was a result of slight misinformation from my ISP and a change to the code in 23.1.7_3 that means a PPPoEv6 request is no longer sent by OPNsense when assigning a static IPv6 address to the WAN interface, when before it used to be but really why would it be needed? And further it was enirely invisible to me until this problem occured.
Well as it turns out, if my ISP's equipment didn't recieve that request at the time of the PPPoE initiation, they were disabling IPv6, which was why I lost outbound IPv6 routing.
I have been able to resolve by changing my WAN interface IPv6 setting to PPPoEv6 and deleting the old now non-needed gateway.
I did a full regression to 23.1, applied no patches and performed the above steps, then updated to 23.1.7_3 and this resolved my issue fully.
I hope this gives some help to others experiencing the same or similar issues.
I also wanted to take the opportunity to thank Franco for his amazing help, it's no wonder he is a hero member!!!
And further thanks to everybody involved with OPNsense, I personally am hugely grateful for the great support, guidance and great product you provide.
Thanks
Gareth
-
Hi Franco,
I think you were right in the end. I had a configuration error. So here what I did to get thing working again. So I activated the "Dynamic gateway policy : This interface does not require an intermediate system to act as a gateway" in the Lan interface. Then, in Gateways, I checked the box to disable the GATEWAY monitoring for this Gateway so the dping would alwas show that interface as active. Left everything the same in my "FAR GATEWAY" single gateway. In my static route I then changed the GATEWAY to network 192.168.12.0/24 to "LAN_GW - inet" and reloaded / rebooted and it all started to work again like before. Cant explain why it does do, was looking foward for you input on this?
Please accept my appologies,
Nic
-
Hi Nic,
No worries. It's just a little difficult to get the full picture in community support as we don't do remote sessions gathering all necessary input like we do in commercial support resulting in a swifter resolution in most cases.
I think if it works that's ok, though "Dynamic gateway policy" is meant for dynamic VPN connections more than anything else. I'd still add a VIP to the LAN network that is in the segment of 192.168.12.0/24 which would allow the routing to pick up the static route more easily (and make that message for the gateway monitor not binding to the right address go away as well).
Cheers,
Franco
-
Hi Franco,
Im sorry to bother your again with this issue. But it is not resolved. I tought is was but it's not. It's been kind of working fine using this: "So I activated the "Dynamic gateway policy : This interface does not require an intermediate system to act as a gateway" in the Lan interface. Then, in Gateways, I checked the box to disable the GATEWAY monitoring for this Gateway so the dping would alwas show that interface as active. Left everything the same in my "FAR GATEWAY" single gateway. In my static route I then changed the GATEWAY to network 192.168.12.0/24 to "LAN_GW - inet" and reloaded / rebooted ". But im experimenting hickup and wierd "BUGS" like MS Teams not working but fine on 23.1.6..
Any way, I found out that I can not have My "LAN_GATEWAY" 192.168.15.1 and my "CISCO_WAN" 192.168.12.1 witch is the FAR GATEWAY running at the same time on 23.1.7_3 and 23.1.8 for that mather. For it to work on 23.1.7_3, the LAN_GATEWAY need's to be in "dynamic" for it's IP with "DISABLE GATEWAY MONITORING" turned on. Plus the "LAN_GATEWAY" alson need's to be set as the default gateway of my "CISCO_WAN" static route, witch make zero sence..
Rollback to 23.1.6 form 23.1.8 and everything is working 100% again. :o
Thank you,
Nick
-
i am facing the same issue with one box 23.1.7_3.
the gateway keeps showing offline but everything is working. only sometimes it has to think about reloading websites.
when i tried the revert back to opnsense-revert -r 23.1.6 on the shell nothing happens on the shell.
supposed to reboot after the command line opnsense-revert -r 23.1.6 ?
-
@My_Network
https://github.com/opnsense/core/commit/25e2c0a30
# opnsense-patch 25e2c0a30
Cheers,
Franco
-
@My_Network
https://github.com/opnsense/core/commit/25e2c0a30
# opnsense-patch 25e2c0a30
Cheers,
Franco
Hi Franco,
thank you so much for your answer.
when i enter the command line nothing happens with this machine.
i have the feeling this one is broke.
i tried it on a different one 23.1.8 and it seems to be applied succecefully.
Patching file etc/inc/system.inc using Plan A...
Hunk #1 succeeded at 677 (offset -10 lines).
done
All patches have been applied successfully. Have a nice day.
-
@Julien
I have the feeling you mistook my note to be for you but I was replying to My_Network/Nick here.