23.1.7_1 broke my Firewall (Fixed)

[gazd25]

@Gareth

> ROUTING: refusing to set inet6 gateway on addressless wan
> ROUTING: refusing to set inet6 gateway on addressless wan

It looks like the newly added log message didn't trigger on the reload here, not sure why or otherwise it would have said "wan(xxx)" to point to the device it's been using.

Looking further and hearing about PPPoE I think you don't have "Use IPv4 connectivity" set for your WAN IPv6 configuration? That's why it wants to use igb2 instead of pppoeX and here we can se igb2 does not have an address indeed.

If, however, the option is set then at least we know where to look further.


Cheers,
Franco

PS: Which WAN IPv6 mode are you using?

Hi Franco,

I ran the procedure a couple of times to make sure it wasn't me, but output was the same on both occasions, so just posted.

In terms of "Use IPv4 connectivity" it is set and my IPv6 mode is set to static because my ISP does not support PPPoEv6 or DHCPv6, few pictures attached showing further info.

I've kept the static IPv6 for the WAN interface and the gateways in the same /64 range so it's basically:

WAN Interface: face:face:face:face::254/64
WAN Gateway: face:face:face:face::50/64

As I recall, when I configured this a pretty long time back now, this was the only way that worked for me and I can see both before the upgrade to 23.1.7_3 and after that the face:face:face:face::254 gateway is being automatically selected, it's just that this gateway refuses to come up after the upgrade, but strangely dpinger shows it as green even though it's offline, I have attached a picture of this also.

The internal clients are given IPv6 addresses using a manually configured radvd since there is no interface to track here.

Thanks for the help

Gareth

[franco]

@Gareth

Thanks for your help! At first glance I was unable to reproduce but it looks like PPPoE Use IPv4 connectivity with static setup is a bit of a loophole with regard to bringing up the IPv6 default route since the IPv4 is dynamically assigned to a spawned PPPoE interface and only then the IPv6 address is added.

The proposed patch is:

https://github.com/opnsense/core/commit/766f1f0c5a3

# opnsense-patch 766f1f0c5a3


Cheers,
Franco

[gazd25]

Hi Franco,

So just applied the patch and I'm afraid it doesn't seem to alter the behaviour at all.

I applied the 766f1f0c5a3 patch to a newly updated 23.1.7_3 and it didn't seem to have any impact, rebooted a couple of times to rule out any issue with timing.

Next to make double sure, I reverted to my working 23.1 using a VMware snapshot, installed the 23.1.7_3 update and then added the 766f1f0c5a3 patch again and rebooted a few more times to check if the issue was timing, but behaviour remained consistent.

Whilst doing this, I did remember something that happens rarely but occassionally and all this rebooting has made the issue more obvious. Sometimes I would say maybe once out of every 5 boots, I will have to click the dpinger play button manually to start the IPv6 gateway. It had almost slipped my mind since it's rare I'll need to do it, but it does occasionally happen when running an update or rebooting for some reason and when I manually click the dpinger button the gateway had always previously started correctly.

Have a feeling that behaviour has been there for a while but would happen so rarely that it was simple just to click the button and not worry.

I wonder if some of the new code you have introduced has crystalised that issue in a way it hadn't on the previous version?

In any case, I also attempted the same thing to manually click the dpinger button, after the 23.1.7_3 update and the patch and it doesn't start the gateway I'm afraid so no IPv6 traffic.

I did also run the "opnsense-log | grep refusing" command against the updated and patched version and it shows the same error:

<11>1 2023-05-16T10:08:45+01:00 OPNSense.local opnsense 78232 - [meta sequenceId="8"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet6 gateway on addressless wan

Let me know what else I can do to help.

Thanks

Gareth

[franco]

It's still missing the crucial bit of https://github.com/opnsense/core/commit/48855143b to know it's not reading the wrong interface information.

# opnsense-patch 48855143b

I'm sure 766f1f0c5a3 already helps with the edge case. But to sure let me state the goal of the session here: the default route for IPv6 is not set on reboot / or not at all (even when reapplying routes from the GUI).


Cheers,
Franco

[gazd25]

Hi Franco,

So just applied the 48855143b patch on top of the 766f1f0c5a3.

Same as before, no change to the IPv6 gateway which remains offline.

Thanks

Gareth

[franco]

It would be best if you could reboot and let me have the recent "ROUTING" output of that boot (can also share via mail franco AT opnsense DOT org or private message).


Cheers,
Franco

[gazd25]

Hi Franco,

Just rebooted and taken a copy of the routing table from System > Routes > Status, which is hopefully what you are after and emailed it over to you.

Thanks

Gareth

[franco]

Hi Gareth,

Just to clarify I'm looking for:

# opnsense-log | grep ROUTING

(Email not received yet -- if it's in there nevermind.)


Cheers,
Franco

[My_Network]

Hi Franco,

Here is an output of the asked LOG with all proposed patch:


<13>1 2023-05-16T18:34:57-04:00 OPNsense.localdomain opnsense 63457 - [meta sequenceId="3"] /usr/local/sbin/pluginctl: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:34:57-04:00 OPNsense.localdomain opnsense 63457 - [meta sequenceId="4"] /usr/local/sbin/pluginctl: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:34:57-04:00 OPNsense.localdomain opnsense 63457 - [meta sequenceId="5"] /usr/local/sbin/pluginctl: ROUTING: keeping current inet default gateway 'xxxxx'
<13>1 2023-05-16T18:34:59-04:00 OPNsense.localdomain opnsense 89022 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:34:59-04:00 OPNsense.localdomain opnsense 89022 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:34:59-04:00 OPNsense.localdomain opnsense 89022 - [meta sequenceId="11"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:12-04:00 OPNsense.localdomain opnsense 93565 - [meta sequenceId="3"] /usr/local/sbin/pluginctl: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:12-04:00 OPNsense.localdomain opnsense 93565 - [meta sequenceId="4"] /usr/local/sbin/pluginctl: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:12-04:00 OPNsense.localdomain opnsense 93565 - [meta sequenceId="5"] /usr/local/sbin/pluginctl: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:14-04:00 OPNsense.localdomain opnsense 54979 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:14-04:00 OPNsense.localdomain opnsense 54979 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:14-04:00 OPNsense.localdomain opnsense 54979 - [meta sequenceId="11"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:50-04:00 OPNsense.localdomain opnsense 27513 - [meta sequenceId="12"] /usr/local/sbin/pluginctl: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:50-04:00 OPNsense.localdomain opnsense 27513 - [meta sequenceId="13"] /usr/local/sbin/pluginctl: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:50-04:00 OPNsense.localdomain opnsense 27513 - [meta sequenceId="14"] /usr/local/sbin/pluginctl: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:52-04:00 OPNsense.localdomain opnsense 48269 - [meta sequenceId="18"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:52-04:00 OPNsense.localdomain opnsense 48269 - [meta sequenceId="19"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:52-04:00 OPNsense.localdomain opnsense 48269 - [meta sequenceId="20"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:38:53-04:00 OPNsense.localdomain opnsense 68645 - [meta sequenceId="21"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:38:53-04:00 OPNsense.localdomain opnsense 68645 - [meta sequenceId="22"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:38:53-04:00 OPNsense.localdomain opnsense 68645 - [meta sequenceId="23"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:39:01-04:00 OPNsense.localdomain opnsense 56798 - [meta sequenceId="25"] /usr/local/sbin/pluginctl: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:39:01-04:00 OPNsense.localdomain opnsense 56798 - [meta sequenceId="26"] /usr/local/sbin/pluginctl: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:39:01-04:00 OPNsense.localdomain opnsense 56798 - [meta sequenceId="27"] /usr/local/sbin/pluginctl: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'
<13>1 2023-05-16T18:39:03-04:00 OPNsense.localdomain opnsense 76629 - [meta sequenceId="31"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-05-16T18:39:03-04:00 OPNsense.localdomain opnsense 76629 - [meta sequenceId="32"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
<13>1 2023-05-16T18:39:03-04:00 OPNsense.localdomain opnsense 76629 - [meta sequenceId="33"] /usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway 'xx.xx.xx.xx'


My "FAR GATEWAY" is still not working on 27.1.7_3..

Log message:

2023-05-16T18:39:01-04:00
opnsense   /usr/local/sbin/pluginctl: Chose to bind CISCO_WAN on 192.168.15.1 since we could not find a proper match.


Reverting back to 23.1.6

Nic

[franco]

I'm really not sure what is wrong. The code didn't even change for dpinger between 23.1.6 and 23.1.7 and the routing code is not relevant here (only VIPs are):

2023-05-16T18:39:01-04:00
opnsense   /usr/local/sbin/pluginctl: Chose to bind CISCO_WAN on 192.168.15.1 since we could not find a proper match.

If you want to reach a monitor 192.168.12.1 you just need a VIP on CISCO_WAN for 192.168.12.XX/24 and it should work... but that was always the case?


Cheers,
Franco

[My_Network]

Hi Franco,

I kind of politely disagre that the new code in src/etc/inc/filter.inc and  /usr/local/sbin/pluginct dont have any impact on routing desision because as soon as we upgrade to the new firmware it fails to reach 192.168.12.0/24 that is perfecly reachable on 23.1.6 via my static route and to make the matter worse we are seeing error messages that are only present in the new code. On another note, I never had to make Virtual IP's to make my setup work.

Regards,

Nic

[franco]

We can disagree on this but if you insist without pointing to a line of code where the bug is you will have to trust me that this might be configuration-related.

Sometimes we fix consistency issues that will highlight misconfigurations as "bugs" on the user end.

I'm only here trying to help which is a bit of a time sink at the moment (lot of time spent, not much progress made).


Cheers,
Franco

[My_Network]

Hi Franco,

Im sorry I was out of line. On a previous post you asked to show us our routing table before and after the upgrade. On 23.1.7_3 does not contain what 23.1.6 does.  :o In 23.1.6 you can clearly see that 192.168.12.0/24 is the gateway for my network.

Nic

[gazd25]

Hi All,

I realise I've gone a bit quiet since Franco has been helping me get to the bottom of the problem I was experiencing so wanted to provide an update.

After a couple more patches and troubleshooting steps supplied by Franco, we were able to identify that the problem was a result of slight misinformation from my ISP and a change to the code in 23.1.7_3 that means a PPPoEv6 request is no longer sent by OPNsense when assigning a static IPv6 address to the WAN interface, when before it used to be but really why would it be needed? And further it was enirely invisible to me until this problem occured.

Well as it turns out, if my ISP's equipment didn't recieve that request at the time of the PPPoE initiation, they were disabling IPv6, which was why I lost outbound IPv6 routing.

I have been able to resolve by changing my WAN interface IPv6 setting to PPPoEv6 and deleting the old now non-needed gateway.

I did a full regression to 23.1, applied no patches and performed the above steps, then updated to 23.1.7_3 and this resolved my issue fully.

I hope this gives some help to others experiencing the same or similar issues.

I also wanted to take the opportunity to thank Franco for his amazing help, it's no wonder he is a hero member!!!

And further thanks to everybody involved with OPNsense, I personally am hugely grateful for the great support, guidance and great product you provide.

Thanks

Gareth

[My_Network]

Hi Franco,

I think you were right in the end. I had a configuration error. So here what I did to get thing working again. So I activated the "Dynamic gateway policy : This interface does not require an intermediate system to act as a gateway" in the Lan interface. Then, in Gateways, I checked the box to disable the GATEWAY monitoring for this Gateway so the dping would alwas show that interface as active. Left everything the same in my "FAR GATEWAY" single gateway. In my static route I then changed the GATEWAY to network 192.168.12.0/24 to "LAN_GW - inet" and reloaded / rebooted and it all started to work again like before. Cant explain why it does do, was looking foward for you input on this?

Please accept my appologies,

Nic