Messages

I was investigating further this issue and after setting up a new server with 22.1.10 and imported the latest backup taken before upgrading to 22.7 everything is working fine on the same network.
The issue with 22.7 is still persisting so the upgrade from 22.1.10 to 22.7 is broken somewhere as there are no changes in the configuration and the network is the same.
Any help in sorting out this issue would be appreciated.

Sorry, I had a bit of an issue understanding what you are saying here... are you saying that the restoration of the latest backup corrected the issue? Or are you saying that it failed to correct the issue?

Also, this seems to be pretty similar to what I described in https://forum.opnsense.org/index.php?topic=29776.0 ... did you have similar findings throughout the course of your investigation?

[Interfaces > Diagnostics > Ping]

Hey all,

We upgraded to 22.7_4 and promptly lost some network connectivity after the upgrade. But this wasn't an all-or-nothing loss-- a few strange "patches" seemed to have corrected part of the issue, though not all. To explain:

Initially, we lost connection to the firewall itself via the OpenVPN firewall. This was because OPNSense was unable to query the FQDN of our IdP and we were able to regain access by adding pointing OPNSense to our IdP's DNS servers, allowing their IP addresses to resolve. In fact, from the console, the only IPs that can be pinged at all are those that are set in /etc/resolv.conf (set indirectly by the web interface).

From the web interface, we are unable to ping any external host when we use Interfaces > Diagnostics > Ping with Source Address set to Default. However, if we set Source Address to any of our other interfaces (including our LAN and WAN interfaces), we receive a successful ping result. This is reflected on the console-- for example, if we execute ping www.google.com -- we receive the following.

Code Select Expand

PING www.google.com (142.250.138.103): 56 data bytes

The process hangs until it times out. The good news is that it appears that the IP itself was resolved-- it's the actual ping that's failing. However, if we explicitly set any of the interface IPs, i.e. ping -S 10.0.0.1 www.google.com -- we receive the following expected output.

Code Select Expand

PING www.google.com (142.250.138.147) from 10.0.0.1: 56 data bytes
64 bytes from 142.250.138.147: icmp_seq=0 ttl=105 time=9.189 ms
64 bytes from 142.250.138.147: icmp_seq=1 ttl=105 time=9.136 ms
64 bytes from 142.250.138.147: icmp_seq=2 ttl=105 time=9.166 ms
64 bytes from 142.250.138.147: icmp_seq=3 ttl=105 time=9.067 ms
...

Again, this is successful with all explicitly-defined interfaces. The use of curl and other similar tools is also successful-- it fails when we don't explicitly specify an interface, and it succeeds when we do. Likewise, all machines that are behind the firewall have maintained their network connectivity, can reach out to the internet, and are otherwise operating normally. Tunnels that reach out to other datacenters are likewise operational.

When we looked at the firewall log live view while using the failed ping command (the one that doesn't specify an interface), we notice that the Source appears to be 0.0.0.0 for whatever reason. My guess is that this is the issue, but I don't really know how to resolve that.

As it stands, we can no longer check for updates from OPNSense itself-- both the console and the web UI have lost the ability to pull data externally. I will note that the console can still ping machines that are on the LAN so my guess is that this issue has something to do with the gateway itself (possibly), but we hadn't changed any of the settings prior to update.

We did execute a connectivity audit... it was painfully slow. The current output is as follows.

Code Select Expand

***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 22.7_4 (amd64/OpenSSL) at Wed Aug 10 04:12:52 UTC 2022
Checking connectivity for host: pkg.opnsense.org -> 89.149.211.205
PING 89.149.211.205 (89.149.211.205): 1500 data bytes

--- 89.149.211.205 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv4): https://pkg.opnsense.org/FreeBSD:13:amd64/22.7
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/meta.txz: Operation timed out
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/packagesite.pkg: Operation timed out
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/packagesite.txz: Operation timed out
Unable to update repository OPNsense
Error updating repositories!
Checking connectivity for host: pkg.opnsense.org -> 2001:1af8:4f00:a005:5::
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://pkg.opnsense.org/FreeBSD:13:amd64/22.7
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/meta.txz: Non-recoverable resolver failure
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/packagesite.pkg: Non-recoverable resolver failure
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/packagesite.txz: Non-recoverable resolver failure
Unable to update repository OPNsense
Error updating repositories!
***DONE***

Any help that y'all could give on this would be phenomenal. Thanks in advance!