1
22.7 Legacy Series / OPNsense 22.7_4: Loss of Network Connectivity
« on: August 10, 2022, 06:25:05 am »
Hey all,
We upgraded to 22.7_4 and promptly lost some network connectivity after the upgrade. But this wasn't an all-or-nothing loss-- a few strange "patches" seemed to have corrected part of the issue, though not all. To explain:
Initially, we lost connection to the firewall itself via the OpenVPN firewall. This was because OPNSense was unable to query the FQDN of our IdP and we were able to regain access by adding pointing OPNSense to our IdP's DNS servers, allowing their IP addresses to resolve. In fact, from the console, the only IPs that can be pinged at all are those that are set in /etc/resolv.conf (set indirectly by the web interface).
From the web interface, we are unable to ping any external host when we use Interfaces > Diagnostics > Ping with Source Address set to Default. However, if we set Source Address to any of our other interfaces (including our LAN and WAN interfaces), we receive a successful ping result. This is reflected on the console-- for example, if we execute ping www.google.com -- we receive the following.
The process hangs until it times out. The good news is that it appears that the IP itself was resolved-- it's the actual ping that's failing. However, if we explicitly set any of the interface IPs, i.e. ping -S 10.0.0.1 www.google.com -- we receive the following expected output.
Again, this is successful with all explicitly-defined interfaces. The use of curl and other similar tools is also successful-- it fails when we don't explicitly specify an interface, and it succeeds when we do. Likewise, all machines that are behind the firewall have maintained their network connectivity, can reach out to the internet, and are otherwise operating normally. Tunnels that reach out to other datacenters are likewise operational.
When we looked at the firewall log live view while using the failed ping command (the one that doesn't specify an interface), we notice that the Source appears to be 0.0.0.0 for whatever reason. My guess is that this is the issue, but I don't really know how to resolve that.
As it stands, we can no longer check for updates from OPNSense itself-- both the console and the web UI have lost the ability to pull data externally. I will note that the console can still ping machines that are on the LAN so my guess is that this issue has something to do with the gateway itself (possibly), but we hadn't changed any of the settings prior to update.
We did execute a connectivity audit... it was painfully slow. The current output is as follows.
Any help that y'all could give on this would be phenomenal. Thanks in advance!
We upgraded to 22.7_4 and promptly lost some network connectivity after the upgrade. But this wasn't an all-or-nothing loss-- a few strange "patches" seemed to have corrected part of the issue, though not all. To explain:
Initially, we lost connection to the firewall itself via the OpenVPN firewall. This was because OPNSense was unable to query the FQDN of our IdP and we were able to regain access by adding pointing OPNSense to our IdP's DNS servers, allowing their IP addresses to resolve. In fact, from the console, the only IPs that can be pinged at all are those that are set in /etc/resolv.conf (set indirectly by the web interface).
From the web interface, we are unable to ping any external host when we use Interfaces > Diagnostics > Ping with Source Address set to Default. However, if we set Source Address to any of our other interfaces (including our LAN and WAN interfaces), we receive a successful ping result. This is reflected on the console-- for example, if we execute ping www.google.com -- we receive the following.
Code: [Select]
PING www.google.com (142.250.138.103): 56 data bytes
The process hangs until it times out. The good news is that it appears that the IP itself was resolved-- it's the actual ping that's failing. However, if we explicitly set any of the interface IPs, i.e. ping -S 10.0.0.1 www.google.com -- we receive the following expected output.
Code: [Select]
PING www.google.com (142.250.138.147) from 10.0.0.1: 56 data bytes
64 bytes from 142.250.138.147: icmp_seq=0 ttl=105 time=9.189 ms
64 bytes from 142.250.138.147: icmp_seq=1 ttl=105 time=9.136 ms
64 bytes from 142.250.138.147: icmp_seq=2 ttl=105 time=9.166 ms
64 bytes from 142.250.138.147: icmp_seq=3 ttl=105 time=9.067 ms
...
Again, this is successful with all explicitly-defined interfaces. The use of curl and other similar tools is also successful-- it fails when we don't explicitly specify an interface, and it succeeds when we do. Likewise, all machines that are behind the firewall have maintained their network connectivity, can reach out to the internet, and are otherwise operating normally. Tunnels that reach out to other datacenters are likewise operational.
When we looked at the firewall log live view while using the failed ping command (the one that doesn't specify an interface), we notice that the Source appears to be 0.0.0.0 for whatever reason. My guess is that this is the issue, but I don't really know how to resolve that.
As it stands, we can no longer check for updates from OPNSense itself-- both the console and the web UI have lost the ability to pull data externally. I will note that the console can still ping machines that are on the LAN so my guess is that this issue has something to do with the gateway itself (possibly), but we hadn't changed any of the settings prior to update.
We did execute a connectivity audit... it was painfully slow. The current output is as follows.
Code: [Select]
***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 22.7_4 (amd64/OpenSSL) at Wed Aug 10 04:12:52 UTC 2022
Checking connectivity for host: pkg.opnsense.org -> 89.149.211.205
PING 89.149.211.205 (89.149.211.205): 1500 data bytes
--- 89.149.211.205 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv4): https://pkg.opnsense.org/FreeBSD:13:amd64/22.7
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/meta.txz: Operation timed out
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/packagesite.pkg: Operation timed out
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/packagesite.txz: Operation timed out
Unable to update repository OPNsense
Error updating repositories!
Checking connectivity for host: pkg.opnsense.org -> 2001:1af8:4f00:a005:5::
ping: UDP connect: No route to host
Checking connectivity for repository (IPv6): https://pkg.opnsense.org/FreeBSD:13:amd64/22.7
Updating OPNsense repository catalogue...
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/meta.txz: Non-recoverable resolver failure
repository OPNsense has no meta file, using default settings
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/packagesite.pkg: Non-recoverable resolver failure
pkg: https://pkg.opnsense.org/FreeBSD:13:amd64/22.7/latest/packagesite.txz: Non-recoverable resolver failure
Unable to update repository OPNsense
Error updating repositories!
***DONE***
Any help that y'all could give on this would be phenomenal. Thanks in advance!