My upgrade experience so far

Started by edhill3, January 30, 2025, 11:46:55 AM

Previous topic - Next topic
January 30, 2025, 11:46:55 AM Last Edit: January 30, 2025, 12:28:59 PM by edhill3
I initiated the upgrade to 25.1 from the UI. I got the normal status messages but the update hung for hours.
I let to run overnight but still did not complete. Unable to SSH to console, I hooked up a monitor to directly login to console.
I could see a bunch of service stop processes hanging so I figured the auto reboot got stuck somehow so I ran the reboot command.
The reboot was successful and 25.1 is up and running now.

I'm observing slower web page loads on 25.1 vs 24.7. Seeing increased RTT on v25.1, not sure why yet.
I have identical rigs in HA setup. I updated the standby box to v25.1, main still on v24.7.

ping 9.9.9.9 I see a difference in RTT but interestingly enough ping 1.1.1.1 is roughly the same RTT. I don't know but I can definitely notice slower web page loads when using the v25.1 box. Admittedly, I didn't do this test between the two nodes before the upgrade so I'll revert the backup rig back to v24.7 to check for hardware issue.

v24.7:
PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data.
64 bytes from 9.9.9.9: icmp_seq=1 ttl=47 time=12.0 ms
64 bytes from 9.9.9.9: icmp_seq=2 ttl=47 time=11.9 ms
64 bytes from 9.9.9.9: icmp_seq=3 ttl=47 time=12.0 ms
64 bytes from 9.9.9.9: icmp_seq=4 ttl=47 time=12.0 ms
64 bytes from 9.9.9.9: icmp_seq=5 ttl=47 time=12.0 ms

--- 9.9.9.9 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 11.898/11.978/12.042/0.051 ms

v25.1:
ING 9.9.9.9 (9.9.9.9) 56(84) bytes of data.
64 bytes from 9.9.9.9: icmp_seq=1 ttl=50 time=62.4 ms
64 bytes from 9.9.9.9: icmp_seq=2 ttl=50 time=78.8 ms
64 bytes from 9.9.9.9: icmp_seq=3 ttl=50 time=66.1 ms
64 bytes from 9.9.9.9: icmp_seq=4 ttl=50 time=108 ms
64 bytes from 9.9.9.9: icmp_seq=5 ttl=50 time=71.0 ms

--- 9.9.9.9 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 62.418/77.319/108.224/16.400 ms


After reverting the backup rig to v24.7 I see the same increased RTT on ping to 9.9.9.9 so not an issue with v25.1 with respect to my observation. I suppose I never noticed before because I rarely run on the network on the backup.
Something else going on here I need to figure out. I was mistaken before - the backup rig has 8G RAM vs 16GB for the main box. Otherwise, identical CPU, storage and NICs.

Different TTL, likely some change in the path. Is that known/deliberate on your end? Traces might be enlightening.

Thanks @pfry. I did a ping followed by traceroute first with the primary firewall active followed by the same after failover to the backup firewall. Let me know if there are other commands I can use to troubleshoot. Both firewalls are in close proximity connected to the same switch device.