Cannot Traceroute WAN Connection?

[incirrata]

My environment has two Netgate XG-2758 firewalls; one is running OPNsense 18.1 and the other is still on PFsense. We also have two ISPs coming in. While both ISP WAN connections work great on the PFsense firewall, they do not work properly on OPNsense despite identical upstream gateway, netmask and IPs confirmed in our block. The gateways and interfaces do not appear to go down, the daemons don't seem to crash, there is nothing unusual in the logs as far as I can tell, but the IPsec VPN tunnel has a weird flickering problem and when I try to traceroute to the firewall it just hits the upstream gateway again and again.

I have already tried all of the following:

• A laptop using the same IP, netmask and gateway as the OPNsense firewall works as expected.
• I have tried using different IPs in our WAN blocks; same results.
• I have tried using different firewall interfaces as WAN; same results.
• I have tried connecting to only one ISP at a time; same results.
• I have tried setting up multi-WAN; same results.
• I have tried disabling IPsec, but this problem was evident from traceroute before IPsec was configured.
• Sticky connections is disabled.

At this point I am not sure what else to do. Does anyone have any idea how to fix this?

[CloudHoppingFlowerChild]

What is "a weird flickering problem" and what does "hits the upstream gateway again and again" mean?

Can you actually provide a sample output of a traceroute because your descriptions probably only make sense to you.

[incirrata]

Sure, here's an example of traceroute from behind the PFsense firewall (external IP would be X.X.X.74). I hope it will make sense to you. Traceroute from a different ISP network is more or less the same, it just takes 12 hops to get to "isp-upstream-fateway".

Code: [Select]

traceroute to OPNsense-firewall (X.X.X.75), 30 hops max, 60 byte packets
 1  PFsense-firewall (192.168.76.254)  0.220 ms  0.217 ms  0.213 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  isp-upstream-gateway (X.X.X.73)  3.376 ms  3.261 ms  3.259 ms
 7  isp-upstream-gateway (X.X.X.73) 3.861 ms  3.801 ms  3.851 ms
 8  isp-upstream-gateway (X.X.X.73) 4.415 ms  4.398 ms  4.385 ms
 9  isp-upstream-gateway (X.X.X.73) 4.926 ms  4.971 ms  4.961 ms
10  isp-upstream-gateway (X.X.X.73) 5.523 ms  5.559 ms  5.628 ms
11  isp-upstream-gateway (X.X.X.73) 6.155 ms  6.135 ms  6.108 ms
12  isp-upstream-gateway (X.X.X.73) 6.723 ms  6.719 ms  6.771 ms
13  isp-upstream-gateway (X.X.X.73) 7.355 ms  7.262 ms  7.298 ms
14  isp-upstream-gateway (X.X.X.73) 7.926 ms  7.795 ms  7.845 ms
15  isp-upstream-gateway (X.X.X.73) 8.461 ms  8.456 ms  8.511 ms
16  isp-upstream-gateway (X.X.X.73) 9.009 ms  9.167 ms  9.075 ms
17  isp-upstream-gateway (X.X.X.73) 9.690 ms  9.687 ms  9.681 ms
18  isp-upstream-gateway (X.X.X.73) 10.235 ms  10.233 ms  10.205 ms
19  isp-upstream-gateway (X.X.X.73) 10.838 ms  10.857 ms  10.856 ms
20  isp-upstream-gateway (X.X.X.73) 11.448 ms  11.441 ms  11.380 ms
21  isp-upstream-gateway (X.X.X.73) 11.894 ms  11.898 ms  11.895 ms
22  isp-upstream-gateway (X.X.X.73) 12.537 ms  12.518 ms  12.515 ms
23  isp-upstream-gateway (X.X.X.73) 13.125 ms  13.172 ms  13.049 ms
24  isp-upstream-gateway (X.X.X.73) 13.719 ms  13.671 ms  13.664 ms
25  isp-upstream-gateway (X.X.X.73) 14.246 ms  14.278 ms  14.271 ms
26  isp-upstream-gateway (X.X.X.73) 14.822 ms  14.849 ms  14.841 ms
27  isp-upstream-gateway (X.X.X.73) 15.406 ms  15.417 ms  15.415 ms
28  isp-upstream-gateway (X.X.X.73) 15.975 ms  15.880 ms  16.000 ms
29  isp-upstream-gateway (X.X.X.73) 16.536 ms  16.619 ms  16.567 ms
30  isp-upstream-gateway (X.X.X.73) 17.119 ms  17.032 ms  17.029 ms