Show Posts
1
20.7 Legacy Series / How to troubleshoot odd IPv6 performance problem
« on: December 16, 2020, 03:44:11 am »
Hello,
I have a very odd performance problem that I'm struggling to track down, and I was wondering if anyone might suggest any diagnostic steps that might help me make sense of this.
In simple terms, when I enable IPv6 on OPNSense globally (in Firewall -> Settings -> Advanced) I see no problem, but when I setup DHCPv6 on the WAN and "Track Interface" on the LAN, I see a major but intermittent performance problem which degrades any audio traversing the link in a noticeable way. I've been able to simplify my test case and I can actually see the issue when I'm using OPNsense's own tools to ping it's own WAN IP. I'm using the web based diagnostic tool to ping the WAN IP address using the WAN interface, and here's what I see:
# /sbin/ping -S '10.0.0.86' -c '10' '10.0.0.86'
PING 10.0.0.86 (10.0.0.86) from 10.0.0.86: 56 data bytes
64 bytes from 10.0.0.86: icmp_seq=0 ttl=64 time=0.121 ms
64 bytes from 10.0.0.86: icmp_seq=1 ttl=64 time=0.093 ms
64 bytes from 10.0.0.86: icmp_seq=2 ttl=64 time=0.098 ms
64 bytes from 10.0.0.86: icmp_seq=3 ttl=64 time=0.110 ms
64 bytes from 10.0.0.86: icmp_seq=4 ttl=64 time=265.842 ms <--- OUCH
64 bytes from 10.0.0.86: icmp_seq=5 ttl=64 time=0.133 ms
64 bytes from 10.0.0.86: icmp_seq=6 ttl=64 time=0.081 ms
64 bytes from 10.0.0.86: icmp_seq=7 ttl=64 time=124.593 ms <--- OUCH
64 bytes from 10.0.0.86: icmp_seq=8 ttl=64 time=0.124 ms
64 bytes from 10.0.0.86: icmp_seq=9 ttl=64 time=0.090 ms
--- 10.0.0.86 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.081/39.129/265.842/84.193 ms
This isn't limited to IPv4, it's seen in IPv6 WAN IP pings too:
# /sbin/ping6 -S '2601:45:4003:4590:2e0:67ff:fe09:6ee8' -c '10' '2601:45:4003:4590:2e0:67ff:fe09:6ee8'
PING6(56=40+8+8 bytes) 2601:45:4003:4590:2e0:67ff:fe09:6ee8 --> 2601:45:4003:4590:2e0:67ff:fe09:6ee8
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=0 hlim=64 time=0.201 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=1 hlim=64 time=0.116 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=2 hlim=64 time=0.091 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=3 hlim=64 time=0.132 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=4 hlim=64 time=0.095 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=5 hlim=64 time=0.127 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=6 hlim=64 time=0.143 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=7 hlim=64 time=235.450 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=8 hlim=64 time=0.132 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=9 hlim=64 time=0.106 ms
--- 2601:45:4003:4590:2e0:67ff:fe09:6ee8 ping6 statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.091/23.659/235.450/70.597 ms
Those long pings correspond with the audio degradation. Sometimes I'll only see one in 10 packets, often I'll see two and occasionally even three. Strange problem, right? I can't recall the last time I ever saw a router or firewall have trouble talking to ITSELF!
Here's the kicker, this only happens when I have a Comcast/Xfinity cable modem in front of OPNSense operating in normal mode. If I bridge the cable modem/router combo they gave me, the performance problem DISAPPEARS! Seriously, ... but you'll note my test does not leave OPNsense so the Comcast/Xfinity device is not adding the delay, somehow it's internal to OPNSense.
I don't have any problems at all with IPv6 setup. It works as advertised. Xfinity delegates a /60 to me and I parcel it up into /64s (presently only using one of my 16 /64s for the LAN). I don't think I have a configuration issue but honestly, I'm really not sure how to investigate this. I feel I should report it, and I'm happy to dig into diagnostics anyone asks for, but I'm at a loss myself.
Anyone have ANY ideas on how to shine more light on this?
-Darren
I have a very odd performance problem that I'm struggling to track down, and I was wondering if anyone might suggest any diagnostic steps that might help me make sense of this.
In simple terms, when I enable IPv6 on OPNSense globally (in Firewall -> Settings -> Advanced) I see no problem, but when I setup DHCPv6 on the WAN and "Track Interface" on the LAN, I see a major but intermittent performance problem which degrades any audio traversing the link in a noticeable way. I've been able to simplify my test case and I can actually see the issue when I'm using OPNsense's own tools to ping it's own WAN IP. I'm using the web based diagnostic tool to ping the WAN IP address using the WAN interface, and here's what I see:
# /sbin/ping -S '10.0.0.86' -c '10' '10.0.0.86'
PING 10.0.0.86 (10.0.0.86) from 10.0.0.86: 56 data bytes
64 bytes from 10.0.0.86: icmp_seq=0 ttl=64 time=0.121 ms
64 bytes from 10.0.0.86: icmp_seq=1 ttl=64 time=0.093 ms
64 bytes from 10.0.0.86: icmp_seq=2 ttl=64 time=0.098 ms
64 bytes from 10.0.0.86: icmp_seq=3 ttl=64 time=0.110 ms
64 bytes from 10.0.0.86: icmp_seq=4 ttl=64 time=265.842 ms <--- OUCH
64 bytes from 10.0.0.86: icmp_seq=5 ttl=64 time=0.133 ms
64 bytes from 10.0.0.86: icmp_seq=6 ttl=64 time=0.081 ms
64 bytes from 10.0.0.86: icmp_seq=7 ttl=64 time=124.593 ms <--- OUCH
64 bytes from 10.0.0.86: icmp_seq=8 ttl=64 time=0.124 ms
64 bytes from 10.0.0.86: icmp_seq=9 ttl=64 time=0.090 ms
--- 10.0.0.86 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.081/39.129/265.842/84.193 ms
This isn't limited to IPv4, it's seen in IPv6 WAN IP pings too:
# /sbin/ping6 -S '2601:45:4003:4590:2e0:67ff:fe09:6ee8' -c '10' '2601:45:4003:4590:2e0:67ff:fe09:6ee8'
PING6(56=40+8+8 bytes) 2601:45:4003:4590:2e0:67ff:fe09:6ee8 --> 2601:45:4003:4590:2e0:67ff:fe09:6ee8
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=0 hlim=64 time=0.201 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=1 hlim=64 time=0.116 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=2 hlim=64 time=0.091 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=3 hlim=64 time=0.132 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=4 hlim=64 time=0.095 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=5 hlim=64 time=0.127 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=6 hlim=64 time=0.143 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=7 hlim=64 time=235.450 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=8 hlim=64 time=0.132 ms
16 bytes from 2601:45:4003:4590:2e0:67ff:fe09:6ee8, icmp_seq=9 hlim=64 time=0.106 ms
--- 2601:45:4003:4590:2e0:67ff:fe09:6ee8 ping6 statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.091/23.659/235.450/70.597 ms
Those long pings correspond with the audio degradation. Sometimes I'll only see one in 10 packets, often I'll see two and occasionally even three. Strange problem, right? I can't recall the last time I ever saw a router or firewall have trouble talking to ITSELF!
Here's the kicker, this only happens when I have a Comcast/Xfinity cable modem in front of OPNSense operating in normal mode. If I bridge the cable modem/router combo they gave me, the performance problem DISAPPEARS! Seriously, ... but you'll note my test does not leave OPNsense so the Comcast/Xfinity device is not adding the delay, somehow it's internal to OPNSense.
I don't have any problems at all with IPv6 setup. It works as advertised. Xfinity delegates a /60 to me and I parcel it up into /64s (presently only using one of my 16 /64s for the LAN). I don't think I have a configuration issue but honestly, I'm really not sure how to investigate this. I feel I should report it, and I'm happy to dig into diagnostics anyone asks for, but I'm at a loss myself.
Anyone have ANY ideas on how to shine more light on this?
-Darren