OPNsense Forum
Archive => 23.1 Legacy Series => Topic started by: gunnarf on July 17, 2023, 10:03:30 pm
-
Hi!
I've very well working ipv6, and one of the peers provided by pool.ntp.org happens to be a ipv6 server. But it never reaches Active or Candidate peer.
Status from the firewall in attached file
It is not of very big importance to have ipv6 peers, just a bit fun if it works
-
I use Chrony with NTS and that works against both Cloudflare and Netnod over IPv6 so I guess you can try switching to that
It's a safer protocol anyway so no downside afaik.
-
Do you have IPv6 connectivity? I can assure you that IPv6 NTP servers generally do work ;)
-
Did you have to set any rules in the firewall? I did a outbound allow NTP from WAN to any
I have very well functioning native IPv6.
-
No - there is an automatic floating rule named "let out anything from firewall host itself". That takes care of that. Generally you practically never need outbound rules on an interface.
Do you see any blocked NTP packets in the firewall live view?
-
No - there is an automatic floating rule named "let out anything from firewall host itself". That takes care of that. Generally you practically never need outbound rules on an interface.
Do you see any blocked NTP packets in the firewall live view?
I didn't find any filtering options in live view, so I could filter for port 123
Didn't see anything floating by.
For example DNS requests flows nicely through fw:
wan 2023-07-18T15:03:21 [2001:9b0:40::xxxx:xxxx]:29118 [2001:4860:4860::8888]:53 udp let out anything from firewall host itself (force gw)
-
What do you have set in Services > Network Time > General > Interfaces?
-
What do you have set in Services > Network Time > General > Interfaces?
LAN, WAN for some reason. Don't remind me changing that, so maybe default
Switched to WAN only
Ran ntpq -p on the firewall:
*gbg2.ntp.netnod .PPS. 1 u 8 64 17 10.053 -5.162 0.231
+mmo1.ntp.netnod .PPS. 1 u 9 64 17 11.270 -5.156 0.446
any.time.nl .INIT. 16 u - 64 0 0.000 +0.000 0.000
lul2.ntp.netnod .PPS. 1 u 7 64 17 13.938 -5.031 0.324
and then a ping6 on the IPv6 site:
root@OPNsense:~ # ping6 any.time.nl
PING6(56=40+8+8 bytes) 2001:9b0:40::967c:56c9 --> 2001:678:8::123
16 bytes from 2001:678:8::123, icmp_seq=0 hlim=48 time=283.389 ms
16 bytes from 2001:678:8::123, icmp_seq=1 hlim=48 time=283.196 ms
-
Leave it at All (recommended) and try again, please.
-
Leave it at All (recommended) and try again, please.
OK I reverted to LAN, WAN and added some ipv6 NTP servers. Only result is:
root@OPNsense:~ # ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
trf.clearnet.pw 65.21.63.130 3 u 9 64 7 42.221 -4.482 0.318
ntp5.flashdance 194.58.202.20 2 u 5 64 7 3.206 -2.867 0.272
ntp-b.0x5e.se .INIT. 16 u - 64 0 0.000 +0.000 0.000
2a01:4f8:c17:ef .INIT. 16 u - 64 0 0.000 +0.000 0.000
ntp2.time.nl .INIT. 16 u - 64 0 0.000 +0.000 0.000
-
What happens if you try to open a connection to the IPv6 server on port 123 with UDP with netcat or similar?
-
What happens if you try to open a connection to the IPv6 server on port 123 with UDP with netcat or similar?
trying. It just stands there. I'm not good at using nc I tried: nc -6uD ntp-b.0x5e.se 123
-
Please use "All (recommended)" and do not select any individual interfaces.
-
Please use "All (recommended)" and do not select any individual interfaces.
I do and the result is consistent No ipv6 peers
-
OK, do you see any requests going out on port 123 with tcpdump when you restart ntpd?
-
OK, do you see any requests going out on port 123 with tcpdump when you restart ntpd?
I ran tcpdump -v -i igb0 | grep NTP:
20:43:31.748331 IP6 (class 0xb8, hlim 64, next-header UDP (17) payload length: 56) h-2001-9b1-10d-39--1-bed3.na.bahnhof.se.ntp > ntp1.time.nl.ntp: [udp sum ok] NTPv4, length 48
20:43:47.660810 IP6 (flowlabel 0x10d00, hlim 63, next-header UDP (17) payload length: 56) h-2001-9b1-c395-d000-cf5-588-2910-7645.na.bahnhof.se.65048 > 2a01:b740:a30:4000::1f2.ntp: [udp sum ok] NTPv4, length 48
20:43:49.662217 IP6 (flowlabel 0xe0b00, hlim 63, next-header UDP (17) payload length: 56) h-2001-9b1-c395-d000-cf5-588-2910-7645.na.bahnhof.se.57976 > 2a01:b740:a30:4000::1f2.ntp: [udp sum ok] NTPv4, length 48
20:43:51.663633 IP6 (flowlabel 0x40e00, hlim 63, next-header UDP (17) payload length: 56) h-2001-9b1-c395-d000-cf5-588-2910-7645.na.bahnhof.se.62243 > 2a01:b740:a08:4000::1f2.ntp: [udp sum ok] NTPv4, length 48
198.235.24.175.50674 > h-82-196-108-106.A980.priv.bahnhof.se.ntp: NTPv4, length 48
h-82-196-108-106.A980.priv.bahnhof.se.ntp > sth1.ntp.netnod.se.ntp: NTPv4, length 48
sth1.ntp.netnod.se.ntp > h-82-196-108-106.A980.priv.bahnhof.se.ntp: NTPv4, length 48
h-82-196-108-106.A980.priv.bahnhof.se.ntp > ntp1.flashdance.cx.ntp: NTPv4, length 48
ntp1.flashdance.cx.ntp > h-82-196-108-106.A980.priv.bahnhof.se.ntp: NTPv4, length 48
20:44:35.791691 IP6 (class 0xb8, hlim 64, next-header UDP (17) payload length: 56) h-2001-9b1-10d-39--1-bed3.na.bahnhof.se.ntp > ntp1.time.nl.ntp: [udp sum ok] NTPv4, length 48
h-82-196-108-106.A980.priv.bahnhof.se.ntp > sth2.ntp.netnod.se.ntp: NTPv4, length 48
sth2.ntp.netnod.se.ntp > h-82-196-108-106.A980.priv.bahnhof.se.ntp: NTPv4, length 48
-
Could you do that with -n and show an ifconfig output of your WAN interface and the netstat -rn output?
-
Could you do that with -n and show an ifconfig output of your WAN interface and the netstat -rn output?
tcpdump: listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
82.196.108.106.123 > 147.78.228.41.123: NTPv4, length 48
147.78.228.41.123 > 82.196.108.106.123: NTPv4, length 48
09:56:45.292265 IP6 (flowlabel 0x533a5, hlim 63, next-header UDP (17) payload length: 56) 2001:9b1:c395:d000:e51:1ff:fee2:88b4.65486 > 2a01:b740:a08:3000::1f2.123: [udp sum ok] NTPv3, length 48
82.196.108.106.29485 > 120.25.115.20.123: NTPv4, length 48
120.25.115.20.123 > 82.196.108.106.29485: NTPv4, length 48
09:57:05.312632 IP6 (flowlabel 0x0f8ea, hlim 63, next-header UDP (17) payload length: 56) 2001:9b1:c395:d000:e51:1ff:fee2:88b4.65485 > 2a01:b740:a08:4000::1f2.123: [udp sum ok] NTPv3, length 48
82.196.108.106.19893 > 194.58.206.148.123: NTPv4, length 48
194.58.206.148.123 > 82.196.108.106.19893: NTPv4, length 48
09:57:25.333826 IP6 (flowlabel 0x47b58, hlim 63, next-header UDP (17) payload length: 56) 2001:9b1:c395:d000:e51:1ff:fee2:88b4.65484 > 2a01:b740:a08:3000::1f2.123: [udp sum ok] NTPv3, length 48
82.196.108.106.123 > 194.58.202.20.123: NTPv4, length 48
194.58.202.20.123 > 82.196.108.106.123: NTPv4, length 48
82.196.108.106.16979 > 216.239.35.8.123: NTPv4, length 48
82.196.108.106.35606 > 216.239.35.0.123: NTPv4, length 48
216.239.35.8.123 > 82.196.108.106.16979: NTPv4, length 48
216.239.35.0.123 > 82.196.108.106.35606: NTPv4, length 48
82.196.108.106.42605 > 216.239.35.4.123: NTPv4, length 48
root@OPNsense:~ # ifconfig igb0
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN (wan)
options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether 00:0d:b9:50:53:68
inet 82.196.108.106 netmask 0xffffffc0 broadcast 82.196.108.127
inet6 fe80::20d:b9ff:fe50:5368%igb0 prefixlen 64 scopeid 0x1
inet6 2001:9b1:10d:39::1:bed3 prefixlen 128
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
And this is just from ordinary running system. No restart of NTP service
-
Could you do that with -n and show an ifconfig output of your WAN interface and the netstat -rn output?
Here is my second firewall. Clearly no respons from the ipv6 NTP servers. And also the "bad udp cksum"!
10:10:35.177188 IP6 (class 0xb8, hlim 64, next-header UDP (17) payload length: 56) 2001:9b0:40::967c:56c9.123 > 2606:4700:f1::1.123: [bad udp cksum 0x8578 -> 0x089e!] NTPv4, length 48
10:10:38.170181 IP6 (class 0xb8, hlim 64, next-header UDP (17) payload length: 56) 2001:9b0:40::967c:56c9.123 > 2003:a:87f:c37c::1.123: [bad udp cksum 0x038a -> 0xa30b!] NTPv4, length 48
10:10:39.122574 IP6 (class 0xb8, hlim 64, next-header UDP (17) payload length: 56) 2001:9b0:40::967c:56c9.123 > 2a00:d78:0:712:94:198:159:10.123: [bad udp cksum 0x599f -> 0xf7bb!] NTPv4, length 48
46.59.40.76.123 > 91.209.0.19.123: NTPv4, length 48
91.209.0.19.123 > 46.59.40.76.123: NTPv4, length 48
and ifconfig from that box
root@OPNsense:~ # ifconfig igb0
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN (wan)
options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 00:0d:b9:51:6d:a8
inet 46.59.40.76 netmask 0xffffff00 broadcast 46.59.40.255
inet6 fe80::20d:b9ff:fe51:6da8%igb0 prefixlen 64 scopeid 0x1
inet6 2001:9b0:40::967c:56c9 prefixlen 128
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
-
The "bad checksum" can be an artifact of tcpdump itself. But it might be worth a try to disable hardware offloading for that interface.
Also what does ntpdate -q for these servers result in? Also no answer at all?
And last - is this OPNsense a hosted service or is "Bahnhof" the company responsible for the OPNsense? If hosted, is it remotely possibly they are blocking NTP? It could be abused for amplification attacks and many providers used to do this.
-
The "bad checksum" can be an artifact of tcpdump itself. But it might be worth a try to disable hardware offloading for that interface.
Also what does ntpdate -q for these servers result in? Also no answer at all?
And last - is this OPNsense a hosted service or is "Bahnhof" the company responsible for the OPNsense? If hosted, is it remotely possibly they are blocking NTP? It could be abused for amplification attacks and many providers used to do this.
The OPNsense fw's are mine. Bahnhof is my isp for both.
output from ntpdate -q on the said servers:
root@OPNsense:~ # ntpq -pnw
remote refid st t when poll reach delay offset jitter
==============================================================================
*91.209.0.19 194.58.204.148 2 u 60 64 377 13.413 -5.595 0.304
+194.58.205.20 .PPS. 1 u 59 64 377 7.151 -5.286 0.329
2606:4700:f1::1 .INIT. 16 u - 512 0 0.000 +0.000 0.000
2003:a:87f:c37c::1
.INIT. 16 u - 512 0 0.000 +0.000 0.000
2a00:d78:0:712:94:198:159:10
.INIT. 16 u - 512 0 0.000 +0.000 0.000
root@OPNsense:~ # ntpdate -q 2606:4700:f1::1
server 2606:4700:f1::1, stratum 3, offset -0.005963, delay 0.02797
19 Jul 10:35:44 ntpdate[6654]: adjust time server 2606:4700:f1::1 offset -0.005963 sec
root@OPNsense:~ # ntpdate -q 2003:a:87f:c37c::1
server 2003:a:87f:c37c::1, stratum 2, offset -0.007790, delay 0.05881
19 Jul 10:36:03 ntpdate[64091]: adjust time server 2003:a:87f:c37c::1 offset -0.007790 sec
root@OPNsense:~ # ntpdate -q 2a00:d78:0:712:94:198:159:10
server 2a00:d78:0:712:94:198:159:10, stratum 1, offset -0.007000, delay 0.05038
19 Jul 10:36:23 ntpdate[30047]: adjust time server 2a00:d78:0:712:94:198:159:10 offset -0.007000 sec
-
So ntpdate works but ntpd doesn't? WTF?
Ah ... one moment.
Are you running ntpdate -q as root? Can you verify with tcpdump that it is also using port 123 as the source port? If it doesn't, then that might hint at your ISP or somebody else blocking port 123 for IPv6.
Also, did you try disabling hardware offloading?
-
So ntpdate works but ntpd doesn't? WTF?
Ah ... one moment.
Are you running ntpdate -q as root? Can you verify with tcpdump that it is also using port 123 as the source port? If it doesn't, then that might hint at your ISP or somebody else blocking port 123 for IPv6.
Also, did you try disabling hardware offloading?
I don't know where to disable hardware offloading.
The login is as root, so Yes I'm running not-date -q as root
I'm waiting for my ISP to answer me on chat whether they are blocking 123 for ipv6 for some reason
-
See screenshot.
-
Thanks. Do I have to restart the fw for these settings to be disabled?
I rebooted the fw. will check after
-
After reboot the only difference is that in the status window for NTP it says .STEP. instead of .INIT. But still no contact
-
The only difference from before is that now the cksum says OK
46.59.40.76.34685 > 203.107.6.88.123: NTPv4, length 48
203.107.6.88.123 > 46.59.40.76.34685: NTPv4, length 48
11:26:47.165656 IP6 (class 0xb8, hlim 64, next-header UDP (17) payload length: 56) 2001:9b0:40::967c:56c9.123 > 2003:a:87f:c37c::4.123: [udp sum ok] NTPv4, length 48
11:26:59.193757 IP6 (class 0xb8, hlim 64, next-header UDP (17) payload length: 56) 2001:9b0:40::967c:56c9.123 > 2001:440:1880:7373::2.123: [udp sum ok] NTPv4, length 48
11:27:04.182473 IP6 (class 0xb8, hlim 64, next-header UDP (17) payload length: 56) 2001:9b0:40::967c:56c9.123 > 2a00:d78:0:712:94:198:159:10.123: [udp sum ok] NTPv4, length 48
46.59.40.76.123 > 194.58.207.20.123: NTPv4, length 48
194.58.207.20.123 > 46.59.40.76.123: NTPv4, length 48
-
Next step: your ISP.
-
I had a chat with the ISP. They are not blocking port 123
-
So possibly the NTP server in question does not like your source network? Definitely running out of ideas, now.
You could try the public NTP servers of Physikalisch-Technische Bundesanstalt, the official time source of Germany. They are open to the public, their only request is that one only points firewalls and other central systems at them and not each and every single desktop system.
ptbtime1.ptb.de: 2001:638:610:be01::108, 192.53.103.108
ptbtime2.ptb.de: 2001:638:610:be01::104, 192.53.103.104
ptbtime3.ptb.de: 2001:638:610:be01::103, 192.53.103.103
ptbtime4.ptb.de: 2001:638:610:cecf::7b, 194.94.95.123
-
I’ve tried a lot of servers including 2.pool.ntp.org and swedish Stupi servers, with the same result. So I guess nothing will be better with the ones you suggest. After restarting the fw I got new peers. Tried ntpdate -q with all these new servers. That works like a charm
-
You could DM me the output of "pfctl -s all" if you like.
-
You could DM me the output of "pfctl -s all" if you like.
Done
-
I will have a look at it after work.
-
I will have a look at it after work.
Did you get time to look at my pfctl output?
-
Sorry, took longer than expected. I cannot see anything obvious in the rules. You seem to have two states for packets sent out to NTP peers with your IPv6 address as the source, but no answer received:
all udp 2001:9b0:40::967c:56c9[123] -> 2003:a:87f:c37c::4[123] SINGLE:NO_TRAFFIC
all udp 2001:9b0:40::967c:56c9[123] -> 2001:440:1880:7373::2[123] SINGLE:NO_TRAFFIC
Have you looked at the firewall live view while e.g. restarting ntpd?
Kind regards,
Patrick
-
Have you looked at the firewall live view while e.g. restarting ntpd?
Kind regards,
Patrick
I'll give it a try. Is there some filtering options while watching? There is a lot of traffic going on, since the server is remote via vpn. Some live "grep" for wanted packages. Or can I record the session and watch in wireshark?
-
I'll make two quick suggestions:
1) For testing, remove all but one NTP source in your config, one of the PTB sources Patrick suggested earlier in the thread will suffice. Remove DNS of the equation as well, use on the IPv6 IP.
2) Consider NTS, all the PTB servers support it and a few others. There's no justification for UDP/123 over the Internet. This chrony directive can help where a battery is not present on the device and it is only used for the initial synchronization due to SSL constraints <ntsnocert>1</ntsnocert>
-
I'll make two quick suggestions:
1) For testing, remove all but one NTP source in your config, one of the PTB sources Patrick suggested earlier in the thread will suffice. Remove DNS of the equation as well, use on the IPv6 IP.
2) Consider NTS, all the PTB servers support it and a few others. There's no justification for UDP/123 over the Internet. This chrony directive can help where a battery is not present on the device and it is only used for the initial synchronization due to SSL constraints <ntsnocert>1</ntsnocert>
The only result I get from only setting up NTP-servers with only ipv6, is that I get no time sync at all.
And setting up NTPsec in OPNsense, that obviously doesn't support it, seems a little to much effort.Digging into setup files that only should be touched by OPNsense, seems a little bit too much interfering with the system for my taste.
-
Just as an experiment, I set up my laptop with Ubuntu, to check if I can get ipv6 NTP responds from there. I can not! Setting up ipv6-enabled NTP-servers in the ntp.conf gives that they stop at .INIT. No sync! When I change to ipv4 NTP-servers, I can immediately get things started.
And for fun I tried ntpdate -d against time.cloudflare.com
root@OPNsense:~ # ntpdate -d time.cloudflare.com
23 Jul 16:20:54 ntpdate[24260]: ntpdate 4.2.8p17@1.4004-o Wed Jun 21 00:58:29 UTC 2023 (1)
transmit(2606:4700:f1::1)
transmit(2606:4700:f1::123)
transmit(162.159.200.123)
receive(162.159.200.123)
transmit(162.159.200.1)
receive(162.159.200.1)
transmit(2606:4700:f1::1)
transmit(2606:4700:f1::123)
transmit(162.159.200.123)
receive(162.159.200.123)
transmit(162.159.200.1)
receive(162.159.200.1)
transmit(2606:4700:f1::1)
transmit(2606:4700:f1::123)
transmit(162.159.200.123)
receive(162.159.200.123)
transmit(162.159.200.1)
receive(162.159.200.1)
transmit(2606:4700:f1::1)
transmit(2606:4700:f1::123)
transmit(162.159.200.123)
receive(162.159.200.123)
transmit(162.159.200.1)
receive(162.159.200.1)
2606:4700:f1::1: Server dropped: no data
2606:4700:f1::123: Server dropped: no data
server 162.159.200.123, port 123
stratum 3, precision -25, leap 00, trust 000
refid [10.128.9.203], root delay 0.000763, root dispersion 0.000137
reference time: e867b598.6ca44811 Sun, Jul 23 2023 16:20:08.424
originate timestamp: e867b5cd.18a0dc10 Sun, Jul 23 2023 16:21:01.096
transmit timestamp: e867b5cd.185138a4 Sun, Jul 23 2023 16:21:01.094
filter delay: 0.02754 0.02740 0.02715 0.02718
---- ---- ---- ----
filter offset: +0.000395 +0.000298 +0.000386 +0.000387
---- ---- ---- ----
delay 0.02715, dispersion 0.00002, offset +0.000386
server 162.159.200.1, port 123
stratum 3, precision -25, leap 00, trust 000
refid [10.128.9.203], root delay 0.000748, root dispersion 0.000092
reference time: e867b5bc.f698a7b4 Sun, Jul 23 2023 16:20:44.963
originate timestamp: e867b5cd.4a023365 Sun, Jul 23 2023 16:21:01.289
transmit timestamp: e867b5cd.49ba9672 Sun, Jul 23 2023 16:21:01.288
filter delay: 0.02727 0.02716 0.02722 0.02719
---- ---- ---- ----
filter offset: +0.000294 +0.000235 +0.000236 +0.000256
---- ---- ---- ----
delay 0.02716, dispersion 0.00000, offset +0.000235
23 Jul 16:21:02 ntpdate[24260]: adjust time server 162.159.200.123 offset +0.000386 sec
-
Here are the steps for NTS on OPNsense:
1. Remove all NTP sources from Services-NetworkTIme-General - Save changes >> Service is now stopped.
2. Install os-chrony plugin
3. Configure Chrony, enable both NTS checkboxes, set port to 123, add preferred NTS Peers and Allowed Networks - Save changes
-
Here are the steps for NTS on OPNsense:
1. Remove all NTP sources from Services-NetworkTIme-General - Save changes >> Service is now stopped.
2. Install os-chrony plugin
3. Configure Chrony, enable both NTS checkboxes, set port to 123, add preferred NTS Peers and Allowed Networks - Save changes
That will do nothing. Tried with its on a raspberry pi. It is the fact that communication is ipv6 that stops it from working. As soon as I remove the ipv6-enabled NTP servers it works like a charm. So no meaning to install crony for NTS support.
-
If you can reproduce the issue with Ubuntu it's probably going back to your ISP ...
-
If you can reproduce the issue with Ubuntu it's probably going back to your ISP ...
I thought so too, but I had a conversation with my ISP technical support, and they claim they have no block on port 123 with ipv6. And as you saw, when I ran ntpdate -q to an ipv6 server, I got a connection Weird is the word. Maybe I should let them make a ticket on the issue, so they test by themselves?
-
I have found that ISP support usually doesn't know what they're talking about, especially when involving IPv6.
If you have access to a remote VM you can try something like netcat, etc and see if you can connect to it over 123. You can get some VMs for free but I'm not sure what the limitations are.
-
So now I got the answers from the ISP "expert". They say that it must be my firewall blocking port 123.
Just plain stupidity. So I'll leave it there. I'm glad I can run ipv6 on all other services.
>:(
-
Just wanted to add myself onto this..
using time.aws.com (opnsense is a bare metal protectli.. if it matters)
Using local unbound..
This happens and then I have to restart ntpd.. will probably switch to chrony but that brings other issues..
System / Settings / General / Prefer to use IPv4 even if IPv6 is available is checked as well
-
Just wanted to add myself onto this..
using time.aws.com (opnsense is a bare metal protectli.. if it matters)
Using local unbound..
This happens and then I have to restart ntpd.. will probably switch to chrony but that brings other issues..
System / Settings / General / Prefer to use IPv4 even if IPv6 is available is checked as well
Is there a reason you only have one time server?
I'm using the default pool entries along with having prefer ip4 checked and I see both ip6 and ip4 peers being used.
In the past I had a lot of issues with time sync because the default hwclock selection drifted too much. Once I fixed that by changing to a more stable clock it's all been working fine.