NTP-Server problem

Started by hansdampf, February 10, 2024, 12:01:16 PM

Previous topic - Next topic
After upgrading to 24.1 and 24.1_1 i encountred a strange problem with the ntpd:
Sometimes i got answers from the public servers, time got synced (Sync Source   2001:638:610:be (stratum 1, .PTB.)
But sometimes there was no sync. Looking at the logs i saw
2024-02-10T11:53:08 Informational ntpd failed to init interface for address fd00::225:90ff:fea8:83
2024-02-10T11:53:08 Error ntpd unable to create socket on em0 (7) for fd00::225:90ff:fea8:83#123
2024-02-10T11:53:08 Error ntpd bind(25) AF_INET6 fd00::225:90ff:fea8:83#123 flags 0x11 failed: Address already in use
2024-02-10T11:53:08 Informational ntpd failed to init interface for address fe80::225:90ff:fea8:83%1
2024-02-10T11:53:08 Error ntpd unable to create socket on em0 (6) for fe80::225:90ff:fea8:83%1#123
2024-02-10T11:53:08 Error ntpd bind(25) AF_INET6 fe80::225:90ff:fea8:83%1#123 flags 0x11 failed: Address already in use


So i stopped the daemon and realized that it was started twice: Had to stop it 2 times via the Stop-button.
After stopping and waiting a moment, i started it again and it works again and syncs.

I am on 24.1.4, this problem still exists!

I think these errors are caused by multiple ntpd daemons being launched. If you kill them all and then restart the logging errors stop. However the issue will return eventually. I do not know yet what causes this. My WAN constantly goes down or drops packets (this is the real silicon valley) and my primary suspicion is that this is related to a forced restart of the server when the WAN goes down. Not exactly sure because I don't see anything in the log that suggests a restart was attempted.

root@blah:/home/user # ps auxf|grep ntp
root    17315   0.0  0.1   23664    8620  -  Ss   13:59      0:11.72 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
root    18836   0.0  0.1   23260    8456  -  Ss   13:59      0:07.52 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
root    25015   0.0  0.1   23132    8420  -  Ss   13:57      0:11.02 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
root    26552   0.0  0.1   23664    8568  -  Ss   13:57      0:11.35 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
root    67431   0.0  0.1   23664    8604  -  Ss   13:59      0:11.26 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
root    87718   0.0  0.1   23664    8572  -  Ss   14:01      0:11.82 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
root    94110   0.0  0.0   12720    2384  0  S+   09:53      0:00.00 grep ntp

NTP does appear to be working so at the moment this is little more than an annoyance. But this is causing the system to log >80MB per day and makes the logs unreadable (without parsing) so it would be pretty useful to resolve this.

I'm not that famliar with FreeBSD (coming from Linux) but let me know where to look and I can collect some data.

Did you change the Interfaces setting to anything different from "All (recommended)"?
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)

I did: interfaces is currently set to LAN.

This can be one cause of what you are observing. If the LAN interface is not 100% stable like in a data centre environment, don't do that. There's a reason for "recommended" in that setting.
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)

The LAN connection should be stable. It's connected to a Cisco SG200 switch, good cable, short length, etc.

The WAN connection is not stable.

fwiw I am dual stack WAN and LAN w/ ipv6 set to tracking.

Can you explain what this setting is for? I thought I was setting which interfaces NTP would listen (and answer) on.

My intent was to prevent exposing my NTP to the world. (fwiw I do not have this port open on the fw).

I will flip this to the recommended value to see if it changes the behavior.

Quote from: wderousse on May 26, 2024, 09:06:02 PM
Can you explain what this setting is for? I thought I was setting which interfaces NTP would listen (and answer) on.
My intent was to prevent exposing my NTP to the world. (fwiw I do not have this port open on the fw).
I will flip this to the recommended value to see if it changes the behavior.
It is. But to not expose NTP to the world it is completely sufficient not to permit port 123 inbound on WAN via firewall rules.

Setting to "All" instructs ntpd to bind to the special address 0.0.0.0 aka INADDR_ANY. This address is stable no matter how interface come an go.

Binding to any particular interface is known to have ill effects specifically for BIND, don't know about ntpd, though, but I think it's always a good idea to bind to 100% stable addresses like either 127.0.0.1 or 0.0.0.0.

Kind regards,
Patrick
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)

OK changing to the default setting has cleared up the errors. It looks like NTP still tries to spawn multiple servers but the behavior is different (it gives up after a few tries rather than trying over and over):

2024-05-26T16:19:53-07:00   Error   ntpd   daemon child exited with code 1   
2024-05-26T16:19:53-07:00   Error   ntpd   unable to bind to wildcard address :: - another process may be running - EXITING   

This appears to be a limitation in the NTP server itself.

I think we can mark this resolved.

Thanks so much for your help!

June 17, 2024, 03:40:15 PM #9 Last Edit: June 17, 2024, 06:08:47 PM by zoechi
Just for anyone else landing here.

I get the same error when interface Vlan03 is included either by individually selecting interfaces or by unselecting all for "All (recommended)". It also doesn't matter if I only select the IPv4 or IPv6 variant.

I have several interfaces which are configured the same (just different IP and VLAN ID) and selecting these doesn't cause issues.

There is only one instance of ntpd running even when I select several interfaces.

Just for anyone else landing here.

I get the same error when interface Vlan03 is included either by individually selecting interfaces or by unselecting all for "All (recommended)". It also doesn't matter if I only select the IPv4 or IPv6 variant.

I have several interfaces which are configured the same (just different IP and VLAN ID) and selecting these doesn't cause issues.

There is only one instance of ntpd running even when I select several interfaces.

When Vlan03 is selected

[15:56:43] ops: <102>1 2024-06-17T15:53:37+02:00 firewall.example.com ntpd 12199 - [meta sequenceId="191"] Listen normally on 3 vlan03 [fe80::6662:66ff:fe21:833%11]:123
<99>1 2024-06-17T15:53:37+02:00 firewall.example.com ntpd 12199 - [meta sequenceId="192"] bind(24) AF_INET6 [fd00:0:0:23::]:123 flags 0x11 failed: Can't assign requested address


When Vlan03 is not selected
[15:59:21] ops: <102>1 2024-06-17T15:58:41+02:00 firewall.example.com ntpd 16772 - [meta sequenceId="143"] Listen normally on 0 lo0 [::1]:123
<102>1 2024-06-17T15:58:41+02:00 firewall.example.com ntpd 16772 - [meta sequenceId="144"] Listen normally on 1 lo0 127.0.0.1:123
<102>1 2024-06-17T15:58:41+02:00 firewall.example.com ntpd 16772 - [meta sequenceId="145"] Listen normally on 2 vlan02 10.22.0.254:123
<102>1 2024-06-17T15:58:41+02:00 firewall.example.com ntpd 16772 - [meta sequenceId="146"] Listen normally on 3 vlan02 [fe80::6662:66ff:fe21:833%10]:123
<102>1 2024-06-17T15:58:41+02:00 firewall.example.com ntpd 16772 - [meta sequenceId="147"] Listen normally on 4 vlan02 [fd00:0:0:22::]:123
<102>1 2024-06-17T15:58:41+02:00 firewall.example.com ntpd 16772 - [meta sequenceId="148"] Listen normally on 5 vlan02 [2a0d:xxxx:xxxx:xxxx:6662:66ff:fe21:833]:123



It looks like there is some duplication, but I haven't figured out yet what or why (this line `inet6 fd00:0:0:23:: prefixlen 64 duplicated`)

vlan03: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT (opt5)
        options=4100000<NETMAP,NOMAP>
        ether 64:62:66:21:08:33
        inet6 fe80::6662:66ff:fe21:833%vlan03 prefixlen 64 scopeid 0xb
        inet6 fd00:0:0:23:: prefixlen 64 duplicated
        inet6 2a0d:xxxx:xxxx:xxxx:6662:66ff:fe21:833 prefixlen 64
        inet 10.23.0.254 netmask 0xffff0000 broadcast 10.23.255.255
        groups: vlan
        vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: igb3
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


Disabling and enabling the interface assigned to vlan3 allowed me to start ntpd bound to all interfaces, but after a restart the error was back.

June 17, 2024, 03:57:26 PM #10 Last Edit: June 17, 2024, 04:04:25 PM by zoechi
deleted

I have the problem with version 24.1.9 seen.
The settings were set to "All (recommendet)" and it was reported to me that the IP address of the FW is already occupied by another service. The log showed that ntp sees the same IP address for ipsec and the internal LAN. After changing the settings on all ports except ipsec, the ntp service was able to start.

I had similar.....

I just binned NTPd by removing all time servers, and installed Chrony from plugins, changed the port to 123, and time.cloudflare.com......sorted. Works perfectly.