Mailing List Archive

IOS-XR Vs. NTP in a duel to the death.
Okay so I've been trying to figure out exactly what IOS XR has a problem with regarding our internal NTP servers.

I was seeing things like this:

RP/0/RSP0/CPU0:Oct 31 15:18:37.422 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
RP/0/RSP0/CPU0:Oct 31 15:34:14.370 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
RP/0/RSP0/CPU0:Oct 31 16:09:25.511 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
RP/0/RSP0/CPU0:Oct 31 16:09:25.511 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
RP/0/RSP0/CPU0:Oct 31 16:30:30.792 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
RP/0/RSP0/CPU0:Oct 31 16:43:10.566 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
RP/0/RSP0/CPU0:Oct 31 16:43:10.566 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
RP/0/RSP0/CPU0:Oct 31 20:35:05.657 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
RP/0/RSP0/CPU0:Oct 31 21:10:08.789 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
RP/0/RSP0/CPU0:Oct 31 21:10:08.789 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
RP/0/RSP0/CPU0:Oct 31 21:45:00.912 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
RP/0/RSP0/CPU0:Oct 31 22:04:17.083 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
RP/0/RSP0/CPU0:Oct 31 22:04:17.083 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
RP/0/RSP0/CPU0:Oct 31 22:19:58.036 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
RP/0/RSP0/CPU0:Oct 31 22:51:33.959 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed

I did some research and Cisco states that the error means what it says: It can't keep the clock synced with the configured NTP server.

So I created 5 more NTP servers and configured it to use all 6 of them.

It is actually now having sync issues MORE often.

I am considering just suppressing the logs for this particular type of log message but before I do that, has anyone ever actually figured out what problem IOS XR has with standards compliant and fully functional NTP servers?

Is there a special way that you have to configure ntpd on a linux host to get this to.. stop?
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: IOS-XR Vs. NTP in a duel to the death. [ In reply to ]
On 2/11/21 2:01 am, Drew Weaver wrote:
> Okay so I've been trying to figure out exactly what IOS XR has a problem with regarding our internal NTP servers.
>
> I was seeing things like this:
>
> RP/0/RSP0/CPU0:Oct 31 15:18:37.422 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 15:34:14.370 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 16:09:25.511 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
> RP/0/RSP0/CPU0:Oct 31 16:09:25.511 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 16:30:30.792 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 16:43:10.566 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
> RP/0/RSP0/CPU0:Oct 31 16:43:10.566 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 20:35:05.657 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 21:10:08.789 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
> RP/0/RSP0/CPU0:Oct 31 21:10:08.789 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 21:45:00.912 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 22:04:17.083 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
> RP/0/RSP0/CPU0:Oct 31 22:04:17.083 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 22:19:58.036 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 22:51:33.959 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed
>
> I did some research and Cisco states that the error means what it says: It can't keep the clock synced with the configured NTP server.
>
> So I created 5 more NTP servers and configured it to use all 6 of them.
>
> It is actually now having sync issues MORE often.
>
> I am considering just suppressing the logs for this particular type of log message but before I do that, has anyone ever actually figured out what problem IOS XR has with standards compliant and fully functional NTP servers?
>
> Is there a special way that you have to configure ntpd on a linux host to get this to.. stop?

*which* NTPd?

The classic ntpd, the ntpsec fork, or something else?

I don't happen to have the full details to hand, but one OS upgrade
(classic, IIRC) ntpd stopped talking to (at least some of) the various
NTP-synced analog clocks I have, I ended up reducing the restrict on
them and things were much happier.

From a config file archive the relevant part of ntp.conf:

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Don't rate limit local subnet, NTP clock is a bad actor
restrict 10.0.0.0 mask 255.255.255.0 notrap nomodify nopeer
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: IOS-XR Vs. NTP in a duel to the death. [ In reply to ]
To answer your question:

ntpd 4.2.6p5 on RHEL 7.

I think what I have learned most of all in 2021 is that I don't really like any of Cisco's operating systems very much.

Thanks,
-Drew



-----Original Message-----
From: Julien Goodwin <jgoodwin@studio442.com.au>
Sent: Tuesday, November 2, 2021 4:04 AM
To: Drew Weaver <drew.weaver@thenap.com>; cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] IOS-XR Vs. NTP in a duel to the death.



On 2/11/21 2:01 am, Drew Weaver wrote:
> Okay so I've been trying to figure out exactly what IOS XR has a problem with regarding our internal NTP servers.
>
> I was seeing things like this:
>
> RP/0/RSP0/CPU0:Oct 31 15:18:37.422 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 15:34:14.370 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 16:09:25.511 EDT: ntpd[263]:
> %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed RP/0/RSP0/CPU0:Oct 31 16:09:25.511 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 16:30:30.792 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 16:43:10.566 EDT: ntpd[263]:
> %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed RP/0/RSP0/CPU0:Oct 31 16:43:10.566 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 20:35:05.657 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 21:10:08.789 EDT: ntpd[263]:
> %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed RP/0/RSP0/CPU0:Oct 31 21:10:08.789 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 21:45:00.912 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 22:04:17.083 EDT: ntpd[263]:
> %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed RP/0/RSP0/CPU0:Oct 31 22:04:17.083 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 22:19:58.036 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 22:51:33.959 EDT: ntpd[263]:
> %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System
> clock selection failed
>
> I did some research and Cisco states that the error means what it says: It can't keep the clock synced with the configured NTP server.
>
> So I created 5 more NTP servers and configured it to use all 6 of them.
>
> It is actually now having sync issues MORE often.
>
> I am considering just suppressing the logs for this particular type of log message but before I do that, has anyone ever actually figured out what problem IOS XR has with standards compliant and fully functional NTP servers?
>
> Is there a special way that you have to configure ntpd on a linux host to get this to.. stop?

*which* NTPd?

The classic ntpd, the ntpsec fork, or something else?

I don't happen to have the full details to hand, but one OS upgrade (classic, IIRC) ntpd stopped talking to (at least some of) the various NTP-synced analog clocks I have, I ended up reducing the restrict on them and things were much happier.

From a config file archive the relevant part of ntp.conf:

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Don't rate limit local subnet, NTP clock is a bad actor restrict 10.0.0.0 mask 255.255.255.0 notrap nomodify nopeer
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: IOS-XR Vs. NTP in a duel to the death. [ In reply to ]
I don't think you will get anywhere without actually capturing the
entire NTP traffic between the host and the NTP server and analyzing
it.

Lukas
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: IOS-XR Vs. NTP in a duel to the death. [ In reply to ]
After checking in multiple times throughout the day yesterday and seeing that the clock is synchronized still even though those error messages keep popping up I just filtered the log messages.

I don't really understand why IOS XR is so aggressive with NTP requests. All I am trying to do is sync the time for the logs. Sometimes the way Cisco does things is just hilarious.

Thanks,
-Drew

-----Original Message-----
From: Lukas Tribus <lukas@ltri.eu>
Sent: Tuesday, November 2, 2021 8:31 AM
To: Drew Weaver <drew.weaver@thenap.com>
Cc: Julien Goodwin <jgoodwin@studio442.com.au>; cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] IOS-XR Vs. NTP in a duel to the death.

I don't think you will get anywhere without actually capturing the entire NTP traffic between the host and the NTP server and analyzing it.

Lukas
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: IOS-XR Vs. NTP in a duel to the death. [ In reply to ]
I typically use 3 external as servers and then have the core peer with
themselves. I don't think that will help in this case but it does seem like
something is borked.


On Tue, Nov 2, 2021 at 8:42 AM Lukas Tribus <lukas@ltri.eu> wrote:

> I don't think you will get anywhere without actually capturing the
> entire NTP traffic between the host and the NTP server and analyzing
> it.
>
> Lukas
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: IOS-XR Vs. NTP in a duel to the death. [ In reply to ]
On Tue, 2 Nov 2021 at 09:00, Drew Weaver <drew.weaver@thenap.com> wrote:

> RP/0/RSP0/CPU0:Oct 31 15:18:37.422 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 2->3.
> RP/0/RSP0/CPU0:Oct 31 15:34:14.370 EDT: ntpd[263]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 3->2.
> RP/0/RSP0/CPU0:Oct 31 16:09:25.511 EDT: ntpd[263]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 192.168.123.6 : System clock selection failed

This may be CSCsz22456 or CSCvr17937. You can probably ignore it, but
it might go away if you upgrade. I do not think this communicates any
problem in your specific NTP setup.

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/