Mailing List Archive

----- Original Message -----
> From: "Forrest Christian (List Account)" <lists@packetflux.com>

> Let me address your points:
[ ... ]
> Let's assume you have a typical GPS-derived NTP server using a typical
> commercially available timing GNSS module. To convince that receiver that
> it was a different time, I'd need to have an SDR that would operate in the
> GPS band. These are widely available for under $500. You'd also need a
> laptop and a download of a GPS simulator from GitLab. With a total
> investment of $500 (assuming I already have a laptop), I now have a system
> that can generate a GPS signal to convince your GPS receiver that it's any
> time at all. If you're a tech neophyte, there are youtube videos on how to
> do this.
>
> All I need to do now is add appropriate antennas and/or amplifiers to
> overcome the official GNSS signals. As you pointed out, depending on the
> location and directivity of your antenna, this is either trivial or becomes
> slightly more difficult. If I can see your antenna, it becomes a lot
> cheaper as I just need a relatively low-powered amplifier and a highly
> directional antenna. If I can't see your antenna, I would opt for a
> higher-power amplifier and a less directional transmit antenna to blanket a
> wide area with the spoofed signal.

If I'm trying to get time out of a NAVSTAR (yes, I know, shut up) receiver,
it can see like 8-20 birds, right? Is there not some voting and such inside
such a receiver? Just letting it see one 'bird' with spoofed time doesn't
seem like it ought to work, to me; what don't I know?

Cheers,
-- jra
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

If I'm spoofing time, I'm going to produce an entire constellation of
satellites. That is, I'm going to provide a signal which looks like all
of the satellites in view providing their timing signals on whatever time I
want your GPS receiver to think it is. All I have to do is ensure that
your receiver receives my signal loud enough that it thinks the real
satellites are noise, and my signal is the real one.

This isn't that hard to accomplish, especially since there are youtube
videos showing you how.

On Sun, Aug 13, 2023 at 6:03?PM Jay R. Ashworth <jra@baylink.com> wrote:

> ----- Original Message -----
> > From: "Forrest Christian (List Account)" <lists@packetflux.com>
>
> > Let me address your points:
> [ ... ]
> > Let's assume you have a typical GPS-derived NTP server using a typical
> > commercially available timing GNSS module. To convince that receiver
> that
> > it was a different time, I'd need to have an SDR that would operate in
> the
> > GPS band. These are widely available for under $500. You'd also need a
> > laptop and a download of a GPS simulator from GitLab. With a total
> > investment of $500 (assuming I already have a laptop), I now have a
> system
> > that can generate a GPS signal to convince your GPS receiver that it's
> any
> > time at all. If you're a tech neophyte, there are youtube videos on how
> to
> > do this.
> >
> > All I need to do now is add appropriate antennas and/or amplifiers to
> > overcome the official GNSS signals. As you pointed out, depending on
> the
> > location and directivity of your antenna, this is either trivial or
> becomes
> > slightly more difficult. If I can see your antenna, it becomes a lot
> > cheaper as I just need a relatively low-powered amplifier and a highly
> > directional antenna. If I can't see your antenna, I would opt for a
> > higher-power amplifier and a less directional transmit antenna to
> blanket a
> > wide area with the spoofed signal.
>
> If I'm trying to get time out of a NAVSTAR (yes, I know, shut up) receiver,
> it can see like 8-20 birds, right? Is there not some voting and such
> inside
> such a receiver? Just letting it see one 'bird' with spoofed time doesn't
> seem like it ought to work, to me; what don't I know?
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink
> jra@baylink.com
> Designer The Things I Think RFC
> 2100
> Ashworth & Associates http://www.bcp38.info 2000 Land
> Rover DII
> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647
> 1274
>


--
- Forrest

Gotcha. The Bad Guys are smarter than me. :-)

Cheers,
-- jra

----- Original Message -----
> From: "Forrest Christian (List Account)" <lists@packetflux.com>
> To: "jra" <jra@baylink.com>
> Cc: "nanog list" <nanog@nanog.org>
> Sent: Sunday, August 13, 2023 8:06:30 PM
> Subject: Re: NTP Sync Issue Across Tata (Europe)

> If I'm spoofing time, I'm going to produce an entire constellation of
> satellites. That is, I'm going to provide a signal which looks like all
> of the satellites in view providing their timing signals on whatever time I
> want your GPS receiver to think it is. All I have to do is ensure that
> your receiver receives my signal loud enough that it thinks the real
> satellites are noise, and my signal is the real one.
>
> This isn't that hard to accomplish, especially since there are youtube
> videos showing you how.
>
> On Sun, Aug 13, 2023 at 6:03?PM Jay R. Ashworth <jra@baylink.com> wrote:
>
>> ----- Original Message -----
>> > From: "Forrest Christian (List Account)" <lists@packetflux.com>
>>
>> > Let me address your points:
>> [ ... ]
>> > Let's assume you have a typical GPS-derived NTP server using a typical
>> > commercially available timing GNSS module. To convince that receiver
>> that
>> > it was a different time, I'd need to have an SDR that would operate in
>> the
>> > GPS band. These are widely available for under $500. You'd also need a
>> > laptop and a download of a GPS simulator from GitLab. With a total
>> > investment of $500 (assuming I already have a laptop), I now have a
>> system
>> > that can generate a GPS signal to convince your GPS receiver that it's
>> any
>> > time at all. If you're a tech neophyte, there are youtube videos on how
>> to
>> > do this.
>> >
>> > All I need to do now is add appropriate antennas and/or amplifiers to
>> > overcome the official GNSS signals. As you pointed out, depending on
>> the
>> > location and directivity of your antenna, this is either trivial or
>> becomes
>> > slightly more difficult. If I can see your antenna, it becomes a lot
>> > cheaper as I just need a relatively low-powered amplifier and a highly
>> > directional antenna. If I can't see your antenna, I would opt for a
>> > higher-power amplifier and a less directional transmit antenna to
>> blanket a
>> > wide area with the spoofed signal.
>>
>> If I'm trying to get time out of a NAVSTAR (yes, I know, shut up) receiver,
>> it can see like 8-20 birds, right? Is there not some voting and such
>> inside
>> such a receiver? Just letting it see one 'bird' with spoofed time doesn't
>> seem like it ought to work, to me; what don't I know?
>>
>> Cheers,
>> -- jra
>> --
>> Jay R. Ashworth Baylink
>> jra@baylink.com
>> Designer The Things I Think RFC
>> 2100
>> Ashworth & Associates http://www.bcp38.info 2000 Land
>> Rover DII
>> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647
>> 1274
>>
>
>
> --
> - Forrest

--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

" As such, the ultimate (a little expensive) solution is to have
your own Rb clocks locally."


Yeah, that's a reasonable course of action for most networks. *sigh*



-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

----- Original Message -----

From: "Masataka Ohta" <mohta@necom830.hpcl.titech.ac.jp>
To: nanog@nanog.org
Sent: Friday, August 11, 2023 4:33:20 AM
Subject: Re: NTP Sync Issue Across Tata (Europe)

Forrest Christian (List Account) wrote:

> The recommendation tends to be the following:
>
> 1) Run your GPS-derived NTP appliances, but DO NOT point end-user
> clients at it. 2) Run a set of internal NTPd servers, and configure
> them to pull time from all of your GPS-derived NTP servers, AND
> trusted public NTP servers 3) Point your clients at the internal NTPd
> servers.

That is not a very good recommendation. See below.

> At some point, using publicly available NTP sources is redundant
> unless one wants to mitigate away the risks behind failure of the GPS
> system itself.

Your assumption that public NTP servers were not GPS-derived NTP
servers is just wrong.

> What I'm advocating against is the seemingly common practice to go
> buy an off-the-shelf lower-cost GPS-NTP appliance (under $1K or so),
> stick an antenna in a window or maybe on the rooftop, and point all
> your devices at that device.

Relying on a local expensive GPS appliance does not improve
security so much and is the worst thing to do.

But, additionally relying on remote servers (including those
provided by NIST) is subject to DOS attacks.

As such, the ultimate (a little expensive) solution is to have
your own Rb clocks locally.

Masataka Ohta

I've responded in bits and pieces to this thread and haven't done an
excellent job expressing my overall opinion. This is probably because my
initial goal was to point out that GPS-transmitted time is no less subject
to being attacked than your garden variety NTP-transmitted time. Since this
thread has evolved, I'd like to describe my overall position to be a bit
clearer.

To start, we need a somewhat simplified version of how UTC is created so I
can refer to it later:

Across the globe, approximately 85 research and standards institutions run
a set of freestanding atomic clocks that contribute to UTC. The number of
atomic clocks across all these institutions totals around 450. Each
institution also produces a version of UTC based on its own set of
atomic clocks. In the international timekeeping world, this is designated
as UTC(Laboratory), where Laboratory is replaced with the abbreviation for
the lab producing that version of UTC. So UTC(NIST) is the version that
NIST produces at Boulder, Colorado, NICT produces UTC(NICT) in Tokyo, and
so on.

Because no clock is perfectly accurate, all of these versions of UTC drift
in relation to each other, and you could have significant differences in
time between different labs. As a result, there has to be a way to
synchronize them. Each month, the standards organization BIPM collects
relative time measurements and other statistics from each
institution described above. This data is then used to determine the
actual value of UTC. BIPM then produces a report detailing each
organization's difference from the correct representation of UTC. Each
institution uses this data to adjust its UTC representation, and the cycle
repeats the next month. In this way, all of the representations of UTC end
up being pretty close to each other. The document BIPM produces is titled
"Circular T." The most recent version indicates that most of the
significant standards institutions maintain a UTC version that differs by
less than 10ns from the official version of UTC.

Note that 10ns is far more accurate than we need for NTP, so most of the
UTC representations can be considered identical as far as this discussion
goes. Still, it is essential to realize that UTC(NIST) is generated
separately from UTC(USNO) or other UTC implementations. For example, a
UTC(NIST) failure should not cause UTC(USNO) to fail as they utilize
separate hardware and systems.

Each of these versions of UTC is also disseminated in various ways.
UTC(NIST) goes out via the "WWV" radio stations, NTP, and other esoteric
methods. GPS primarily distributes UTC(USNO), which is also available
directly via NTP. UTC(SU) is the timescale for GLONASS. And so on.

So, back to NTP and the accuracy required:

Most end users (people running everyday web applications or streaming video
or similar) don't need precisely synchronized time. The most sensitive
application I'm aware of in this space is likely TOTP, which often needs
time on the server and time on the client (or hardware key) within 90
seconds of each other. In addition, having NTP time fail usually isn't
the end of the world for these users. The best way to synchronize their
computers (including desktop and server systems) to UTC is to point their
computer time synchronization service (whatever that is) at pool.ntp.org,
time.windows.com, their ISP's time server, or similar. Or, with
modern OS'es, you can leave the time configured to whatever server the OS
manufacturer preconfigured. As an aside, one should note that
historically windows ticked at 15ms or so, so trying to synchronize most
windows closer than 15ms was futile.

On the other hand, large ISPs or other service providers (including content
providers) see real benefits to having systems synchronized to fractions of
seconds of UTC. Comparing logs and traces becomes much easier when you
know that something logged at 10:02:23.1 on one device came before
something logged at 10:02:23.5 on another. Various server-to-server
protocols and software implementations need time to be synchronized to
sub-second intervals since they rely on timestamps to determine the latest
copy of data, and so on. In addition, as an ISP, you'll often provide
time services to downstream customers who demand more accuracy and
reliability than is strictly necessary.

As a result, one wants to ensure that all time servers are synchronized
within some reasonable standard of accuracy. Within 100ms is acceptable
for most applications but a goal of under 50ms is better. If you have
local GPS receivers, times down to around 1ms is achievable with careful
design. Beyond that, you're chasing unnecessary accuracy. Note that loss
of precision is somewhat cumulative here - running a time server
synchronized to within 100ms will ensure that no client can be synchronized
to better than within 100ms from that server. Generally, you'll want your
time server to be synchronized much better than needed to avoid the time
server being the limiting factor.

In a perfect world with no bad actors and where all links ran perfectly,
one could set up an NTP server that pulled from pool.ntp.org or used GPS
and essentially acted as a proxy. Unfortunately, we don't live in this
world. So one has to ask how you build a system that meets at least the
following goals:

* Synchronized to UTC within 50ms, with lower being better.
* Not subject to a reasonable set of attacks (typical DoS attacks, RF
signal attacks, spoofing, etc).
* Able to be run by typical network operations staff

In addition, an ideal server setup would be made up of redundant servers in
case one piece of hardware fails. I will ignore this part, as it's usually
just setting up multiple copies of the same thing.

The two most straightforward options are using a GPS-based NTP appliance or
installing an NTP server and pointing it at pool.ntp.org. Under normal
circumstances, both options will be synchronized to UTC with enough
accuracy for most applications, and both are easy to run by typical network
operations staff. This assumes reasonably consistent network latency in
the NTP case and a good sky view in the GPS case. The GPS-based appliance
is, however, subject to spoofing or jamming, as I've discussed earlier.
The NTP server is at the mercy of the quality of the servers it picked
from pool.ntp.org and is also subject to various outside attacks (spoofing,
etc.). One must decide how critical time is to them before deciding
whether this option is valid.

The other end of the scale is the "develop your own offline version of UTC
using atomic clocks" methodology. This fixes the attack issue but
introduces several others. The main one is that you are now relying on
the clock's accuracy. Admittedly rubidium and especially cesium clocks
tend to be sufficiently reliable and stable. However, one has to ensure
the frequency is accurate initially and stays that way. You must also wire
the clock to an NTP Server and calibrate the initial UTC offset. If the
clock goes haywire or is less accurate than is required, your in-house
version of UTC will drift in relation to real UTC. This means you may
need 2 or 3 or more atomic clocks to be sufficiently reliable. You'll then
need to regularly take an average, compare it to UTC, and adjust if it's
drifted too much. This quickly becomes more of a science project than
something you want network operations staff to deal with on an ongoing
basis. To be clear: If you need robust time not subject to outside
forces and have or can obtain the skill set to pull this off internally, I
won't argue that this is a bad option. However, I feel this isn't the type
of service most providers want to run internally.

So, looking at some middle-ground options that trade a bit of robustness
for ease of use is reasonable.

My lowest cost preference has always been to use a set of in-house NTP
servers pointed at a carefully curated collection of NTP servers. Your
curation strategy should depend on network connectivity, the reliability of
the time sources, etc. In North America, picking one or two NIST servers
from each NIST location is a good starting point. That is one or two from
each of Maryland, Fort Collins, Boulder, and the University of Colorado.
One may want to add some servers from other timekeeping organizations
(such as USNO). Note that there is one commonality: These time servers
are run by organizations listed in circular T as contributing to UTC, and
the servers are tied to the atomic clocks. That way, we ensure that the
servers are not subject to inaccuracies caused by time transfer from an
authoritative source for UTC. What is left is any potential attack on the
time transfer over NTP itself. I would argue that with a curated list of
enough NTP servers, this risk can be pushed down to where it is low enough
for many use cases. A lot will depend on the quantity and quality of NTP
servers you select and the robustness of the network path to those
servers. If the packets between your NTP server and the NTP servers you
choose traverse a relatively secure and short path with plenty of
bandwidth, and the paths to differing NTP servers are diverse, many attacks
will become harder to implement. In addition, the more NTP servers you
add, the more likely it is that NTP will be able to correctly pick the
servers providing the correct time, even if an attacker is successfully
spoofing one or more sources. In some cases it may make sense to add
additional servers which are run by third parties if it gains additional
robustness based on network architecture. This is especially true if
you're closely connected network-wise with the third party and they run a
good quality NTP service as well.

As I've mentioned, a good middle-of-the-road solution is adding various
sources of time derived via GPS. Note I said, "to add." Start with the
carefully curated NTP server set, then install one or more GPS-based NTP
Servers polled by your NTP server. Adding these GPS time sources to your
NTP servers does three things: First, it provides another source of time
NTP can use to determine the correct time. Second, we're now using a
different time transmission method with different vulnerabilities. And
finally, it will significantly improve the accuracy of the time the NTP
server produces as NTPd will generally prefer it to do the final trimming
to UTC. The strength of the combination of both terrestrial transmitted
time via NTP and the precision of rf-transmitted GPS time ensures that time
is both correct and precise. There are still attack vectors here, but as
you add more time sources, the complexity of pulling off a successful
attack increases. This is especially true if you can monitor the NTP
server for signs of stress, such as time servers that are not telling the
correct time or GPS signals which are inconsistent with the NTP-derived
time. A successful attack would require simultaneous NTP (network) and
GPS (rf) attacks.

Other options or blends of options are also possible. With a reasonably
large network, putting enough GPS receivers into place would significantly
reduce the possibility of a spoofer or jammer taking out your entire GPS
infrastructure. Reducing or eliminating external NTP time sources might be
reasonable in that case. The theory is that attacking GPS receivers at
one location is easy. Doing it at dozens simultaneously is much more
difficult. To use an exaggeration to make a point: If you had 100
different GPS receivers spread across 100 widely geographically diverse
locations, and all of your NTP servers were able to poll all of them for
time, the chances that an attacker would be able to take out or spoof
enough GPS receivers to make a difference would be close to zero. Your
failure point becomes UTC(USNO) and the GPS constellation itself. The same
argument would apply to NTP servers regarding quantity and diversity.

Other options involve adding additional technologies. For example, some
appliances use GPS to discipline (adjust) an internal atomic clock. Once
the atomic clock is locked to UTC, the GPS can fail for extended periods
without affecting NTP output. In addition, some of these will filter
updates from the GPS based on the appliance's internal atomic time. That
way, a spoofer would be ignored, jammers would have to continue for hours
or days, and so on. Of course, these solutions' reliability depends on
the implementation quality. If I had the budget to implement something
like this in a network, I'd likely scatter a few of these around the
network and then still use garden variety NTPd servers which would be
pointed at these appliances. I might even consider buying solutions from
multiple vendors to ensure a bug in one implementation was filtered out and
ignored.

I can't cover every option here, but balancing security, cost, operational
complexity, and application needs is the key. Some solutions are cheap
and easy but not robust. Some are highly robust but expensive and not
easy. Somewhere in the middle is probably where most real implementations
should lie.

Now, to address a couple of specific items:

1) Additional GPS and commercial time distribution systems will likely
improve reliability. However, only GPS and GALILEO are available for free
in the US. I'm ignoring GLONASS for various legal and political reasons.
GALILEO is a valid option but it lives in the same band as GPS, so jamming
GPS will usually also jam GALILEO. Utilizing GNSS receivers that use the
civilian signals in the newer bands would also help. Some commercial
solutions are available that don't require GNSS, but they're relatively new
and not as commonly available as one would like.

2) For running my own time servers in a service-provider environment, I'd
rather specifically designate the exact NTP server I want to utilize and
not rely on a third party to give me a pool of servers. It's more about
ensuring the server I use is running a trusted server, and if I delegate
the server selection, I lose this ability. On the other hand, where I'm
not running a NTP server that is critical for many clients, I'll just point
it at pool.ntp.org, or north-america.pool.ntp.org and skip all of the
recommendations that I've made above. I would be cautious about
requesting pool.ntp.org add entries for "stratum of server" or "origin of
time" as this seems like it would tend to overload the stratum one servers
in the pool with people "optimizing" their configuration to use only
stratum one servers. Remember that pool.ntp.org is generally intended as
an end-user-device service, and providing methods that end users can bypass
the robustness that a fully distributed pool will provide is probably not a
great idea.

3) This all should hopefully sort itself out over the next few years. GPS
and GALILEO are flying new birds that have changes designed to improve
attack resilience by using cryptography to ensure authentic transmissions
(which may rely on ground transmission of cryptographic keys). NTP
already supports manual cryptographic keys that work, but NTS is a pain in
the rear. Hopefully, NTPv5 will have a better security mechanism. Other,
more secure, time sources are on the horizon as the cybersecurity crowd is
aware of the issues.

And finally, as a sort of a tl;dr; Summary: Each operator needs to decide
how critical time is to their network and pick a solution that works for
them and fits the organization's budget. Some operators might point
everything at pool.ntp.org and not run their own servers. Others might run
their own time lab and use that time to provide NTP time and precision time
and frequency via various methods. Most will be somewhere in between. But
regardless of which you choose, please be aware that GPS isn't 100% secure,
and neither is NTP. If attack resilience matters to you, you should think
about all of the attack vectors and design something that is robust enough
to meet your use case.

Mike Hammett wrote:

> " As such, the ultimate (a little expensive) solution is to have
> your own Rb clocks locally."

> Yeah, that's a reasonable course of action for most networks.

For most data centers with time sensitive transactions, at least.

> *sigh*

https://en.wikipedia.org/wiki/Atomic_clock
Modern rubidium standard tubes last more than ten years,
and can cost as little as US$50.

https://www.ebay.com/sch/i.html?_nkw=rubidium

Masataka Ohta

Forrest,

I think you’re gilding the lilly. My original recommendation was to use GPS as primary, for its superior accuracy and resistance to attack, and have anti-GPS back up. If you want automatic fail over, do that in an intermediate server on your site that makes a conscious test and decision to fail over to Internet NTP.

You’re mistaken to say that the vulnerability of GPS is remotely comparable to the vulnerability of Internet-based NTP. To interfere with a GPS-derived clock, an attacker has to physically be present. That’s a huge expense — and risk — that hackers are really not interested in undertaking. They would much rather sit in Russia or China and attack NTP servers remotely using any of the several attack methodologies I’ve cited previously.

So curate Internet NTP or not (personally, that seems like just another thing to monitor and maintain), but make GPS your primary time standard. You’re much better off staying air-gapped from Internet NTP until you detect a GPS failure. All the other machinations are pointless while GPS is working, because GPS gives you by far the best accuracy and security for the buck. Like I said, spend $400 on a commercial GPS time server and timing problems are solved. Or use facility-provided GPS if you can’t get an antenna up.

-mel

On Aug 14, 2023, at 12:10 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:

?
I've responded in bits and pieces to this thread and haven't done an excellent job expressing my overall opinion. This is probably because my initial goal was to point out that GPS-transmitted time is no less subject to being attacked than your garden variety NTP-transmitted time. Since this thread has evolved, I'd like to describe my overall position to be a bit clearer.

To start, we need a somewhat simplified version of how UTC is created so I can refer to it later:

Across the globe, approximately 85 research and standards institutions run a set of freestanding atomic clocks that contribute to UTC. The number of atomic clocks across all these institutions totals around 450. Each institution also produces a version of UTC based on its own set of atomic clocks. In the international timekeeping world, this is designated as UTC(Laboratory), where Laboratory is replaced with the abbreviation for the lab producing that version of UTC. So UTC(NIST) is the version that NIST produces at Boulder, Colorado, NICT produces UTC(NICT) in Tokyo, and so on.

Because no clock is perfectly accurate, all of these versions of UTC drift in relation to each other, and you could have significant differences in time between different labs. As a result, there has to be a way to synchronize them. Each month, the standards organization BIPM collects relative time measurements and other statistics from each institution described above. This data is then used to determine the actual value of UTC. BIPM then produces a report detailing each organization's difference from the correct representation of UTC. Each institution uses this data to adjust its UTC representation, and the cycle repeats the next month. In this way, all of the representations of UTC end up being pretty close to each other. The document BIPM produces is titled "Circular T." The most recent version indicates that most of the significant standards institutions maintain a UTC version that differs by less than 10ns from the official version of UTC.

Note that 10ns is far more accurate than we need for NTP, so most of the UTC representations can be considered identical as far as this discussion goes. Still, it is essential to realize that UTC(NIST) is generated separately from UTC(USNO) or other UTC implementations. For example, a UTC(NIST) failure should not cause UTC(USNO) to fail as they utilize separate hardware and systems.

Each of these versions of UTC is also disseminated in various ways. UTC(NIST) goes out via the "WWV" radio stations, NTP, and other esoteric methods. GPS primarily distributes UTC(USNO), which is also available directly via NTP. UTC(SU) is the timescale for GLONASS. And so on.

So, back to NTP and the accuracy required:

Most end users (people running everyday web applications or streaming video or similar) don't need precisely synchronized time. The most sensitive application I'm aware of in this space is likely TOTP, which often needs time on the server and time on the client (or hardware key) within 90 seconds of each other. In addition, having NTP time fail usually isn't the end of the world for these users. The best way to synchronize their computers (including desktop and server systems) to UTC is to point their computer time synchronization service (whatever that is) at pool.ntp.org<http://pool.ntp.org>, time.windows.com<http://time.windows.com>, their ISP's time server, or similar. Or, with modern OS'es, you can leave the time configured to whatever server the OS manufacturer preconfigured. As an aside, one should note that historically windows ticked at 15ms or so, so trying to synchronize most windows closer than 15ms was futile.

On the other hand, large ISPs or other service providers (including content providers) see real benefits to having systems synchronized to fractions of seconds of UTC. Comparing logs and traces becomes much easier when you know that something logged at 10:02:23.1 on one device came before something logged at 10:02:23.5 on another. Various server-to-server protocols and software implementations need time to be synchronized to sub-second intervals since they rely on timestamps to determine the latest copy of data, and so on. In addition, as an ISP, you'll often provide time services to downstream customers who demand more accuracy and reliability than is strictly necessary.

As a result, one wants to ensure that all time servers are synchronized within some reasonable standard of accuracy. Within 100ms is acceptable for most applications but a goal of under 50ms is better. If you have local GPS receivers, times down to around 1ms is achievable with careful design. Beyond that, you're chasing unnecessary accuracy. Note that loss of precision is somewhat cumulative here - running a time server synchronized to within 100ms will ensure that no client can be synchronized to better than within 100ms from that server. Generally, you'll want your time server to be synchronized much better than needed to avoid the time server being the limiting factor.

In a perfect world with no bad actors and where all links ran perfectly, one could set up an NTP server that pulled from pool.ntp.org<http://pool.ntp.org> or used GPS and essentially acted as a proxy. Unfortunately, we don't live in this world. So one has to ask how you build a system that meets at least the following goals:

* Synchronized to UTC within 50ms, with lower being better.
* Not subject to a reasonable set of attacks (typical DoS attacks, RF signal attacks, spoofing, etc).
* Able to be run by typical network operations staff

In addition, an ideal server setup would be made up of redundant servers in case one piece of hardware fails. I will ignore this part, as it's usually just setting up multiple copies of the same thing.

The two most straightforward options are using a GPS-based NTP appliance or installing an NTP server and pointing it at pool.ntp.org<http://pool.ntp.org>. Under normal circumstances, both options will be synchronized to UTC with enough accuracy for most applications, and both are easy to run by typical network operations staff. This assumes reasonably consistent network latency in the NTP case and a good sky view in the GPS case. The GPS-based appliance is, however, subject to spoofing or jamming, as I've discussed earlier. The NTP server is at the mercy of the quality of the servers it picked from pool.ntp.org<http://pool.ntp.org> and is also subject to various outside attacks (spoofing, etc.). One must decide how critical time is to them before deciding whether this option is valid.

The other end of the scale is the "develop your own offline version of UTC using atomic clocks" methodology. This fixes the attack issue but introduces several others. The main one is that you are now relying on the clock's accuracy. Admittedly rubidium and especially cesium clocks tend to be sufficiently reliable and stable. However, one has to ensure the frequency is accurate initially and stays that way. You must also wire the clock to an NTP Server and calibrate the initial UTC offset. If the clock goes haywire or is less accurate than is required, your in-house version of UTC will drift in relation to real UTC. This means you may need 2 or 3 or more atomic clocks to be sufficiently reliable. You'll then need to regularly take an average, compare it to UTC, and adjust if it's drifted too much. This quickly becomes more of a science project than something you want network operations staff to deal with on an ongoing basis. To be clear: If you need robust time not subject to outside forces and have or can obtain the skill set to pull this off internally, I won't argue that this is a bad option. However, I feel this isn't the type of service most providers want to run internally.

So, looking at some middle-ground options that trade a bit of robustness for ease of use is reasonable.

My lowest cost preference has always been to use a set of in-house NTP servers pointed at a carefully curated collection of NTP servers. Your curation strategy should depend on network connectivity, the reliability of the time sources, etc. In North America, picking one or two NIST servers from each NIST location is a good starting point. That is one or two from each of Maryland, Fort Collins, Boulder, and the University of Colorado. One may want to add some servers from other timekeeping organizations (such as USNO). Note that there is one commonality: These time servers are run by organizations listed in circular T as contributing to UTC, and the servers are tied to the atomic clocks. That way, we ensure that the servers are not subject to inaccuracies caused by time transfer from an authoritative source for UTC. What is left is any potential attack on the time transfer over NTP itself. I would argue that with a curated list of enough NTP servers, this risk can be pushed down to where it is low enough for many use cases. A lot will depend on the quantity and quality of NTP servers you select and the robustness of the network path to those servers. If the packets between your NTP server and the NTP servers you choose traverse a relatively secure and short path with plenty of bandwidth, and the paths to differing NTP servers are diverse, many attacks will become harder to implement. In addition, the more NTP servers you add, the more likely it is that NTP will be able to correctly pick the servers providing the correct time, even if an attacker is successfully spoofing one or more sources. In some cases it may make sense to add additional servers which are run by third parties if it gains additional robustness based on network architecture. This is especially true if you're closely connected network-wise with the third party and they run a good quality NTP service as well.

As I've mentioned, a good middle-of-the-road solution is adding various sources of time derived via GPS. Note I said, "to add." Start with the carefully curated NTP server set, then install one or more GPS-based NTP Servers polled by your NTP server. Adding these GPS time sources to your NTP servers does three things: First, it provides another source of time NTP can use to determine the correct time. Second, we're now using a different time transmission method with different vulnerabilities. And finally, it will significantly improve the accuracy of the time the NTP server produces as NTPd will generally prefer it to do the final trimming to UTC. The strength of the combination of both terrestrial transmitted time via NTP and the precision of rf-transmitted GPS time ensures that time is both correct and precise. There are still attack vectors here, but as you add more time sources, the complexity of pulling off a successful attack increases. This is especially true if you can monitor the NTP server for signs of stress, such as time servers that are not telling the correct time or GPS signals which are inconsistent with the NTP-derived time. A successful attack would require simultaneous NTP (network) and GPS (rf) attacks.

Other options or blends of options are also possible. With a reasonably large network, putting enough GPS receivers into place would significantly reduce the possibility of a spoofer or jammer taking out your entire GPS infrastructure. Reducing or eliminating external NTP time sources might be reasonable in that case. The theory is that attacking GPS receivers at one location is easy. Doing it at dozens simultaneously is much more difficult. To use an exaggeration to make a point: If you had 100 different GPS receivers spread across 100 widely geographically diverse locations, and all of your NTP servers were able to poll all of them for time, the chances that an attacker would be able to take out or spoof enough GPS receivers to make a difference would be close to zero. Your failure point becomes UTC(USNO) and the GPS constellation itself. The same argument would apply to NTP servers regarding quantity and diversity.

Other options involve adding additional technologies. For example, some appliances use GPS to discipline (adjust) an internal atomic clock. Once the atomic clock is locked to UTC, the GPS can fail for extended periods without affecting NTP output. In addition, some of these will filter updates from the GPS based on the appliance's internal atomic time. That way, a spoofer would be ignored, jammers would have to continue for hours or days, and so on. Of course, these solutions' reliability depends on the implementation quality. If I had the budget to implement something like this in a network, I'd likely scatter a few of these around the network and then still use garden variety NTPd servers which would be pointed at these appliances. I might even consider buying solutions from multiple vendors to ensure a bug in one implementation was filtered out and ignored.

I can't cover every option here, but balancing security, cost, operational complexity, and application needs is the key. Some solutions are cheap and easy but not robust. Some are highly robust but expensive and not easy. Somewhere in the middle is probably where most real implementations should lie.

Now, to address a couple of specific items:

1) Additional GPS and commercial time distribution systems will likely improve reliability. However, only GPS and GALILEO are available for free in the US. I'm ignoring GLONASS for various legal and political reasons. GALILEO is a valid option but it lives in the same band as GPS, so jamming GPS will usually also jam GALILEO. Utilizing GNSS receivers that use the civilian signals in the newer bands would also help. Some commercial solutions are available that don't require GNSS, but they're relatively new and not as commonly available as one would like.

2) For running my own time servers in a service-provider environment, I'd rather specifically designate the exact NTP server I want to utilize and not rely on a third party to give me a pool of servers. It's more about ensuring the server I use is running a trusted server, and if I delegate the server selection, I lose this ability. On the other hand, where I'm not running a NTP server that is critical for many clients, I'll just point it at pool.ntp.org<http://pool.ntp.org>, or north-america.pool.ntp.org<http://north-america.pool.ntp.org> and skip all of the recommendations that I've made above. I would be cautious about requesting pool.ntp.org<http://pool.ntp.org> add entries for "stratum of server" or "origin of time" as this seems like it would tend to overload the stratum one servers in the pool with people "optimizing" their configuration to use only stratum one servers. Remember that pool.ntp.org<http://pool.ntp.org> is generally intended as an end-user-device service, and providing methods that end users can bypass the robustness that a fully distributed pool will provide is probably not a great idea.

3) This all should hopefully sort itself out over the next few years. GPS and GALILEO are flying new birds that have changes designed to improve attack resilience by using cryptography to ensure authentic transmissions (which may rely on ground transmission of cryptographic keys). NTP already supports manual cryptographic keys that work, but NTS is a pain in the rear. Hopefully, NTPv5 will have a better security mechanism. Other, more secure, time sources are on the horizon as the cybersecurity crowd is aware of the issues.

And finally, as a sort of a tl;dr; Summary: Each operator needs to decide how critical time is to their network and pick a solution that works for them and fits the organization's budget. Some operators might point everything at pool.ntp.org<http://pool.ntp.org> and not run their own servers. Others might run their own time lab and use that time to provide NTP time and precision time and frequency via various methods. Most will be somewhere in between. But regardless of which you choose, please be aware that GPS isn't 100% secure, and neither is NTP. If attack resilience matters to you, you should think about all of the attack vectors and design something that is robust enough to meet your use case.

On Mon, 14 Aug 2023, Masataka Ohta wrote:
> Mike Hammett wrote:
>> " As such, the ultimate (a little expensive) solution is to have
>> your own Rb clocks locally."
>
>> Yeah, that's a reasonable course of action for most networks.
>
> For most data centers with time sensitive transactions, at least.
>
>> *sigh*
>
> https://en.wikipedia.org/wiki/Atomic_clock
> Modern rubidium standard tubes last more than ten years,
> and can cost as little as US$50.
>
> https://www.ebay.com/sch/i.html?_nkw=rubidium

From this discussion it seems there is very little overlap between nanog
membership and time-nuts.

Cheap Rb GPSDO are well known there. Even a bottom barrel OCXO GPSDO would
provide significant protection against determined GPS attacker.

-Dan

We're going to have to somewhat disagree here...

I may not have been 100% clear about what I see as the most common risks
for GPS. The reason I suggest that NTP risks and GPS risks are similar is
not primarily due to intentional time injection hacks (although that is a
risk). Instead, it's related to GPS failure modes, and the increased
commonality of GPS jamming causing those failures. I will 100% concede
that NTP carries far more spoofing or intentional DoS risks. But GPS is
far more likely to suffer a failure in the absence of a bad actor than NTP.

The reason for this is that the GPS signal is incredibly weak, and it's
incredibly easy to break GPS reception. Good antenna placement and
antennas that try to reject terrestrial signals help but don't always
prevent the failures from happening.

Because GPS is used more and more to track objects and people, people who
don't want to be tracked are starting to buy and use jammers. In addition,
it's becoming increasingly common for gamers to spoof their GPS location
(and, as a result, time) via GPS injection. So the kid down the street
trying to cheat at pokemon go or the truck driver not wanting to get in
trouble for speeding may unintentionally cause your time server to quit
working correctly. Not to mention the random piece of electronic gear
which malfunctions and spews noise across the GPS band.

So, yes, I will 100% agree with you that NTP carries more intentional
hacking risk. But I'm going to argue that GPS carries a significantly
higher risk of a jamming-related failure. Without good statistics, it's
hard to tell which is more prevalent. I see a lot more GPS failures from
my viewpoint, but I also have to talk to customers who are having precision
timing issues due to GPS failures.

My intuitive feeling is that in the absence of bad actors, NTP is
significantly more reliable than GPS. In the presence of remote bad
actors, I'll grant that NTP is 100% the loser here. When everything is
working, GPS will provide better time. Adding a holdover oscillator to GPS
does help in marginal situations, but doesn't resolve all of the GPS issues.

In those situations where time is not critical, either NTP or GPS is a good
solution, and it largely comes down to which you prefer. I deal with way
too many antennas so I'd rather just harden a NTP server. You might deal
with way too many hackers getting in your systems so you might prefer
relying on a GPS antenna. Either way, most of the time you're going to
get decent time service. We could go into a lot of details about how each
system can fail, but for non-time critical applications I'm not sure either
would come out a clear winner. I know you believe GPS does, and I believe
that it isn't 100% clear which one is better for those "just want time that
works most of the time" applications. We could argue all day about this
and we won't get anywhere beyond us disagreeing about this.

Once you get to more time-critical apps where actual budget is going to be
expended on ensuring reliable NTP services are available 24x7, then neither
a default configuration NTP server nor a single GPS receiver will provide
reliable time. Selecting servers and hardening firewalls to limit the
likelihood of time injection can work wonders on NTP robustness. GPS
works too if you provide enough GPS timing sources that multiple locations
would have to be jammed at once. Providing a mix of these is even
better. If I was to go GPS-only I'd probably try to ensure a minimum of 3
different GPS receivers at 3 different locations, with internal NTP servers
pulling from each of the GPS-connected NTP servers. 5 would even be
better. An even more robust option would be to go with 5 GPS receivers
and 2 or 3 NTP-connected stratum 1 time sources. In this last case, you
could spoof ALL of the NTP servers and the GPS would still be in control.
You could also have signal failures at 3 of the GPS sites and the NTP
connections would provide redundant time sources. Only with GPS failures
at multiple sites AND NTP failures or spoofing happening at the same time
would one have an issue where the NTP servers could possibly fail to
receive correct time.


On Mon, Aug 14, 2023 at 2:00?AM Mel Beckman <mel@beckman.org> wrote:

> Forrest,
>
> I think you’re gilding the lilly. My original recommendation was to use
> GPS as primary, for its superior accuracy and resistance to attack, and
> have anti-GPS back up. If you want automatic fail over, do that in an
> intermediate server on your site that makes a conscious test and decision
> to fail over to Internet NTP.
>
> You’re mistaken to say that the vulnerability of GPS is remotely
> comparable to the vulnerability of Internet-based NTP. To interfere with a
> GPS-derived clock, an attacker has to physically be present. That’s a huge
> expense — and risk — that hackers are really not interested in undertaking.
> They would much rather sit in Russia or China and attack NTP servers
> remotely using any of the several attack methodologies I’ve cited
> previously.
>
> So curate Internet NTP or not (personally, that seems like just another
> thing to monitor and maintain), but make GPS your primary time standard.
> You’re much better off staying air-gapped from Internet NTP until you
> detect a GPS failure. All the other machinations are pointless while GPS
> is working, because GPS gives you by far the best accuracy and security for
> the buck. Like I said, spend $400 on a commercial GPS time server and
> timing problems are solved. Or use facility-provided GPS if you can’t get
> an antenna up.
>
> -mel
>
> On Aug 14, 2023, at 12:10 AM, Forrest Christian (List Account) <
> lists@packetflux.com> wrote:
>
> ?
> I've responded in bits and pieces to this thread and haven't done an
> excellent job expressing my overall opinion. This is probably because my
> initial goal was to point out that GPS-transmitted time is no less subject
> to being attacked than your garden variety NTP-transmitted time. Since this
> thread has evolved, I'd like to describe my overall position to be a bit
> clearer.
>
> To start, we need a somewhat simplified version of how UTC is created so I
> can refer to it later:
>
> Across the globe, approximately 85 research and standards institutions run
> a set of freestanding atomic clocks that contribute to UTC. The number of
> atomic clocks across all these institutions totals around 450. Each
> institution also produces a version of UTC based on its own set of
> atomic clocks. In the international timekeeping world, this is designated
> as UTC(Laboratory), where Laboratory is replaced with the abbreviation for
> the lab producing that version of UTC. So UTC(NIST) is the version that
> NIST produces at Boulder, Colorado, NICT produces UTC(NICT) in Tokyo, and
> so on.
>
> Because no clock is perfectly accurate, all of these versions of UTC drift
> in relation to each other, and you could have significant differences in
> time between different labs. As a result, there has to be a way to
> synchronize them. Each month, the standards organization BIPM collects
> relative time measurements and other statistics from each
> institution described above. This data is then used to determine the
> actual value of UTC. BIPM then produces a report detailing each
> organization's difference from the correct representation of UTC. Each
> institution uses this data to adjust its UTC representation, and the cycle
> repeats the next month. In this way, all of the representations of UTC end
> up being pretty close to each other. The document BIPM produces is titled
> "Circular T." The most recent version indicates that most of the
> significant standards institutions maintain a UTC version that differs by
> less than 10ns from the official version of UTC.
>
> Note that 10ns is far more accurate than we need for NTP, so most of the
> UTC representations can be considered identical as far as this discussion
> goes. Still, it is essential to realize that UTC(NIST) is generated
> separately from UTC(USNO) or other UTC implementations. For example, a
> UTC(NIST) failure should not cause UTC(USNO) to fail as they utilize
> separate hardware and systems.
>
> Each of these versions of UTC is also disseminated in various ways.
> UTC(NIST) goes out via the "WWV" radio stations, NTP, and other esoteric
> methods. GPS primarily distributes UTC(USNO), which is also available
> directly via NTP. UTC(SU) is the timescale for GLONASS. And so on.
>
> So, back to NTP and the accuracy required:
>
> Most end users (people running everyday web applications or streaming
> video or similar) don't need precisely synchronized time. The most
> sensitive application I'm aware of in this space is likely TOTP, which
> often needs time on the server and time on the client (or hardware key)
> within 90 seconds of each other. In addition, having NTP time fail
> usually isn't the end of the world for these users. The best way to
> synchronize their computers (including desktop and server systems) to UTC
> is to point their computer time synchronization service (whatever that is)
> at pool.ntp.org, time.windows.com, their ISP's time server, or similar.
> Or, with modern OS'es, you can leave the time configured to whatever server
> the OS manufacturer preconfigured. As an aside, one should note that
> historically windows ticked at 15ms or so, so trying to synchronize most
> windows closer than 15ms was futile.
>
> On the other hand, large ISPs or other service providers (including
> content providers) see real benefits to having systems synchronized to
> fractions of seconds of UTC. Comparing logs and traces becomes much
> easier when you know that something logged at 10:02:23.1 on one device came
> before something logged at 10:02:23.5 on another. Various
> server-to-server protocols and software implementations need time to be
> synchronized to sub-second intervals since they rely on timestamps to
> determine the latest copy of data, and so on. In addition, as an ISP,
> you'll often provide time services to downstream customers who demand more
> accuracy and reliability than is strictly necessary.
>
> As a result, one wants to ensure that all time servers are synchronized
> within some reasonable standard of accuracy. Within 100ms is acceptable
> for most applications but a goal of under 50ms is better. If you have
> local GPS receivers, times down to around 1ms is achievable with careful
> design. Beyond that, you're chasing unnecessary accuracy. Note that loss
> of precision is somewhat cumulative here - running a time server
> synchronized to within 100ms will ensure that no client can be synchronized
> to better than within 100ms from that server. Generally, you'll want your
> time server to be synchronized much better than needed to avoid the time
> server being the limiting factor.
>
> In a perfect world with no bad actors and where all links ran perfectly,
> one could set up an NTP server that pulled from pool.ntp.org or used GPS
> and essentially acted as a proxy. Unfortunately, we don't live in this
> world. So one has to ask how you build a system that meets at least the
> following goals:
>
> * Synchronized to UTC within 50ms, with lower being better.
> * Not subject to a reasonable set of attacks (typical DoS attacks, RF
> signal attacks, spoofing, etc).
> * Able to be run by typical network operations staff
>
> In addition, an ideal server setup would be made up of redundant servers
> in case one piece of hardware fails. I will ignore this part, as it's
> usually just setting up multiple copies of the same thing.
>
> The two most straightforward options are using a GPS-based NTP appliance
> or installing an NTP server and pointing it at pool.ntp.org. Under
> normal circumstances, both options will be synchronized to UTC with enough
> accuracy for most applications, and both are easy to run by typical network
> operations staff. This assumes reasonably consistent network latency in
> the NTP case and a good sky view in the GPS case. The GPS-based appliance
> is, however, subject to spoofing or jamming, as I've discussed earlier.
> The NTP server is at the mercy of the quality of the servers it picked
> from pool.ntp.org and is also subject to various outside attacks
> (spoofing, etc.). One must decide how critical time is to them before
> deciding whether this option is valid.
>
> The other end of the scale is the "develop your own offline version of UTC
> using atomic clocks" methodology. This fixes the attack issue but
> introduces several others. The main one is that you are now relying on
> the clock's accuracy. Admittedly rubidium and especially cesium clocks
> tend to be sufficiently reliable and stable. However, one has to ensure
> the frequency is accurate initially and stays that way. You must also wire
> the clock to an NTP Server and calibrate the initial UTC offset. If the
> clock goes haywire or is less accurate than is required, your in-house
> version of UTC will drift in relation to real UTC. This means you may
> need 2 or 3 or more atomic clocks to be sufficiently reliable. You'll then
> need to regularly take an average, compare it to UTC, and adjust if it's
> drifted too much. This quickly becomes more of a science project than
> something you want network operations staff to deal with on an ongoing
> basis. To be clear: If you need robust time not subject to outside
> forces and have or can obtain the skill set to pull this off internally, I
> won't argue that this is a bad option. However, I feel this isn't the type
> of service most providers want to run internally.
>
> So, looking at some middle-ground options that trade a bit of robustness
> for ease of use is reasonable.
>
> My lowest cost preference has always been to use a set of in-house NTP
> servers pointed at a carefully curated collection of NTP servers. Your
> curation strategy should depend on network connectivity, the reliability of
> the time sources, etc. In North America, picking one or two NIST servers
> from each NIST location is a good starting point. That is one or two from
> each of Maryland, Fort Collins, Boulder, and the University of Colorado.
> One may want to add some servers from other timekeeping organizations
> (such as USNO). Note that there is one commonality: These time servers
> are run by organizations listed in circular T as contributing to UTC, and
> the servers are tied to the atomic clocks. That way, we ensure that the
> servers are not subject to inaccuracies caused by time transfer from an
> authoritative source for UTC. What is left is any potential attack on the
> time transfer over NTP itself. I would argue that with a curated list of
> enough NTP servers, this risk can be pushed down to where it is low enough
> for many use cases. A lot will depend on the quantity and quality of NTP
> servers you select and the robustness of the network path to those
> servers. If the packets between your NTP server and the NTP servers you
> choose traverse a relatively secure and short path with plenty of
> bandwidth, and the paths to differing NTP servers are diverse, many attacks
> will become harder to implement. In addition, the more NTP servers you
> add, the more likely it is that NTP will be able to correctly pick the
> servers providing the correct time, even if an attacker is successfully
> spoofing one or more sources. In some cases it may make sense to add
> additional servers which are run by third parties if it gains additional
> robustness based on network architecture. This is especially true if
> you're closely connected network-wise with the third party and they run a
> good quality NTP service as well.
>
> As I've mentioned, a good middle-of-the-road solution is adding various
> sources of time derived via GPS. Note I said, "to add." Start with the
> carefully curated NTP server set, then install one or more GPS-based NTP
> Servers polled by your NTP server. Adding these GPS time sources to your
> NTP servers does three things: First, it provides another source of time
> NTP can use to determine the correct time. Second, we're now using a
> different time transmission method with different vulnerabilities. And
> finally, it will significantly improve the accuracy of the time the NTP
> server produces as NTPd will generally prefer it to do the final trimming
> to UTC. The strength of the combination of both terrestrial transmitted
> time via NTP and the precision of rf-transmitted GPS time ensures that time
> is both correct and precise. There are still attack vectors here, but as
> you add more time sources, the complexity of pulling off a successful
> attack increases. This is especially true if you can monitor the NTP
> server for signs of stress, such as time servers that are not telling the
> correct time or GPS signals which are inconsistent with the NTP-derived
> time. A successful attack would require simultaneous NTP (network) and
> GPS (rf) attacks.
>
> Other options or blends of options are also possible. With a reasonably
> large network, putting enough GPS receivers into place would significantly
> reduce the possibility of a spoofer or jammer taking out your entire GPS
> infrastructure. Reducing or eliminating external NTP time sources might be
> reasonable in that case. The theory is that attacking GPS receivers at
> one location is easy. Doing it at dozens simultaneously is much more
> difficult. To use an exaggeration to make a point: If you had 100
> different GPS receivers spread across 100 widely geographically diverse
> locations, and all of your NTP servers were able to poll all of them for
> time, the chances that an attacker would be able to take out or spoof
> enough GPS receivers to make a difference would be close to zero. Your
> failure point becomes UTC(USNO) and the GPS constellation itself. The same
> argument would apply to NTP servers regarding quantity and diversity.
>
> Other options involve adding additional technologies. For example, some
> appliances use GPS to discipline (adjust) an internal atomic clock. Once
> the atomic clock is locked to UTC, the GPS can fail for extended periods
> without affecting NTP output. In addition, some of these will filter
> updates from the GPS based on the appliance's internal atomic time. That
> way, a spoofer would be ignored, jammers would have to continue for hours
> or days, and so on. Of course, these solutions' reliability depends on
> the implementation quality. If I had the budget to implement something
> like this in a network, I'd likely scatter a few of these around the
> network and then still use garden variety NTPd servers which would be
> pointed at these appliances. I might even consider buying solutions from
> multiple vendors to ensure a bug in one implementation was filtered out and
> ignored.
>
> I can't cover every option here, but balancing security, cost, operational
> complexity, and application needs is the key. Some solutions are cheap
> and easy but not robust. Some are highly robust but expensive and not
> easy. Somewhere in the middle is probably where most real implementations
> should lie.
>
> Now, to address a couple of specific items:
>
> 1) Additional GPS and commercial time distribution systems will likely
> improve reliability. However, only GPS and GALILEO are available for free
> in the US. I'm ignoring GLONASS for various legal and political reasons.
> GALILEO is a valid option but it lives in the same band as GPS, so jamming
> GPS will usually also jam GALILEO. Utilizing GNSS receivers that use the
> civilian signals in the newer bands would also help. Some commercial
> solutions are available that don't require GNSS, but they're relatively new
> and not as commonly available as one would like.
>
> 2) For running my own time servers in a service-provider environment, I'd
> rather specifically designate the exact NTP server I want to utilize and
> not rely on a third party to give me a pool of servers. It's more about
> ensuring the server I use is running a trusted server, and if I delegate
> the server selection, I lose this ability. On the other hand, where I'm
> not running a NTP server that is critical for many clients, I'll just point
> it at pool.ntp.org, or north-america.pool.ntp.org and skip all of the
> recommendations that I've made above. I would be cautious about
> requesting pool.ntp.org add entries for "stratum of server" or "origin of
> time" as this seems like it would tend to overload the stratum one servers
> in the pool with people "optimizing" their configuration to use only
> stratum one servers. Remember that pool.ntp.org is generally intended
> as an end-user-device service, and providing methods that end users can
> bypass the robustness that a fully distributed pool will provide is
> probably not a great idea.
>
> 3) This all should hopefully sort itself out over the next few years.
> GPS and GALILEO are flying new birds that have changes designed to improve
> attack resilience by using cryptography to ensure authentic transmissions
> (which may rely on ground transmission of cryptographic keys). NTP
> already supports manual cryptographic keys that work, but NTS is a pain in
> the rear. Hopefully, NTPv5 will have a better security mechanism. Other,
> more secure, time sources are on the horizon as the cybersecurity crowd is
> aware of the issues.
>
> And finally, as a sort of a tl;dr; Summary: Each operator needs to decide
> how critical time is to their network and pick a solution that works for
> them and fits the organization's budget. Some operators might point
> everything at pool.ntp.org and not run their own servers. Others might
> run their own time lab and use that time to provide NTP time and precision
> time and frequency via various methods. Most will be somewhere in between.
> But regardless of which you choose, please be aware that GPS isn't 100%
> secure, and neither is NTP. If attack resilience matters to you, you should
> think about all of the attack vectors and design something that is robust
> enough to meet your use case.
>
>
>
>

--
- Forrest

Replying to two posts at once...

One can definitely get inexpensive and high-quality rubidiums for dirt
cheap on the second-hand market. I've specifically ignored those when
discussing price as options as one can never be sure about their accuracy
or long-term reliability, and I try to filter my suggestions on NANOG based
on what I'd want to put in my production network (putting on my ISP network
administrator hat for a second). Many of those rubidiums have lived their
lives in the harsh environment of a cellular tower's equipment enclosure
and were pulled out after many years of service.

A brand-new rubidium oscillator is typically just under $2K (A common
brand is $1695 as we speak). This is just the oscillator. To this you
have to add various support items such as power supplies and signal
conditioning and heatsinks and enclosure to end up with something
which will connect to some sort of NTP server. Note that this doesn't
provide any way to actually phase-align that rubidium oscillator to UTC.
For that, you'd have to add more hardware.

Professionally, I keep looking for a supplier of lower cost rubidum
oscillators but have not found anything much cheaper than the SRS PRS10's
at the $1695 I mentioned above.

There are lots of ways to improve a GPS-based NTP server. Better antenna
positioning. Better GPS chipset. Paying attention to antenna patterns.
Adding notch filters to the GPS feed. And so on. Once you get the 1PPS
out of the GPS receiver you then can utilize the 1PPS signal to discipline
some sort of clock. Maybe a TCXO for a day or so of NTP holdover. Maybe
a rubidium for a year or more of holdover depending on the accuracy you
need. You can add software to filter the GPS signal to limit the
likelihood of time injection attacks. And on and on. It really comes
down to how much money you want to put into the "appliance" you're
building. But, in the end, there is nothing better than adding a second
GPS source at a diverse location as far as improving reliability, provided
that's an option based on timing needs.

I can also attest that there is at least one overlap between time-nuts and
NANOG.... occupational hazard here. From my desk I can reach out and touch
two rubidium oscillators, one is GPS synchronized and one is freerunning.
There are a couple others unpowered in the box of spares in the other
room. There are also at least 3-4 TCXO-based GPSDOs floating around
(including one I use for my reference source), and don't get me started on
the T&M equipment I have collected for comparing various time sources. So
far I've avoided spending the mid-5 figures to get a decent cesium
oscillator.

On Mon, Aug 14, 2023 at 2:55?AM goemon--- via NANOG <nanog@nanog.org> wrote:

> On Mon, 14 Aug 2023, Masataka Ohta wrote:
> > Mike Hammett wrote:
> >> " As such, the ultimate (a little expensive) solution is to have
> >> your own Rb clocks locally."
> >
> >> Yeah, that's a reasonable course of action for most networks.
> >
> > For most data centers with time sensitive transactions, at least.
> >
> >> *sigh*
> >
> > https://en.wikipedia.org/wiki/Atomic_clock
> > Modern rubidium standard tubes last more than ten years,
> > and can cost as little as US$50.
> >
> > https://www.ebay.com/sch/i.html?_nkw=rubidium
>
> From this discussion it seems there is very little overlap between nanog
> membership and time-nuts.
>
> Cheap Rb GPSDO are well known there. Even a bottom barrel OCXO GPSDO would
> provide significant protection against determined GPS attacker.
>
> -Dan
>


--
- Forrest

> On Aug 14, 2023, at 3:07 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:
>
> I've responded in bits and pieces to this thread and haven't done an excellent job expressing my overall opinion. This is probably because my initial goal was to point out that GPS-transmitted time is no less subject to being attacked than your garden variety NTP-transmitted time. Since this thread has evolved, I'd like to describe my overall position to be a bit clearer.
>
<SNIP/>
>
> And finally, as a sort of a tl;dr; Summary: Each operator needs to decide how critical time is to their network and pick a solution that works for them and fits the organization's budget. Some operators might point everything at pool.ntp.org <http://pool.ntp.org/> and not run their own servers. Others might run their own time lab and use that time to provide NTP time and precision time and frequency via various methods. Most will be somewhere in between. But regardless of which you choose, please be aware that GPS isn't 100% secure, and neither is NTP. If attack resilience matters to you, you should think about all of the attack vectors and design something that is robust enough to meet your use case.
>
This has been an interesting thread. I consider Forrest Christian’s note to be most cogent. Much of the GPS vs Internet sourcing arguments can probably be found in NANOG archives from many years ago. The threat list is longer now, but the problem of providing Time Service is still the same.

Twenty-five or so years ago my design process for providing Network Time Service to a large company intranet started with the business requirements for time service. The Management practice of “Not in my cost center” was fundamental to NOT attempting GPS-based deployment. The internal enterprise network provided a set of geographically distributed Stratum 2 servers having carefully firewalled access to a similar set of Stratum 1 servers with Internet access. The Stratum 0 server set list included NIST, USNO, and other similar sources distributed globally.

The magic of Dr. Mills algorithm made truechimers of the intranet NTP server set which did serve well for the lifetime of the company.

-
James R. Cutler

Forrest seems to have posted a good general overview and perspectives about "good enough for the use case" while others continue to be pedantic about nuances that don't seem to be relevant to most use cases.




-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

----- Original Message -----

From: "Forrest Christian (List Account)" <lists@packetflux.com>
To: "nanog list" <nanog@nanog.org>
Sent: Monday, August 14, 2023 2:07:14 AM
Subject: Re: NTP Sync Issue Across Tata (Europe)



I've responded in bits and pieces to this thread and haven't done an excellent job expressing my overall opinion. This is probably because my initial goal was to point out that GPS-transmitted time is no less subject to being attacked than your garden variety NTP-transmitted time. Since this thread has evolved, I'd like to describe my overall position to be a bit clearer.


To start, we need a somewhat simplified version of how UTC is created so I can refer to it later:


Across the globe, approximately 85 research and standards institutions run a set of freestanding atomic clocks that contribute to UTC. The number of atomic clocks across all these institutions totals around 450. Each institution also produces a version of UTC based on its own set of atomic clocks. In the international timekeeping world, this is designated as UTC(Laboratory), where Laboratory is replaced with the abbreviation for the lab producing that version of UTC. So UTC(NIST) is the version that NIST produces at Boulder, Colorado, NICT produces UTC(NICT) in Tokyo, and so on.


Because no clock is perfectly accurate, all of these versions of UTC drift in relation to each other, and you could have significant differences in time between different labs. As a result, there has to be a way to synchronize them. Each month, the standards organization BIPM collects relative time measurements and other statistics from each institution described above. This data is then used to determine the actual value of UTC. BIPM then produces a report detailing each organization's difference from the correct representation of UTC. Each institution uses this data to adjust its UTC representation, and the cycle repeats the next month. In this way, all of the representations of UTC end up being pretty close to each other. The document BIPM produces is titled "Circular T." The most recent version indicates that most of the significant standards institutions maintain a UTC version that differs by less than 10ns from the official version of UTC.


Note that 10ns is far more accurate than we need for NTP, so most of the UTC representations can be considered identical as far as this discussion goes. Still, it is essential to realize that UTC(NIST) is generated separately from UTC(USNO) or other UTC implementations. For example, a UTC(NIST) failure should not cause UTC(USNO) to fail as they utilize separate hardware and systems.


Each of these versions of UTC is also disseminated in various ways. UTC(NIST) goes out via the "WWV" radio stations, NTP, and other esoteric methods. GPS primarily distributes UTC(USNO), which is also available directly via NTP. UTC(SU) is the timescale for GLONASS. And so on.


So, back to NTP and the accuracy required:


Most end users (people running everyday web applications or streaming video or similar) don't need precisely synchronized time. The most sensitive application I'm aware of in this space is likely TOTP, which often needs time on the server and time on the client (or hardware key) within 90 seconds of each other. In addition, having NTP time fail usually isn't the end of the world for these users. The best way to synchronize their computers (including desktop and server systems) to UTC is to point their computer time synchronization service (whatever that is) at pool.ntp.org , time.windows.com , their ISP's time server, or similar. Or, with modern OS'es, you can leave the time configured to whatever server the OS manufacturer preconfigured. As an aside, one should note that historically windows ticked at 15ms or so, so trying to synchronize most windows closer than 15ms was futile.


On the other hand, large ISPs or other service providers (including content providers) see real benefits to having systems synchronized to fractions of seconds of UTC. Comparing logs and traces becomes much easier when you know that something logged at 10:02:23.1 on one device came before something logged at 10:02:23.5 on another. Various server-to-server protocols and software implementations need time to be synchronized to sub-second intervals since they rely on timestamps to determine the latest copy of data, and so on. In addition, as an ISP, you'll often provide time services to downstream customers who demand more accuracy and reliability than is strictly necessary.


As a result, one wants to ensure that all time servers are synchronized within some reasonable standard of accuracy. Within 100ms is acceptable for most applications but a goal of under 50ms is better. If you have local GPS receivers, times down to around 1ms is achievable with careful design. Beyond that, you're chasing unnecessary accuracy. Note that loss of precision is somewhat cumulative here - running a time server synchronized to within 100ms will ensure that no client can be synchronized to better than within 100ms from that server. Generally, you'll want your time server to be synchronized much better than needed to avoid the time server being the limiting factor.


In a perfect world with no bad actors and where all links ran perfectly, one could set up an NTP server that pulled from pool.ntp.org or used GPS and essentially acted as a proxy. Unfortunately, we don't live in this world. So one has to ask how you build a system that meets at least the following goals:


* Synchronized to UTC within 50ms, with lower being better.
* Not subject to a reasonable set of attacks (typical DoS attacks, RF signal attacks, spoofing, etc).
* Able to be run by typical network operations staff


In addition, an ideal server setup would be made up of redundant servers in case one piece of hardware fails. I will ignore this part, as it's usually just setting up multiple copies of the same thing.


The two most straightforward options are using a GPS-based NTP appliance or installing an NTP server and pointing it at pool.ntp.org . Under normal circumstances, both options will be synchronized to UTC with enough accuracy for most applications, and both are easy to run by typical network operations staff. This assumes reasonably consistent network latency in the NTP case and a good sky view in the GPS case. The GPS-based appliance is, however, subject to spoofing or jamming, as I've discussed earlier. The NTP server is at the mercy of the quality of the servers it picked from pool.ntp.org and is also subject to various outside attacks (spoofing, etc.). One must decide how critical time is to them before deciding whether this option is valid.


The other end of the scale is the "develop your own offline version of UTC using atomic clocks" methodology. This fixes the attack issue but introduces several others. The main one is that you are now relying on the clock's accuracy. Admittedly rubidium and especially cesium clocks tend to be sufficiently reliable and stable. However, one has to ensure the frequency is accurate initially and stays that way. You must also wire the clock to an NTP Server and calibrate the initial UTC offset. If the clock goes haywire or is less accurate than is required, your in-house version of UTC will drift in relation to real UTC. This means you may need 2 or 3 or more atomic clocks to be sufficiently reliable. You'll then need to regularly take an average, compare it to UTC, and adjust if it's drifted too much. This quickly becomes more of a science project than something you want network operations staff to deal with on an ongoing basis. To be clear: If you need robust time not subject to outside forces and have or can obtain the skill set to pull this off internally, I won't argue that this is a bad option. However, I feel this isn't the type of service most providers want to run internally.


So, looking at some middle-ground options that trade a bit of robustness for ease of use is reasonable.


My lowest cost preference has always been to use a set of in-house NTP servers pointed at a carefully curated collection of NTP servers. Your curation strategy should depend on network connectivity, the reliability of the time sources, etc. In North America, picking one or two NIST servers from each NIST location is a good starting point. That is one or two from each of Maryland, Fort Collins, Boulder, and the University of Colorado. One may want to add some servers from other timekeeping organizations (such as USNO). Note that there is one commonality: These time servers are run by organizations listed in circular T as contributing to UTC, and the servers are tied to the atomic clocks. That way, we ensure that the servers are not subject to inaccuracies caused by time transfer from an authoritative source for UTC. What is left is any potential attack on the time transfer over NTP itself. I would argue that with a curated list of enough NTP servers, this risk can be pushed down to where it is low enough for many use cases. A lot will depend on the quantity and quality of NTP servers you select and the robustness of the network path to those servers. If the packets between your NTP server and the NTP servers you choose traverse a relatively secure and short path with plenty of bandwidth, and the paths to differing NTP servers are diverse, many attacks will become harder to implement. In addition, the more NTP servers you add, the more likely it is that NTP will be able to correctly pick the servers providing the correct time, even if an attacker is successfully spoofing one or more sources. In some cases it may make sense to add additional servers which are run by third parties if it gains additional robustness based on network architecture. This is especially true if you're closely connected network-wise with the third party and they run a good quality NTP service as well.


As I've mentioned, a good middle-of-the-road solution is adding various sources of time derived via GPS. Note I said, "to add." Start with the carefully curated NTP server set, then install one or more GPS-based NTP Servers polled by your NTP server. Adding these GPS time sources to your NTP servers does three things: First, it provides another source of time NTP can use to determine the correct time. Second, we're now using a different time transmission method with different vulnerabilities. And finally, it will significantly improve the accuracy of the time the NTP server produces as NTPd will generally prefer it to do the final trimming to UTC. The strength of the combination of both terrestrial transmitted time via NTP and the precision of rf-transmitted GPS time ensures that time is both correct and precise. There are still attack vectors here, but as you add more time sources, the complexity of pulling off a successful attack increases. This is especially true if you can monitor the NTP server for signs of stress, such as time servers that are not telling the correct time or GPS signals which are inconsistent with the NTP-derived time. A successful attack would require simultaneous NTP (network) and GPS (rf) attacks.


Other options or blends of options are also possible. With a reasonably large network, putting enough GPS receivers into place would significantly reduce the possibility of a spoofer or jammer taking out your entire GPS infrastructure. Reducing or eliminating external NTP time sources might be reasonable in that case. The theory is that attacking GPS receivers at one location is easy. Doing it at dozens simultaneously is much more difficult. To use an exaggeration to make a point: If you had 100 different GPS receivers spread across 100 widely geographically diverse locations, and all of your NTP servers were able to poll all of them for time, the chances that an attacker would be able to take out or spoof enough GPS receivers to make a difference would be close to zero. Your failure point becomes UTC(USNO) and the GPS constellation itself. The same argument would apply to NTP servers regarding quantity and diversity.


Other options involve adding additional technologies. For example, some appliances use GPS to discipline (adjust) an internal atomic clock. Once the atomic clock is locked to UTC, the GPS can fail for extended periods without affecting NTP output. In addition, some of these will filter updates from the GPS based on the appliance's internal atomic time. That way, a spoofer would be ignored, jammers would have to continue for hours or days, and so on. Of course, these solutions' reliability depends on the implementation quality. If I had the budget to implement something like this in a network, I'd likely scatter a few of these around the network and then still use garden variety NTPd servers which would be pointed at these appliances. I might even consider buying solutions from multiple vendors to ensure a bug in one implementation was filtered out and ignored.


I can't cover every option here, but balancing security, cost, operational complexity, and application needs is the key. Some solutions are cheap and easy but not robust. Some are highly robust but expensive and not easy. Somewhere in the middle is probably where most real implementations should lie.


Now, to address a couple of specific items:


1) Additional GPS and commercial time distribution systems will likely improve reliability. However, only GPS and GALILEO are available for free in the US. I'm ignoring GLONASS for various legal and political reasons. GALILEO is a valid option but it lives in the same band as GPS, so jamming GPS will usually also jam GALILEO. Utilizing GNSS receivers that use the civilian signals in the newer bands would also help. Some commercial solutions are available that don't require GNSS, but they're relatively new and not as commonly available as one would like.


2) For running my own time servers in a service-provider environment, I'd rather specifically designate the exact NTP server I want to utilize and not rely on a third party to give me a pool of servers. It's more about ensuring the server I use is running a trusted server, and if I delegate the server selection, I lose this ability. On the other hand, where I'm not running a NTP server that is critical for many clients, I'll just point it at pool.ntp.org , or north-america.pool.ntp.org and skip all of the recommendations that I've made above. I would be cautious about requesting pool.ntp.org add entries for "stratum of server" or "origin of time" as this seems like it would tend to overload the stratum one servers in the pool with people "optimizing" their configuration to use only stratum one servers. Remember that pool.ntp.org is generally intended as an end-user-device service, and providing methods that end users can bypass the robustness that a fully distributed pool will provide is probably not a great idea.


3) This all should hopefully sort itself out over the next few years. GPS and GALILEO are flying new birds that have changes designed to improve attack resilience by using cryptography to ensure authentic transmissions (which may rely on ground transmission of cryptographic keys). NTP already supports manual cryptographic keys that work, but NTS is a pain in the rear. Hopefully, NTPv5 will have a better security mechanism. Other, more secure, time sources are on the horizon as the cybersecurity crowd is aware of the issues.


And finally, as a sort of a tl;dr; Summary: Each operator needs to decide how critical time is to their network and pick a solution that works for them and fits the organization's budget. Some operators might point everything at pool.ntp.org and not run their own servers. Others might run their own time lab and use that time to provide NTP time and precision time and frequency via various methods. Most will be somewhere in between. But regardless of which you choose, please be aware that GPS isn't 100% secure, and neither is NTP. If attack resilience matters to you, you should think about all of the attack vectors and design something that is robust enough to meet your use case.

Forrest Christian (List Account) wrote:

> There are lots of ways to improve a GPS-based NTP server. Better antenna
> positioning. Better GPS chipset. Paying attention to antenna patterns.
> Adding notch filters to the GPS feed. And so on.

They are not a very meaningful improvement.

> But, in the end, there is nothing better than adding a second
> GPS source at a diverse location as far as improving reliability, provided
> that's an option based on timing needs.

You keep ignoring DOS attacks.

Though you wrote:

: If I just want to deny you time, it gets cheaper and
: easier. All I need is a 1.2 GHz oscillator coupled to an
: antenna. There are units like this available for under $10,
: delivered. These block GPS trackers on trucks and/or private
: automobiles. Build your own and you can get a watt or two
: to shove into a tiny antenna for not a lot more. Guaranteed
: to Jam anything within a couple of blocks.

you don't understand similar effectiveness by DOS.

> I can also attest that there is at least one overlap between time-nuts and
> NANOG....

See above.

Masataka Ohta

Mel Beckman wrote:-

>Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I?d like to see the documentation.

The event took place on the evening of Sunday 12 July 2020, and seems NOT
to have been due to an issue caused directly by GPS, but rather to
misbehaviour of a GPS NTP server relating to week numbers. Our regulator
subsequently issued the following comprehensive document:-

https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf

By way of summary, JT operated two GPS derived NTP servers, with all of
their routers were pointing to both. On the evening in question, one of
the two reset its clock back to 27 November 2000.

Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication. The authentication [section 4.19]
was described having a "password validity start date" of 01 July 2012.
Thus, any routers which had picked up the time from the faulty source no
longer had valid IS-IS authentication and were thus isolated.

Whilst only 15% of their routers were affected, this was enough to cause an
almost total failure in their network, affecting telephony (fixed & mobile)
and Internet. For foreign readers (this is NANOG!) "999" calls refer to
the emergency services in these parts, where any failures attract the
attention of our regulator.

The details of why the clock "failed" start at section 4.23, and seem to
relate a GPS week number rollover.

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

One curious aspect is that some routers followed the "bad" time, which is
alluded to in section 4.31.

Something not discussed in that report is that JT's email failed during the
incident despite its being hosted on Office365. The reason was that the
two authoritative DNS servers for jtglobal.com were hosted in Jersey inside
their network. As that network was wholly disconnected, there was no DNS
and hence no email. Despite my having raised this since with their senior
management, their DNS remains hosted in this way:-

>matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @ns1.jtibs.net
>;; Got answer:
>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
>;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
>
>;; QUESTION SECTION:
>;jtglobal.com. IN NS
>
>;; ANSWER SECTION:
>jtglobal.com. 60 IN NS ns2.jtibs.net.
>jtglobal.com. 60 IN NS ns1.jtibs.net.
>
>;; ADDITIONAL SECTION:
>ns1.jtibs.net. 60 IN A 212.9.0.135
>ns2.jtibs.net. 60 IN A 212.9.0.136
>ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1
>ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2

Rediculously (and again despite my agitation to their management) our
government domain gov.je has similar DNS fragility:-

>matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
>;; Got answer:
>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
>;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
>
>;; QUESTION SECTION:
>;gov.je. IN NS
>
>;; ANSWER SECTION:
>gov.je. 3600 IN NS ns2.gov.je.
>gov.je. 3600 IN NS ns1.gov.je.
>
>;; ADDITIONAL SECTION:
>ns2.gov.je. 3600 IN A 212.9.21.137
>ns1.gov.je. 3600 IN A 212.9.21.9

--
Best wishes,
Matthew

------
>From: Mel Beckman <mel@beckman.org>
>To: Matthew Richardson <matthew-l@itconsult.co.uk>
>Cc: Nanog <nanog@nanog.org>
>Date: Tue, 8 Aug 2023 15:12:29 +0000
>Subject: Re: NTP Sync Issue Across Tata (Europe)

>Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I?d like to see the documentation.
>
>Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical ? but conceivable ? meteor storm). But that would be a fall-back. I would not mix the systems.
>
> -mel
>
>> On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:
>>
>> ?Mel Beckman wrote:-
>>
>>> It's a problem that has received a lot of attention in both NTP and
>>> aviation navigation circles. What is hard to defend against is total signal
>>> suppression via high powered jamming. But that you can do with a
>>> geographically diverse GPS NTP network.
>>
>> Whilst looking forward to being corrected, GPS (even across multiple
>> locations) seems to be a SINGLE source of time. You seem (have I
>> misunderstood?) to be a proponent of using GPS exclusively as the external
>> clock source.
>>
>> Might it be preferable to have a mixture of GPS (perhaps with another GNSS)
>> together with carefully selected Internet-based NTP servers?
>>
>> I recall an incident over here in Jersey (the one they named New Jersey
>> after!) where our primary telco had a substantial time shift on one of
>> their two GPS synced servers. This managed to adjust the clock on enough
>> of their routers that the certificate-based OSPF authentication considered
>> the certificates invalid, and caused a failure of almost their whole
>> network.
>>
>> This is, of course, not to say that GPS is not a very good clock source,
>> but rather to wonder whether more diversity would be preferable than using
>> it as a single source.
>>
>> --
>> Best wishes,
>> Matthew
>>
>> ------
>>> From: Mel Beckman <mel@beckman.org>
>>> To: "Forrest Christian (List Account)" <lists@packetflux.com>
>>> Cc: Nanog <nanog@nanog.org>
>>> Date: Mon, 7 Aug 2023 14:03:30 +0000
>>> Subject: Re: NTP Sync Issue Across Tata (Europe)
>>
>>> Forrest,
>>>
>>> GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
>>>
>>> -mel
>>>
>>> On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:
>>>
>>> ?
>>> The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.
>>>
>>> With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.
>>>
>>> In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.
>>>
>>> Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service .
>>>
>>>
>>>
>>> On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
>>> GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
>>>
>>> DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
>>>
>>> Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager.
>>>
>>> I sense hand-waving :)
>>>
>>> -mel via cell
>>>
>>> On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote:
>>>
>>> ?
>>>
>>>
>>> On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
>>> Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
>>>
>>> The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
>>>
>>> BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
>>>
>>> OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
>>>
>>> Rubens
>>

Throw PTP in the mix for the greater accuracy required for some wireless (5G) configurations, and the situation becomes even more complicated.

On Aug 14, 2023, at 6:55 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:

?We're going to have to somewhat disagree here...
I may not have been 100% clear about what I see as the most common risks for GPS. The reason I suggest that NTP risks and GPS risks are similar is not primarily due to intentional time injection hacks (although that is a risk). Instead, it's related to GPS failure modes, and the increased commonality of GPS jamming causing those failures. I will 100% concede that NTP carries far more spoofing or intentional DoS risks. But GPS is far more likely to suffer a failure in the absence of a bad actor than NTP.
The reason for this is that the GPS signal is incredibly weak, and it's incredibly easy to break GPS reception. Good antenna placement and antennas that try to reject terrestrial signals help but don't always prevent the failures from happening.
Because GPS is used more and more to track objects and people, people who don't want to be tracked are starting to buy and use jammers. In addition, it's becoming increasingly common for gamers to spoof their GPS location (and, as a result, time) via GPS injection. So the kid down the street trying to cheat at pokemon go or the truck driver not wanting to get in trouble for speeding may unintentionally cause your time server to quit working correctly. Not to mention the random piece of electronic gear which malfunctions and spews noise across the GPS band.
So, yes, I will 100% agree with you that NTP carries more intentional hacking risk. But I'm going to argue that GPS carries a significantly higher risk of a jamming-related failure. Without good statistics, it's hard to tell which is more prevalent. I see a lot more GPS failures from my viewpoint, but I also have to talk to customers who are having precision timing issues due to GPS failures.
My intuitive feeling is that in the absence of bad actors, NTP is significantly more reliable than GPS. In the presence of remote bad actors, I'll grant that NTP is 100% the loser here. When everything is working, GPS will provide better time. Adding a holdover oscillator to GPS does help in marginal situations, but doesn't resolve all of the GPS issues.
In those situations where time is not critical, either NTP or GPS is a good solution, and it largely comes down to which you prefer. I deal with way too many antennas so I'd rather just harden a NTP server. You might deal with way too many hackers getting in your systems so you might prefer relying on a GPS antenna. Either way, most of the time you're going to get decent time service. We could go into a lot of details about how each system can fail, but for non-time critical applications I'm not sure either would come out a clear winner. I know you believe GPS does, and I believe that it isn't 100% clear which one is better for those "just want time that works most of the time" applications. We could argue all day about this and we won't get anywhere beyond us disagreeing about this.
Once you get to more time-critical apps where actual budget is going to be expended on ensuring reliable NTP services are available 24x7, then neither a default configuration NTP server nor a single GPS receiver will provide reliable time. Selecting servers and hardening firewalls to limit the likelihood of time injection can work wonders on NTP robustness. GPS works too if you provide enough GPS timing sources that multiple locations would have to be jammed at once. Providing a mix of these is even better. If I was to go GPS-only I'd probably try to ensure a minimum of 3 different GPS receivers at 3 different locations, with internal NTP servers pulling from each of the GPS-connected NTP servers. 5 would even be better. An even more robust option would be to go with 5 GPS receivers and 2 or 3 NTP-connected stratum 1 time sources. In this last case, you could spoof ALL of the NTP servers and the GPS would still be in control. You could also have signal failures at 3 of the GPS sites and the NTP connections would provide redundant time sources. Only with GPS failures at multiple sites AND NTP failures or spoofing happening at the same time would one have an issue where the NTP servers could possibly fail to receive correct time.


On Mon, Aug 14, 2023 at 2:00?AM Mel Beckman <mel@beckman.org> wrote:

Forrest,
I think you’re gilding the lilly. My original recommendation was to use GPS as primary, for its superior accuracy and resistance to attack, and have anti-GPS back up. If you want automatic fail over, do that in an intermediate server on your site that makes a conscious test and decision to fail over to Internet NTP.
You’re mistaken to say that the vulnerability of GPS is remotely comparable to the vulnerability of Internet-based NTP. To interfere with a GPS-derived clock, an attacker has to physically be present. That’s a huge expense — and risk — that hackers are really not interested in undertaking. They would much rather sit in Russia or China and attack NTP servers remotely using any of the several attack methodologies I’ve cited previously.
So curate Internet NTP or not (personally, that seems like just another thing to monitor and maintain), but make GPS your primary time standard. You’re much better off staying air-gapped from Internet NTP until you detect a GPS failure. All the other machinations are pointless while GPS is working, because GPS gives you by far the best accuracy and security for the buck. Like I said, spend $400 on a commercial GPS time server and timing problems are solved. Or use facility-provided GPS if you can’t get an antenna up.

-mel

On Aug 14, 2023, at 12:10 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:

? I've responded in bits and pieces to this thread and haven't done an excellent job expressing my overall opinion. This is probably because my initial goal was to point out that GPS-transmitted time is no less subject to being attacked than your garden variety NTP-transmitted time. Since this thread has evolved, I'd like to describe my overall position to be a bit clearer.
To start, we need a somewhat simplified version of how UTC is created so I can refer to it later:
Across the globe, approximately 85 research and standards institutions run a set of freestanding atomic clocks that contribute to UTC. The number of atomic clocks across all these institutions totals around 450. Each institution also produces a version of UTC based on its own set of atomic clocks. In the international timekeeping world, this is designated as UTC(Laboratory), where Laboratory is replaced with the abbreviation for the lab producing that version of UTC. So UTC(NIST) is the version that NIST produces at Boulder, Colorado, NICT produces UTC(NICT) in Tokyo, and so on.
Because no clock is perfectly accurate, all of these versions of UTC drift in relation to each other, and you could have significant differences in time between different labs. As a result, there has to be a way to synchronize them. Each month, the standards organization BIPM collects relative time measurements and other statistics from each institution described above. This data is then used to determine the actual value of UTC. BIPM then produces a report detailing each organization's difference from the correct representation of UTC. Each institution uses this data to adjust its UTC representation, and the cycle repeats the next month. In this way, all of the representations of UTC end up being pretty close to each other. The document BIPM produces is titled "Circular T." The most recent version indicates that most of the significant standards institutions maintain a UTC version that differs by less than 10ns from the official version of UTC.
Note that 10ns is far more accurate than we need for NTP, so most of the UTC representations can be considered identical as far as this discussion goes. Still, it is essential to realize that UTC(NIST) is generated separately from UTC(USNO) or other UTC implementations. For example, a UTC(NIST) failure should not cause UTC(USNO) to fail as they utilize separate hardware and systems.
Each of these versions of UTC is also disseminated in various ways. UTC(NIST) goes out via the "WWV" radio stations, NTP, and other esoteric methods. GPS primarily distributes UTC(USNO), which is also available directly via NTP. UTC(SU) is the timescale for GLONASS. And so on.
So, back to NTP and the accuracy required:
Most end users (people running everyday web applications or streaming video or similar) don't need precisely synchronized time. The most sensitive application I'm aware of in this space is likely TOTP, which often needs time on the server and time on the client (or hardware key) within 90 seconds of each other. In addition, having NTP time fail usually isn't the end of the world for these users. The best way to synchronize their computers (including desktop and server systems) to UTC is to point their computer time synchronization service (whatever that is) at http://pool.ntp.org"] pool.ntp.org, http://time.windows.com"]time.windows.com, their ISP's time server, or similar. Or, with modern OS'es, you can leave the time configured to whatever server the OS manufacturer preconfigured. As an aside, one should note that historically windows ticked at 15ms or so, so trying to synchronize most windows closer than 15ms was futile.
On the other hand, large ISPs or other service providers (including content providers) see real benefits to having systems synchronized to fractions of seconds of UTC. Comparing logs and traces becomes much easier when you know that something logged at 10:02:23.1 on one device came before something logged at 10:02:23.5 on another. Various server-to-server protocols and software implementations need time to be synchronized to sub-second intervals since they rely on timestamps to determine the latest copy of data, and so on. In addition, as an ISP, you'll often provide time services to downstream customers who demand more accuracy and reliability than is strictly necessary.
As a result, one wants to ensure that all time servers are synchronized within some reasonable standard of accuracy. Within 100ms is acceptable for most applications but a goal of under 50ms is better. If you have local GPS receivers, times down to around 1ms is achievable with careful design. Beyond that, you're chasing unnecessary accuracy. Note that loss of precision is somewhat cumulative here - running a time server synchronized to within 100ms will ensure that no client can be synchronized to better than within 100ms from that server. Generally, you'll want your time server to be synchronized much better than needed to avoid the time server being the limiting factor.
In a perfect world with no bad actors and where all links ran perfectly, one could set up an NTP server that pulled from http://pool.ntp.org"]pool.ntp.org or used GPS and essentially acted as a proxy. Unfortunately, we don't live in this world. So one has to ask how you build a system that meets at least the following goals:
* Synchronized to UTC within 50ms, with lower being better. * Not subject to a reasonable set of attacks (typical DoS attacks, RF signal attacks, spoofing, etc). * Able to be run by typical network operations staff
In addition, an ideal server setup would be made up of redundant servers in case one piece of hardware fails. I will ignore this part, as it's usually just setting up multiple copies of the same thing.
The two most straightforward options are using a GPS-based NTP appliance or installing an NTP server and pointing it at http://pool.ntp.org"]pool.ntp.org. Under normal circumstances, both options will be synchronized to UTC with enough accuracy for most applications, and both are easy to run by typical network operations staff. This assumes reasonably consistent network latency in the NTP case and a good sky view in the GPS case. The GPS-based appliance is, however, subject to spoofing or jamming, as I've discussed earlier. The NTP server is at the mercy of the quality of the servers it picked from http://pool.ntp.org"]pool.ntp.org and is also subject to various outside attacks (spoofing, etc.). One must decide how critical time is to them before deciding whether this option is valid.
The other end of the scale is the "develop your own offline version of UTC using atomic clocks" methodology. This fixes the attack issue but introduces several others. The main one is that you are now relying on the clock's accuracy. Admittedly rubidium and especially cesium clocks tend to be sufficiently reliable and stable. However, one has to ensure the frequency is accurate initially and stays that way. You must also wire the clock to an NTP Server and calibrate the initial UTC offset. If the clock goes haywire or is less accurate than is required, your in-house version of UTC will drift in relation to real UTC. This means you may need 2 or 3 or more atomic clocks to be sufficiently reliable. You'll then need to regularly take an average, compare it to UTC, and adjust if it's drifted too much. This quickly becomes more of a science project than something you want network operations staff to deal with on an ongoing basis. To be clear: If you need robust time not subject to outside forces and have or can obtain the skill set to pull this off internally, I won't argue that this is a bad option. However, I feel this isn't the type of service most providers want to run internally.
So, looking at some middle-ground options that trade a bit of robustness for ease of use is reasonable.
My lowest cost preference has always been to use a set of in-house NTP servers pointed at a carefully curated collection of NTP servers. Your curation strategy should depend on network connectivity, the reliability of the time sources, etc. In North America, picking one or two NIST servers from each NIST location is a good starting point. That is one or two from each of Maryland, Fort Collins, Boulder, and the University of Colorado. One may want to add some servers from other timekeeping organizations (such as USNO). Note that there is one commonality: These time servers are run by organizations listed in circular T as contributing to UTC, and the servers are tied to the atomic clocks. That way, we ensure that the servers are not subject to inaccuracies caused by time transfer from an authoritative source for UTC. What is left is any potential attack on the time transfer over NTP itself. I would argue that with a curated list of enough NTP servers, this risk can be pushed down to where it is low enough for many use cases. A lot will depend on the quantity and quality of NTP servers you select and the robustness of the network path to those servers. If the packets between your NTP server and the NTP servers you choose traverse a relatively secure and short path with plenty of bandwidth, and the paths to differing NTP servers are diverse, many attacks will become harder to implement. In addition, the more NTP servers you add, the more likely it is that NTP will be able to correctly pick the servers providing the correct time, even if an attacker is successfully spoofing one or more sources. In some cases it may make sense to add additional servers which are run by third parties if it gains additional robustness based on network architecture. This is especially true if you're closely connected network-wise with the third party and they run a good quality NTP service as well.
As I've mentioned, a good middle-of-the-road solution is adding various sources of time derived via GPS. Note I said, "to add." Start with the carefully curated NTP server set, then install one or more GPS-based NTP Servers polled by your NTP server. Adding these GPS time sources to your NTP servers does three things: First, it provides another source of time NTP can use to determine the correct time. Second, we're now using a different time transmission method with different vulnerabilities. And finally, it will significantly improve the accuracy of the time the NTP server produces as NTPd will generally prefer it to do the final trimming to UTC. The strength of the combination of both terrestrial transmitted time via NTP and the precision of rf-transmitted GPS time ensures that time is both correct and precise. There are still attack vectors here, but as you add more time sources, the complexity of pulling off a successful attack increases. This is especially true if you can monitor the NTP server for signs of stress, such as time servers that are not telling the correct time or GPS signals which are inconsistent with the NTP-derived time. A successful attack would require simultaneous NTP (network) and GPS (rf) attacks.
Other options or blends of options are also possible. With a reasonably large network, putting enough GPS receivers into place would significantly reduce the possibility of a spoofer or jammer taking out your entire GPS infrastructure. Reducing or eliminating external NTP time sources might be reasonable in that case. The theory is that attacking GPS receivers at one location is easy. Doing it at dozens simultaneously is much more difficult. To use an exaggeration to make a point: If you had 100 different GPS receivers spread across 100 widely geographically diverse locations, and all of your NTP servers were able to poll all of them for time, the chances that an attacker would be able to take out or spoof enough GPS receivers to make a difference would be close to zero. Your failure point becomes UTC(USNO) and the GPS constellation itself. The same argument would apply to NTP servers regarding quantity and diversity.
Other options involve adding additional technologies. For example, some appliances use GPS to discipline (adjust) an internal atomic clock. Once the atomic clock is locked to UTC, the GPS can fail for extended periods without affecting NTP output. In addition, some of these will filter updates from the GPS based on the appliance's internal atomic time. That way, a spoofer would be ignored, jammers would have to continue for hours or days, and so on. Of course, these solutions' reliability depends on the implementation quality. If I had the budget to implement something like this in a network, I'd likely scatter a few of these around the network and then still use garden variety NTPd servers which would be pointed at these appliances. I might even consider buying solutions from multiple vendors to ensure a bug in one implementation was filtered out and ignored.
I can't cover every option here, but balancing security, cost, operational complexity, and application needs is the key. Some solutions are cheap and easy but not robust. Some are highly robust but expensive and not easy. Somewhere in the middle is probably where most real implementations should lie.
Now, to address a couple of specific items:
1) Additional GPS and commercial time distribution systems will likely improve reliability. However, only GPS and GALILEO are available for free in the US. I'm ignoring GLONASS for various legal and political reasons. GALILEO is a valid option but it lives in the same band as GPS, so jamming GPS will usually also jam GALILEO. Utilizing GNSS receivers that use the civilian signals in the newer bands would also help. Some commercial solutions are available that don't require GNSS, but they're relatively new and not as commonly available as one would like.
2) For running my own time servers in a service-provider environment, I'd rather specifically designate the exact NTP server I want to utilize and not rely on a third party to give me a pool of servers. It's more about ensuring the server I use is running a trusted server, and if I delegate the server selection, I lose this ability. On the other hand, where I'm not running a NTP server that is critical for many clients, I'll just point it at http://pool.ntp.org"]pool.ntp.org, or http://north-america.pool.ntp.org"] north-america.pool.ntp.org and skip all of the recommendations that I've made above. I would be cautious about requesting http://pool.ntp.org"]pool.ntp.org add entries for "stratum of server" or "origin of time" as this seems like it would tend to overload the stratum one servers in the pool with people "optimizing" their configuration to use only stratum one servers. Remember that http://pool.ntp.org"]pool.ntp.org is generally intended as an end-user-device service, and providing methods that end users can bypass the robustness that a fully distributed pool will provide is probably not a great idea.
3) This all should hopefully sort itself out over the next few years. GPS and GALILEO are flying new birds that have changes designed to improve attack resilience by using cryptography to ensure authentic transmissions (which may rely on ground transmission of cryptographic keys). NTP already supports manual cryptographic keys that work, but NTS is a pain in the rear. Hopefully, NTPv5 will have a better security mechanism. Other, more secure, time sources are on the horizon as the cybersecurity crowd is aware of the issues.
And finally, as a sort of a tl;dr; Summary: Each operator needs to decide how critical time is to their network and pick a solution that works for them and fits the organization's budget. Some operators might point everything at http://pool.ntp.org"]pool.ntp.org and not run their own servers. Others might run their own time lab and use that time to provide NTP time and precision time and frequency via various methods. Most will be somewhere in between. But regardless of which you choose, please be aware that GPS isn't 100% secure, and neither is NTP. If attack resilience matters to you, you should think about all of the attack vectors and design something that is robust enough to meet your use case.

--
- Forrest

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.



-mel beckman

On Aug 16, 2023, at 3:51 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:

?Mel Beckman wrote:-

Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation.

The event took place on the evening of Sunday 12 July 2020, and seems NOT
to have been due to an issue caused directly by GPS, but rather to
misbehaviour of a GPS NTP server relating to week numbers. Our regulator
subsequently issued the following comprehensive document:-

https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf

By way of summary, JT operated two GPS derived NTP servers, with all of
their routers were pointing to both. On the evening in question, one of
the two reset its clock back to 27 November 2000.

Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication. The authentication [section 4.19]
was described having a "password validity start date" of 01 July 2012.
Thus, any routers which had picked up the time from the faulty source no
longer had valid IS-IS authentication and were thus isolated.

Whilst only 15% of their routers were affected, this was enough to cause an
almost total failure in their network, affecting telephony (fixed & mobile)
and Internet. For foreign readers (this is NANOG!) "999" calls refer to
the emergency services in these parts, where any failures attract the
attention of our regulator.

The details of why the clock "failed" start at section 4.23, and seem to
relate a GPS week number rollover.

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

One curious aspect is that some routers followed the "bad" time, which is
alluded to in section 4.31.

Something not discussed in that report is that JT's email failed during the
incident despite its being hosted on Office365. The reason was that the
two authoritative DNS servers for jtglobal.com were hosted in Jersey inside
their network. As that network was wholly disconnected, there was no DNS
and hence no email. Despite my having raised this since with their senior
management, their DNS remains hosted in this way:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @ns1.jtibs.net
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;jtglobal.com. IN NS

;; ANSWER SECTION:
jtglobal.com. 60 IN NS ns2.jtibs.net.
jtglobal.com. 60 IN NS ns1.jtibs.net.

;; ADDITIONAL SECTION:
ns1.jtibs.net. 60 IN A 212.9.0.135
ns2.jtibs.net. 60 IN A 212.9.0.136
ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1
ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2

Rediculously (and again despite my agitation to their management) our
government domain gov.je has similar DNS fragility:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;gov.je. IN NS

;; ANSWER SECTION:
gov.je. 3600 IN NS ns2.gov.je.
gov.je. 3600 IN NS ns1.gov.je.

;; ADDITIONAL SECTION:
ns2.gov.je. 3600 IN A 212.9.21.137
ns1.gov.je. 3600 IN A 212.9.21.9

--
Best wishes,
Matthew

------
From: Mel Beckman <mel@beckman.org>
To: Matthew Richardson <matthew-l@itconsult.co.uk>
Cc: Nanog <nanog@nanog.org>
Date: Tue, 8 Aug 2023 15:12:29 +0000
Subject: Re: NTP Sync Issue Across Tata (Europe)

Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation.

Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical - but conceivable - meteor storm). But that would be a fall-back. I would not mix the systems.

-mel

On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:

?Mel Beckman wrote:-

It's a problem that has received a lot of attention in both NTP and
aviation navigation circles. What is hard to defend against is total signal
suppression via high powered jamming. But that you can do with a
geographically diverse GPS NTP network.

Whilst looking forward to being corrected, GPS (even across multiple
locations) seems to be a SINGLE source of time. You seem (have I
misunderstood?) to be a proponent of using GPS exclusively as the external
clock source.

Might it be preferable to have a mixture of GPS (perhaps with another GNSS)
together with carefully selected Internet-based NTP servers?

I recall an incident over here in Jersey (the one they named New Jersey
after!) where our primary telco had a substantial time shift on one of
their two GPS synced servers. This managed to adjust the clock on enough
of their routers that the certificate-based OSPF authentication considered
the certificates invalid, and caused a failure of almost their whole
network.

This is, of course, not to say that GPS is not a very good clock source,
but rather to wonder whether more diversity would be preferable than using
it as a single source.

--
Best wishes,
Matthew

------
From: Mel Beckman <mel@beckman.org>
To: "Forrest Christian (List Account)" <lists@packetflux.com>
Cc: Nanog <nanog@nanog.org>
Date: Mon, 7 Aug 2023 14:03:30 +0000
Subject: Re: NTP Sync Issue Across Tata (Europe)

Forrest,

GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.

-mel

On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:

?
The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.

With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.

In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.

Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service .



On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.

DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.

Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager.

I sense hand-waving :)

-mel via cell

On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote:

?


On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:

The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.

BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).

Rubens

Thanks for that link.

This is jumping out at me though :

Their interior routing protocol used amongst their mesh of routers was
> IS-IS which was using authentication. The authentication [section 4.19]
> was described having a "password validity start date" of 01 July 2012.
> Thus, any routers which had picked up the time from the faulty source no
> longer had valid IS-IS authentication and were thus isolated.
>

It's been a while, but last time I remember diving into the IS-IS weeds ,
the time of the transmitting system wasn't part of a Hello. Is this a
Cisco specific option they toss in a TLV?

On Wed, Aug 16, 2023 at 9:04?AM Matthew Richardson via NANOG <
nanog@nanog.org> wrote:

> Mel Beckman wrote:-
>
> >Do you have a citation for your Jersey event? I doubt GPS caused the
> problem, but I’d like to see the documentation.
>
> The event took place on the evening of Sunday 12 July 2020, and seems NOT
> to have been due to an issue caused directly by GPS, but rather to
> misbehaviour of a GPS NTP server relating to week numbers. Our regulator
> subsequently issued the following comprehensive document:-
>
>
> https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf
>
> By way of summary, JT operated two GPS derived NTP servers, with all of
> their routers were pointing to both. On the evening in question, one of
> the two reset its clock back to 27 November 2000.
>
> Their interior routing protocol used amongst their mesh of routers was
> IS-IS which was using authentication. The authentication [section 4.19]
> was described having a "password validity start date" of 01 July 2012.
> Thus, any routers which had picked up the time from the faulty source no
> longer had valid IS-IS authentication and were thus isolated.
>
> Whilst only 15% of their routers were affected, this was enough to cause an
> almost total failure in their network, affecting telephony (fixed & mobile)
> and Internet. For foreign readers (this is NANOG!) "999" calls refer to
> the emergency services in these parts, where any failures attract the
> attention of our regulator.
>
> The details of why the clock "failed" start at section 4.23, and seem to
> relate a GPS week number rollover.
>
> So, probably not a failure "caused by GPS", rather one caused by poor
> design (only two clock sources) combined with unsupported and buggy
> devices.
>
> One curious aspect is that some routers followed the "bad" time, which is
> alluded to in section 4.31.
>
> Something not discussed in that report is that JT's email failed during the
> incident despite its being hosted on Office365. The reason was that the
> two authoritative DNS servers for jtglobal.com were hosted in Jersey
> inside
> their network. As that network was wholly disconnected, there was no DNS
> and hence no email. Despite my having raised this since with their senior
> management, their DNS remains hosted in this way:-
>
> >matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @
> ns1.jtibs.net
> >;; Got answer:
> >;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
> >;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
> >
> >;; QUESTION SECTION:
> >;jtglobal.com. IN NS
> >
> >;; ANSWER SECTION:
> >jtglobal.com. 60 IN NS ns2.jtibs.net.
> >jtglobal.com. 60 IN NS ns1.jtibs.net.
> >
> >;; ADDITIONAL SECTION:
> >ns1.jtibs.net. 60 IN A 212.9.0.135
> >ns2.jtibs.net. 60 IN A 212.9.0.136
> >ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1
> >ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2
>
> Rediculously (and again despite my agitation to their management) our
> government domain gov.je has similar DNS fragility:-
>
> >matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @
> ns1.gov.je
> >;; Got answer:
> >;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
> >;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
> >
> >;; QUESTION SECTION:
> >;gov.je. IN NS
> >
> >;; ANSWER SECTION:
> >gov.je. 3600 IN NS ns2.gov.je.
> >gov.je. 3600 IN NS ns1.gov.je.
> >
> >;; ADDITIONAL SECTION:
> >ns2.gov.je. 3600 IN A 212.9.21.137
> >ns1.gov.je. 3600 IN A 212.9.21.9
>
> --
> Best wishes,
> Matthew
>
> ------
> >From: Mel Beckman <mel@beckman.org>
> >To: Matthew Richardson <matthew-l@itconsult.co.uk>
> >Cc: Nanog <nanog@nanog.org>
> >Date: Tue, 8 Aug 2023 15:12:29 +0000
> >Subject: Re: NTP Sync Issue Across Tata (Europe)
>
> >Until the Internet NTP network can be made secure, no. Do you have a
> citation for your Jersey event? I doubt GPS caused the problem, but I’d
> like to see the documentation.
> >
> >Using GPS for time sync is simple risk management: the risk of Internet
> NTP with known, well documented vulnerabilities and many security
> incidents, versus the risk of some theoretical GPS-based vulnerability, for
> which mitigations such as geographic diversity are readily available. Sure,
> you could use Internet NTP as a last resort should GPS fail globally
> (perhaps due to a theoretical — but conceivable — meteor storm). But that
> would be a fall-back. I would not mix the systems.
> >
> > -mel
> >
> >> On Aug 8, 2023, at 1:36 AM, Matthew Richardson <
> matthew-l@itconsult.co.uk> wrote:
> >>
> >> ?Mel Beckman wrote:-
> >>
> >>> It's a problem that has received a lot of attention in both NTP and
> >>> aviation navigation circles. What is hard to defend against is total
> signal
> >>> suppression via high powered jamming. But that you can do with a
> >>> geographically diverse GPS NTP network.
> >>
> >> Whilst looking forward to being corrected, GPS (even across multiple
> >> locations) seems to be a SINGLE source of time. You seem (have I
> >> misunderstood?) to be a proponent of using GPS exclusively as the
> external
> >> clock source.
> >>
> >> Might it be preferable to have a mixture of GPS (perhaps with another
> GNSS)
> >> together with carefully selected Internet-based NTP servers?
> >>
> >> I recall an incident over here in Jersey (the one they named New Jersey
> >> after!) where our primary telco had a substantial time shift on one of
> >> their two GPS synced servers. This managed to adjust the clock on
> enough
> >> of their routers that the certificate-based OSPF authentication
> considered
> >> the certificates invalid, and caused a failure of almost their whole
> >> network.
> >>
> >> This is, of course, not to say that GPS is not a very good clock source,
> >> but rather to wonder whether more diversity would be preferable than
> using
> >> it as a single source.
> >>
> >> --
> >> Best wishes,
> >> Matthew
> >>
> >> ------
> >>> From: Mel Beckman <mel@beckman.org>
> >>> To: "Forrest Christian (List Account)" <lists@packetflux.com>
> >>> Cc: Nanog <nanog@nanog.org>
> >>> Date: Mon, 7 Aug 2023 14:03:30 +0000
> >>> Subject: Re: NTP Sync Issue Across Tata (Europe)
> >>
> >>> Forrest,
> >>>
> >>> GPS spoofing may work with a primitive Raspberry Pi-based NTP server,
> but commercial industrial NTP servers have specific anti-spoofing
> mitigations. There are also antenna diversity strategies that vendors
> support to ensure the signal being relied upon is coming from the right
> direction. It's a problem that has received a lot of attention in both NTP
> and aviation navigation circles. What is hard to defend against is total
> signal suppression via high powered jamming. But that you can do with a
> geographically diverse GPS NTP network.
> >>>
> >>> -mel
> >>>
> >>> On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <
> lists@packetflux.com> wrote:
> >>>
> >>> ?
> >>> The problem with relying exclusively on GPS to do time distribution is
> the ease with which one can spoof the GPS signals.
> >>>
> >>> With a budget of around $1K, not including a laptop, anyone with
> decent technical skills could convince a typical GPS receiver it was at any
> position and was at any time in the world. All it takes is a decent
> directional antenna, some SDR hardware, and depending on the location and
> directivity of your antenna maybe a smallish amplifier. There is much
> discussion right now in the PNT (Position, Navigation and Timing) community
> as to how best to secure the GNSS network, but right now one should
> consider the data from GPS to be no more trustworthy than some random NTP
> server on the internet.
> >>>
> >>> In order to build a resilient NTP server infrastructure you need
> multiple sources of time distributed by multiple methods - typically both
> via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty
> good job of sorting out multiple time servers and discarding sources that
> are lying. But to do this you need multiple time sources. A common
> recommendation is to run a couple/few NTP servers which only get time from
> a GPS receiver and only serve time to a second tier of servers that pull
> from both those in-house GPS-timed-NTP servers and other trusted NTP
> servers. I'd recommend selecting the time servers to gain geographic
> diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly
> both.
> >>>
> >>> Note that NIST will exchange (via mail) a set of keys with you to talk
> encrypted NTP with you. See
> https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service
> .
> >>>
> >>>
> >>>
> >>> On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:
> mel@beckman.org>> wrote:
> >>> GPS Selective Availability did not disrupt the timing chain of GPS,
> only the ephemeris (position information). But a government-disrupted
> timebase scenario has never occurred, while hackers are a documented threat.
> >>>
> >>> DNS has DNSSec, which while not deployed as broadly as we might like,
> at least lets us know which servers we can trust.
> >>>
> >>> Your own atomic clocks still have to be synced to a common standard to
> be useful. To what are they sync'd? GPS, I'll wager.
> >>>
> >>> I sense hand-waving :)
> >>>
> >>> -mel via cell
> >>>
> >>> On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:
> rubensk@gmail.com>> wrote:
> >>>
> >>> ?
> >>>
> >>>
> >>> On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:
> mel@beckman.org>> wrote:
> >>> Or one can read recent research papers that thoroughly document the
> incredible fragility of the existing NTP hierarchy and soberly consider
> their recommendations for remediation:
> >>>
> >>> The paper suggests the compromise of critical infrastructure. So,
> besides not using NTP, why not stop using DNS ? Just populate a hosts file
> with all you need.
> >>>
> >>> BTW, the stratum-0 source you suggested is known to have been
> manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/),
> so you need to bet on that specific state actor not returning to old habits.
> >>>
> >>> OTOH, 4 of the 5 servers I suggested have their own atomic clock, and
> you can keep using GPS as well. If GPS goes bananas on timing, that source
> will just be disregarded (one of the features of the NTP architecture that
> has been pointed out over and over in this thread and you keep ignoring it).
> >>>
> >>> Rubens
> >>
>
>

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

Interesting software bug, but not really germane to this discussion, other than as a cautionary tale about time distribution architectures.


-mel beckman

On Aug 16, 2023, at 3:51 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:

?Mel Beckman wrote:-

Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation.

The event took place on the evening of Sunday 12 July 2020, and seems NOT
to have been due to an issue caused directly by GPS, but rather to
misbehaviour of a GPS NTP server relating to week numbers. Our regulator
subsequently issued the following comprehensive document:-

https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf

By way of summary, JT operated two GPS derived NTP servers, with all of
their routers were pointing to both. On the evening in question, one of
the two reset its clock back to 27 November 2000.

Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication. The authentication [section 4.19]
was described having a "password validity start date" of 01 July 2012.
Thus, any routers which had picked up the time from the faulty source no
longer had valid IS-IS authentication and were thus isolated.

Whilst only 15% of their routers were affected, this was enough to cause an
almost total failure in their network, affecting telephony (fixed & mobile)
and Internet. For foreign readers (this is NANOG!) "999" calls refer to
the emergency services in these parts, where any failures attract the
attention of our regulator.

The details of why the clock "failed" start at section 4.23, and seem to
relate a GPS week number rollover.

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

One curious aspect is that some routers followed the "bad" time, which is
alluded to in section 4.31.

Something not discussed in that report is that JT's email failed during the
incident despite its being hosted on Office365. The reason was that the
two authoritative DNS servers for jtglobal.com were hosted in Jersey inside
their network. As that network was wholly disconnected, there was no DNS
and hence no email. Despite my having raised this since with their senior
management, their DNS remains hosted in this way:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @ns1.jtibs.net
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;jtglobal.com. IN NS

;; ANSWER SECTION:
jtglobal.com. 60 IN NS ns2.jtibs.net.
jtglobal.com. 60 IN NS ns1.jtibs.net.

;; ADDITIONAL SECTION:
ns1.jtibs.net. 60 IN A 212.9.0.135
ns2.jtibs.net. 60 IN A 212.9.0.136
ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1
ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2

Rediculously (and again despite my agitation to their management) our
government domain gov.je has similar DNS fragility:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;gov.je. IN NS

;; ANSWER SECTION:
gov.je. 3600 IN NS ns2.gov.je.
gov.je. 3600 IN NS ns1.gov.je.

;; ADDITIONAL SECTION:
ns2.gov.je. 3600 IN A 212.9.21.137
ns1.gov.je. 3600 IN A 212.9.21.9

--
Best wishes,
Matthew

------
From: Mel Beckman <mel@beckman.org>
To: Matthew Richardson <matthew-l@itconsult.co.uk>
Cc: Nanog <nanog@nanog.org>
Date: Tue, 8 Aug 2023 15:12:29 +0000
Subject: Re: NTP Sync Issue Across Tata (Europe)

Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation.

Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical - but conceivable - meteor storm). But that would be a fall-back. I would not mix the systems.

-mel

On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:

?Mel Beckman wrote:-

It's a problem that has received a lot of attention in both NTP and
aviation navigation circles. What is hard to defend against is total signal
suppression via high powered jamming. But that you can do with a
geographically diverse GPS NTP network.

Whilst looking forward to being corrected, GPS (even across multiple
locations) seems to be a SINGLE source of time. You seem (have I
misunderstood?) to be a proponent of using GPS exclusively as the external
clock source.

Might it be preferable to have a mixture of GPS (perhaps with another GNSS)
together with carefully selected Internet-based NTP servers?

I recall an incident over here in Jersey (the one they named New Jersey
after!) where our primary telco had a substantial time shift on one of
their two GPS synced servers. This managed to adjust the clock on enough
of their routers that the certificate-based OSPF authentication considered
the certificates invalid, and caused a failure of almost their whole
network.

This is, of course, not to say that GPS is not a very good clock source,
but rather to wonder whether more diversity would be preferable than using
it as a single source.

--
Best wishes,
Matthew

------
From: Mel Beckman <mel@beckman.org>
To: "Forrest Christian (List Account)" <lists@packetflux.com>
Cc: Nanog <nanog@nanog.org>
Date: Mon, 7 Aug 2023 14:03:30 +0000
Subject: Re: NTP Sync Issue Across Tata (Europe)

Forrest,

GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.

-mel

On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:

?
The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.

With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.

In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.

Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service .



On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.

DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.

Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager.

I sense hand-waving :)

-mel via cell

On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote:

?


On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:

The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.

BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).

Rubens

>
> So, probably not a failure "caused by GPS", rather one caused by poor
> design (only two clock sources) combined with unsupported and buggy
> devices.


100% correct. From the PDF :

4.31 JT summarised its findings in relation to the ‘Panic Timer’ on the
> Cisco IOS XR NTP Client, namely that: JT’s efforts in understanding the
> root cause, and mitigation steps to take to avoid future incidents have
> focused on the Cisco NTP Client behaviour, and notably Cisco’s decision to
> not implement the ‘Panic Timer’ on their IOS XR operating system. Arguably,
> whilst the NTP server injected an invalid time into the network, it is the
> NTP Clients filtering and selection algorithms which are responsible for
> detecting and disregarding falsetickers, and it was the Cisco NTP Clients
> failure to appropriately handle this which triggered the network incident.
> 43 […] Further detailed soak testing, log analysis and debug analysis
> corroborated that the Cisco IOS XR NTP Client did not implement the ‘Panic
> Timer’ that would normally cause an NTP Client to ignore an NTP Server
> exceeding 1000 seconds variance.


On Wed, Aug 16, 2023 at 10:50?AM Mel Beckman <mel@beckman.org> wrote:

> So, probably not a failure "caused by GPS", rather one caused by poor
> design (only two clock sources) combined with unsupported and buggy
> devices.
>
>
>
> -mel beckman
>
> On Aug 16, 2023, at 3:51 AM, Matthew Richardson <matthew-l@itconsult.co.uk>
> wrote:
>
> ?Mel Beckman wrote:-
>
> Do you have a citation for your Jersey event? I doubt GPS caused the
> problem, but I'd like to see the documentation.
>
>
> The event took place on the evening of Sunday 12 July 2020, and seems NOT
> to have been due to an issue caused directly by GPS, but rather to
> misbehaviour of a GPS NTP server relating to week numbers. Our regulator
> subsequently issued the following comprehensive document:-
>
>
> https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf
>
> By way of summary, JT operated two GPS derived NTP servers, with all of
> their routers were pointing to both. On the evening in question, one of
> the two reset its clock back to 27 November 2000.
>
> Their interior routing protocol used amongst their mesh of routers was
> IS-IS which was using authentication. The authentication [section 4.19]
> was described having a "password validity start date" of 01 July 2012.
> Thus, any routers which had picked up the time from the faulty source no
> longer had valid IS-IS authentication and were thus isolated.
>
> Whilst only 15% of their routers were affected, this was enough to cause an
> almost total failure in their network, affecting telephony (fixed & mobile)
> and Internet. For foreign readers (this is NANOG!) "999" calls refer to
> the emergency services in these parts, where any failures attract the
> attention of our regulator.
>
> The details of why the clock "failed" start at section 4.23, and seem to
> relate a GPS week number rollover.
>
> So, probably not a failure "caused by GPS", rather one caused by poor
> design (only two clock sources) combined with unsupported and buggy
> devices.
>
> One curious aspect is that some routers followed the "bad" time, which is
> alluded to in section 4.31.
>
> Something not discussed in that report is that JT's email failed during the
> incident despite its being hosted on Office365. The reason was that the
> two authoritative DNS servers for jtglobal.com were hosted in Jersey
> inside
> their network. As that network was wholly disconnected, there was no DNS
> and hence no email. Despite my having raised this since with their senior
> management, their DNS remains hosted in this way:-
>
> matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @
> ns1.jtibs.net
>
> ;; Got answer:
>
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
>
> ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
>
>
> ;; QUESTION SECTION:
>
> ;jtglobal.com. IN NS
>
>
> ;; ANSWER SECTION:
>
> jtglobal.com. 60 IN NS ns2.jtibs.net.
>
> jtglobal.com. 60 IN NS ns1.jtibs.net.
>
>
> ;; ADDITIONAL SECTION:
>
> ns1.jtibs.net. 60 IN A 212.9.0.135
>
> ns2.jtibs.net. 60 IN A 212.9.0.136
>
> ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1
>
> ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2
>
>
> Rediculously (and again despite my agitation to their management) our
> government domain gov.je has similar DNS fragility:-
>
> matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
>
> ;; Got answer:
>
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
>
> ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
>
>
> ;; QUESTION SECTION:
>
> ;gov.je. IN NS
>
>
> ;; ANSWER SECTION:
>
> gov.je. 3600 IN NS ns2.gov.je.
>
> gov.je. 3600 IN NS ns1.gov.je.
>
>
> ;; ADDITIONAL SECTION:
>
> ns2.gov.je. 3600 IN A 212.9.21.137
>
> ns1.gov.je. 3600 IN A 212.9.21.9
>
>
> --
> Best wishes,
> Matthew
>
> ------
>
> From: Mel Beckman <mel@beckman.org>
>
> To: Matthew Richardson <matthew-l@itconsult.co.uk>
>
> Cc: Nanog <nanog@nanog.org>
>
> Date: Tue, 8 Aug 2023 15:12:29 +0000
>
> Subject: Re: NTP Sync Issue Across Tata (Europe)
>
>
> Until the Internet NTP network can be made secure, no. Do you have a
> citation for your Jersey event? I doubt GPS caused the problem, but I'd
> like to see the documentation.
>
>
> Using GPS for time sync is simple risk management: the risk of Internet
> NTP with known, well documented vulnerabilities and many security
> incidents, versus the risk of some theoretical GPS-based vulnerability, for
> which mitigations such as geographic diversity are readily available. Sure,
> you could use Internet NTP as a last resort should GPS fail globally
> (perhaps due to a theoretical - but conceivable - meteor storm). But that
> would be a fall-back. I would not mix the systems.
>
>
> -mel
>
>
> On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk>
> wrote:
>
>
> ?Mel Beckman wrote:-
>
>
> It's a problem that has received a lot of attention in both NTP and
>
> aviation navigation circles. What is hard to defend against is total signal
>
> suppression via high powered jamming. But that you can do with a
>
> geographically diverse GPS NTP network.
>
>
> Whilst looking forward to being corrected, GPS (even across multiple
>
> locations) seems to be a SINGLE source of time. You seem (have I
>
> misunderstood?) to be a proponent of using GPS exclusively as the external
>
> clock source.
>
>
> Might it be preferable to have a mixture of GPS (perhaps with another GNSS)
>
> together with carefully selected Internet-based NTP servers?
>
>
> I recall an incident over here in Jersey (the one they named New Jersey
>
> after!) where our primary telco had a substantial time shift on one of
>
> their two GPS synced servers. This managed to adjust the clock on enough
>
> of their routers that the certificate-based OSPF authentication considered
>
> the certificates invalid, and caused a failure of almost their whole
>
> network.
>
>
> This is, of course, not to say that GPS is not a very good clock source,
>
> but rather to wonder whether more diversity would be preferable than using
>
> it as a single source.
>
>
> --
>
> Best wishes,
>
> Matthew
>
>
> ------
>
> From: Mel Beckman <mel@beckman.org>
>
> To: "Forrest Christian (List Account)" <lists@packetflux.com>
>
> Cc: Nanog <nanog@nanog.org>
>
> Date: Mon, 7 Aug 2023 14:03:30 +0000
>
> Subject: Re: NTP Sync Issue Across Tata (Europe)
>
>
> Forrest,
>
>
> GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but
> commercial industrial NTP servers have specific anti-spoofing mitigations.
> There are also antenna diversity strategies that vendors support to ensure
> the signal being relied upon is coming from the right direction. It's a
> problem that has received a lot of attention in both NTP and aviation
> navigation circles. What is hard to defend against is total signal
> suppression via high powered jamming. But that you can do with a
> geographically diverse GPS NTP network.
>
>
> -mel
>
>
> On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <
> lists@packetflux.com> wrote:
>
>
> ?
>
> The problem with relying exclusively on GPS to do time distribution is the
> ease with which one can spoof the GPS signals.
>
>
> With a budget of around $1K, not including a laptop, anyone with decent
> technical skills could convince a typical GPS receiver it was at any
> position and was at any time in the world. All it takes is a decent
> directional antenna, some SDR hardware, and depending on the location and
> directivity of your antenna maybe a smallish amplifier. There is much
> discussion right now in the PNT (Position, Navigation and Timing) community
> as to how best to secure the GNSS network, but right now one should
> consider the data from GPS to be no more trustworthy than some random NTP
> server on the internet.
>
>
> In order to build a resilient NTP server infrastructure you need multiple
> sources of time distributed by multiple methods - typically both via
> satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good
> job of sorting out multiple time servers and discarding sources that are
> lying. But to do this you need multiple time sources. A common
> recommendation is to run a couple/few NTP servers which only get time from
> a GPS receiver and only serve time to a second tier of servers that pull
> from both those in-house GPS-timed-NTP servers and other trusted NTP
> servers. I'd recommend selecting the time servers to gain geographic
> diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly
> both.
>
>
> Note that NIST will exchange (via mail) a set of keys with you to talk
> encrypted NTP with you. See
> https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service
> .
>
>
>
>
> On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:
> mel@beckman.org>> wrote:
>
> GPS Selective Availability did not disrupt the timing chain of GPS, only
> the ephemeris (position information). But a government-disrupted timebase
> scenario has never occurred, while hackers are a documented threat.
>
>
> DNS has DNSSec, which while not deployed as broadly as we might like, at
> least lets us know which servers we can trust.
>
>
> Your own atomic clocks still have to be synced to a common standard to be
> useful. To what are they sync'd? GPS, I'll wager.
>
>
> I sense hand-waving :)
>
>
> -mel via cell
>
>
> On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:
> rubensk@gmail.com>> wrote:
>
>
> ?
>
>
>
> On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:
> mel@beckman.org>> wrote:
>
> Or one can read recent research papers that thoroughly document the
> incredible fragility of the existing NTP hierarchy and soberly consider
> their recommendations for remediation:
>
>
> The paper suggests the compromise of critical infrastructure. So, besides
> not using NTP, why not stop using DNS ? Just populate a hosts file with all
> you need.
>
>
> BTW, the stratum-0 source you suggested is known to have been manipulated
> in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you
> need to bet on that specific state actor not returning to old habits.
>
>
> OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you
> can keep using GPS as well. If GPS goes bananas on timing, that source will
> just be disregarded (one of the features of the NTP architecture that has
> been pointed out over and over in this thread and you keep ignoring it).
>
>
> Rubens
>
>
>
>