Mailing List Archive

Hi Mark.

Wouldn't a local server be more optimal?

IE:

server 0.nl.pool.ntp.org
server 1.nl.pool.ntp.org
server 2.nl.pool.ntp.org
server 3.nl.pool.ntp.org

Or for Africa
server 0.za.pool.ntp.org
server 1.za.pool.ntp.org
server 2.za.pool.ntp.org
server 3.za.pool.ntp.org

Matthew

On 8/5/2023 10:41 AM, Mark Tinka wrote:
> Hi all.
>
> I have NTP servers in Europe that are choosing Tata (6453) to get to
> 0.freebsd.pool.ntp.org which lives on 197.224.66.40:
>
> traceroute -I 0.freebsd.pool.ntp.org
> traceroute to 0.freebsd.pool.ntp.org (197.224.66.40), 64 hops max, 48
> byte packets
>
>
> Mark.

On 8/5/23 18:30, Matthew McGehrin wrote:
>
> Hi Mark.
>
> Wouldn't a local server be more optimal?
>
> IE:
>
> server 0.nl.pool.ntp.org
> server 1.nl.pool.ntp.org
> server 2.nl.pool.ntp.org
> server 3.nl.pool.ntp.org

I have a bunch of servers all over Europe I'd prefer not to touch if
this is just a transient issue.


> Or for Africa
> server 0.za.pool.ntp.org
> server 1.za.pool.ntp.org
> server 2.za.pool.ntp.org
> server 3.za.pool.ntp.org

I was speaking about Europe.

There is no problem in Africa.

Mark.

See for yourself how his pool server scores at
https://www.ntppool.org/scores/197.224.66.40

I am not sure why it would be inserted into DNS answers for a worldwide
pool like 0.freebsd as it clearly does have connectivity issues from some
of the pool project's own sensors.
-andreas

On Sat, Aug 5, 2023 at 9:37?AM Mark Tinka <mark@tinka.africa> wrote:

>
> I was speaking about Europe.
>
> There is no problem in Africa.
>
>

On 8/5/23 19:51, Andreas Ott wrote:

> See for yourself how his pool server scores at
> https://www.ntppool.org/scores/197.224.66.40
>
> I am not sure why it would be inserted into DNS answers for a
> worldwide pool like 0.freebsd as it clearly does have connectivity
> issues from some of the pool project's own sensors.

Many thanks, Andreas.

I'll take this up with the FreeBSD folk.

Mark.

Once upon a time, Mark Tinka <mark@tinka.africa> said:
> On 8/5/23 19:51, Andreas Ott wrote:
> >See for yourself how his pool server scores at
> >https://www.ntppool.org/scores/197.224.66.40
> >
> >I am not sure why it would be inserted into DNS answers for a
> >worldwide pool like 0.freebsd as it clearly does have connectivity
> >issues from some of the pool project's own sensors.
>
> Many thanks, Andreas.
>
> I'll take this up with the FreeBSD folk.

It's the NTP pool people you need to talk to - the .freebsd. bit is just
a vendored entry into the pool (more for load tracking and management).

--
Chris Adams <cma@cmadams.net>

On 8/5/23 20:17, Chris Adams wrote:

> It's the NTP pool people you need to talk to - the .freebsd. bit is just
> a vendored entry into the pool (more for load tracking and management).

Yes, Andreas clarified in unicast. Will do. Thanks.

Mark.

Mark,

You might consider setting up your own GPS-based NTP network. Commercial Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as little as $400. Or you can roll your own using a Raspberry Pi or similar nano computer with a GPS module and antenna. We use these exclusively in data centers now rather than depending on Internet NTP servers, primarily for security, because financial transactions in e-commerce can be sensitive to false time information. There are also a variety of NTP-based Internet attacks, so if you can block NTP at your border you’ve eliminated another attack surface.

-mel via cell

> On Aug 5, 2023, at 11:22 AM, Mark Tinka <mark@tinka.africa> wrote:
>
> ?
>
>> On 8/5/23 20:17, Chris Adams wrote:
>>
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just
>> a vendored entry into the pool (more for load tracking and management).
>
> Yes, Andreas clarified in unicast. Will do. Thanks.
>
> Mark.

If the path has simmetric one way latencies you don't have to pick a lower
latency faulty one. Perhaps creating a selection at startup and then using
that collection ?

Rubens

Em sáb., 5 de ago. de 2023 12:42, Mark Tinka <mark@tinka.africa> escreveu:

> Hi all.
>
> I have NTP servers in Europe that are choosing Tata (6453) to get to
> 0.freebsd.pool.ntp.org which lives on 197.224.66.40:
>
> traceroute -I 0.freebsd.pool.ntp.org
> traceroute to 0.freebsd.pool.ntp.org (197.224.66.40), 64 hops max, 48
> byte packets
> 1 ae-2-24.er-01-ams.nl.seacomnet.com (105.26.64.13) 0.300 ms 0.301
> ms 0.215 ms
> 2 ce-0-0-11.cr-01-mrs.fr.seacomnet.com (105.16.8.201) 22.163 ms
> 22.370 ms 22.084 ms
> 3 ce-0-0-3.br-01-mrs.fr.seacomnet.com (105.16.32.254) 20.230 ms
> 20.243 ms 20.139 ms
> 4 ix-hge-0-0-0-28.ecore3.emrs2-marseille.as6453.net (80.231.165.52)
> 21.875 ms 21.679 ms 21.762 ms
> 5 * if-be-3-2.ecore2.emrs2-marseille.as6453.net (195.219.175.0) 42.751
> ms *
> 6 if-ae-25-2.tcore1.ldn-london.as6453.net (195.219.175.5) 43.509 ms
> 43.280 ms 43.353 ms
> 7 195.219.83.158 (195.219.83.158) 203.310 ms 203.452 ms 203.209 ms
> 8 196.20.225.84 (196.20.225.84) 208.289 ms 208.637 ms 208.374 ms
> 9 197.226.230.13 (197.226.230.13) 209.657 ms 209.658 ms 209.830 ms
> 10 197.224.66.40 (197.224.66.40) 208.638 ms 208.632 ms 208.712 ms
>
> NTP is not sync'ing to that address, and sessions stay in an Init state.
>
> Other NTP servers I have in Africa are reaching the same address via local
> Anycast hosters, and sync'ing just fine.
>
> Anyone else seeing this particular issue across Tata in Europe?
>
> Mark.
>

On Sat, Aug 5, 2023 at 12:26?PM Mel Beckman <mel@beckman.org> wrote:
> You might consider setting up your own GPS-based NTP network.

GPS time is monitored (and when necessary, adjusted) from the U.S.
Naval Observatory Master Clock, which is -the- authoritative time
source for the United States. The USNO also provides an NTP time
source from the same master clock:

https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-Naval-Observatory/Precise-Time-Department/Network-Time-Protocol-NTP/

You -should not- just point your servers there, but it's useful to
point a few servers each at one of them in order to serve as your
network stratum 2 sources that keep the rest of your machines in sync
with each other.

That last point is key. You don't want your servers in sync with
random Internet time sources. You want them in sync with each other.

Regards,
Bill Herrin



--
William Herrin
bill@herrin.us
https://bill.herrin.us/

Bill,

That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.

<https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/>
[ddos-lc.png]
NTP amplification DDoS attack<https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/>
cloudflare.com<https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/>


There are also replay and Man in the middle attacks (MITM) which can corrupt local NTP servers’ time basis. Worse, security flaws in NTP make others security protocols, such as SSL, vulnerable.

https://www.sidn.nl/en/news-and-blogs/security-flaws-in-network-time-protocol-make-other-security-protocols-vulnerable

if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

-mel

On Aug 5, 2023, at 6:18 PM, William Herrin <bill@herrin.us> wrote:

?On Sat, Aug 5, 2023 at 12:26?PM Mel Beckman <mel@beckman.org> wrote:
You might consider setting up your own GPS-based NTP network.

GPS time is monitored (and when necessary, adjusted) from the U.S.
Naval Observatory Master Clock, which is -the- authoritative time
source for the United States. The USNO also provides an NTP time
source from the same master clock:

https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-Naval-Observatory/Precise-Time-Department/Network-Time-Protocol-NTP/

You -should not- just point your servers there, but it's useful to
point a few servers each at one of them in order to serve as your
network stratum 2 sources that keep the rest of your machines in sync
with each other.

That last point is key. You don't want your servers in sync with
random Internet time sources. You want them in sync with each other.

Regards,
Bill Herrin



--
William Herrin
bill@herrin.us
https://bill.herrin.us/

* mel@beckman.org (Mel Beckman) [Sun 06 Aug 2023, 04:26 CEST]:
>if you can eliminate such security problems for $400, I say it’s
>cheap at twice the price.

You must be unfamiliar with the prices neutral colocation facilities
charge for roof access.


-- Niels.

?Niels,

You’re the first person to mention neutral collocation facilities as a requirement. The OP only talked about servers generally. Obviously, building your own GPS-based NTP network requires you have visibility to the sky. However, that need not be rooftop access. We routinely locate GPS antennae for time synch at CLEC colos in an available window, behind the glass. Nobody has yet charged us anything more than a one-time cabling fee for that.

But most carrier-neutral colos already have their own GPS-derived, non-Internet time source to which you can subscribe. There is thus no reason to build your own airgapped network, as one is already available from the facility. If you don’t need PTP 50-?s-precise timing for SONET and whatnot, ordinary NTP is very cost effective. For example:

<https://docs.equinix.com/en-us/Content/Edge-Services/EPT/EPT.htm>
About Equinix Precision Time<https://docs.equinix.com/en-us/Content/Edge-Services/EPT/EPT.htm>
docs.equinix.com<https://docs.equinix.com/en-us/Content/Edge-Services/EPT/EPT.htm>
[favicon-16x16.png]<https://docs.equinix.com/en-us/Content/Edge-Services/EPT/EPT.htm>

Most colos also offer their own Stratum-1 public NTP servers for free. For example, Hurricane Electric (https://www.he.net/adm/ntp.html). Being you’re already on-net in the colo, that would still give you NTP that doesn’t transit the Internet. Still way better than pool.ntp.org!


-mel

On Aug 6, 2023, at 4:53 AM, Niels Bakker <niels=nanog@bakker.net> wrote:

?* mel@beckman.org (Mel Beckman) [Sun 06 Aug 2023, 04:26 CEST]:
if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

You must be unfamiliar with the prices neutral colocation facilities charge for roof access.


-- Niels.

On Sat, Aug 5, 2023 at 7:24?PM Mel Beckman <mel@beckman.org> wrote:
> That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.

Hi Mel,

From what I can tell, a fairly simple firewall policy of allow UDP 123
from known NTP clients and established connections (I sent them a UDP
packet recently) stops every one of those attacks (that's actually an
NTP attack and not something else like a DNS attack) except for
upstream address hijack that happens to coincide with your system
boot. And it still depends on the attacker executing an additional
sophisticated attack to do more than cause you a denial of service.

The links you sent are very interesting, at least in an academic
sense, but they don't cause me to be unduly concerned about employing
NTP.


> if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

Except you can't. Redundancy is required for any critical service. At
the $400 price point, your approach has multiple
single-points-of-failure. The device itself of course. Your ability to
receive continuous non-jammed GPS signals at the location where you're
able to place an antenna. And in your plan you'll need one of these in
every discontiguous network where you have equipment since you're not
doing NTP over the Internet.

Not to mention the operations cost. Keeping track of a six inch brick
with a wall wart and an antenna installed at a remote site is... not
entirely abnormal but it's a one-off that consumes manpower.

And then you're only vulnerable to the litany of Internet attacks
which don't involve NTP. Yay!

Don't get me wrong: the Time Machines TM1000A you recommended looks
like a cool little device well worth checking into. As a supplement to
Internet NTP, not a replacement.

Regards,
Bill Herrin


--
William Herrin
bill@herrin.us
https://bill.herrin.us/

William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.

-mel

> On Aug 6, 2023, at 10:53 AM, William Herrin <bill@herrin.us> wrote:
>
> ?On Sat, Aug 5, 2023 at 7:24?PM Mel Beckman <mel@beckman.org> wrote:
>> That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.
>
> Hi Mel,
>
> From what I can tell, a fairly simple firewall policy of allow UDP 123
> from known NTP clients and established connections (I sent them a UDP
> packet recently) stops every one of those attacks (that's actually an
> NTP attack and not something else like a DNS attack) except for
> upstream address hijack that happens to coincide with your system
> boot. And it still depends on the attacker executing an additional
> sophisticated attack to do more than cause you a denial of service.
>
> The links you sent are very interesting, at least in an academic
> sense, but they don't cause me to be unduly concerned about employing
> NTP.
>
>
>> if you can eliminate such security problems for $400, I say it’s cheap at twice the price.
>
> Except you can't. Redundancy is required for any critical service. At
> the $400 price point, your approach has multiple
> single-points-of-failure. The device itself of course. Your ability to
> receive continuous non-jammed GPS signals at the location where you're
> able to place an antenna. And in your plan you'll need one of these in
> every discontiguous network where you have equipment since you're not
> doing NTP over the Internet.
>
> Not to mention the operations cost. Keeping track of a six inch brick
> with a wall wart and an antenna installed at a remote site is... not
> entirely abnormal but it's a one-off that consumes manpower.
>
> And then you're only vulnerable to the litany of Internet attacks
> which don't involve NTP. Yay!
>
> Don't get me wrong: the Time Machines TM1000A you recommended looks
> like a cool little device well worth checking into. As a supplement to
> Internet NTP, not a replacement.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> bill@herrin.us
> https://bill.herrin.us/

Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering
a reasonable mitigation for this, as designed?

Royce

On Sun, Aug 6, 2023 at 10:21?AM Mel Beckman <mel@beckman.org> wrote:

> William,
>
> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These
> flaws make it trivial to spoof NTP packets, and many firewalls have no
> specific protection against this. in one attack the malefactor simply fires
> a continuous stream of NTP packets with invalid time at your firewall. When
> your NTP client queries the spoofed server, the malicious packet is the one
> you likely receive.
>
> That’s just one attack vector. There are several others, and all have
> complex remediation. Why should people bother being exposed to the risk at
> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve
> already described. Having suffered through such attacks more than once, I
> can say from personal experience that you don’t want to risk it.
>
>

A carefully selected set of stratum 0 sources for a set of stratum 1 servers is the heart of good NTP source design. With at least four “local” stratum 1 servers, Dr. Mills algorithm is excellent at distinguishing truechimers from falsetickers and providing a reliable source of monotonic time. DOS is a separate problem.

My NTP network deployment experience for a major auto manufacturer, among others, is in agreement with William Herin. A GPS NTP source is a valid Stratum 0 source, but relying on a single instance for local time is not exceedingly better than querying time.apple.com <http://time.apple.com/> or a similar source.
-
James R. Cutler
>
> William,
>
> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive.
>
> That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.
>
> -mel
>
>> On Aug 6, 2023, at 10:53 AM, William Herrin <bill@herrin.us> wrote:
>>
>> ?On Sat, Aug 5, 2023 at 7:24?PM Mel Beckman <mel@beckman.org> wrote:
>>> That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.
>>
>> Hi Mel,
>>
>> From what I can tell, a fairly simple firewall policy of allow UDP 123
>> from known NTP clients and established connections (I sent them a UDP
>> packet recently) stops every one of those attacks (that's actually an
>> NTP attack and not something else like a DNS attack) except for
>> upstream address hijack that happens to coincide with your system
>> boot. And it still depends on the attacker executing an additional
>> sophisticated attack to do more than cause you a denial of service.
>>
>> The links you sent are very interesting, at least in an academic
>> sense, but they don't cause me to be unduly concerned about employing
>> NTP.
>>
>>
>>> if you can eliminate such security problems for $400, I say it’s cheap at twice the price.
>>
>> Except you can't. Redundancy is required for any critical service. At
>> the $400 price point, your approach has multiple
>> single-points-of-failure. The device itself of course. Your ability to
>> receive continuous non-jammed GPS signals at the location where you're
>> able to place an antenna. And in your plan you'll need one of these in
>> every discontiguous network where you have equipment since you're not
>> doing NTP over the Internet.
>>
>> Not to mention the operations cost. Keeping track of a six inch brick
>> with a wall wart and an antenna installed at a remote site is... not
>> entirely abnormal but it's a one-off that consumes manpower.
>>
>> And then you're only vulnerable to the litany of Internet attacks
>> which don't involve NTP. Yay!
>>
>> Don't get me wrong: the Time Machines TM1000A you recommended looks
>> like a cool little device well worth checking into. As a supplement to
>> Internet NTP, not a replacement.
>>
>> Regards,
>> Bill Herrin
>>
>>
>> --
>> William Herrin
>> bill@herrin.us
>> https://bill.herrin.us/

A carefully selected set of stratum 0 sources for a set of stratum 1 servers is the heart of good NTP source design. With at least four “local” stratum 1 servers, Dr. Mills algorithm is excellent at distinguishing truechimers from falsetickers and providing a reliable source of monotonic time. DOS is a separate problem.

My NTP network deployment experience for a major auto manufacturer, among others, is in agreement with William Herin. A GPS NTP source is a valid Stratum 0 source, but relying on a single instance for local time is not exceedingly better than querying time.apple.com <http://time.apple.com/> or a similar source.
-
James R. Cutler
>
> William,
>
> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive.
>
> That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.
>
> -mel
>
>> On Aug 6, 2023, at 10:53 AM, William Herrin <bill@herrin.us> wrote:
>>
>> ?On Sat, Aug 5, 2023 at 7:24?PM Mel Beckman <mel@beckman.org> wrote:
>>> That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.
>>
>> Hi Mel,
>>
>> From what I can tell, a fairly simple firewall policy of allow UDP 123
>> from known NTP clients and established connections (I sent them a UDP
>> packet recently) stops every one of those attacks (that's actually an
>> NTP attack and not something else like a DNS attack) except for
>> upstream address hijack that happens to coincide with your system
>> boot. And it still depends on the attacker executing an additional
>> sophisticated attack to do more than cause you a denial of service.
>>
>> The links you sent are very interesting, at least in an academic
>> sense, but they don't cause me to be unduly concerned about employing
>> NTP.
>>
>>
>>> if you can eliminate such security problems for $400, I say it’s cheap at twice the price.
>>
>> Except you can't. Redundancy is required for any critical service. At
>> the $400 price point, your approach has multiple
>> single-points-of-failure. The device itself of course. Your ability to
>> receive continuous non-jammed GPS signals at the location where you're
>> able to place an antenna. And in your plan you'll need one of these in
>> every discontiguous network where you have equipment since you're not
>> doing NTP over the Internet.
>>
>> Not to mention the operations cost. Keeping track of a six inch brick
>> with a wall wart and an antenna installed at a remote site is... not
>> entirely abnormal but it's a one-off that consumes manpower.
>>
>> And then you're only vulnerable to the litany of Internet attacks
>> which don't involve NTP. Yay!
>>
>> Don't get me wrong: the Time Machines TM1000A you recommended looks
>> like a cool little device well worth checking into. As a supplement to
>> Internet NTP, not a replacement.
>>
>> Regards,
>> Bill Herrin
>>
>>
>> --
>> William Herrin
>> bill@herrin.us
>> https://bill.herrin.us/

In a nutshell, no. Refer to my prior cites for detailed explanations. For a list of real-world attack incidents, see

https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#<https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#:~:text=NTP%20server%20misuse%20and%20abuse%20covers%20a%20number%20of%20practices,the%20NTP%20rules%20of%20engagement.>


-mel

On Aug 6, 2023, at 12:03 PM, Royce Williams <royce@techsolvency.com> wrote:

?
Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a reasonable mitigation for this, as designed?

Royce

On Sun, Aug 6, 2023 at 10:21?AM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.

Respectfully, that Wikipedia article (which is mostly about legit but
unauthorized clients overwhelming a given peer) and your cites don't seem
to cover what I was responding to - the "don't peer with public NTP because
someone can flood your firewall and spoof the responses" problem. I just
want to make sure that I'm understanding the distinction betwen this class
and other classes of attack.

Wouldn't a robust implementation of peering - say, seven peers, with the
NTP algorithm handily selecting a subset to peer with for each cycle -
require an attacker to know and overwhelm not just one, but a majority of
the peer IPs?

This is *not* to say that I'm advocating against maintaining local stratum
0s as part of a proper operator-grade NTP mesh. (My definition of "robust
implementation" would include both local stratum 0 and a healthy serving of
Internet stratum 1s). I'm just suggesting "don't peer with public servers"
seems draconian given the deliberate design choices of the protocol.

For this audience, this would recommend a tiered system where multiple
mixed-stratum internal stratrum 0+1s would peer with each other, and
maintain internal-facing consensus for all other downstream internal
devices - such that the vast majority of internal peers would indeed not be
talking to the public Internet (but the "parent" peers would, and have the
mesh and mix that I describe).

And I'm well aware of who I'm saying this to ... so I expect to find out
where I'm wrong, as I keep digging. :D

--
Royce

On Sun, Aug 6, 2023 at 11:40?AM Mel Beckman <mel@beckman.org> wrote:

> In a nutshell, no. Refer to my prior cites for detailed explanations. For
> a list of real-world attack incidents, see
>
> https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#
> <https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#:~:text=NTP%20server%20misuse%20and%20abuse%20covers%20a%20number%20of%20practices,the%20NTP%20rules%20of%20engagement.>
>
>
> -mel
>
> On Aug 6, 2023, at 12:03 PM, Royce Williams <royce@techsolvency.com>
> wrote:
>
> ?
> Naively, instead of abstaining ;) ... isn't robust diversity of NTP
> peering a reasonable mitigation for this, as designed?
>
> Royce
>
> On Sun, Aug 6, 2023 at 10:21?AM Mel Beckman <mel@beckman.org> wrote:
>
>> William,
>>
>> Due to flaws in the NTP protocol, a simple UDP filter is not enough.
>> These flaws make it trivial to spoof NTP packets, and many firewalls have
>> no specific protection against this. in one attack the malefactor simply
>> fires a continuous stream of NTP packets with invalid time at your
>> firewall. When your NTP client queries the spoofed server, the malicious
>> packet is the one you likely receive.
>>
>> That’s just one attack vector. There are several others, and all have
>> complex remediation. Why should people bother being exposed to the risk at
>> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve
>> already described. Having suffered through such attacks more than once, I
>> can say from personal experience that you don’t want to risk it.
>>
>>

This entirely discounts the fact that bcp-38 and bcp-84 which, more or
less, eliminate this "problem space" entirely.

I find it hard to believe ntp reflection is actually a problem in the year
2023, assuming you're not running a ridiculously old ntp client and have
taken really simple steps to protect your network.

On Sun, Aug 6, 2023, 15:42 Mel Beckman <mel@beckman.org> wrote:

> In a nutshell, no. Refer to my prior cites for detailed explanations. For
> a list of real-world attack incidents, see
>
> https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#
> <https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#:~:text=NTP%20server%20misuse%20and%20abuse%20covers%20a%20number%20of%20practices,the%20NTP%20rules%20of%20engagement.>
>
>
> -mel
>
> On Aug 6, 2023, at 12:03 PM, Royce Williams <royce@techsolvency.com>
> wrote:
>
> ?
> Naively, instead of abstaining ;) ... isn't robust diversity of NTP
> peering a reasonable mitigation for this, as designed?
>
> Royce
>
> On Sun, Aug 6, 2023 at 10:21?AM Mel Beckman <mel@beckman.org> wrote:
>
>> William,
>>
>> Due to flaws in the NTP protocol, a simple UDP filter is not enough.
>> These flaws make it trivial to spoof NTP packets, and many firewalls have
>> no specific protection against this. in one attack the malefactor simply
>> fires a continuous stream of NTP packets with invalid time at your
>> firewall. When your NTP client queries the spoofed server, the malicious
>> packet is the one you likely receive.
>>
>> That’s just one attack vector. There are several others, and all have
>> complex remediation. Why should people bother being exposed to the risk at
>> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve
>> already described. Having suffered through such attacks more than once, I
>> can say from personal experience that you don’t want to risk it.
>>
>>

Or one can select NTS-capable NTP servers, like those 5:
a.st1.ntp.br
b.st1.ntp.br
c.st1.ntp.br
d.st1.ntp.br
gps.ntp.br

Or any other NTP server that has NTS deployed. Game-over for NTP impersonation.


Rubens

On Sun, Aug 6, 2023 at 4:41?PM Mel Beckman <mel@beckman.org> wrote:
>
> In a nutshell, no. Refer to my prior cites for detailed explanations. For a list of real-world attack incidents, see
>
> https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#
>
>
> -mel
>
> On Aug 6, 2023, at 12:03 PM, Royce Williams <royce@techsolvency.com> wrote:
>
> ?
> Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a reasonable mitigation for this, as designed?
>
> Royce
>
> On Sun, Aug 6, 2023 at 10:21?AM Mel Beckman <mel@beckman.org> wrote:
>>
>> William,
>>
>> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive.
>>
>> That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.
>>

On Sun, Aug 6, 2023 at 1:19?PM Royce Williams <royce@techsolvency.com> wrote:
> Wouldn't a robust implementation of peering - say, seven peers, with the NTP algorithm handily selecting a subset to peer with for each cycle - require an attacker to know and overwhelm not just one, but a majority of the peer IPs?

Hi Royce,

More or less, yes. There are some corner cases where that may not be
true, but in terms of designing a useful attack, the attacker has to
be able to figure out which NTP servers you're talking to, flood you
with enough packets forged from all of them that the forged packet
arrives ahead of the real one, generally no more than 10s of
milliseconds behind the sent packet that only happens once every 15
minutes or so. And then it has to move you off real time gradually
enough for the NTP daemon not to decide the peer has become a false
ticker.

It's like DNS cache poisoning but a few orders of magnitude harder.


> This is *not* to say that I'm advocating against maintaining local stratum 0s as part of a proper operator-grade NTP mesh. (My definition of "robust implementation" would include both local stratum 0 and a healthy serving of Internet stratum 1s). I'm just suggesting "don't peer with public servers" seems draconian given the deliberate design choices of the protocol.
>
> For this audience, this would recommend a tiered system where multiple mixed-stratum internal stratrum 0+1s would peer with each other, and maintain internal-facing consensus for all other downstream internal devices - such that the vast majority of internal peers would indeed not be talking to the public Internet (but the "parent" peers would, and have the mesh and mix that I describe).


Well, it's not really one-size-fits-all.

Some systems consist of a well defined network containing routers and
servers with clear boundaries between self and Internet. Your approach
would work well there.

Some systems consist of equipment scattered at various data centers,
interconnected by the Internet itself. Implementing a GPS time
receiver at every one could get very expensive very fast. Standard
security rules apply: don't spend more protecting yourself than the
risk-cost. Risk is threat times vulnerability. Given the complexity of
the attack, you have to consider whether you're enough of a target
(threat) for anyone to bother.

Some systems are heavily virtualized, scattered over many vendors and
locations. You could -try- to track down a "secured" NTP source from
the vendor at each location, but that way lies configuration insanity.

Some systems are air-gapped from the Internet. You're going to get
time from GPS or the cellular phone network. GPS devices like the one
Mel pointed out are probably cheaper and more accurate.

Regards,
Bill Herrin


--
William Herrin
bill@herrin.us
https://bill.herrin.us/

The paper suggests the compromise of critical infrastructure. So, besides
not using NTP, why not stop using DNS ? Just populate a hosts file with all
you need.

BTW, the stratum-0 source you suggested is known to have been manipulated
in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you
need to bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you
can keep using GPS as well. If GPS goes bananas on timing, that source will
just be disregarded (one of the features of the NTP architecture that has
been pointed out over and over in this thread and you keep ignoring it).

Rubens



Rubens



On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org> wrote:

> Or one can read recent research papers that thoroughly document the
> incredible fragility of the existing NTP hierarchy and soberly consider
> their recommendations for remediation:
>
> [image: preview.png]
>
> ndss2021_1A-2_24302_paper
> <https://www.ndss-symposium.org/wp-content/uploads/ndss2021_1A-2_24302_paper.pdf>
> PDF Document · 1.7 MB
> <https://www.ndss-symposium.org/wp-content/uploads/ndss2021_1A-2_24302_paper.pdf>
>
> <https://www.ndss-symposium.org/wp-content/uploads/ndss2021_1A-2_24302_paper.pdf>
>
> Or simply use non-Internet NTP servers based on a Stratum-0 GPS source for
> mission-critical network timing.
>
> Until then, we may all wake up one morning and discover massive data
> breaches traced to an unfounded reliance on insecure public NTP servers.
> Then the game truly will be over, but not in our favor.
>
> -mel
>
> On Aug 6, 2023, at 2:35 PM, Rubens Kuhl <rubensk@gmail.com> wrote:
>
> ?Or one can select NTS-capable NTP servers, like those 5:
> a.st1.ntp.br
> b.st1.ntp.br
> c.st1.ntp.br
> d.st1.ntp.br
> gps.ntp.br
>
> Or any other NTP server that has NTS deployed. Game-over for NTP
> impersonation.
>
>
> Rubens
>
> On Sun, Aug 6, 2023 at 4:41?PM Mel Beckman <mel@beckman.org> wrote:
>
>
> In a nutshell, no. Refer to my prior cites for detailed explanations. For
> a list of real-world attack incidents, see
>
>
> https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#
>
>
>
> -mel
>
>
> On Aug 6, 2023, at 12:03 PM, Royce Williams <royce@techsolvency.com>
> wrote:
>
>
> ?
>
> Naively, instead of abstaining ;) ... isn't robust diversity of NTP
> peering a reasonable mitigation for this, as designed?
>
>
> Royce
>
>
> On Sun, Aug 6, 2023 at 10:21?AM Mel Beckman <mel@beckman.org> wrote:
>
>
> William,
>
>
> Due to flaws in the NTP protocol, a simple UDP filter is not enough. These
> flaws make it trivial to spoof NTP packets, and many firewalls have no
> specific protection against this. in one attack the malefactor simply fires
> a continuous stream of NTP packets with invalid time at your firewall. When
> your NTP client queries the spoofed server, the malicious packet is the one
> you likely receive.
>
>
> That’s just one attack vector. There are several others, and all have
> complex remediation. Why should people bother being exposed to the risk at
> all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve
> already described. Having suffered through such attacks more than once, I
> can say from personal experience that you don’t want to risk it.
>
>
>

Bill,

You’re mistaking targeted NTP attacks with global ones. Yes, to attack your specific NTP client, the attacker has to know which NTP servers you’re using. But to simply succeed at random attacks, the attacker need only spoof popular servers. This is how time-shifting attacks work. Once an attack succeeds with a random victim, the hacker goes to work compromising time-sensitive security services.

And don’t forget that anyone can join pool.ntp.org. Including hackers. There is no credentialed vetting process.

-mel

> On Aug 6, 2023, at 4:30 PM, William Herrin <bill@herrin.us> wrote:
>
> ?On Sun, Aug 6, 2023 at 1:19?PM Royce Williams <royce@
> .com> wrote:
>> Wouldn't a robust implementation of peering - say, seven peers, with the NTP algorithm handily selecting a subset to peer with for each cycle - require an attacker to know and overwhelm not just one, but a majority of the peer IPs?
>
> Hi Royce,
>
> More or less, yes. There are some corner cases where that may not be
> true, but in terms of designing a useful attack, the attacker has to
> be able to figure out which NTP servers you're talking to, flood you
> with enough packets forged from all of them that the forged packet
> arrives ahead of the real one, generally no more than 10s of
> milliseconds behind the sent packet that only happens once every 15
> minutes or so. And then it has to move you off real time gradually
> enough for the NTP daemon not to decide the peer has become a false
> ticker.
>
> It's like DNS cache poisoning but a few orders of magnitude harder.
>
>
>> This is *not* to say that I'm advocating against maintaining local stratum 0s as part of a proper operator-grade NTP mesh. (My definition of "robust implementation" would include both local stratum 0 and a healthy serving of Internet stratum 1s). I'm just suggesting "don't peer with public servers" seems draconian given the deliberate design choices of the protocol.
>>
>> For this audience, this would recommend a tiered system where multiple mixed-stratum internal stratrum 0+1s would peer with each other, and maintain internal-facing consensus for all other downstream internal devices - such that the vast majority of internal peers would indeed not be talking to the public Internet (but the "parent" peers would, and have the mesh and mix that I describe).
>
>
> Well, it's not really one-size-fits-all.
>
> Some systems consist of a well defined network containing routers and
> servers with clear boundaries between self and Internet. Your approach
> would work well there.
>
> Some systems consist of equipment scattered at various data centers,
> interconnected by the Internet itself. Implementing a GPS time
> receiver at every one could get very expensive very fast. Standard
> security rules apply: don't spend more protecting yourself than the
> risk-cost. Risk is threat times vulnerability. Given the complexity of
> the attack, you have to consider whether you're enough of a target
> (threat) for anyone to bother.
>
> Some systems are heavily virtualized, scattered over many vendors and
> locations. You could -try- to track down a "secured" NTP source from
> the vendor at each location, but that way lies configuration insanity.
>
> Some systems are air-gapped from the Internet. You're going to get
> time from GPS or the cellular phone network. GPS devices like the one
> Mel pointed out are probably cheaper and more accurate.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> bill@herrin.us
> https://bill.herrin.us/