Mailing List Archive

DNS SERVFAIL for nist.gov
We have to query and compare against NIST time servers for FINRA compliance. This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work.

[root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;time-a-g.nist.gov. IN A

;; Query time: 6 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:27:45 EDT 2021
;; MSG SIZE rcvd: 46

[root@bacall log]# dig @8.8.8.8 nist.gov in soa

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;nist.gov. IN SOA

;; Query time: 5 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:31:59 EDT 2021
;; MSG SIZE rcvd: 37

The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi

Using the IP addresses work, it look like the nist.gov domain is offline.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mhuff@ox.com |?www.ox.com
...........................................................................................................................................

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: [EXTERNAL] DNS SERVFAIL for nist.gov [ In reply to ]
are your local resolvers forwarding to 8.8.8.8?

I tried a small sample of public resolvers and only the google once
failed. Maybe DNSSEC? (looks like the NIST signature rotated yesterday)


1.1.1.1
gm.nist.gov. netops.nist.gov. 2889174 10800 1080 2419200 300

8.8.8.8
failed

8.8.4.4
failed

75.75.75.75
gm.nist.gov. netops.nist.gov. 2889174 10800 1080 2419200 300

9.9.9.9
gm.nist.gov. netops.nist.gov. 2889174 10800 1080 2419200 300






On 6/14/21 6:35 AM, Matthew Huff via Outages wrote:
> External email warning - This email originated outside the company. Please do not click links or open attachments unless you were expecting this communication. - SANS Security Team -
>
> We have to query and compare against NIST time servers for FINRA compliance. This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work.
>
> [root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov
>
> ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;time-a-g.nist.gov. IN A
>
> ;; Query time: 6 msec
> ;; SERVER: 8.8.8.8#53(8.8.8.8)
> ;; WHEN: Mon Jun 14 06:27:45 EDT 2021
> ;; MSG SIZE rcvd: 46
>
> [root@bacall log]# dig @8.8.8.8 nist.gov in soa
>
> ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;nist.gov. IN SOA
>
> ;; Query time: 5 msec
> ;; SERVER: 8.8.8.8#53(8.8.8.8)
> ;; WHEN: Mon Jun 14 06:31:59 EDT 2021
> ;; MSG SIZE rcvd: 37
>
> The time servers are documented here: https://urldefense.com/v3/__https://tf.nist.gov/tf-cgi/servers.cgi__;!!MlQdS1fu!DZRm9lRTouO4RyYpsdoZy2u792hhsKWBND7n9t0k6c_z15nmXjO3j7ufO18Zog$
>
> Using the IP addresses work, it look like the nist.gov domain is offline.
>
> Matthew Huff | Director of Technical Operations | OTA Management LLC
>
> Office: 914-460-4039
> mhuff@ox.com | https://urldefense.com/v3/__http://www.ox.com__;!!MlQdS1fu!DZRm9lRTouO4RyYpsdoZy2u792hhsKWBND7n9t0k6c_z15nmXjO3j7s2_kXJlQ$
> ...........................................................................................................................................
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!MlQdS1fu!DZRm9lRTouO4RyYpsdoZy2u792hhsKWBND7n9t0k6c_z15nmXjO3j7vlVeiO4w$
>
Re: DNS SERVFAIL for nist.gov [ In reply to ]
There's been some chatter on NANOG - they're under DDOS attack apparently.

________________________________
From: Outages <outages-bounces@outages.org> on behalf of Matthew Huff via Outages <outages@outages.org>
Sent: 14 June 2021 11:35
To: outages@outages.org <outages@outages.org>
Subject: [outages] DNS SERVFAIL for nist.gov

We have to query and compare against NIST time servers for FINRA compliance. This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work.

[root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;time-a-g.nist.gov. IN A

;; Query time: 6 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:27:45 EDT 2021
;; MSG SIZE rcvd: 46

[root@bacall log]# dig @8.8.8.8 nist.gov in soa

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;nist.gov. IN SOA

;; Query time: 5 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:31:59 EDT 2021
;; MSG SIZE rcvd: 37

The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi

Using the IP addresses work, it look like the nist.gov domain is offline.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mhuff@ox.com | www.ox.com
...........................................................................................................................................

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: DNS SERVFAIL for nist.gov [ In reply to ]
On Mon, Jun 14, 2021 at 10:52:43AM +0000,
Geoff Snowdon via Outages <outages@outages.org> wrote
a message of 200 lines which said:

> There's been some chatter on NANOG - they're under DDOS attack apparently.

Indeed, most of the authoritative name servers timeout.

% check-soa -i nist.gov
bea.nist.gov.
132.163.4.10: ERROR: read udp 193.70.85.11:60102->132.163.4.10:53: i/o timeout
2610:20:6b01:4::10: ERROR: read udp [2001:41d0:302:2200::180]:35149->[2610:20:6b01:4::10]:53: i/o timeout
bea2.nist.gov.
132.163.4.11: ERROR: read udp 193.70.85.11:34369->132.163.4.11:53: i/o timeout
2610:20:6b01:4::11: ERROR: read udp [2001:41d0:302:2200::180]:47271->[2610:20:6b01:4::11]:53: i/o timeout
gea.nist.gov.
129.6.13.3: ERROR: read udp 193.70.85.11:36517->129.6.13.3:53: i/o timeout
2610:20:6005:13::3: ERROR: read udp [2001:41d0:302:2200::180]:60519->[2610:20:6005:13::3]:53: i/o timeout
gea2.nist.gov.
2610:20:6005:13::4: ERROR: read udp [2001:41d0:302:2200::180]:57731->[2610:20:6005:13::4]:53: i/o timeout
129.6.13.4: ERROR: read udp 193.70.85.11:38669->129.6.13.4:53: i/o timeout
gea3.nist.gov.
129.6.14.193: OK: 2889174 (84 ms)
2610:20:6005:2192::193: ERROR: read udp [2001:41d0:302:2200::180]:39170->[2610:20:6005:2192::193]:53: i/o timeout

The RIPE Atlas probes show some SERVFAILs:

% blaeu-resolve -r 100 -c US --type A nist.gov
[129.6.13.49] : 62 occurrences
[ERROR: SERVFAIL] : 15 occurrences
[198.105.244.131 198.105.254.131] : 1 occurrences
Test #30851192 done at 2021-06-14T11:22:04Z

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: DNS SERVFAIL for nist.gov [ In reply to ]
You should consider eliminating dependence on Internet-delivered NIST time and switch to GPS-based time servers. The GPS network has its own airborne atomic clocks that use a well-disciplined protocol to synchronize to NIST reference atomic time without transiting the Internet.

According to NIST’s documentation:

“Currently, the GPS system provides time to the general public with uncertainties measured in nanoseconds. With a well-designed receiver system the user can obtain the time to better than 100 ns in a few minutes, and to about +/- 10 ns with a 24 hour average (and a good local clock).”

All sources of error in GPS time propagation total less than one millisecond, well within your 50ms tolerance.

https://www.nist.gov/pml/time-and-frequency-division/time-services/one-way-gps-time-transfer

NIST maintains publicly-accessible logs of al clock differences to provide documented compliance under FINRA clock synchronization rules.

[cid:54276CD7-0F1E-4855-BA0B-BFE406A63AB5-L0-001]

The log has a one-hour resolution, satisfying the FINRA requirement to verify synchronization “throughout the day”.

IP-based GPS clocks are widely available with low-drift oven-controlled crystal oscillators (OXCO), or even internal cesium-based atomic clocks, for as little as a few thousand dollars. This lets you ride out time signal outages of days or even weeks.

The US DHS recommends discontinuation of unauthenticated Internet-based reference clocks, owing to their vulnerability to IP address spoofing:

https://www.dhs.gov/sites/default/files/publications/GPS-PNT-Best-Practices-Time-Frequency-Sources-Fixed-Locations-508.pdf


-mel via cell

On Jun 14, 2021, at 3:51 AM, Matthew Huff via Outages <outages@outages.org> wrote:

?We have to query and compare against NIST time servers for FINRA compliance This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work.

[root@bacall log]# dig @8.8.8.8 time-a-g.nist.gov

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;time-a-g.nist.gov. IN A

;; Query time: 6 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:27:45 EDT 2021
;; MSG SIZE rcvd: 46

[root@bacall log]# dig @8.8.8.8 nist.gov in soa

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;nist.gov. IN SOA

;; Query time: 5 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:31:59 EDT 2021
;; MSG SIZE rcvd: 37

The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi

Using the IP addresses work, it look like the nist.gov domain is offline.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mhuff@ox.com | www.ox.com
.........................................................................................................................................

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: DNS SERVFAIL for nist.gov [ In reply to ]
Of course.

Like I stated in the original email, we don’t use NIST for time sync. We actually have a GPS and a PTP feed.

WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA FINRA REGULATIONS

Yes, it is stupid
No, it isn’t a good idea
FINRA specifically state that we MUST run a comparison and log any differences between our current time and Internet NIST servers. I have challenged them on this a number of times, but with no luck.

if we don’t want to get fined and prevented from trading, we have to follow FINRA regulations..


Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mhuff@ox.com<mailto:mhuff@ox.com> | www.ox.com<http://www.ox.com>
Re: DNS SERVFAIL for nist.gov [ In reply to ]
----- Original Message -----
> From: "Matthew Huff via Outages" <outages@outages.org>

> Of course.
>
> Like I stated in the original email, we don’t use NIST for time sync. We
> actually have a GPS and a PTP feed.
>
> WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA
> FINRA REGULATIONS
>
> Yes, it is stupid
>
> No, it isn’t a good idea
>
> FINRA specifically state that we MUST run a comparison and log any differences
> between our current time and Internet NIST servers. I have challenged them on
> this a number of times, but with no luck.
>
> if we don’t want to get fined and prevented from trading, we have to follow
> FINRA regulations..

Sure. Your system for this is correctly logging the failures *at the other
end outside your administrative span of control*, correct?

That's Force Majeure, as far as I can see; you can't compare to a server
inaccessible through no fault of your own.

Find the NANOG messages, print them out, sign and date, put in your logbook.

If you're really worried, consult your inside counsel to see if there's anything
else you can do. And surely FINRA itself has counsel for this, too?

Oh, and let's take this to -discuss. :-)

Cheers,
-- jr '<admin/>' a
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: DNS SERVFAIL for nist.gov [ In reply to ]
On 6/14/21 8:58 AM, Jay R. Ashworth via Outages wrote:
> ----- Original Message -----
>> From: "Matthew Huff via Outages" <outages@outages.org>
>> Of course.
>>
>> Like I stated in the original email, we don’t use NIST for time sync. We
>> actually have a GPS and a PTP feed.
>>
>> WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA
>> FINRA REGULATIONS
>>
>> Yes, it is stupid
>>
>> No, it isn’t a good idea

In prior job, I did work for a datacenter.  Datacenter had both GPS and
NTP.  We monitored drift.  We saw drift.  At first, I was dumbfounded
that our NTP pool could be drifting.  Then I traced it back.  Found out
that there was a misconfiguration on the GPS Receiver side instead.

Lesson learned: run both, track both, use each to confirm the other. 
Set one as priority over the other.

>>
>> FINRA specifically state that we MUST run a comparison and log any differences
>> between our current time and Internet NIST servers. I have challenged them on
>> this a number of times, but with no luck.
>>
>> if we don’t want to get fined and prevented from trading, we have to follow
>> FINRA regulations..
>

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: DNS SERVFAIL for nist.gov [ In reply to ]
----- Original Message -----
> From: "Mel Beckman via Outages" <outages@outages.org>
> Subject: Re: [outages] DNS SERVFAIL for nist.gov

> I support many financial networks, and comply with the same FINRA rule 6820 you
> do. This rule doesn’t state that the time must be synchronized over the
> Internet using public IP addresses, but only “to within a fifty (50)
> millisecond tolerance of the time maintained by the atomic clock” at NIST .
> Because GPS clock times are legally traceable to NIST, and deviation logs are

They are? *Legally* traceable? I've just read the first couple pages of Judah's
2941 PDF, and the things it says to me suggest that you cannot make that assertion
*about the GPS system*, only about specific measurements it produces, and that
distinction seems material in this context.

> available in real-time, there is no reason to depend on IP-based NTP over the
> Internet, and good reason not to as today’s event demonstrates. The 1ms
> accuracy is well within the 50ms limit.

Are you missing it, or are you purposely ignoring what he said?

He is *not* "depending on [...] NTP". He merely has to have it and log it
to comply with FINRA, or so his counsel tells him.

I'm declaring this part of the thread out-of-bounds for outages anyway; take it
to -discuss please, if you want to continue ignoring Matt. :-)

Cheers,
-- jr '<admin/>' a
--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274