Mailing List Archive

Oh my. Seeing the same from the Northeast.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Outages <outages-bounces@outages.org> on behalf of Michael Loftis via Outages <outages@outages.org>
Sent: Saturday, March 25, 2023 9:34:32 PM
To: outages <outages@outages.org>
Subject: [outages] FAA.gov nameserver outage

Looks like faa.gov's nameservers are all having a bad time, only
occasionally responding right now from multiple tests from my home
network, datacenter POPs (Seattle, Chicago), and 8.8.8.8

--

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Seeing the same here across all our POPs.

Has anyone notified FAA NOC yet?
--
Sent from Gmail Mobile

Was down 60 seconds ago;
checked https://nasstatus.faa.gov/ and that was up;
seems main faa.gov is operational again.



On Sat, Mar 25, 2023 at 9:51?PM Michael B. Williams via Outages <
outages@outages.org> wrote:

> Seeing the same here across all our POPs.
>
> Has anyone notified FAA NOC yet?
> --
> Sent from Gmail Mobile
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

mloftis> Looks like faa.gov's nameservers are all having a bad time,
mloftis> only occasionally responding right now from multiple tests from
mloftis> my home network, datacenter POPs (Seattle, Chicago), and
mloftis> 8.8.8.8

They only seem to have two auth nameservers for faa, both within the
faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks
the servers are in all die just within each block run by the FAA.

Seems like an internal routing meltdown making the only 2 nameservers
unreachbable reliably.
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On Sat, 2023-03-25 at 19:34 -0600, Michael Loftis via Outages wrote:
> Looks like faa.gov's nameservers are all having a bad time, only
> occasionally responding right now from multiple tests from my home
> network, datacenter POPs (Seattle, Chicago), and 8.8.8.8
>
-----
faa.gov is CDN (akamaiedge) supported. Latency appears to be off the
roof.


/vrode

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Hi, I'm a researcher of DNS vulnerabilities.

It loos like random subdomain attacks (water tourtue attack).

This is the data of my rate-limitted openresolver as a honeypot.
http://www.e-ontap.com/dns/todaydowngov.txt
http://www.e-ontap.com/dns/todaydown.txt
(You can not view these page if you are using 8.8.8.8, sorry.)

Raw logs of my Unbound (Time is JST)
local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head -5
Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210 unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN SERVFAIL 15.112813 0 30
Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
local/etc/unbound%
local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | head -5
Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. from 2620:74:27::2:30 no server to query nameserver addresses not usable
Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout
Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. upstream server timeout
local/etc/unbound%
local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" | tail -5
Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
local/etc/unbound%
local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail -5
Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34
local/etc/unbound%
local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc -l
1408

--
T.Suzuki
--
T.Suzuki / E.F.$B%7%e!<%^%C%O!<$H(BI.$B%$%j%$%A$rFI$b$&(B
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

What would be the symptoms here of a "water torture attack" rather than
what John had indicated as a firewall failure in their infrastructure:

> Initial looks from the firewall team point to an automatic failover event
and the secondary failed.

And the symptoms of which lined up with network level info from Paul
earlier:

> They only seem to have two auth nameservers for faa, both within the
faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
servers are in all die just within each block run by the FAA.
>
> Seems like an internal routing meltdown making the only 2 nameservers
unreachable reliably.

Are you saying that your open resolvers have a per client rate limit
applied, that rate limit got tripped, and shortly thereafter the resolvers
became unavailable, suggesting query floods for the domain(s) that knocked
the resolvers offline (or from the other discussion, possibly was the thing
that overwhelmed that firewall layer, causing the initial failover and
possibly also causing the firewall secondary to fail to come online)?

On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org>
wrote:

> Hi, I'm a researcher of DNS vulnerabilities.
>
> It loos like random subdomain attacks (water tourtue attack).
>
> This is the data of my rate-limitted openresolver as a honeypot.
> http://www.e-ontap.com/dns/todaydowngov.txt
> http://www.e-ontap.com/dns/todaydown.txt
> (You can not view these page if you are using 8.8.8.8, sorry.)
>
> Raw logs of my Unbound (Time is JST)
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
> -5
> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
> exceeded ratelimit for zone faa.gov.
> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
> SERVFAIL 15.112813 0 30
> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
> exceeded ratelimit for zone faa.gov.
> local/etc/unbound%
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> head -5
> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
> all servers for this domain failed, at zone faa.gov. from
> 2620:74:27::2:30 no server to query nameserver addresses not usable
> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
> A IN>: all servers for this domain failed, at zone faa.gov. no server to
> query nameserver addresses not usable
> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
> servers for this domain failed, at zone faa.gov. no server to query
> nameserver addresses not usable
> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
> all servers for this domain failed, at zone faa.gov. upstream server
> timeout
> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
> A IN>: all servers for this domain failed, at zone faa.gov. upstream
> server timeout
> local/etc/unbound%
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> tail -5
> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
> servers for this domain failed, at zone faa.gov. no server to query
> nameserver addresses not usable
> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
> A IN>: all servers for this domain failed, at zone faa.gov. no server to
> query nameserver addresses not usable
> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
> zone faa.gov. no server to query nameserver addresses not usable
> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> at zone faa.gov. no server to query nameserver addresses not usable
> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> IN>: all servers for this domain failed, at zone faa.gov. no server to
> query nameserver addresses not usable
> local/etc/unbound%
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
> -5
> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> at zone faa.gov. no server to query nameserver addresses not usable
> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> IN>: all servers for this domain failed, at zone faa.gov. no server to
> query nameserver addresses not usable
> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
> IN SERVFAIL 0.000000 0 34
> local/etc/unbound%
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
> -l
> 1408
>
> --
> T.Suzuki
> --
> T.Suzuki / E.F.????????I.????????
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

On Sun, 26 Mar 2023 08:35:29 -0700
Hugo Slabbert <hugo@slabnet.com> wrote:

> What would be the symptoms here of a "water torture attack" rather than
> what John had indicated as a firewall failure in their infrastructure:
>
> > Initial looks from the firewall team point to an automatic failover event
> and the secondary failed.
>
> And the symptoms of which lined up with network level info from Paul
> earlier:
>
> > They only seem to have two auth nameservers for faa, both within the
> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
> servers are in all die just within each block run by the FAA.
> >
> > Seems like an internal routing meltdown making the only 2 nameservers
> unreachable reliably.
>
> Are you saying that your open resolvers have a per client rate limit
> applied, that rate limit got tripped, and shortly thereafter the resolvers
> became unavailable, suggesting query floods for the domain(s) that knocked
> the resolvers offline (or from the other discussion, possibly was the thing
> that overwhelmed that firewall layer, causing the initial failover and
> possibly also causing the firewall secondary to fail to come online)?

Yes. (limitting per client, and per second for all)
Perhaps, large numbers open resolvers including no ratelimit are used.
Then massive random subdomain queries caused the firewall symptoms.
(It's only my guess.)

> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org>
> wrote:
>
> > Hi, I'm a researcher of DNS vulnerabilities.
> >
> > It loos like random subdomain attacks (water tourtue attack).
> >
> > This is the data of my rate-limitted openresolver as a honeypot.
> > http://www.e-ontap.com/dns/todaydowngov.txt
> > http://www.e-ontap.com/dns/todaydown.txt
> > (You can not view these page if you are using 8.8.8.8, sorry.)
> >
> > Raw logs of my Unbound (Time is JST)
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
> > -5
> > Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
> > unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
> > Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
> > unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
> > Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
> > exceeded ratelimit for zone faa.gov.
> > Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
> > SERVFAIL 15.112813 0 30
> > Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
> > exceeded ratelimit for zone faa.gov.
> > local/etc/unbound%
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> > head -5
> > Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
> > all servers for this domain failed, at zone faa.gov. from
> > 2620:74:27::2:30 no server to query nameserver addresses not usable
> > Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
> > A IN>: all servers for this domain failed, at zone faa.gov. no server to
> > query nameserver addresses not usable
> > Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
> > servers for this domain failed, at zone faa.gov. no server to query
> > nameserver addresses not usable
> > Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
> > all servers for this domain failed, at zone faa.gov. upstream server
> > timeout
> > Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
> > A IN>: all servers for this domain failed, at zone faa.gov. upstream
> > server timeout
> > local/etc/unbound%
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> > tail -5
> > Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
> > servers for this domain failed, at zone faa.gov. no server to query
> > nameserver addresses not usable
> > Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
> > A IN>: all servers for this domain failed, at zone faa.gov. no server to
> > query nameserver addresses not usable
> > Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
> > eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
> > zone faa.gov. no server to query nameserver addresses not usable
> > Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> > faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> > at zone faa.gov. no server to query nameserver addresses not usable
> > Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> > IN>: all servers for this domain failed, at zone faa.gov. no server to
> > query nameserver addresses not usable
> > local/etc/unbound%
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
> > -5
> > Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
> > eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
> > Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> > faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> > at zone faa.gov. no server to query nameserver addresses not usable
> > Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
> > faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
> > Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> > IN>: all servers for this domain failed, at zone faa.gov. no server to
> > query nameserver addresses not usable
> > Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
> > IN SERVFAIL 0.000000 0 34
> > local/etc/unbound%
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
> > -l
> > 1408
> >
> > --
> > T.Suzuki
> > --
> > T.Suzuki / E.F.$B%7%e!<%^%C%O!<$H(BI.$B%$%j%$%A$rFI$b$&(B
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >


--
T.Suzuki / E.F.$B%7%e!<%^%C%O!<$H(BI.$B%$%j%$%A$rFI$b$&(B
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

This belongs on the outages discussion list, NOT here. This is only for outages and the immediate outage info. If you don’t understand why, go ask on THAT list.

--
Carlos Alvarez
602-368-6403
On Mar 26, 2023 at 5:13 PM -0700, T.Suzuki via Outages <outages@outages.org>, wrote:
> On Sun, 26 Mar 2023 08:35:29 -0700
> Hugo Slabbert <hugo@slabnet.com> wrote:
>
> > What would be the symptoms here of a "water torture attack" rather than
> > what John had indicated as a firewall failure in their infrastructure:
> >
> > > Initial looks from the firewall team point to an automatic failover event
> > and the secondary failed.
> >
> > And the symptoms of which lined up with network level info from Paul
> > earlier:
> >
> > > They only seem to have two auth nameservers for faa, both within the
> > faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
> > servers are in all die just within each block run by the FAA.
> > >
> > > Seems like an internal routing meltdown making the only 2 nameservers
> > unreachable reliably.
> >
> > Are you saying that your open resolvers have a per client rate limit
> > applied, that rate limit got tripped, and shortly thereafter the resolvers
> > became unavailable, suggesting query floods for the domain(s) that knocked
> > the resolvers offline (or from the other discussion, possibly was the thing
> > that overwhelmed that firewall layer, causing the initial failover and
> > possibly also causing the firewall secondary to fail to come online)?
>
> Yes. (limitting per client, and per second for all)
> Perhaps, large numbers open resolvers including no ratelimit are used.
> Then massive random subdomain queries caused the firewall symptoms.
> (It's only my guess.)
>
> > On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org>
> > wrote:
> >
> > > Hi, I'm a researcher of DNS vulnerabilities.
> > >
> > > It loos like random subdomain attacks (water tourtue attack).
> > >
> > > This is the data of my rate-limitted openresolver as a honeypot.
> > > http://www.e-ontap.com/dns/todaydowngov.txt
> > > http://www.e-ontap.com/dns/todaydown.txt
> > > (You can not view these page if you are using 8.8.8.8, sorry.)
> > >
> > > Raw logs of my Unbound (Time is JST)
> > > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
> > > -5
> > > Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
> > > unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
> > > Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
> > > unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
> > > Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
> > > exceeded ratelimit for zone faa.gov.
> > > Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
> > > SERVFAIL 15.112813 0 30
> > > Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
> > > exceeded ratelimit for zone faa.gov.
> > > local/etc/unbound%
> > > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> > > head -5
> > > Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
> > > all servers for this domain failed, at zone faa.gov. from
> > > 2620:74:27::2:30 no server to query nameserver addresses not usable
> > > Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
> > > A IN>: all servers for this domain failed, at zone faa.gov. no server to
> > > query nameserver addresses not usable
> > > Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
> > > servers for this domain failed, at zone faa.gov. no server to query
> > > nameserver addresses not usable
> > > Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
> > > all servers for this domain failed, at zone faa.gov. upstream server
> > > timeout
> > > Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
> > > A IN>: all servers for this domain failed, at zone faa.gov. upstream
> > > server timeout
> > > local/etc/unbound%
> > > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> > > tail -5
> > > Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
> > > servers for this domain failed, at zone faa.gov. no server to query
> > > nameserver addresses not usable
> > > Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
> > > A IN>: all servers for this domain failed, at zone faa.gov. no server to
> > > query nameserver addresses not usable
> > > Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
> > > eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
> > > zone faa.gov. no server to query nameserver addresses not usable
> > > Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> > > faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> > > at zone faa.gov. no server to query nameserver addresses not usable
> > > Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> > > IN>: all servers for this domain failed, at zone faa.gov. no server to
> > > query nameserver addresses not usable
> > > local/etc/unbound%
> > > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
> > > -5
> > > Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
> > > eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
> > > Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> > > faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> > > at zone faa.gov. no server to query nameserver addresses not usable
> > > Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
> > > faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
> > > Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> > > IN>: all servers for this domain failed, at zone faa.gov. no server to
> > > query nameserver addresses not usable
> > > Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
> > > IN SERVFAIL 0.000000 0 34
> > > local/etc/unbound%
> > > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
> > > -l
> > > 1408
> > >
> > > --
> > > T.Suzuki
> > > --
> > > T.Suzuki / E.F.????????I.????????
> > > _______________________________________________
> > > Outages mailing list
> > > Outages@outages.org
> > > https://puck.nether.net/mailman/listinfo/outages
> > >
>
>
> --
> T.Suzuki / E.F.????????I.????????
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

Can’t believe it’s still dead…

-Mike

> On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages@outages.org> wrote:
>
> ?On Sun, 26 Mar 2023 08:35:29 -0700
> Hugo Slabbert <hugo@slabnet.com> wrote:
>
>> What would be the symptoms here of a "water torture attack" rather than
>> what John had indicated as a firewall failure in their infrastructure:
>>
>>> Initial looks from the firewall team point to an automatic failover event
>> and the secondary failed.
>>
>> And the symptoms of which lined up with network level info from Paul
>> earlier:
>>
>>> They only seem to have two auth nameservers for faa, both within the
>> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
>> servers are in all die just within each block run by the FAA.
>>>
>>> Seems like an internal routing meltdown making the only 2 nameservers
>> unreachable reliably.
>>
>> Are you saying that your open resolvers have a per client rate limit
>> applied, that rate limit got tripped, and shortly thereafter the resolvers
>> became unavailable, suggesting query floods for the domain(s) that knocked
>> the resolvers offline (or from the other discussion, possibly was the thing
>> that overwhelmed that firewall layer, causing the initial failover and
>> possibly also causing the firewall secondary to fail to come online)?
>
> Yes. (limitting per client, and per second for all)
> Perhaps, large numbers open resolvers including no ratelimit are used.
> Then massive random subdomain queries caused the firewall symptoms.
> (It's only my guess.)
>
>>> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org>
>>> wrote:
>>>
>>> Hi, I'm a researcher of DNS vulnerabilities.
>>>
>>> It loos like random subdomain attacks (water tourtue attack).
>>>
>>> This is the data of my rate-limitted openresolver as a honeypot.
>>> http://www.e-ontap.com/dns/todaydowngov.txt
>>> http://www.e-ontap.com/dns/todaydown.txt
>>> (You can not view these page if you are using 8.8.8.8, sorry.)
>>>
>>> Raw logs of my Unbound (Time is JST)
>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
>>> -5
>>> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
>>> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
>>> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
>>> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
>>> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
>>> exceeded ratelimit for zone faa.gov.
>>> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
>>> SERVFAIL 15.112813 0 30
>>> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
>>> exceeded ratelimit for zone faa.gov.
>>> local/etc/unbound%
>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
>>> head -5
>>> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
>>> all servers for this domain failed, at zone faa.gov. from
>>> 2620:74:27::2:30 no server to query nameserver addresses not usable
>>> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
>>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
>>> query nameserver addresses not usable
>>> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
>>> servers for this domain failed, at zone faa.gov. no server to query
>>> nameserver addresses not usable
>>> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
>>> all servers for this domain failed, at zone faa.gov. upstream server
>>> timeout
>>> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
>>> A IN>: all servers for this domain failed, at zone faa.gov. upstream
>>> server timeout
>>> local/etc/unbound%
>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
>>> tail -5
>>> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
>>> servers for this domain failed, at zone faa.gov. no server to query
>>> nameserver addresses not usable
>>> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
>>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
>>> query nameserver addresses not usable
>>> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
>>> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
>>> zone faa.gov. no server to query nameserver addresses not usable
>>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
>>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
>>> at zone faa.gov. no server to query nameserver addresses not usable
>>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
>>> IN>: all servers for this domain failed, at zone faa.gov. no server to
>>> query nameserver addresses not usable
>>> local/etc/unbound%
>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
>>> -5
>>> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
>>> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
>>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
>>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
>>> at zone faa.gov. no server to query nameserver addresses not usable
>>> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
>>> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
>>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
>>> IN>: all servers for this domain failed, at zone faa.gov. no server to
>>> query nameserver addresses not usable
>>> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
>>> IN SERVFAIL 0.000000 0 34
>>> local/etc/unbound%
>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
>>> -l
>>> 1408
>>>
>>> --
>>> T.Suzuki
>>> --
>>> T.Suzuki / E.F.????????I.????????
>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>
>
> --
> T.Suzuki / E.F.????????I.????????
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On Sun, 26 Mar 2023 17:17:25 -0700
Mike Lyon <mike.lyon@gmail.com> wrote:

> Can$B!G(Bt believe it$B!G(Bs still dead$B!D(B
>
> -Mike

The attack appears to be over, at Mar 26 13:41:28 JST (GMT +0900)
(This may be specific to my server).
Maybe the cause is something else.
Or the person in charge of manual recovery is on holiday.

Mar 26 13:41:08 unbound[48103:0] reply: 24.199.82.210 asm.faa.gov. A IN SERVFAIL 0.000000 0 29
Mar 26 13:41:15 unbound[48103:0] query: 24.199.82.210 sas-uss.edc.nas.faa.gov. A IN
Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:15 unbound[48103:0] reply: 24.199.82.210 sas-uss.edc.nas.faa.gov. A IN SERVFAIL 0.000000 0 41
Mar 26 13:41:22 unbound[48103:0] query: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN
Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
Mar 26 13:41:23 unbound[48103:0] query: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN
Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
Mar 26 13:41:28 unbound[48103:0] query: 24.199.82.210 chronos3.faa.gov. A IN
Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34

> > On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages@outages.org> wrote:
> >
> > $B".(BOn Sun, 26 Mar 2023 08:35:29 -0700
> > Hugo Slabbert <hugo@slabnet.com> wrote:
> >
> >> What would be the symptoms here of a "water torture attack" rather than
> >> what John had indicated as a firewall failure in their infrastructure:
> >>
> >>> Initial looks from the firewall team point to an automatic failover event
> >> and the secondary failed.
> >>
> >> And the symptoms of which lined up with network level info from Paul
> >> earlier:
> >>
> >>> They only seem to have two auth nameservers for faa, both within the
> >> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
> >> servers are in all die just within each block run by the FAA.
> >>>
> >>> Seems like an internal routing meltdown making the only 2 nameservers
> >> unreachable reliably.
> >>
> >> Are you saying that your open resolvers have a per client rate limit
> >> applied, that rate limit got tripped, and shortly thereafter the resolvers
> >> became unavailable, suggesting query floods for the domain(s) that knocked
> >> the resolvers offline (or from the other discussion, possibly was the thing
> >> that overwhelmed that firewall layer, causing the initial failover and
> >> possibly also causing the firewall secondary to fail to come online)?
> >
> > Yes. (limitting per client, and per second for all)
> > Perhaps, large numbers open resolvers including no ratelimit are used.
> > Then massive random subdomain queries caused the firewall symptoms.
> > (It's only my guess.)
> >
> >>> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org>
> >>> wrote:
> >>>
> >>> Hi, I'm a researcher of DNS vulnerabilities.
> >>>
> >>> It loos like random subdomain attacks (water tourtue attack).
> >>>
> >>> This is the data of my rate-limitted openresolver as a honeypot.
> >>> http://www.e-ontap.com/dns/todaydowngov.txt
> >>> http://www.e-ontap.com/dns/todaydown.txt
> >>> (You can not view these page if you are using 8.8.8.8, sorry.)
> >>>
> >>> Raw logs of my Unbound (Time is JST)
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
> >>> -5
> >>> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
> >>> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
> >>> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
> >>> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
> >>> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
> >>> exceeded ratelimit for zone faa.gov.
> >>> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
> >>> SERVFAIL 15.112813 0 30
> >>> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
> >>> exceeded ratelimit for zone faa.gov.
> >>> local/etc/unbound%
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> >>> head -5
> >>> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
> >>> all servers for this domain failed, at zone faa.gov. from
> >>> 2620:74:27::2:30 no server to query nameserver addresses not usable
> >>> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
> >>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
> >>> query nameserver addresses not usable
> >>> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
> >>> servers for this domain failed, at zone faa.gov. no server to query
> >>> nameserver addresses not usable
> >>> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
> >>> all servers for this domain failed, at zone faa.gov. upstream server
> >>> timeout
> >>> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
> >>> A IN>: all servers for this domain failed, at zone faa.gov. upstream
> >>> server timeout
> >>> local/etc/unbound%
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> >>> tail -5
> >>> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
> >>> servers for this domain failed, at zone faa.gov. no server to query
> >>> nameserver addresses not usable
> >>> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
> >>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
> >>> query nameserver addresses not usable
> >>> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
> >>> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
> >>> zone faa.gov. no server to query nameserver addresses not usable
> >>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> >>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> >>> at zone faa.gov. no server to query nameserver addresses not usable
> >>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> >>> IN>: all servers for this domain failed, at zone faa.gov. no server to
> >>> query nameserver addresses not usable
> >>> local/etc/unbound%
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
> >>> -5
> >>> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
> >>> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
> >>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> >>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> >>> at zone faa.gov. no server to query nameserver addresses not usable
> >>> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
> >>> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
> >>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> >>> IN>: all servers for this domain failed, at zone faa.gov. no server to
> >>> query nameserver addresses not usable
> >>> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
> >>> IN SERVFAIL 0.000000 0 34
> >>> local/etc/unbound%
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
> >>> -l
> >>> 1408
> >>>
> >>> --
> >>> T.Suzuki
> >>> --
> >>> T.Suzuki / E.F.$B%7%e!<%^%C%O!<$H(BI.$B%$%j%$%A$rFI$b$&(B
> >>> _______________________________________________
> >>> Outages mailing list
> >>> Outages@outages.org
> >>> https://puck.nether.net/mailman/listinfo/outages
> >>>
> >
> >
> > --
> > T.Suzuki / E.F.$B%7%e!<%^%C%O!<$H(BI.$B%$%j%$%A$rFI$b$&(B
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
>


--
T.Suzuki / E.F.$B%7%e!<%^%C%O!<$H(BI.$B%$%j%$%A$rFI$b$&(B
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

No worries.  Phil Washington has the ball and will fix this.

https://twitter.com/CitizenFreePres/status/1640243188395831297

-Pete


On 3/25/23 21:34, Michael Loftis via Outages wrote:
> Looks like faa.gov's nameservers are all having a bad time, only
> occasionally responding right now from multiple tests from my home
> network, datacenter POPs (Seattle, Chicago), and 8.8.8.8
>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

As is Generalissimo Francisco Franco.

Indeed, folks; please move these meta conversations to the -discuss list; they
are off topic for the main notification list.

Cheers,
-- jr '<admin/>' a

----- Original Message -----
> From: "Mike Lyon via Outages" <outages@outages.org>
> To: "T.Suzuki" <tss-outage@e-ontap.com>
> Cc: "Michael Loftis via Outages" <outages@outages.org>
> Sent: Sunday, March 26, 2023 8:17:25 PM
> Subject: Re: [outages] FAA.gov nameserver outage

> Can’t believe it’s still dead…
>
> -Mike
>
>> On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages@outages.org> wrote:
>>
>> ?On Sun, 26 Mar 2023 08:35:29 -0700
>> Hugo Slabbert <hugo@slabnet.com> wrote:
>>
>>> What would be the symptoms here of a "water torture attack" rather than
>>> what John had indicated as a firewall failure in their infrastructure:
>>>
>>>> Initial looks from the firewall team point to an automatic failover event
>>> and the secondary failed.
>>>
>>> And the symptoms of which lined up with network level info from Paul
>>> earlier:
>>>
>>>> They only seem to have two auth nameservers for faa, both within the
>>> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
>>> servers are in all die just within each block run by the FAA.
>>>>
>>>> Seems like an internal routing meltdown making the only 2 nameservers
>>> unreachable reliably.
>>>
>>> Are you saying that your open resolvers have a per client rate limit
>>> applied, that rate limit got tripped, and shortly thereafter the resolvers
>>> became unavailable, suggesting query floods for the domain(s) that knocked
>>> the resolvers offline (or from the other discussion, possibly was the thing
>>> that overwhelmed that firewall layer, causing the initial failover and
>>> possibly also causing the firewall secondary to fail to come online)?
>>
>> Yes. (limitting per client, and per second for all)
>> Perhaps, large numbers open resolvers including no ratelimit are used.
>> Then massive random subdomain queries caused the firewall symptoms.
>> (It's only my guess.)
>>
>>>> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages@outages.org>
>>>> wrote:
>>>>
>>>> Hi, I'm a researcher of DNS vulnerabilities.
>>>>
>>>> It loos like random subdomain attacks (water tourtue attack).
>>>>
>>>> This is the data of my rate-limitted openresolver as a honeypot.
>>>> http://www.e-ontap.com/dns/todaydowngov.txt
>>>> http://www.e-ontap.com/dns/todaydown.txt
>>>> (You can not view these page if you are using 8.8.8.8, sorry.)
>>>>
>>>> Raw logs of my Unbound (Time is JST)
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
>>>> -5
>>>> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
>>>> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
>>>> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
>>>> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
>>>> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
>>>> exceeded ratelimit for zone faa.gov.
>>>> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
>>>> SERVFAIL 15.112813 0 30
>>>> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
>>>> exceeded ratelimit for zone faa.gov.
>>>> local/etc/unbound%
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
>>>> head -5
>>>> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
>>>> all servers for this domain failed, at zone faa.gov. from
>>>> 2620:74:27::2:30 no server to query nameserver addresses not usable
>>>> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
>>>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
>>>> query nameserver addresses not usable
>>>> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
>>>> servers for this domain failed, at zone faa.gov. no server to query
>>>> nameserver addresses not usable
>>>> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
>>>> all servers for this domain failed, at zone faa.gov. upstream server
>>>> timeout
>>>> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
>>>> A IN>: all servers for this domain failed, at zone faa.gov. upstream
>>>> server timeout
>>>> local/etc/unbound%
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
>>>> tail -5
>>>> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
>>>> servers for this domain failed, at zone faa.gov. no server to query
>>>> nameserver addresses not usable
>>>> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
>>>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
>>>> query nameserver addresses not usable
>>>> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
>>>> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
>>>> zone faa.gov. no server to query nameserver addresses not usable
>>>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
>>>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
>>>> at zone faa.gov. no server to query nameserver addresses not usable
>>>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
>>>> IN>: all servers for this domain failed, at zone faa.gov. no server to
>>>> query nameserver addresses not usable
>>>> local/etc/unbound%
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
>>>> -5
>>>> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
>>>> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
>>>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
>>>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
>>>> at zone faa.gov. no server to query nameserver addresses not usable
>>>> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
>>>> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
>>>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
>>>> IN>: all servers for this domain failed, at zone faa.gov. no server to
>>>> query nameserver addresses not usable
>>>> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
>>>> IN SERVFAIL 0.000000 0 34
>>>> local/etc/unbound%
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
>>>> -l
>>>> 1408
>>>>
>>>> --
>>>> T.Suzuki
>>>> --
>>>> T.Suzuki / E.F.????????I.????????
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages@outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>
>>
>> --
>> T.Suzuki / E.F.????????I.????????
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages