Mailing List Archive

Facebook.com no records?
Seems the facebook.com domain no longer has any DNS records? Getting
nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc

https://toolbox.googleapps.com/apps/dig/#A/facebook.com

--
Jon Sands
MFI Labs
https://fohdeesha.com/

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: Facebook.com no records? [ In reply to ]
A carrier networking engineer friend of mine observed that the network that those DNS servers live in is no longer in the global routing table. Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.)

  -Rich

Rich Lafferty
rich@lafferty.ca / 647-377-1165
Toronto, Canada
http://www.lafferty.ca


On Oct 4, 2021, 1:14 PM -0300, Jon Sands via Outages <outages@outages.org>, wrote:
> Seems the facebook.com domain no longer has any DNS records? Getting
> nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc
>
> https://toolbox.googleapps.com/apps/dig/#A/facebook.com
>
> --
> Jon Sands
> MFI Labs
> https://fohdeesha.com/
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
Re: Facebook.com no records? [ In reply to ]
I wonder if someone was fired and deleted something.


> host facebook.com
Host facebook.com not found: 2(SERVFAIL)
> dig facebook.com

; <<>> DiG 9.6.1-P1 <<>> facebook.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27869
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;facebook.com. IN A

;; Query time: 0 msec
;; SERVER: 10.64.4.2#53(10.64.4.2)
;; WHEN: Mon Oct 4 11:00:17 2021
;; MSG SIZE rcvd: 30

>


John Quinby
Vision Net | Network Operations Center Lead
p:?? (406) 216-4620
m:? (406) 836-0223
w:?? https://vision.net










-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Jon Sands via
Outages
Sent: Monday, October 4, 2021 9:57 AM
To: outages@outages.org
Subject: [outages] Facebook.com no records?

Seems the facebook.com domain no longer has any DNS records? Getting
nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc

https://toolbox.googleapps.com/apps/dig/#A/facebook.com

--
Jon Sands
MFI Labs
https://fohdeesha.com/

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: Facebook.com no records? [ In reply to ]
According to r/sysadmin, this is a peering issue.


Update 1440 UTC:

As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

Part of this is also due to lower staffing in data centers due to pandemic measures.

See: https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/hfda42z/?context=3


On Oct 4, 2021, at 11:56 AM, Jon Sands via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:

Seems the facebook.com<http://facebook.com> domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc

https://toolbox.googleapps.com/apps/dig/#A/facebook.com

--
Jon Sands
MFI Labs
https://fohdeesha.com/

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: Facebook.com no records? [ In reply to ]
See here (with a grain of salt):
https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/hfda42z/?utm_source=share&utm_medium=web2x&context=3

"DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC)."

Hagen Miller
Senior Systems Engineer
Computer Services, Inc.

-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Jon Sands via Outages
Sent: Monday, October 4, 2021 9:57 AM
To: outages@outages.org
Subject: [outages] Facebook.com no records?

WARNING: This email is from an external source. Do not click links or attachments unless you recognize the sender and know the content is safe.

Seems the facebook.com domain no longer has any DNS records? Getting nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc

https://toolbox.googleapps.com/apps/dig/#A/facebook.com

--
Jon Sands
MFI Labs
https://fohdeesha.com/

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages


________________________________

This e-mail and any files transmitted with it are the property of Computer Services, Inc. and/or its divisions, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. The sender does not accept any responsibility for any loss, disruption or damage to your data or computer systems that may occur from content of any sort contained in, transmitted with, or added to this e-mail. If you are not a named recipient or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited.
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: Facebook.com no records? [ In reply to ]
This reddit thread has comments from someone on the FB recovery team about
BGP peering routers being down and requiring on site resolution.

https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/hfd4dyv/


Nick

On Mon, Oct 4, 2021 at 12:08 PM Jon Sands via Outages <outages@outages.org>
wrote:

> Seems the facebook.com domain no longer has any DNS records? Getting
> nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc
>
> https://toolbox.googleapps.com/apps/dig/#A/facebook.com
>
> --
> Jon Sands
> MFI Labs
> https://fohdeesha.com/
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
Re: Facebook.com no records? [ In reply to ]
I’ve seen this BGP withdrawal cited in a number of places.

However we peer with FB, and all four sessions are up and have been live for days/months (FB had maintenance on one peering router a couple of weeks ago) and all 30 prefixes are consistent and announced.

The NS IP’s are in the announcements, but you can’t ICMP to them, or query them as NS for the auth records either..

Peter Knapp

From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rich Lafferty via Outages
Sent: 04 October 2021 17:27
To: Outages mailing list <outages@outages.org>
Subject: Re: [outages] Facebook.com no records?

A carrier networking engineer friend of mine observed that the network that those DNS servers live in is no longer in the global routing table. Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.)

-Rich

Rich Lafferty
rich@lafferty.ca<mailto:rich@lafferty.ca> / 647-377-1165
Toronto, Canada
http://www.lafferty.ca

[https://s3.amazonaws.com/htmlsig-assets/round/twitter.png]<https://twitter.com/mendel> <https://twitter.com/mendel> [https://s3.amazonaws.com/htmlsig-assets/round/facebook.png] <https://facebook.com/richlafferty> <https://facebook.com/richlafferty> [https://s3.amazonaws.com/htmlsig-assets/round/linkedin.png] <https://linkedin.com/in/richlafferty> <https://linkedin.com/in/richlafferty> [https://s3.amazonaws.com/htmlsig-assets/round/instagram.png] <https://instagram.com/wlonkly> <https://instagram.com/wlonkly> [https://s3.amazonaws.com/htmlsig-assets/round/flickr.png] <https://www.flickr.com/photos/mendel> <https://www.flickr.com/photos/mendel> [https://s3.amazonaws.com/htmlsig-assets/round/github.png] <https://github.com/wlonkly> <https://github.com/wlonkly> [https://s3.amazonaws.com/htmlsig-assets/round/tumblr.png] <https://mendel.tumblr.com> <https://mendel.tumblr.com>
On Oct 4, 2021, 1:14 PM -0300, Jon Sands via Outages <outages@outages.org<mailto:outages@outages.org>>, wrote:

Seems the facebook.com domain no longer has any DNS records? Getting
nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc

https://toolbox.googleapps.com/apps/dig/#A/facebook.com

--
Jon Sands
MFI Labs
https://fohdeesha.com/

_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
Re: Facebook.com no records? [ In reply to ]
Hrm. There were definitely no routes for 129.134.31.0/24 from multiple route servers at around 14:00 UTC today, but I see that there are now routes for a containing /17 in AS 32934 now (20:25 UTC).

You can see the /24 disappear:

https://stat.ripe.net/widget/bgplay#w.resource=129.134.30.0/24&w.ignoreReannouncements=true&w.starttime=1633348800&w.endtime=1633363186&w.rrcs=0,1,2,5,6,7,10,11,13,14,15,16,18,20&w.instant=null&w.type=bgp

  -Rich

Rich Lafferty
rich@lafferty.ca / 647-377-1165
Toronto, Canada
http://www.lafferty.ca


On Oct 4, 2021, 5:01 PM -0300, Peter Knapp <Peter.Knapp@ccsleeds.co.uk>, wrote:
> I’ve seen this BGP withdrawal cited in a number of places.
>
> However we peer with FB, and all four sessions are up and have been live for days/months (FB had maintenance on one peering router a couple of weeks ago) and all 30 prefixes are consistent and announced.
>
> The NS IP’s are in the announcements, but you can’t ICMP to them, or query them as NS for the auth records either..
>
> Peter Knapp
>
> From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rich Lafferty via Outages
> Sent: 04 October 2021 17:27
> To: Outages mailing list <outages@outages.org>
> Subject: Re: [outages] Facebook.com no records?
>
> A carrier networking engineer friend of mine observed that the network that those DNS servers live in is no longer in the global routing table. Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.)
>
>   -Rich
>
> Rich Lafferty
> rich@lafferty.ca / 647-377-1165
> Toronto, Canada
> http://www.lafferty.ca
>
>
> On Oct 4, 2021, 1:14 PM -0300, Jon Sands via Outages <outages@outages.org>, wrote:
>
> > quote_type
> > Seems the facebook.com domain no longer has any DNS records? Getting
> > nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc
> >
> > https://toolbox.googleapps.com/apps/dig/#A/facebook.com
> >
> > --
> > Jon Sands
> > MFI Labs
> > https://fohdeesha.com/
> >
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
Re: Facebook.com no records? [ In reply to ]
Yes, that's the underlying issue. A faulty BGP change caused a bunch of
routes to be withdrawn, including the Facebook DNS servers, which they self
host.

On Mon, Oct 4, 2021 at 5:37 PM Rich Lafferty via Outages <
outages@outages.org> wrote:

> A carrier networking engineer friend of mine observed that the network
> that those DNS servers live in is no longer in the global routing table.
> Sounds like BGP issues rather than DNS ones, I guess. (I won’t admit to
> understanding BGP.)
>
> -Rich
>
> *Rich Lafferty*
> rich@lafferty.ca / 647-377-1165
> Toronto, Canada
> http://www.lafferty.ca
>
> <https://twitter.com/mendel> <https://twitter.com/mendel>
> <https://facebook.com/richlafferty> <https://facebook.com/richlafferty>
> <https://linkedin.com/in/richlafferty>
> <https://linkedin.com/in/richlafferty> <https://instagram.com/wlonkly>
> <https://instagram.com/wlonkly> <https://www.flickr.com/photos/mendel>
> <https://www.flickr.com/photos/mendel> <https://github.com/wlonkly>
> <https://github.com/wlonkly> <https://mendel.tumblr.com>
> <https://mendel.tumblr.com>
> On Oct 4, 2021, 1:14 PM -0300, Jon Sands via Outages <outages@outages.org>,
> wrote:
>
> Seems the facebook.com domain no longer has any DNS records? Getting
> nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc
>
> https://toolbox.googleapps.com/apps/dig/#A/facebook.com
>
> --
> Jon Sands
> MFI Labs
> https://fohdeesha.com/
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
Re: Facebook.com no records? [ In reply to ]
Cloudflare summed it up nicely: https://blog.cloudflare.com/october-2021-facebook-outage/

Rob++

----- Original Message -----
> From: "Jason Kuehl via Outages" <outages@outages.org>
> To: "Jason Kuehl via Outages" <outages@outages.org>
> Sent: Monday, October 4, 2021 11:26:52 AM
> Subject: Re: [outages] Facebook.com no records?

> A carrier networking engineer friend of mine observed that the network that
> those DNS servers live in is no longer in the global routing table. Sounds like
> BGP issues rather than DNS ones, I guess. (I won’t admit to understanding BGP.)
>
> -Rich
>
> Rich Lafferty

--
Internet: windsor@warthog.com
Life: Rob@Van_Alstyne.Texas.USA.Earth

"They couldn't hit an elephant at this distance."
-- Major General John Sedgwick
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: [EXTERNAL] Re: Facebook.com no records? [ In reply to ]
I?d be surprised if a single person had that much authority. Good news is there might be some engineering jobs open up at FB
________________________________
From: Outages <outages-bounces@outages.org> on behalf of John Quinby via Outages <outages@outages.org>
Sent: Monday, October 4, 2021 1:03:11 PM
To: outages@outages.org <outages@outages.org>
Subject: [EXTERNAL] Re: [outages] Facebook.com no records?

CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.

I wonder if someone was fired and deleted something.


> host facebook.com
Host facebook.com not found: 2(SERVFAIL)
> dig facebook.com

; <<>> DiG 9.6.1-P1 <<>> facebook.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27869
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;facebook.com. IN A

;; Query time: 0 msec
;; SERVER: 10.64.4.2#53(10.64.4.2)
;; WHEN: Mon Oct 4 11:00:17 2021
;; MSG SIZE rcvd: 30

>


John Quinby
Vision Net | Network Operations Center Lead
p: (406) 216-4620
m: (406) 836-0223
w: https://vision.net










-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Jon Sands via
Outages
Sent: Monday, October 4, 2021 9:57 AM
To: outages@outages.org
Subject: [outages] Facebook.com no records?

Seems the facebook.com domain no longer has any DNS records? Getting
nxdomain from most clients, same when digging @ 1.1.1.1 8.8.8.8 etc

https://toolbox.googleapps.com/apps/dig/#A/facebook.com

--
Jon Sands
MFI Labs
https://fohdeesha.com/

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

E-MAIL CONFIDENTIALITY NOTICE:
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.