Mailing List Archive

Wow. Their own site is even down.

On Sep 10, 2012, at 2:07 PM, "Aaron C. de Bruyn" <aaron@heyaaron.com> wrote:

> For the last ~15 minutes I've been receiving complaints about DNS
> issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of
> tweets about their hosting and VPS being down. I'm unable to access
> the control panel for one of my customer accounts.
>
>
> -A
>

On Mon, Sep 10, 2012 at 2:07 PM, Aaron C. de Bruyn <aaron@heyaaron.com>wrote:

> For the last ~15 minutes I've been receiving complaints about DNS
> issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of
> tweets about their hosting and VPS being down. I'm unable to access
> the control panel for one of my customer accounts.
>
>
> -A
>
>
This is already being covered on outages list seems be very widespread

Looks like this may be a DDoS attack from Anonymous:

http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/


-----Original Message-----
From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com]
Sent: Monday, September 10, 2012 1:07 PM
To: NANOG mailing list
Subject: Heads-Up: GoDaddy Broke the Interwebs...

For the last ~15 minutes I've been receiving complaints about DNS issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of tweets about their hosting and VPS being down. I'm unable to access the control panel for one of my customer accounts.


-A

Of course, it could be a case of Hanlon's Razor.

-Steve

On Mon, Sep 10, 2012 at 3:13 PM, <Bill.Ingrum@t-systems.com> wrote:

> Looks like this may be a DDoS attack from Anonymous:
>
>
> http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/
>
>
> -----Original Message-----
> From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com]
> Sent: Monday, September 10, 2012 1:07 PM
> To: NANOG mailing list
> Subject: Heads-Up: GoDaddy Broke the Interwebs...
>
> For the last ~15 minutes I've been receiving complaints about DNS issues.
> GoDaddy DNS is apparently b0rked. I'm also seeing a lot of tweets about
> their hosting and VPS being down. I'm unable to access the control panel
> for one of my customer accounts.
>
>
> -A
>
>

Yeah, I heard about downtime from couple of our clients.



Seems like US West coast datacenter has issues specially with DNS. I see
DNS working fine in Asia (via Singapore nodes), and in other parts of US as
well.


Here's a failing DNS node (checking from my Europe test server)



traceroute to 216.69.185.46 (216.69.185.46), 30 hops max, 60 byte packets
1 gw.giga-dns.com (91.194.90.1) 1.141 ms 1.134 ms 1.346 ms
2 62.140.24.125 (62.140.24.125) 1.835 ms 1.830 ms 1.820 ms
3 ae-19-19.ebr1.Frankfurt1.Level3.net (4.69.153.246) 6.560 ms 6.560 ms
6.552 ms
4 ae-46-46.ebr2.Paris1.Level3.net (4.69.143.138) 15.286 ms
ae-47-47.ebr2.Paris1.Level3.net (4.69.143.142) 15.270 ms
ae-45-45.ebr2.Paris1.Level3.net (4.69.143.134) 15.246 ms
5 ae-42-42.ebr2.Washington1.Level3.net (4.69.137.54) 95.934 ms 95.928
ms ae-43-43.ebr2.Washington1.Level3.net (4.69.137.58) 95.820 ms
6 ae-82-82.csw3.Washington1.Level3.net (4.69.134.154) 94.157 ms
ae-92-92.csw4.Washington1.Level3.net (4.69.134.158) 96.707 ms 98.696 ms
7 ae-23-70.car3.Washington1.Level3.net (4.69.149.69) 95.490 ms
ae-43-90.car3.Washington1.Level3.net (4.69.149.197) 93.731 ms 96.204 ms
8 THE-GO-DADD.car3.Washington1.Level3.net (4.79.168.54) 94.708 ms
97.039 ms 95.756 ms
9 208.109.112.253 (208.109.112.253) 95.244 ms 94.401 ms 94.388 ms
10 208.109.112.253 (208.109.112.253) 94.378 ms 93.864 ms 95.172 ms
11 208.109.113.58 (208.109.113.58) 94.387 ms 94.359 ms 94.042 ms
12 * * *





I am very curious to know what prevents Godaddy from withdrawling prefixes
from that specific region and let other working datacenters to handle DNS?
Can that cause issue on other stable nodes during DOS attack?




On Tue, Sep 11, 2012 at 12:43 AM, <Bill.Ingrum@t-systems.com> wrote:

> Looks like this may be a DDoS attack from Anonymous:
>
>
> http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/
>
>
> -----Original Message-----
> From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com]
> Sent: Monday, September 10, 2012 1:07 PM
> To: NANOG mailing list
> Subject: Heads-Up: GoDaddy Broke the Interwebs...
>
> For the last ~15 minutes I've been receiving complaints about DNS issues.
> GoDaddy DNS is apparently b0rked. I'm also seeing a lot of tweets about
> their hosting and VPS being down. I'm unable to access the control panel
> for one of my customer accounts.
>
>
> -A
>
>


--

Anurag Bhatia
anuragbhatia.com

Linkedin <http://in.linkedin.com/in/anuragbhatia21> |
Twitter<https://twitter.com/anurag_bhatia>|
Google+ <https://plus.google.com/118280168625121532854>

I thought I saw an article on routergod.com from Dance Patrick regarding anycast DNS......
~oliver

Sent via DynaTAC. Please forgive spelling and grammar.

-----Original Message-----
From: Bill.Ingrum@t-systems.com
Date: Mon, 10 Sep 2012 19:13:27
To: <aaron@heyaaron.com>; <nanog@nanog.org>
Subject: RE: Heads-Up: GoDaddy Broke the Interwebs...


Looks like this may be a DDoS attack from Anonymous:

http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/


-----Original Message-----
From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com]
Sent: Monday, September 10, 2012 1:07 PM
To: NANOG mailing list
Subject: Heads-Up: GoDaddy Broke the Interwebs...

For the last ~15 minutes I've been receiving complaints about DNS issues.  GoDaddy DNS is apparently b0rked.  I'm also seeing a lot of tweets about their hosting and VPS being down.  I'm unable to access the control panel for one of my customer accounts.


-A

http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

On Mon, Sep 10, 2012 at 1:27 PM, Operations Dallas
<operations.tcdallas@hotmail.com> wrote:
> I thought I saw an article on routergod.com from Dance Patrick regarding anycast DNS......
> ~oliver
>
> Sent via DynaTAC. Please forgive spelling and grammar.
>
> -----Original Message-----
> From: Bill.Ingrum@t-systems.com
> Date: Mon, 10 Sep 2012 19:13:27
> To: <aaron@heyaaron.com>; <nanog@nanog.org>
> Subject: RE: Heads-Up: GoDaddy Broke the Interwebs...
>
>
> Looks like this may be a DDoS attack from Anonymous:
>
> http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/
>
>
> -----Original Message-----
> From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com]
> Sent: Monday, September 10, 2012 1:07 PM
> To: NANOG mailing list
> Subject: Heads-Up: GoDaddy Broke the Interwebs...
>
> For the last ~15 minutes I've been receiving complaints about DNS issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of tweets about their hosting and VPS being down. I'm unable to access the control panel for one of my customer accounts.
>
>
> -A
>



--
Kyle Creyts

Information Assurance Professional
BSidesDetroit Organizer

No DDoS or Anonymous attack appears to have been involved.

On Tue, Sep 11, 2012 at 10:54 AM, Kyle Creyts <kyle.creyts@gmail.com> wrote:
> http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410
>
> On Mon, Sep 10, 2012 at 1:27 PM, Operations Dallas
> <operations.tcdallas@hotmail.com> wrote:
>> I thought I saw an article on routergod.com from Dance Patrick regarding anycast DNS......
>> ~oliver
>>
>> Sent via DynaTAC. Please forgive spelling and grammar.
>>
>> -----Original Message-----
>> From: Bill.Ingrum@t-systems.com
>> Date: Mon, 10 Sep 2012 19:13:27
>> To: <aaron@heyaaron.com>; <nanog@nanog.org>
>> Subject: RE: Heads-Up: GoDaddy Broke the Interwebs...
>>
>>
>> Looks like this may be a DDoS attack from Anonymous:
>>
>> http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-of-sites/
>>
>>
>> -----Original Message-----
>> From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com]
>> Sent: Monday, September 10, 2012 1:07 PM
>> To: NANOG mailing list
>> Subject: Heads-Up: GoDaddy Broke the Interwebs...
>>
>> For the last ~15 minutes I've been receiving complaints about DNS issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of tweets about their hosting and VPS being down. I'm unable to access the control panel for one of my customer accounts.
>>
>>
>> -A
>>
>
>
>
> --
> Kyle Creyts
>
> Information Assurance Professional
> BSidesDetroit Organizer



--
Kyle Creyts

Information Assurance Professional
BSidesDetroit Organizer

Now it's CNN

/Jason


-----Original Message-----
From: Kyle Creyts [mailto:kyle.creyts@gmail.com]
Sent: Tuesday, September 11, 2012 1:55 PM
To: Operations Dallas
Cc: nanog@nanog.org
Subject: Re: Heads-Up: GoDaddy Broke the Interwebs...

No DDoS or Anonymous attack appears to have been involved.

On Tue, Sep 11, 2012 at 10:54 AM, Kyle Creyts <kyle.creyts@gmail.com>
wrote:
> http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410
>
> On Mon, Sep 10, 2012 at 1:27 PM, Operations Dallas
> <operations.tcdallas@hotmail.com> wrote:
>> I thought I saw an article on routergod.com from Dance Patrick
regarding anycast DNS......
>> ~oliver
>>
>> Sent via DynaTAC. Please forgive spelling and grammar.
>>
>> -----Original Message-----
>> From: Bill.Ingrum@t-systems.com
>> Date: Mon, 10 Sep 2012 19:13:27
>> To: <aaron@heyaaron.com>; <nanog@nanog.org>
>> Subject: RE: Heads-Up: GoDaddy Broke the Interwebs...
>>
>>
>> Looks like this may be a DDoS attack from Anonymous:
>>
>> http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-o
>> f-sites/
>>
>>
>> -----Original Message-----
>> From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com]
>> Sent: Monday, September 10, 2012 1:07 PM
>> To: NANOG mailing list
>> Subject: Heads-Up: GoDaddy Broke the Interwebs...
>>
>> For the last ~15 minutes I've been receiving complaints about DNS
issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of
tweets about their hosting and VPS being down. I'm unable to access the
control panel for one of my customer accounts.
>>
>>
>> -A
>>
>
>
>
> --
> Kyle Creyts
>
> Information Assurance Professional
> BSidesDetroit Organizer



--
Kyle Creyts

Information Assurance Professional
BSidesDetroit Organizer

And this is bad why?

Sent from my iPhone

On 2012-09-11, at 1:14 PM, "Jason Bertoch" <Jason_Bertoch@nwrdc.fsu.edu> wrote:

> Now it's CNN
>
> /Jason
>
>
> -----Original Message-----
> From: Kyle Creyts [mailto:kyle.creyts@gmail.com]
> Sent: Tuesday, September 11, 2012 1:55 PM
> To: Operations Dallas
> Cc: nanog@nanog.org
> Subject: Re: Heads-Up: GoDaddy Broke the Interwebs...
>
> No DDoS or Anonymous attack appears to have been involved.
>
> On Tue, Sep 11, 2012 at 10:54 AM, Kyle Creyts <kyle.creyts@gmail.com>
> wrote:
>> http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410
>>
>> On Mon, Sep 10, 2012 at 1:27 PM, Operations Dallas
>> <operations.tcdallas@hotmail.com> wrote:
>>> I thought I saw an article on routergod.com from Dance Patrick
> regarding anycast DNS......
>>> ~oliver
>>>
>>> Sent via DynaTAC. Please forgive spelling and grammar.
>>>
>>> -----Original Message-----
>>> From: Bill.Ingrum@t-systems.com
>>> Date: Mon, 10 Sep 2012 19:13:27
>>> To: <aaron@heyaaron.com>; <nanog@nanog.org>
>>> Subject: RE: Heads-Up: GoDaddy Broke the Interwebs...
>>>
>>>
>>> Looks like this may be a DDoS attack from Anonymous:
>>>
>>> http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-millions-o
>>> f-sites/
>>>
>>>
>>> -----Original Message-----
>>> From: Aaron C. de Bruyn [mailto:aaron@heyaaron.com]
>>> Sent: Monday, September 10, 2012 1:07 PM
>>> To: NANOG mailing list
>>> Subject: Heads-Up: GoDaddy Broke the Interwebs...
>>>
>>> For the last ~15 minutes I've been receiving complaints about DNS
> issues. GoDaddy DNS is apparently b0rked. I'm also seeing a lot of
> tweets about their hosting and VPS being down. I'm unable to access the
> control panel for one of my customer accounts.
>>>
>>>
>>> -A
>>>
>>
>>
>>
>> --
>> Kyle Creyts
>>
>> Information Assurance Professional
>> BSidesDetroit Organizer
>
>
>
> --
> Kyle Creyts
>
> Information Assurance Professional
> BSidesDetroit Organizer
>
>

On Tue, Sep 11, 2012 at 1:54 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:
> http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

"many of our customers experienced intermittent service outages"

Must be that new definition of the word "intermittent." The one
roughly synonymous with "total."

-Bill


--
William D. Herrin ................ herrin@dirtside.com bill@herrin.us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

On Tue, Sep 11, 2012 at 03:15:55PM -0400, William Herrin wrote:
> On Tue, Sep 11, 2012 at 1:54 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:
> > http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410
>
> "many of our customers experienced intermittent service outages"
>
> Must be that new definition of the word "intermittent." The one
> roughly synonymous with "total."

Yeah. Doubleplusungood.

--
Mike Andrews, W5EGO
mikea@mikea.ath.cx
Tired old sysadmin

As someone else nicely pointed out "network problems starting when the anon post said they would, and ending when they said they would stop.... ironic?"

-----Original Message-----
From: William Herrin [mailto:bill@herrin.us]
Sent: Tuesday, September 11, 2012 1:16 PM
To: Kyle Creyts
Cc: nanog@nanog.org
Subject: Re: Heads-Up: GoDaddy Broke the Interwebs...

On Tue, Sep 11, 2012 at 1:54 PM, Kyle Creyts <kyle.creyts@gmail.com> wrote:
> http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410

"many of our customers experienced intermittent service outages"

Must be that new definition of the word "intermittent." The one roughly synonymous with "total."

-Bill


--
William D. Herrin ................ herrin@dirtside.com bill@herrin.us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

What time did the anon first say there would be GoDaddy problems? Earliest
timestamp I can find is 10:45am (pacific) and the problems had started much
earlier, around 10:15. Curious if you found evidence of any anons claiming
an attack or responsibility for the attack within, say, 5 minutes of it
starting.

Also, the time it stopped wasn't exactly tied to anything the anon said,
other than his vague statements like "it can last one hour or one month"
and "soon u guys can acess". And he said that latter statement at 1:59pm
while the outage ended at 3:45pm.

Summary: 30 minutes late on the start time, and off by well over an hour on
the stop time.

Damian

On Tue, Sep 11, 2012 at 12:25 PM, Blake Pfankuch <blake@pfankuch.me> wrote:

> As someone else nicely pointed out "network problems starting when the
> anon post said they would, and ending when they said they would stop....
> ironic?"
>
> -----Original Message-----
> From: William Herrin [mailto:bill@herrin.us]
> Sent: Tuesday, September 11, 2012 1:16 PM
> To: Kyle Creyts
> Cc: nanog@nanog.org
> Subject: Re: Heads-Up: GoDaddy Broke the Interwebs...
>
> On Tue, Sep 11, 2012 at 1:54 PM, Kyle Creyts <kyle.creyts@gmail.com>
> wrote:
> > http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410
>
> "many of our customers experienced intermittent service outages"
>
> Must be that new definition of the word "intermittent." The one roughly
> synonymous with "total."
>
> -Bill
>
>
> --
> William D. Herrin ................ herrin@dirtside.com bill@herrin.us
> 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls
> Church, VA 22042-3004
>
>
>


--
Damian Menscher :: Security Reliability Engineer :: Google

On Tue, Sep 11, 2012 at 3:47 PM, Damian Menscher <damian@google.com> wrote:
>
> Summary: 30 minutes late on the start time, and off by well over an hour on
> the stop time.

even a broken clock is right 2x/day?
nostrodamus was eventually right a few times?
'If you're cold, shoot until you get hot, then keep shooting!' - dick vitale

folk like to look for the most complicated/spooky/crazy reason... most
often it's just a simple reason for failure :(
so far godaddy seems to agree with the 'it was a simple mistake on our
part' (paraphrased, they probably won't say 'simple')

On Sep 11, 2012, at 16:04 , Christopher Morrow <morrowc.lists@gmail.com> wrote:
> On Tue, Sep 11, 2012 at 3:47 PM, Damian Menscher <damian@google.com> wrote:
>>
>> Summary: 30 minutes late on the start time, and off by well over an hour on
>> the stop time.
>
> even a broken clock is right 2x/day?
> nostrodamus was eventually right a few times?
> 'If you're cold, shoot until you get hot, then keep shooting!' - dick vitale
>
> folk like to look for the most complicated/spooky/crazy reason... most
> often it's just a simple reason for failure :(
> so far godaddy seems to agree with the 'it was a simple mistake on our
> part' (paraphrased, they probably won't say 'simple')

No large flows reported to the affected NSes, tweets were suspicious at best, other anon-ops denied the attack was them, and GoDaddy admitted internal error.

I'm going to take GoDaddy at their word, and give them major kudos for owning up to the mistake - in public.

--
TTFN,
patrick

> No large flows reported to the affected NSes, tweets were suspicious at best, other anon-ops denied the attack was them, and GoDaddy admitted internal error.
>
> I'm going to take GoDaddy at their word, and give them major kudos for owning up to the mistake - in public.

That doesn't mean that their description of the internal error fits
what happened. Not to say that there were an attack, just that there
can be more internal failures, including processes, to be accounted
for. Whether they will publish a root-cause analysis/swiss chesse
model/<insert your preferred methodology> or not is up to them, but to
tech-savvy stakeholders I think they are still in debt.


Rubens

The blog says 99.999% uptime, but I'm guessing this "outage" lasted
more them 5.49300000002 minutes and they probably had other issues
during the year.


On 11 September 2012 21:53, Rubens Kuhl <rubensk@gmail.com> wrote:
>
> > No large flows reported to the affected NSes, tweets were suspicious at best, other anon-ops denied the attack was them, and GoDaddy admitted internal error.
> >
> > I'm going to take GoDaddy at their word, and give them major kudos for owning up to the mistake - in public.
>
> That doesn't mean that their description of the internal error fits
> what happened. Not to say that there were an attack, just that there
> can be more internal failures, including processes, to be accounted
> for. Whether they will publish a root-cause analysis/swiss chesse
> model/<insert your preferred methodology> or not is up to them, but to
> tech-savvy stakeholders I think they are still in debt.
>
>
> Rubens
>



--
Äêêêêêçççççêêêêêçççç
Îééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
Æéééééçççççéééééçççççééééééééçççççéééééçççççééééééééçççççéééééçççççééééééééçççççééé

BaconZombie

LOAD "*",8,1

when patrick is referring to "taking their word for it", he's referring to
a post on outages@ by godaddy's network engineering manager that stated
"bgp, and more details to follow".

i tend to align with patrick's thought. i'm also interested to see the
details, which they are really under no obligation to provide.

On Tue, Sep 11, 2012 at 1:53 PM, Rubens Kuhl <rubensk@gmail.com> wrote:

> > No large flows reported to the affected NSes, tweets were suspicious at
> best, other anon-ops denied the attack was them, and GoDaddy admitted
> internal error.
> >
> > I'm going to take GoDaddy at their word, and give them major kudos for
> owning up to the mistake - in public.
>
> That doesn't mean that their description of the internal error fits
> what happened. Not to say that there were an attack, just that there
> can be more internal failures, including processes, to be accounted
> for. Whether they will publish a root-cause analysis/swiss chesse
> model/<insert your preferred methodology> or not is up to them, but to
> tech-savvy stakeholders I think they are still in debt.
>
>
> Rubens
>
>

On Sep 11, 2012, at 4:53 PM, Rubens Kuhl <rubensk@gmail.com> wrote:

> That doesn't mean that their description of the internal error fits
> what happened

Anytime I've seen a real RFO, it takes more than 24 hours to collect data. Sometimes you actually don't know what happened. There's a reason for this comic: http://www.dilbert.com/strips/comic/1999-08-04/ (the reboot cleared the problem).

I've seen many odd behaviors of devices that nobody could explain, including the vendors.. sometimes it takes a few years to understand what happened. I recall a case where 2-3 years after a major outage someone made some minor comment about their architecture and a light came on.

I welcome more information about mistakes/errors that we can all learn from. Sharing that information can be hard or uncomfortable at times, but can help others learn and not make the same mistakes again. I took the recommendation of others and have started to read "Normal Accidents". amazon link: http://tinyurl.com/9dc6x98

The whole multiple-failures problem really makes me concerned about cascading system failures when things go wrong.

- Jared

On Sep 11, 2012, at 17:04 , ryanL <ryan.landry@gmail.com> wrote:

> when patrick is referring to "taking their word for it", he's referring to a post on outages@ by godaddy's network engineering manager that stated "bgp, and more details to follow".

Well, mostly I'm taking GoDaddy at their word that this was not a DoS attack.

I also believe it was related to BGP, and am happy to get more info. But we are discussing Anonymous vs. Self-inflicted wound here.

--
TTFN,
patrick


> i tend to align with patrick's thought. i'm also interested to see the details, which they are really under no obligation to provide.
>
> On Tue, Sep 11, 2012 at 1:53 PM, Rubens Kuhl <rubensk@gmail.com> wrote:
> > No large flows reported to the affected NSes, tweets were suspicious at best, other anon-ops denied the attack was them, and GoDaddy admitted internal error.
> >
> > I'm going to take GoDaddy at their word, and give them major kudos for owning up to the mistake - in public.
>
> That doesn't mean that their description of the internal error fits
> what happened. Not to say that there were an attack, just that there
> can be more internal failures, including processes, to be accounted
> for. Whether they will publish a root-cause analysis/swiss chesse
> model/<insert your preferred methodology> or not is up to them, but to
> tech-savvy stakeholders I think they are still in debt.
>
>
> Rubens
>
>

On Tue, Sep 11, 2012 at 6:04 PM, ryanL <ryan.landry@gmail.com> wrote:
> when patrick is referring to "taking their word for it", he's referring to a
> post on outages@ by godaddy's network engineering manager that stated "bgp,
> and more details to follow".

"more" is the operating word here.

> i tend to align with patrick's thought. i'm also interested to see the
> details, which they are really under no obligation to provide.

They could have said unspecified/yet-to-be-investigated internal
technical failure(s). But when they quoted an specific failure, their
communication(the PR, not the outages@ report) was trying to benefit
from being specific, which is known to gain more trust. That comes
with a responsibility to be precise and opens it to scrutiny by the
technical community.



Rubens

> Well, mostly I'm taking GoDaddy at their word that this was not a DoS attack.
>
> I also believe it was related to BGP, and am happy to get more info. But we are discussing Anonymous vs. Self-inflicted wound here.

I'm skeptical, BGPlay (http://bgplay.routeviews.org/) doesn't show any withdrawn routes for any of their prefixes over Sep 9-11. Infact, their BGP operation looks fairly operational during the time from what I can gather.

So, it would be nice to get more info.

- Naveen

On Tue, Sep 11, 2012 at 11:16 PM, Naveen Nathan <naveen@lastninja.net> wrote:
>> Well, mostly I'm taking GoDaddy at their word that this was not a DoS attack.
>>
>> I also believe it was related to BGP, and am happy to get more info. But we are discussing Anonymous vs. Self-inflicted wound here.
>
> I'm skeptical, BGPlay (http://bgplay.routeviews.org/) doesn't show any withdrawn routes for any of their prefixes over Sep 9-11. Infact, their BGP operation looks fairly operational during the time from what I can gather.

a bgp error doesn't HAVE to mean that they withdrew (or even
re-announced!) anything to the outside world, does it?

for instance:
border-router -> internet
redistribute your aggregate networks from statics to Null0 on the
border-router
accept full routes so you can send them to the other borders and
make good decisions at the external edge

border-router -> internal
send default or some version of default via a fitler to internal
datacenter routers/aggregation/distribution devices.
accept from them (maybe) local subnets that are part of your aggregates

now, accidently remove the filter content for the sessions between the
border and internal ... oops, your internal devices bounce with
'corrupted tables' (blown tables)... you still send your aggs steadily
to the interwebs, wee!

-chris