Mailing List Archive

Yes, we saw a temporary blip on our Level3 at that same time. Our BGP
session to them did not drop, but various sites were blackholed while
things were reconverging.



Ken Matlock
Network Analyst
Exempla Healthcare
(303) 467-4671
matlockk at exempla.org

________________________________

From: outages-bounces@outages.org [mailto:outages-bounces@outages.org]
On Behalf Of randal k
Sent: Tuesday, May 18, 2010 9:52 AM
To: outages at outages.org
Subject: [outages] Denver Level3 problems?



Did anybody else see Denver Level3 service take a hit this morning? Our
logs indicated nothing, but we saw various blackholing all over their
network for ~20 minutes, starting around 9:15am MST.

Everybody have a great day -
Randal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/a1856567/attachment.html>

We've been having issues staying connected to Google Chat for the last hour
or so.
And it just so happens we're connected through you. :-)

Best Wishes,
Jayson A. Baker
Peak Internet


On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com> wrote:

> Did anybody else see Denver Level3 service take a hit this morning? Our
> logs indicated nothing, but we saw various blackholing all over their
> network for ~20 minutes, starting around 9:15am MST.
>
> Everybody have a great day -
> Randal
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/77bfc51e/attachment.html>

We got nailed in Chicago.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/c14f9c06/attachment.html>

Hit hard in Seattle too. Level3 NOC is investigating.

- Kevin Broderick
AS11608

"Beth Johnson" <bethjohnson5060 at gmail.com> wrote:

>We got nailed in Chicago.
>_______________________________________________
>Outages mailing list
>Outages at outages.org
>https://puck.nether.net/mailman/listinfo/outages

We saw a huge hit earlier in Denver (around 9:20am) through Level3, but only
partially dropped our traffic. (about 1/3 was still available)



Shawn



From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On
Behalf Of randal k
Sent: Tuesday, May 18, 2010 9:52 AM
To: outages at outages.org
Subject: [outages] Denver Level3 problems?



Did anybody else see Denver Level3 service take a hit this morning? Our logs
indicated nothing, but we saw various blackholing all over their network for
~20 minutes, starting around 9:15am MST.

Everybody have a great day -
Randal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/2362141b/attachment.html>

We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yet...


From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker
Sent: Tuesday, May 18, 2010 12:31
To: randal k
Cc: outages at outages.org
Subject: Re: [outages] Denver Level3 problems?

We've been having issues staying connected to Google Chat for the last hour or so.
And it just so happens we're connected through you. :-)

Best Wishes,
Jayson A. Baker
Peak Internet

On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com<mailto:rkohutek at gmail.com>> wrote:
Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST.

Everybody have a great day -
Randal

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/31847e02/attachment-0001.html>

Nationwide, huh?

Great...





On Tue, May 18, 2010 at 9:45 AM, Kevin Broderick <
kbroder at accretive-networks.net> wrote:

> Hit hard in Seattle too. Level3 NOC is investigating.
>
> - Kevin Broderick
> AS11608
>
> "Beth Johnson" <bethjohnson5060 at gmail.com> wrote:
>
> >We got nailed in Chicago.
> >_______________________________________________
> >Outages mailing list
> >Outages at outages.org
> >https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/1d8fcb6a/attachment.html>

We saw it too. We're connected to Detroit (Southfield Gateway), and
saw traffic falling off the face of the earth at Chicago.

At 12:49 PM 5/18/2010, Shawn Clemens wrote:
>Content-Type: multipart/alternative;
> boundary="----=_NextPart_000_003F_01CAF677.C644CFE0"
>Content-Language: en-us
>
>We saw a huge hit earlier in Denver (around 9:20am) through Level3,
>but only partially dropped our traffic. (about 1/3 was still available)
>
>Shawn
>
>From: outages-bounces at outages.org
>[mailto:outages-bounces at outages.org] On Behalf Of randal k
>Sent: Tuesday, May 18, 2010 9:52 AM
>To: outages at outages.org
>Subject: [outages] Denver Level3 problems?
>
>Did anybody else see Denver Level3 service take a hit this morning?
>Our logs indicated nothing, but we saw various blackholing all over
>their network for ~20 minutes, starting around 9:15am MST.
>
>Everybody have a great day -
>Randal
>_______________________________________________
>Outages mailing list
>Outages at outages.org
>https://puck.nether.net/mailman/listinfo/outages

---

Clayton Zekelman
Managed Network Systems Inc. (MNSi)
3363 Tecumseh Rd. E
Windsor, Ontario
N8W 1H4

tel. 519-985-8410
fax. 519-985-8409
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/e660201b/attachment.html>

We also saw issues with customer end points that took level 3 around the same time but in NYC - LA and DFW POPS.

Joseph


From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Sean Clark
Sent: Tuesday, May 18, 2010 11:53 AM
To: Jayson Baker; randal k
Cc: outages at outages.org
Subject: Re: [outages] Denver Level3 problems?

We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yet...


From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker
Sent: Tuesday, May 18, 2010 12:31
To: randal k
Cc: outages at outages.org
Subject: Re: [outages] Denver Level3 problems?

We've been having issues staying connected to Google Chat for the last hour or so.
And it just so happens we're connected through you. :-)

Best Wishes,
Jayson A. Baker
Peak Internet
On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com<mailto:rkohutek at gmail.com>> wrote:
Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST.

Everybody have a great day -
Randal

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/7d8a7e3f/attachment.html>

An internal source has informed me there was an issue with a Data Center in
the Denver area in which something ?blew up? and this caused a chain effect
with anything going through Denver to any other pop in the USA.

The problem since has been resolved and was for a very short period.

Brad



On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson at aninetworks.net> wrote:

> We also saw issues with customer end points that took level 3 around the same
> time but in NYC ? LA and DFW POPS.
>
> Joseph
>
>
>
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Sean Clark
> Sent: Tuesday, May 18, 2010 11:53 AM
> To: Jayson Baker; randal k
> Cc: outages at outages.org
> Subject: Re: [outages] Denver Level3 problems?
>
> We had many customer complaints and all our IP SLA tests spiked between 1110
> and 1135 est for Level 3 in our Denver market. But we also saw the problems
> in DAL on our level 3 peer (SLA test as well). Some event happened in the
> Denver area on level 3, we have opened a ticket with them and they reported to
> us they have had a lot of customer call in for the same thing. They have not
> given us an RFO yet?
>
>
>
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Jayson Baker
> Sent: Tuesday, May 18, 2010 12:31
> To: randal k
> Cc: outages at outages.org
> Subject: Re: [outages] Denver Level3 problems?
>
> We've been having issues staying connected to Google Chat for the last hour or
> so.
>
> And it just so happens we're connected through you. :-)
>
>
> Best Wishes,
> Jayson A. Baker
> Peak Internet
>
> On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com> wrote:
> Did anybody else see Denver Level3 service take a hit this morning? Our logs
> indicated nothing, but we saw various blackholing all over their network for
> ~20 minutes, starting around 9:15am MST.
>
> Everybody have a great day -
> Randal
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages


Bradley D. Bopp
Director Of Engineering
http://www.nationalnet.com
http://as22384.peeringdb.net

NationalNet is committed to the highest level of Customer Service available
in the Web Hosting business. If you have any questions, comments or
concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free,
888-4-NATNET).


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/197e1b77/attachment.html>

They have told us that were trying to isolate some dirty circuits from an etherchannel bundle and that went way wrong... In the Denver area... data center/POP .. whatever.. Level 3 rocks...

From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Bradley Bopp
Sent: Tuesday, May 18, 2010 15:04
To: Joseph Jackson
Cc: outages at outages.org
Subject: Re: [outages] Denver Level3 problems?

An internal source has informed me there was an issue with a Data Center in the Denver area in which something "blew up" and this caused a chain effect with anything going through Denver to any other pop in the USA.

The problem since has been resolved and was for a very short period.

Brad



On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson at aninetworks.net> wrote:
We also saw issues with customer end points that took level 3 around the same time but in NYC - LA and DFW POPS.

Joseph



From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Sean Clark
Sent: Tuesday, May 18, 2010 11:53 AM
To: Jayson Baker; randal k
Cc: outages at outages.org
Subject: Re: [outages] Denver Level3 problems?

We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yet...



From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker
Sent: Tuesday, May 18, 2010 12:31
To: randal k
Cc: outages at outages.org
Subject: Re: [outages] Denver Level3 problems?

We've been having issues staying connected to Google Chat for the last hour or so.

And it just so happens we're connected through you. :-)


Best Wishes,
Jayson A. Baker
Peak Internet

On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com> wrote:
Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST.

Everybody have a great day -
Randal

_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

________________________________
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages


Bradley D. Bopp
Director Of Engineering
http://www.nationalnet.com
http://as22384.peeringdb.net

NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100518/ac689674/attachment.html>

L3 NOC called us back with reason for outage yesterday: PEBKAC.

- K

Bradley Bopp wrote:
> An internal source has informed me there was an issue with a Data Center in
> the Denver area in which something ?blew up? and this caused a chain effect
> with anything going through Denver to any other pop in the USA.
>
> The problem since has been resolved and was for a very short period.
>
> Brad
>
>
>
> On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson at aninetworks.net> wrote:
>
>
>> We also saw issues with customer end points that took level 3 around the same
>> time but in NYC ? LA and DFW POPS.
>>
>> Joseph
>>
>>
>>
>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>> Behalf Of Sean Clark
>> Sent: Tuesday, May 18, 2010 11:53 AM
>> To: Jayson Baker; randal k
>> Cc: outages at outages.org
>> Subject: Re: [outages] Denver Level3 problems?
>>
>> We had many customer complaints and all our IP SLA tests spiked between 1110
>> and 1135 est for Level 3 in our Denver market. But we also saw the problems
>> in DAL on our level 3 peer (SLA test as well). Some event happened in the
>> Denver area on level 3, we have opened a ticket with them and they reported to
>> us they have had a lot of customer call in for the same thing. They have not
>> given us an RFO yet?
>>
>>
>>
>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>> Behalf Of Jayson Baker
>> Sent: Tuesday, May 18, 2010 12:31
>> To: randal k
>> Cc: outages at outages.org
>> Subject: Re: [outages] Denver Level3 problems?
>>
>> We've been having issues staying connected to Google Chat for the last hour or
>> so.
>>
>> And it just so happens we're connected through you. :-)
>>
>>
>> Best Wishes,
>> Jayson A. Baker
>> Peak Internet
>>
>> On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com> wrote:
>> Did anybody else see Denver Level3 service take a hit this morning? Our logs
>> indicated nothing, but we saw various blackholing all over their network for
>> ~20 minutes, starting around 9:15am MST.
>>
>> Everybody have a great day -
>> Randal
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
>
> Bradley D. Bopp
> Director Of Engineering
> http://www.nationalnet.com
> http://as22384.peeringdb.net
>
> NationalNet is committed to the highest level of Customer Service available
> in the Web Hosting business. If you have any questions, comments or
> concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free,
> 888-4-NATNET).
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Sorry to dredge up an old issue, but did anyone get anything better than "PEBKAC".

We were having some issues with voice that was going to Level3 last Tuesday and trying to figure out if what you all saw was related or not (I use Level3 for transit and voice termination), but since I didn't open a ticket at the time (we thought it was an internal issue), Level3 says they can't research the issue.

Any help?

-Sean


-----Original Message-----
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Kevin Broderick
Sent: Wednesday, May 19, 2010 10:24 AM
To: outages at outages.org
Subject: Re: [outages] Denver Level3 problems?

L3 NOC called us back with reason for outage yesterday: PEBKAC.

- K

Bradley Bopp wrote:
> An internal source has informed me there was an issue with a Data Center in
> the Denver area in which something ?blew up? and this caused a chain effect
> with anything going through Denver to any other pop in the USA.
>
> The problem since has been resolved and was for a very short period.
>
> Brad
>
>
>
> On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson at aninetworks.net> wrote:
>
>
>> We also saw issues with customer end points that took level 3 around the same
>> time but in NYC ? LA and DFW POPS.
>>
>> Joseph
>>
>>
>>
>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>> Behalf Of Sean Clark
>> Sent: Tuesday, May 18, 2010 11:53 AM
>> To: Jayson Baker; randal k
>> Cc: outages at outages.org
>> Subject: Re: [outages] Denver Level3 problems?
>>
>> We had many customer complaints and all our IP SLA tests spiked between 1110
>> and 1135 est for Level 3 in our Denver market. But we also saw the problems
>> in DAL on our level 3 peer (SLA test as well). Some event happened in the
>> Denver area on level 3, we have opened a ticket with them and they reported to
>> us they have had a lot of customer call in for the same thing. They have not
>> given us an RFO yet?
>>
>>
>>
>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>> Behalf Of Jayson Baker
>> Sent: Tuesday, May 18, 2010 12:31
>> To: randal k
>> Cc: outages at outages.org
>> Subject: Re: [outages] Denver Level3 problems?
>>
>> We've been having issues staying connected to Google Chat for the last hour or
>> so.
>>
>> And it just so happens we're connected through you. :-)
>>
>>
>> Best Wishes,
>> Jayson A. Baker
>> Peak Internet
>>
>> On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com> wrote:
>> Did anybody else see Denver Level3 service take a hit this morning? Our logs
>> indicated nothing, but we saw various blackholing all over their network for
>> ~20 minutes, starting around 9:15am MST.
>>
>> Everybody have a great day -
>> Randal
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
>
> Bradley D. Bopp
> Director Of Engineering
> http://www.nationalnet.com
> http://as22384.peeringdb.net
>
> NationalNet is committed to the highest level of Customer Service available
> in the Web Hosting business. If you have any questions, comments or
> concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free,
> 888-4-NATNET).
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages