We have a ticket open.
MTR between our network (level 3 customer) and another network (level 3
customer) was showing that ae-1-100.ebr2.Denver1.Level3.net was running
50%-90% packet loss for the past 6+ hours. Well, that doesn't really
tell the whole story since routers routinely drop ICMP as a matter of
due course during periods of high load. But, point to point pings to
far end destinations that would normally be 40-60ms were varying
regularly (every 60 seconds) to 240-260ms. And, of course the voice
calls on that path sounded like crap.
What it looks like is a router that is experiencing consistently high
CPU load that is chewing on some process every 60 seconds. That
explains the MTR packet loss numbers that are probably innocuous. But,
while the processor is chewing on that process, the throughput latency
goes through the roof. This is a very common thing that we have seen
before . . . just not in the middle of the Level3 network.
The Level3 tech said that since he could not replicate the problem, he
would leave the ticket open until tomorrow.
Bryan
Sean Crandall wrote:
> Sorry to dredge up an old issue, but did anyone get anything better than "PEBKAC".
>
> We were having some issues with voice that was going to Level3 last Tuesday and trying to figure out if what you all saw was related or not (I use Level3 for transit and voice termination), but since I didn't open a ticket at the time (we thought it was an internal issue), Level3 says they can't research the issue.
>
> Any help?
>
> -Sean
>
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of Kevin Broderick
> Sent: Wednesday, May 19, 2010 10:24 AM
> To: outages at outages.org
> Subject: Re: [outages] Denver Level3 problems?
>
> L3 NOC called us back with reason for outage yesterday: PEBKAC.
>
> - K
>
> Bradley Bopp wrote:
>> An internal source has informed me there was an issue with a Data Center in
>> the Denver area in which something ?blew up? and this caused a chain effect
>> with anything going through Denver to any other pop in the USA.
>>
>> The problem since has been resolved and was for a very short period.
>>
>> Brad
>>
>>
>>
>> On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson at aninetworks.net> wrote:
>>
>>
>>> We also saw issues with customer end points that took level 3 around the same
>>> time but in NYC ? LA and DFW POPS.
>>>
>>> Joseph
>>>
>>>
>>>
>>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>>> Behalf Of Sean Clark
>>> Sent: Tuesday, May 18, 2010 11:53 AM
>>> To: Jayson Baker; randal k
>>> Cc: outages at outages.org
>>> Subject: Re: [outages] Denver Level3 problems?
>>>
>>> We had many customer complaints and all our IP SLA tests spiked between 1110
>>> and 1135 est for Level 3 in our Denver market. But we also saw the problems
>>> in DAL on our level 3 peer (SLA test as well). Some event happened in the
>>> Denver area on level 3, we have opened a ticket with them and they reported to
>>> us they have had a lot of customer call in for the same thing. They have not
>>> given us an RFO yet?
>>>
>>>
>>>
>>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>>> Behalf Of Jayson Baker
>>> Sent: Tuesday, May 18, 2010 12:31
>>> To: randal k
>>> Cc: outages at outages.org
>>> Subject: Re: [outages] Denver Level3 problems?
>>>
>>> We've been having issues staying connected to Google Chat for the last hour or
>>> so.
>>>
>>> And it just so happens we're connected through you. :-)
>>>
>>>
>>> Best Wishes,
>>> Jayson A. Baker
>>> Peak Internet
>>>
>>> On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com> wrote:
>>> Did anybody else see Denver Level3 service take a hit this morning? Our logs
>>> indicated nothing, but we saw various blackholing all over their network for
>>> ~20 minutes, starting around 9:15am MST.
>>>
>>> Everybody have a great day -
>>> Randal
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
>> Bradley D. Bopp
>> Director Of Engineering
>> http://www.nationalnet.com
>> http://as22384.peeringdb.net
>>
>> NationalNet is committed to the highest level of Customer Service available
>> in the Web Hosting business. If you have any questions, comments or
>> concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free,
>> 888-4-NATNET).
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
MTR between our network (level 3 customer) and another network (level 3
customer) was showing that ae-1-100.ebr2.Denver1.Level3.net was running
50%-90% packet loss for the past 6+ hours. Well, that doesn't really
tell the whole story since routers routinely drop ICMP as a matter of
due course during periods of high load. But, point to point pings to
far end destinations that would normally be 40-60ms were varying
regularly (every 60 seconds) to 240-260ms. And, of course the voice
calls on that path sounded like crap.
What it looks like is a router that is experiencing consistently high
CPU load that is chewing on some process every 60 seconds. That
explains the MTR packet loss numbers that are probably innocuous. But,
while the processor is chewing on that process, the throughput latency
goes through the roof. This is a very common thing that we have seen
before . . . just not in the middle of the Level3 network.
The Level3 tech said that since he could not replicate the problem, he
would leave the ticket open until tomorrow.
Bryan
Sean Crandall wrote:
> Sorry to dredge up an old issue, but did anyone get anything better than "PEBKAC".
>
> We were having some issues with voice that was going to Level3 last Tuesday and trying to figure out if what you all saw was related or not (I use Level3 for transit and voice termination), but since I didn't open a ticket at the time (we thought it was an internal issue), Level3 says they can't research the issue.
>
> Any help?
>
> -Sean
>
>
> -----Original Message-----
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of Kevin Broderick
> Sent: Wednesday, May 19, 2010 10:24 AM
> To: outages at outages.org
> Subject: Re: [outages] Denver Level3 problems?
>
> L3 NOC called us back with reason for outage yesterday: PEBKAC.
>
> - K
>
> Bradley Bopp wrote:
>> An internal source has informed me there was an issue with a Data Center in
>> the Denver area in which something ?blew up? and this caused a chain effect
>> with anything going through Denver to any other pop in the USA.
>>
>> The problem since has been resolved and was for a very short period.
>>
>> Brad
>>
>>
>>
>> On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson at aninetworks.net> wrote:
>>
>>
>>> We also saw issues with customer end points that took level 3 around the same
>>> time but in NYC ? LA and DFW POPS.
>>>
>>> Joseph
>>>
>>>
>>>
>>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>>> Behalf Of Sean Clark
>>> Sent: Tuesday, May 18, 2010 11:53 AM
>>> To: Jayson Baker; randal k
>>> Cc: outages at outages.org
>>> Subject: Re: [outages] Denver Level3 problems?
>>>
>>> We had many customer complaints and all our IP SLA tests spiked between 1110
>>> and 1135 est for Level 3 in our Denver market. But we also saw the problems
>>> in DAL on our level 3 peer (SLA test as well). Some event happened in the
>>> Denver area on level 3, we have opened a ticket with them and they reported to
>>> us they have had a lot of customer call in for the same thing. They have not
>>> given us an RFO yet?
>>>
>>>
>>>
>>> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
>>> Behalf Of Jayson Baker
>>> Sent: Tuesday, May 18, 2010 12:31
>>> To: randal k
>>> Cc: outages at outages.org
>>> Subject: Re: [outages] Denver Level3 problems?
>>>
>>> We've been having issues staying connected to Google Chat for the last hour or
>>> so.
>>>
>>> And it just so happens we're connected through you. :-)
>>>
>>>
>>> Best Wishes,
>>> Jayson A. Baker
>>> Peak Internet
>>>
>>> On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek at gmail.com> wrote:
>>> Did anybody else see Denver Level3 service take a hit this morning? Our logs
>>> indicated nothing, but we saw various blackholing all over their network for
>>> ~20 minutes, starting around 9:15am MST.
>>>
>>> Everybody have a great day -
>>> Randal
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
>> Bradley D. Bopp
>> Director Of Engineering
>> http://www.nationalnet.com
>> http://as22384.peeringdb.net
>>
>> NationalNet is committed to the highest level of Customer Service available
>> in the Web Hosting business. If you have any questions, comments or
>> concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free,
>> 888-4-NATNET).
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages