Mailing List Archive

I was hoping somebody was going to climb up to 30,000 feet on this, because it sort of felt to me like it was pervasive as well. I encourage further analysis on this point, and apologize to anybody who would prefer that it be on the discussion list - I think it actually belongs here.
</mod>

On April 21, 2020 11:58:30 AM EDT, Cary Wiedemann via Outages <outages@outages.org> wrote:
>Hey All,
>
>There's a few threads on this already, but they all call out specific
>ISPs
>when this trouble isn't limited to any one last-mile provider. Nearly
>all
>tier 1 transit providers are experiencing loss on the US West Coast
>today.
>Some examples:
>
>Comcast/Level3 peering point in Seattle:
>[image: image.png]
>
>Telia/AT&T in Seattle:
>[image: image.png]
>
>Hurricane Electric/Cox in San Jose:
>[image: image.png]
>
>And this isn't the half of it. Very few tier 1 transit providers are
>unaffected.
>
>So much for carrier redundancy!
>
>- Cary

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

FWIW, I’m not seeing the same sort of loss and extra latency to DC on HE out of Seattle, that I am on CTL out of Spokane.

--Eric

From: Outages <outages-bounces@outages.org> On Behalf Of Cary Wiedemann via Outages
Sent: Tuesday, April 21, 2020 8:59 AM
To: outages <outages@outages.org>
Subject: ADV: [outages] US West Coast Massive Tier 1 Backbone Issues

Hey All,

There's a few threads on this already, but they all call out specific ISPs when this trouble isn't limited to any one last-mile provider. Nearly all tier 1 transit providers are experiencing loss on the US West Coast today. Some examples:

Comcast/Level3 peering point in Seattle:
[cid:image001.png@01D617BE.F3CBE7A0]

Telia/AT&T in Seattle:
[cid:image002.png@01D617BE.F3CBE7A0]

Hurricane Electric/Cox in San Jose:
[cid:image003.png@01D617BE.F3CBE7A0]

And this isn't the half of it. Very few tier 1 transit providers are unaffected.

So much for carrier redundancy!

- Cary

Try the 60,000 foot view; the midwest is having a bad day as well, though
it sounds like a fiber cut is to blame.

https://www.reddit.com/r/sysadmin/comments/g5gyms/midwest_internet_outage/



On Tue, Apr 21, 2020 at 12:38 PM Jay Ashworth via Outages <
outages@outages.org> wrote:

> I was hoping somebody was going to climb up to 30,000 feet on this,
> because it sort of felt to me like it was pervasive as well. I encourage
> further analysis on this point, and apologize to anybody who would prefer
> that it be on the discussion list - I think it actually belongs here.
> </mod>
>
> On April 21, 2020 11:58:30 AM EDT, Cary Wiedemann via Outages <
> outages@outages.org> wrote:
>>
>> Hey All,
>>
>> There's a few threads on this already, but they all call out specific
>> ISPs when this trouble isn't limited to any one last-mile provider. Nearly
>> all tier 1 transit providers are experiencing loss on the US West Coast
>> today. Some examples:
>>
>> Comcast/Level3 peering point in Seattle:
>> [image: image.png]
>>
>> Telia/AT&T in Seattle:
>> [image: image.png]
>>
>> Hurricane Electric/Cox in San Jose:
>> [image: image.png]
>>
>> And this isn't the half of it. Very few tier 1 transit providers are
>> unaffected.
>>
>> So much for carrier redundancy!
>>
>> - Cary
>>
>>
>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Yeah, the CenturyLink ticket is running in another thread on the list, and the fiber cut comments hit the list about 4 minutes before your message.

The internet as a whole has had substantially less Reserve capacity, as the power generation people like to put it, for about the last month, and while I hear tell that excess capacity is being turned up to release some of the pressure, I'm not in a position to know how fast that's happening.

On April 21, 2020 12:44:48 PM EDT, Brett Dikeman <brett.dikeman@gmail.com> wrote:
>Try the 60,000 foot view; the midwest is having a bad day as well,
>though
>it sounds like a fiber cut is to blame.
>
>https://www.reddit.com/r/sysadmin/comments/g5gyms/midwest_internet_outage/
>
>
>
>On Tue, Apr 21, 2020 at 12:38 PM Jay Ashworth via Outages <
>outages@outages.org> wrote:
>
>> I was hoping somebody was going to climb up to 30,000 feet on this,
>> because it sort of felt to me like it was pervasive as well. I
>encourage
>> further analysis on this point, and apologize to anybody who would
>prefer
>> that it be on the discussion list - I think it actually belongs here.
>> </mod>
>>
>> On April 21, 2020 11:58:30 AM EDT, Cary Wiedemann via Outages <
>> outages@outages.org> wrote:
>>>
>>> Hey All,
>>>
>>> There's a few threads on this already, but they all call out
>specific
>>> ISPs when this trouble isn't limited to any one last-mile provider.
>Nearly
>>> all tier 1 transit providers are experiencing loss on the US West
>Coast
>>> today. Some examples:
>>>
>>> Comcast/Level3 peering point in Seattle:
>>> [image: image.png]
>>>
>>> Telia/AT&T in Seattle:
>>> [image: image.png]
>>>
>>> Hurricane Electric/Cox in San Jose:
>>> [image: image.png]
>>>
>>> And this isn't the half of it. Very few tier 1 transit providers
>are
>>> unaffected.
>>>
>>> So much for carrier redundancy!
>>>
>>> - Cary
>>>
>>>
>>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Great read on carrier diversity:

https://transition.fcc.gov/bureaus/pshs/docs/clearinghouse/ATIS_NDAI_Final_Report_2006.pdf

From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jay Ashworth via Outages
Sent: Tuesday, April 21, 2020 9:50 AM
To: Brett Dikeman <brett.dikeman@gmail.com>
Cc: Cary Wiedemann via Outages <outages@outages.org>
Subject: Re: [outages] US West Coast Massive Tier 1 Backbone Issues

Yeah, the CenturyLink ticket is running in another thread on the list, and the fiber cut comments hit the list about 4 minutes before your message.

The internet as a whole has had substantially less Reserve capacity, as the power generation people like to put it, for about the last month, and while I hear tell that excess capacity is being turned up to release some of the pressure, I'm not in a position to know how fast that's happening.
On April 21, 2020 12:44:48 PM EDT, Brett Dikeman <brett.dikeman@gmail.com<mailto:brett.dikeman@gmail.com>> wrote:
Try the 60,000 foot view; the midwest is having a bad day as well, though it sounds like a fiber cut is to blame.

https://www.reddit.com/r/sysadmin/comments/g5gyms/midwest_internet_outage/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.reddit.com_r_sysadmin_comments_g5gyms_midwest-5Finternet-5Foutage_&d=DwMFaQ&c=N13-TaG7c-EYAiUNohBk74oLRjUiBTwVm-KSnr4bPSc&r=-GzOCp0ppLaBQPFaZ7lZ4bUUBQxpFBukitRP75oaRdQ&m=icB-gA7RcG0Wjt9BVrR_NoAKFCXQZQ8DUg68_4AG1Z8&s=ulznjaY0roT-mCOZ_0PosVaKatrqWHLvAef5-2sAaYI&e=>


----------------------------------------------------------------------
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.