Mailing List Archive: Major Level3 (CenturyLink) Issues

Major Level3 (CenturyLink) Issues

Aug 30, 2020, 3:56 AM

Post #1 of 5 (689 views)

Starting around 6:15 Eastern or so noticed that BGP was still up with them, but no traffic between a home fios address in NJ and our L3 Transit. We have since turned down transit and regained connectivity.

Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and Cologix, but now it’s not reachable anywhere I have access.

Lots of chatter on Twitter from geographically diverse accounts...

Additionally, Cloudflare is reporting a problem, but it’s not clear if that’s related to CenturyLink customers reporting issues or not:

+++++
Update - Cloudflare is investigating an increased level of HTTP 5xx class errors.

We are working to understand the full impact and mitigate this problem. More updates to follow shortly.

Aug 30, 10:45 UTC

Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available.

Aug 30, 10:21 UTC
+++++

Charles
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: Major Level3 (CenturyLink) Issues [ In reply to ]

outages at outages

Aug 30, 2020, 4:29 AM

Post #2 of 5 (686 views)

Permalink

Confirm for Europe, too.

On 30 Aug 2020, at 12:56, Charles Sprickman via Outages wrote:

> Starting around 6:15 Eastern or so noticed that BGP was still up with
> them, but no traffic between a home fios address in NJ and our L3
> Transit. We have since turned down transit and regained connectivity.
>
> Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and
> Cologix, but now it’s not reachable anywhere I have access.
>
> Lots of chatter on Twitter from geographically diverse accounts...
>
> Additionally, Cloudflare is reporting a problem, but it’s not clear
> if that’s related to CenturyLink customers reporting issues or not:
>
> +++++
> Update - Cloudflare is investigating an increased level of HTTP 5xx
> class errors.
>
> We are working to understand the full impact and mitigate this
> problem. More updates to follow shortly.
>
> Aug 30, 10:45 UTC
>
> Investigating - Cloudflare is aware of, and investigating an issue
> which potentially impacts multiple customers. Further detail will be
> provided as more information becomes available.
>
> Aug 30, 10:21 UTC
> +++++
>
> Charles
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: Major Level3 (CenturyLink) Issues [ In reply to ]

outages at outages

Aug 30, 2020, 4:47 AM

Post #3 of 5 (686 views)

Permalink

I can confirm issues seen from US and EU continues for our ip traffic
destined to US market, some traffic is dies on their Miami router even
though we're no longer announcing on any of their routers our ip space, it
appears as if the routes are stuck on their routers in certain locations.

On Sun, Aug 30, 2020 at 7:43 AM Jörg Kost via Outages <outages@outages.org>
wrote:

> Confirm for Europe, too.
>
> On 30 Aug 2020, at 12:56, Charles Sprickman via Outages wrote:
>
> > Starting around 6:15 Eastern or so noticed that BGP was still up with
> > them, but no traffic between a home fios address in NJ and our L3
> > Transit. We have since turned down transit and regained connectivity.
> >
> > Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and
> > Cologix, but now it’s not reachable anywhere I have access.
> >
> > Lots of chatter on Twitter from geographically diverse accounts...
> >
> > Additionally, Cloudflare is reporting a problem, but it’s not clear
> > if that’s related to CenturyLink customers reporting issues or not:
> >
> > +++++
> > Update - Cloudflare is investigating an increased level of HTTP 5xx
> > class errors.
> >
> > We are working to understand the full impact and mitigate this
> > problem. More updates to follow shortly.
> >
> > Aug 30, 10:45 UTC
> >
> > Investigating - Cloudflare is aware of, and investigating an issue
> > which potentially impacts multiple customers. Further detail will be
> > provided as more information becomes available.
> >
> > Aug 30, 10:21 UTC
> > +++++
> >
> > Charles
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Re: Major Level3 (CenturyLink) Issues [ In reply to ]

outages at outages

Aug 30, 2020, 5:03 AM

Post #4 of 5 (686 views)

Permalink

Got a trace?

On Sun, Aug 30, 2020 at 8:01 AM Dejan Protich via Outages <
outages@outages.org> wrote:

> I can confirm issues seen from US and EU continues for our ip traffic
> destined to US market, some traffic is dies on their Miami router even
> though we're no longer announcing on any of their routers our ip space, it
> appears as if the routes are stuck on their routers in certain locations.
>
> On Sun, Aug 30, 2020 at 7:43 AM Jörg Kost via Outages <outages@outages.org>
> wrote:
>
>> Confirm for Europe, too.
>>
>> On 30 Aug 2020, at 12:56, Charles Sprickman via Outages wrote:
>>
>> > Starting around 6:15 Eastern or so noticed that BGP was still up with
>> > them, but no traffic between a home fios address in NJ and our L3
>> > Transit. We have since turned down transit and regained connectivity.
>> >
>> > Initially I was able to reach some IPs (eg: 4.2.2.1) from Choopa and
>> > Cologix, but now it’s not reachable anywhere I have access.
>> >
>> > Lots of chatter on Twitter from geographically diverse accounts...
>> >
>> > Additionally, Cloudflare is reporting a problem, but it’s not clear
>> > if that’s related to CenturyLink customers reporting issues or not:
>> >
>> > +++++
>> > Update - Cloudflare is investigating an increased level of HTTP 5xx
>> > class errors.
>> >
>> > We are working to understand the full impact and mitigate this
>> > problem. More updates to follow shortly.
>> >
>> > Aug 30, 10:45 UTC
>> >
>> > Investigating - Cloudflare is aware of, and investigating an issue
>> > which potentially impacts multiple customers. Further detail will be
>> > provided as more information becomes available.
>> >
>> > Aug 30, 10:21 UTC
>> > +++++
>> >
>> > Charles
>> > _______________________________________________
>> > Outages mailing list
>> > Outages@outages.org
>> > https://puck.nether.net/mailman/listinfo/outages
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.kuehl@gmail.com

Re: Major Level3 (CenturyLink) Issues [ In reply to ]

outages at outages

Sep 1, 2020, 11:17 PM

Post #5 of 5 (680 views)

Permalink

the RFO is making the rounds
http://seele.lamehost.it/~marco/blind/Network_Event_Formal_RFO_Multiple_Markets_19543671_19544042_30_August.pdf

it kinda explains the flowspec issue but completely ignores the stuck
routes, which imiho was the more damaging problem.

randy
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages