Mailing List Archive

For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



-------- Original message --------
From: Jay Ashworth via Outages <outages@outages.org>
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages@outages.org
Subject: [outages] 3556 problems

Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Not on that ticket, so thanks a ton for the update! Execs are asking
questions about what's going on and it's nice to give them feedback!

On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <
outages@outages.org> wrote:

> For those who aren't on the master CL ticket, here is the latest from CL.
> Ticket number: 19545403
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
> Protocol (BGP) issues causing service impacts across multiple markets.
> Cooperative escalated investigation efforts with CenturyLink leadership are
> underway to isolate and troubleshoot the issue and expedite service
> restoral.
>
>
>
> -------- Original message --------
> From: Jay Ashworth via Outages <outages@outages.org>
> Date: 8/30/20 9:36 AM (GMT-06:00)
> To: outages@outages.org
> Subject: [outages] 3556 problems
>
> Okay, it's pretty clear that they're having trouble. :-) Let's limit it
> now to actual tickets and reports of clearance.
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Here is their latest update:



The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



Next update by: 2020-08-30 15:50 GMT



This notification is sent from an unmonitored email account. Please click the link below to reply:



mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A



Notes History:



08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.



08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.



08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.

The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.


From: Michael McDonald <mmcdonald@liquidweb.com>
Sent: Sunday, August 30, 2020 9:56 AM
To: Dylan Ebner <dylan.ebner@crlmed.com>
Cc: Jay Ashworth <jra@baylink.com>; outages@outages.org
Subject: Re: [outages] 3556 problems

Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!

On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



-------- Original message --------
From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>>
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] 3556 problems

Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages

Not sure if CenturyLink is facing Bogon routes issue, we are impacted intermittently on AS174 however we are accepting default route from this peer

https://bgp.he.net/AS3356#_graph4

https://bgp.he.net/report/bogons#_bogonsv4asn



Sent from my iPhone
#Covid-19 (Be Safe | Maintain Social Distancing)

On 30-Aug-2020, at 8:33 PM, Dylan Ebner via Outages <outages@outages.org> wrote:

?
Here is their latest update:



The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



Next update by: 2020-08-30 15:50 GMT



This notification is sent from an unmonitored email account. Please click the link below to reply:



mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A



Notes History:



08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.



08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.



08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.

The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.


From: Michael McDonald <mmcdonald@liquidweb.com>
Sent: Sunday, August 30, 2020 9:56 AM
To: Dylan Ebner <dylan.ebner@crlmed.com>
Cc: Jay Ashworth <jra@baylink.com>; outages@outages.org
Subject: Re: [outages] 3556 problems

Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!

On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



-------- Original message --------
From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>>
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] 3556 problems

Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages<https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages


This message is subject to Fortitude Law Associates's email communication policy.
FLA-Disclaimer: https://protect-us.mimecast.com/s/hbb3COYERvH2nNq7hE2Z6w

It's recovering right now (for me , Munich).

On 30 Aug 2020, at 16:59, Dylan Ebner via Outages wrote:

> Here is their latest update:
>
>
>
> The IP NOC with the assistance of the Operations Engineering team
> confirmed a routing issue to be preventing BGP sessions from
> establishing correctly. A configuration adjustment was deployed at a
> high level, and sessions began to re-establish with stability. As the
> change propagates through the affected devices, service affecting
> alarms continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP
> session. If after that action has been performed a service issue
> prevails, please contact the CenturyLink Repair Center for
> troubleshooting assistance.
>
>
>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Sent from my iPhone

On Aug 30, 2020, at 10:21 AM, Jörg Kost via Outages <outages@outages.org> wrote:

?It's recovering right now (for me , Munich).

> On 30 Aug 2020, at 16:59, Dylan Ebner via Outages wrote:
>
> Here is their latest update:
>
>
>
> The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.
>
>
>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Latest update from CenturyLink


Notes History:



08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided.



08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.



08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.



08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.



The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.



Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn@staff.atlantic.net<mailto:sflynn@staff.atlantic.net>
www.atlantic.net<http://www.atlantic.net/>

From: Outages <outages-bounces@outages.org> On Behalf Of Dylan Ebner via Outages
Sent: Sunday, August 30, 2020 11:00 AM
To: Michael McDonald <mmcdonald@liquidweb.com>
Cc: outages@outages.org
Subject: Re: [outages] 3556 problems


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Here is their latest update:



The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



Next update by: 2020-08-30 15:50 GMT



This notification is sent from an unmonitored email account. Please click the link below to reply:



mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A



Notes History:



08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.



08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.



08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.



The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.


From: Michael McDonald <mmcdonald@liquidweb.com<mailto:mmcdonald@liquidweb.com>>
Sent: Sunday, August 30, 2020 9:56 AM
To: Dylan Ebner <dylan.ebner@crlmed.com<mailto:dylan.ebner@crlmed.com>>
Cc: Jay Ashworth <jra@baylink.com<mailto:jra@baylink.com>>; outages@outages.org<mailto:outages@outages.org>
Subject: Re: [outages] 3556 problems

Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!

On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



-------- Original message --------
From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>>
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] 3556 problems

Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages

According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back?



From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Stephen Flynn via Outages
Sent: Sunday, August 30, 2020 8:41 AM
To: outages@outages.org
Subject: Re: [outages] 3556 problems



Latest update from CenturyLink



Notes History:



08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided.



08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.



08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.



08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.



The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.







Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn@staff.atlantic.net <mailto:sflynn@staff.atlantic.net>
www.atlantic.net <http://www.atlantic.net/>



From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On Behalf Of Dylan Ebner via Outages
Sent: Sunday, August 30, 2020 11:00 AM
To: Michael McDonald <mmcdonald@liquidweb.com <mailto:mmcdonald@liquidweb.com> >
Cc: outages@outages.org <mailto:outages@outages.org>
Subject: Re: [outages] 3556 problems






CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.



Here is their latest update:





The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



Next update by: 2020-08-30 15:50 GMT



This notification is sent from an unmonitored email account. Please click the link below to reply:



mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403, <mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A> &body=Customer%20Reply%3A%0D%0A



Notes History:



08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.



08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.



08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.



The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.





From: Michael McDonald <mmcdonald@liquidweb.com <mailto:mmcdonald@liquidweb.com> >
Sent: Sunday, August 30, 2020 9:56 AM
To: Dylan Ebner <dylan.ebner@crlmed.com <mailto:dylan.ebner@crlmed.com> >
Cc: Jay Ashworth <jra@baylink.com <mailto:jra@baylink.com> >; outages@outages.org <mailto:outages@outages.org>
Subject: Re: [outages] 3556 problems



Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!



On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org <mailto:outages@outages.org> > wrote:

For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403






08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.







-------- Original message --------

From: Jay Ashworth via Outages <outages@outages.org <mailto:outages@outages.org> >

Date: 8/30/20 9:36 AM (GMT-06:00)

To: outages@outages.org <mailto:outages@outages.org>

Subject: [outages] 3556 problems



Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages

We were effected by this in reaching the Google SAS. It's been working
without problems for 100 minutes.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages@outages.org>
wrote:

> According to Centurylink tweet, their service has been restored. Can
> someone help me confirm that this is true before I start shifting the
> traffic back?
>
>
>
> *From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Stephen
> Flynn via Outages
> *Sent:* Sunday, August 30, 2020 8:41 AM
> *To:* outages@outages.org
> *Subject:* Re: [outages] 3556 problems
>
>
>
> Latest update from CenturyLink
>
>
>
> Notes History:
>
>
>
> 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service
> affecting alarms clearing as the configuration that was deployed continues
> to propagate through the affected devices. Once affected services have been
> confirmed restored and stable, a final notification will be provided.
>
>
>
> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations
> Engineering team confirmed a routing issue to be preventing BGP sessions
> from establishing correctly. A configuration adjustment was deployed at a
> high level, and sessions began to re-establish with stability. As the
> change propagates through the affected devices, service affecting alarms
> continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP session. If
> after that action has been performed a service issue prevails, please
> contact the CenturyLink Repair Center for troubleshooting assistance.
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
> Protocol (BGP) issues causing service impacts across multiple markets.
> Cooperative escalated investigation efforts with CenturyLink leadership
> are underway to isolate and troubleshoot the issue and expedite service
> restoral.
>
>
>
> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
> investigation and troubleshooting efforts remain ongoing this time.
>
>
>
> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated
> investigations to isolate and troubleshoot the fault at this time.
>
>
>
> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
> identified a market wide. As this network fault is impacting multiple
> clients, the event has increased visibility with CenturyLink leadership. As
> such, client trouble tickets associated to this fault have been
> automatically escalated to higher priority.
>
>
>
> The NOC is engaged and investigating in order to isolate the cause. Please
> be advised that updates for this event will be relayed at a minimum of
> hourly unless otherwise noted. The information conveyed hereafter is
> associated to live troubleshooting effort and as the discovery process
> evolves through to service resolution, ticket closure, or post incident
> review, details may evolve.
>
>
>
>
>
>
>
> Regards,
>
> Stephen Flynn
> Atlantic.Net
> Direct: (321) 206-1390
> sflynn@staff.atlantic.net
> www.atlantic.net
>
>
>
> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *Dylan Ebner
> via Outages
> *Sent:* Sunday, August 30, 2020 11:00 AM
> *To:* Michael McDonald <mmcdonald@liquidweb.com>
> *Cc:* outages@outages.org
> *Subject:* Re: [outages] 3556 problems
>
>
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Here is their latest update:
>
>
>
>
>
> The IP NOC with the assistance of the Operations Engineering team
> confirmed a routing issue to be preventing BGP sessions from establishing
> correctly. A configuration adjustment was deployed at a high level, and
> sessions began to re-establish with stability. As the change propagates
> through the affected devices, service affecting alarms continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP session. If
> after that action has been performed a service issue prevails, please
> contact the CenturyLink Repair Center for troubleshooting assistance.
>
>
>
> Next update by: 2020-08-30 15:50 GMT
>
>
>
> This notification is sent from an unmonitored email account. Please click
> the link below to reply:
>
>
>
>
> mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A
> <SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A>
>
>
>
> Notes History:
>
>
>
> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations
> Engineering team confirmed a routing issue to be preventing BGP sessions
> from establishing correctly. A configuration adjustment was deployed at a
> high level, and sessions began to re-establish with stability. As the
> change propagates through the affected devices, service affecting alarms
> continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP session. If
> after that action has been performed a service issue prevails, please
> contact the CenturyLink Repair Center for troubleshooting assistance.
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
> Protocol (BGP) issues causing service impacts across multiple markets.
> Cooperative escalated investigation efforts with CenturyLink leadership
> are underway to isolate and troubleshoot the issue and expedite service
> restoral.
>
>
>
> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
> investigation and troubleshooting efforts remain ongoing this time.
>
>
>
> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated
> investigations to isolate and troubleshoot the fault at this time.
>
>
>
> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
> identified a market wide. As this network fault is impacting multiple
> clients, the event has increased visibility with CenturyLink leadership. As
> such, client trouble tickets associated to this fault have been
> automatically escalated to higher priority.
>
>
>
> The NOC is engaged and investigating in order to isolate the cause. Please
> be advised that updates for this event will be relayed at a minimum of
> hourly unless otherwise noted. The information conveyed hereafter is
> associated to live troubleshooting effort and as the discovery process
> evolves through to service resolution, ticket closure, or post incident
> review, details may evolve.
>
>
>
>
>
> *From:* Michael McDonald <mmcdonald@liquidweb.com>
> *Sent:* Sunday, August 30, 2020 9:56 AM
> *To:* Dylan Ebner <dylan.ebner@crlmed.com>
> *Cc:* Jay Ashworth <jra@baylink.com>; outages@outages.org
> *Subject:* Re: [outages] 3556 problems
>
>
>
> Not on that ticket, so thanks a ton for the update! Execs are asking
> questions about what's going on and it's nice to give them feedback!
>
>
>
> On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <
> outages@outages.org> wrote:
>
> For those who aren't on the master CL ticket, here is the latest from CL.
> Ticket number: 19545403
>
>
>
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
> Protocol (BGP) issues causing service impacts across multiple markets.
> Cooperative escalated investigation efforts with CenturyLink leadership are
> underway to isolate and troubleshoot the issue and expedite service
> restoral.
>
>
>
>
>
>
>
> -------- Original message --------
>
> From: Jay Ashworth via Outages <outages@outages.org>
>
> Date: 8/30/20 9:36 AM (GMT-06:00)
>
> To: outages@outages.org
>
> Subject: [outages] 3556 problems
>
>
>
> Okay, it's pretty clear that they're having trouble. :-) Let's limit it
> now to actual tickets and reports of clearance.
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

I just shifted back and everything looks normal. I'm in Waltham MA.

On Sun, Aug 30, 2020 at 12:44 PM Josh Luthman via Outages <
outages@outages.org> wrote:

> We were effected by this in reaching the Google SAS. It's been working
> without problems for 100 minutes.
>
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>
> On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages@outages.org>
> wrote:
>
>> According to Centurylink tweet, their service has been restored. Can
>> someone help me confirm that this is true before I start shifting the
>> traffic back?
>>
>>
>>
>> *From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Stephen
>> Flynn via Outages
>> *Sent:* Sunday, August 30, 2020 8:41 AM
>> *To:* outages@outages.org
>> *Subject:* Re: [outages] 3556 problems
>>
>>
>>
>> Latest update from CenturyLink
>>
>>
>>
>> Notes History:
>>
>>
>>
>> 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service
>> affecting alarms clearing as the configuration that was deployed continues
>> to propagate through the affected devices. Once affected services have been
>> confirmed restored and stable, a final notification will be provided.
>>
>>
>>
>> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the
>> Operations Engineering team confirmed a routing issue to be preventing BGP
>> sessions from establishing correctly. A configuration adjustment was
>> deployed at a high level, and sessions began to re-establish with
>> stability. As the change propagates through the affected devices, service
>> affecting alarms continue to clear.
>>
>>
>>
>> Due to the nature of this outage, it may be necessary to reset your
>> services locally at your equipment, or manually reset your BGP session. If
>> after that action has been performed a service issue prevails, please
>> contact the CenturyLink Repair Center for troubleshooting assistance.
>>
>>
>>
>> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
>> Protocol (BGP) issues causing service impacts across multiple markets.
>> Cooperative escalated investigation efforts with CenturyLink leadership
>> are underway to isolate and troubleshoot the issue and expedite service
>> restoral.
>>
>>
>>
>> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
>> investigation and troubleshooting efforts remain ongoing this time.
>>
>>
>>
>> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated
>> investigations to isolate and troubleshoot the fault at this time.
>>
>>
>>
>> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
>> identified a market wide. As this network fault is impacting multiple
>> clients, the event has increased visibility with CenturyLink leadership. As
>> such, client trouble tickets associated to this fault have been
>> automatically escalated to higher priority.
>>
>>
>>
>> The NOC is engaged and investigating in order to isolate the cause.
>> Please be advised that updates for this event will be relayed at a minimum
>> of hourly unless otherwise noted. The information conveyed hereafter is
>> associated to live troubleshooting effort and as the discovery process
>> evolves through to service resolution, ticket closure, or post incident
>> review, details may evolve.
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>> Stephen Flynn
>> Atlantic.Net
>> Direct: (321) 206-1390
>> sflynn@staff.atlantic.net
>> www.atlantic.net
>>
>>
>>
>> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *Dylan Ebner
>> via Outages
>> *Sent:* Sunday, August 30, 2020 11:00 AM
>> *To:* Michael McDonald <mmcdonald@liquidweb.com>
>> *Cc:* outages@outages.org
>> *Subject:* Re: [outages] 3556 problems
>>
>>
>>
>>
>>
>> *CAUTION:* This email originated from outside of the organization. Do
>> not click links or open attachments unless you recognize the sender and
>> know the content is safe.
>>
>>
>>
>> Here is their latest update:
>>
>>
>>
>>
>>
>> The IP NOC with the assistance of the Operations Engineering team
>> confirmed a routing issue to be preventing BGP sessions from establishing
>> correctly. A configuration adjustment was deployed at a high level, and
>> sessions began to re-establish with stability. As the change propagates
>> through the affected devices, service affecting alarms continue to clear.
>>
>>
>>
>> Due to the nature of this outage, it may be necessary to reset your
>> services locally at your equipment, or manually reset your BGP session. If
>> after that action has been performed a service issue prevails, please
>> contact the CenturyLink Repair Center for troubleshooting assistance.
>>
>>
>>
>> Next update by: 2020-08-30 15:50 GMT
>>
>>
>>
>> This notification is sent from an unmonitored email account. Please
>> click the link below to reply:
>>
>>
>>
>>
>> mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A
>> <SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A>
>>
>>
>>
>> Notes History:
>>
>>
>>
>> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the
>> Operations Engineering team confirmed a routing issue to be preventing BGP
>> sessions from establishing correctly. A configuration adjustment was
>> deployed at a high level, and sessions began to re-establish with
>> stability. As the change propagates through the affected devices, service
>> affecting alarms continue to clear.
>>
>>
>>
>> Due to the nature of this outage, it may be necessary to reset your
>> services locally at your equipment, or manually reset your BGP session. If
>> after that action has been performed a service issue prevails, please
>> contact the CenturyLink Repair Center for troubleshooting assistance.
>>
>>
>>
>> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
>> Protocol (BGP) issues causing service impacts across multiple markets.
>> Cooperative escalated investigation efforts with CenturyLink leadership
>> are underway to isolate and troubleshoot the issue and expedite service
>> restoral.
>>
>>
>>
>> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
>> investigation and troubleshooting efforts remain ongoing this time.
>>
>>
>>
>> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated
>> investigations to isolate and troubleshoot the fault at this time.
>>
>>
>>
>> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
>> identified a market wide. As this network fault is impacting multiple
>> clients, the event has increased visibility with CenturyLink leadership. As
>> such, client trouble tickets associated to this fault have been
>> automatically escalated to higher priority.
>>
>>
>>
>> The NOC is engaged and investigating in order to isolate the cause.
>> Please be advised that updates for this event will be relayed at a minimum
>> of hourly unless otherwise noted. The information conveyed hereafter is
>> associated to live troubleshooting effort and as the discovery process
>> evolves through to service resolution, ticket closure, or post incident
>> review, details may evolve.
>>
>>
>>
>>
>>
>> *From:* Michael McDonald <mmcdonald@liquidweb.com>
>> *Sent:* Sunday, August 30, 2020 9:56 AM
>> *To:* Dylan Ebner <dylan.ebner@crlmed.com>
>> *Cc:* Jay Ashworth <jra@baylink.com>; outages@outages.org
>> *Subject:* Re: [outages] 3556 problems
>>
>>
>>
>> Not on that ticket, so thanks a ton for the update! Execs are asking
>> questions about what's going on and it's nice to give them feedback!
>>
>>
>>
>> On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <
>> outages@outages.org> wrote:
>>
>> For those who aren't on the master CL ticket, here is the latest from CL.
>> Ticket number: 19545403
>>
>>
>>
>>
>>
>>
>> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
>> Protocol (BGP) issues causing service impacts across multiple markets.
>> Cooperative escalated investigation efforts with CenturyLink leadership are
>> underway to isolate and troubleshoot the issue and expedite service
>> restoral.
>>
>>
>>
>>
>>
>>
>>
>> -------- Original message --------
>>
>> From: Jay Ashworth via Outages <outages@outages.org>
>>
>> Date: 8/30/20 9:36 AM (GMT-06:00)
>>
>> To: outages@outages.org
>>
>> Subject: [outages] 3556 problems
>>
>>
>>
>> Okay, it's pretty clear that they're having trouble. :-) Let's limit it
>> now to actual tickets and reports of clearance.
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.kuehl@gmail.com

“Can someone help me confirm that this is true before I start shifting the traffic back?”

I won’t be activating our links until this evenings maintenance window.


Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn@staff.atlantic.net<mailto:sflynn@staff.atlantic.net>
www.atlantic.net<http://www.atlantic.net/>

From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages
Sent: Sunday, August 30, 2020 12:51 PM
To: Josh Luthman <josh@imaginenetworksllc.com>
Cc: OUTAGES Mailing List <outages@outages.org>
Subject: Re: [outages] 3556 problems


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I just shifted back and everything looks normal. I'm in Waltham MA.

On Sun, Aug 30, 2020 at 12:44 PM Josh Luthman via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
We were effected by this in reaching the Google SAS. It's been working without problems for 100 minutes.
Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back?

From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Stephen Flynn via Outages
Sent: Sunday, August 30, 2020 8:41 AM
To: outages@outages.org<mailto:outages@outages.org>
Subject: Re: [outages] 3556 problems

Latest update from CenturyLink


Notes History:



08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided.



08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.



08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.



08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.



The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.



Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn@staff.atlantic.net<mailto:sflynn@staff.atlantic.net>
www.atlantic.net<http://www.atlantic.net/>

From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Dylan Ebner via Outages
Sent: Sunday, August 30, 2020 11:00 AM
To: Michael McDonald <mmcdonald@liquidweb.com<mailto:mmcdonald@liquidweb.com>>
Cc: outages@outages.org<mailto:outages@outages.org>
Subject: Re: [outages] 3556 problems


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Here is their latest update:



The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



Next update by: 2020-08-30 15:50 GMT



This notification is sent from an unmonitored email account. Please click the link below to reply:



mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A



Notes History:



08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.



Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.



08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.



08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.



The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.


From: Michael McDonald <mmcdonald@liquidweb.com<mailto:mmcdonald@liquidweb.com>>
Sent: Sunday, August 30, 2020 9:56 AM
To: Dylan Ebner <dylan.ebner@crlmed.com<mailto:dylan.ebner@crlmed.com>>
Cc: Jay Ashworth <jra@baylink.com<mailto:jra@baylink.com>>; outages@outages.org<mailto:outages@outages.org>
Subject: Re: [outages] 3556 problems

Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!

On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403



08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.



-------- Original message --------
From: Jay Ashworth via Outages <outages@outages.org<mailto:outages@outages.org>>
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] 3556 problems

Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages


--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com>

Stable,

latest statement (Flowspec):

Summary: On August 30, 2020 10:04 GMT, CenturyLink identified an issue
to be affecting users across multiple markets. The IP Network Operations
Center (NOC) was engaged, and initial research identified that an
offending flowspec announcement prevented Border Gateway Protocol (BGP)
from establishing across multiple elements throughout the CenturyLink
Network. The IP NOC deployed a global configuration change to block the
offending flowspec announcement, which allowed BGP to begin to correctly
establish. As the change propagated through the network, the IP NOC
observed all associated service affecting alarms clearing and services
returning to a stable state.

On 30 Aug 2020, at 18:33, Tim Choe via Outages wrote:

> According to Centurylink tweet, their service has been restored. Can
> someone help me confirm that this is true before I start shifting the
> traffic back?
>
>
>
> From: Outages [mailto:outages-bounces@outages.org] On Behalf Of
> Stephen Flynn via Outages
> Sent: Sunday, August 30, 2020 8:41 AM
> To: outages@outages.org
> Subject: Re: [outages] 3556 problems
>
>
>
> Latest update from CenturyLink
>
>
>
> Notes History:
>
>
>
> 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service
> affecting alarms clearing as the configuration that was deployed
> continues to propagate through the affected devices. Once affected
> services have been confirmed restored and stable, a final notification
> will be provided.
>
>
>
> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the
> Operations Engineering team confirmed a routing issue to be preventing
> BGP sessions from establishing correctly. A configuration adjustment
> was deployed at a high level, and sessions began to re-establish with
> stability. As the change propagates through the affected devices,
> service affecting alarms continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP
> session. If after that action has been performed a service issue
> prevails, please contact the CenturyLink Repair Center for
> troubleshooting assistance.
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border
> Gateway Protocol (BGP) issues causing service impacts across multiple
> markets. Cooperative escalated investigation efforts with CenturyLink
> leadership are underway to isolate and troubleshoot the issue and
> expedite service restoral.
>
>
>
> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
> investigation and troubleshooting efforts remain ongoing this time.
>
>
>
> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative
> escalated investigations to isolate and troubleshoot the fault at this
> time.
>
>
>
> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
> identified a market wide. As this network fault is impacting multiple
> clients, the event has increased visibility with CenturyLink
> leadership. As such, client trouble tickets associated to this fault
> have been automatically escalated to higher priority.
>
>
>
> The NOC is engaged and investigating in order to isolate the cause.
> Please be advised that updates for this event will be relayed at a
> minimum of hourly unless otherwise noted. The information conveyed
> hereafter is associated to live troubleshooting effort and as the
> discovery process evolves through to service resolution, ticket
> closure, or post incident review, details may evolve.
>
>
>
>
>
>
>
> Regards,
>
> Stephen Flynn
> Atlantic.Net
> Direct: (321) 206-1390
> sflynn@staff.atlantic.net <mailto:sflynn@staff.atlantic.net>
> www.atlantic.net <http://www.atlantic.net/>
>
>
>
> From: Outages <outages-bounces@outages.org
> <mailto:outages-bounces@outages.org> > On Behalf Of Dylan Ebner via
> Outages
> Sent: Sunday, August 30, 2020 11:00 AM
> To: Michael McDonald <mmcdonald@liquidweb.com
> <mailto:mmcdonald@liquidweb.com> >
> Cc: outages@outages.org <mailto:outages@outages.org>
> Subject: Re: [outages] 3556 problems
>
>
>
>
>
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender
> and know the content is safe.
>
>
>
> Here is their latest update:
>
>
>
>
>
> The IP NOC with the assistance of the Operations Engineering team
> confirmed a routing issue to be preventing BGP sessions from
> establishing correctly. A configuration adjustment was deployed at a
> high level, and sessions began to re-establish with stability. As the
> change propagates through the affected devices, service affecting
> alarms continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP
> session. If after that action has been performed a service issue
> prevails, please contact the CenturyLink Repair Center for
> troubleshooting assistance.
>
>
>
> Next update by: 2020-08-30 15:50 GMT
>
>
>
> This notification is sent from an unmonitored email account. Please
> click the link below to reply:
>
>
>
> mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,
> <mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A>
> &body=Customer%20Reply%3A%0D%0A
>
>
>
> Notes History:
>
>
>
> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the
> Operations Engineering team confirmed a routing issue to be preventing
> BGP sessions from establishing correctly. A configuration adjustment
> was deployed at a high level, and sessions began to re-establish with
> stability. As the change propagates through the affected devices,
> service affecting alarms continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP
> session. If after that action has been performed a service issue
> prevails, please contact the CenturyLink Repair Center for
> troubleshooting assistance.
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border
> Gateway Protocol (BGP) issues causing service impacts across multiple
> markets. Cooperative escalated investigation efforts with CenturyLink
> leadership are underway to isolate and troubleshoot the issue and
> expedite service restoral.
>
>
>
> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
> investigation and troubleshooting efforts remain ongoing this time.
>
>
>
> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative
> escalated investigations to isolate and troubleshoot the fault at this
> time.
>
>
>
> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
> identified a market wide. As this network fault is impacting multiple
> clients, the event has increased visibility with CenturyLink
> leadership. As such, client trouble tickets associated to this fault
> have been automatically escalated to higher priority.
>
>
>
> The NOC is engaged and investigating in order to isolate the cause.
> Please be advised that updates for this event will be relayed at a
> minimum of hourly unless otherwise noted. The information conveyed
> hereafter is associated to live troubleshooting effort and as the
> discovery process evolves through to service resolution, ticket
> closure, or post incident review, details may evolve.
>
>
>
>
>
> From: Michael McDonald <mmcdonald@liquidweb.com
> <mailto:mmcdonald@liquidweb.com> >
> Sent: Sunday, August 30, 2020 9:56 AM
> To: Dylan Ebner <dylan.ebner@crlmed.com
> <mailto:dylan.ebner@crlmed.com> >
> Cc: Jay Ashworth <jra@baylink.com <mailto:jra@baylink.com> >;
> outages@outages.org <mailto:outages@outages.org>
> Subject: Re: [outages] 3556 problems
>
>
>
> Not on that ticket, so thanks a ton for the update! Execs are asking
> questions about what's going on and it's nice to give them feedback!
>
>
>
> On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages
> <outages@outages.org <mailto:outages@outages.org> > wrote:
>
> For those who aren't on the master CL ticket, here is the latest from
> CL. Ticket number: 19545403
>
>
>
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border
> Gateway Protocol (BGP) issues causing service impacts across multiple
> markets. Cooperative escalated investigation efforts with CenturyLink
> leadership are underway to isolate and troubleshoot the issue and
> expedite service restoral.
>
>
>
>
>
>
>
> -------- Original message --------
>
> From: Jay Ashworth via Outages <outages@outages.org
> <mailto:outages@outages.org> >
>
> Date: 8/30/20 9:36 AM (GMT-06:00)
>
> To: outages@outages.org <mailto:outages@outages.org>
>
> Subject: [outages] 3556 problems
>
>
>
> Okay, it's pretty clear that they're having trouble. :-) Let's limit
> it now to actual tickets and reports of clearance.
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org <mailto:Outages@outages.org>
> https://puck.nether.net/mailman/listinfo/outages


> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

On Sun 30 Aug 2020 09:33:33 GMT, Tim Choe via Outages wrote:
> According to Centurylink tweet, their service has been restored. Can
> someone help me confirm that this is true before I start shifting the
> traffic back?

I’ve enabled back sessions at Ashburn and Paris, both are stable.

--
Alarig
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

> initial research identified that an offending flowspec announcement
> prevented Border Gateway Protocol (BGP) from establishing

opsec paranoid here wonders about the content of the flowspec
announcement and whence it originated.

randy
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Randy,

I'm almost wondering if the FlowSpec didn't inadvertently target control plane or MP-BGP traffic inadvertently causing the undulating and unrecoverable flapping of their network.

Alfredo Rodriguez

PS: Good to see a PSC staffer here (from a former CMU student and 'Burgh resident).

-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Randy Bush via Outages
Sent: Sunday, August 30, 2020 1:14 PM
To: outages@outages.org
Subject: Re: [outages] 3556 problems

Sabre Email Notification: This email is from an EXTERNAL source. Please use caution when clicking on links or opening attachments from an unknown or suspicious sender.To report a suspected phishing email, browse to: help.sabre.com > Risk & Security > Report a Security Issue

______________________________________________________________________
> initial research identified that an offending flowspec announcement
> prevented Border Gateway Protocol (BGP) from establishing

opsec paranoid here wonders about the content of the flowspec announcement and whence it originated.

randy
_______________________________________________
Outages mailing list
Outages@outages.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=FXJfUb1oWgygD0uNz-ujnA&r=YIif9c589xQ1-VT6Qqb1xgLuFPoYqoCsrVQEY29DkTM&m=i5c-xR5FjiDRhNkvsFY650mOQHWb0Tx3ST0PjtXHxFY&s=bjT6DVwICEkyj2ROw07oUzlc0eKV_Fqug5wxV5xeavc&e=
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Apologies if this has been posted already - I found this interesting:

https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/


Hagen Miller
Senior Systems Engineer
Computer Services, Inc.

-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Rodriguez, Alfredo via Outages
Sent: Monday, August 31, 2020 12:43 PM
To: outages@outages.org
Subject: Re: [outages] 3556 problems

WARNING: This email is from an external source. Do not click links or attachments unless you recognize the sender and know the content is safe.

Randy,

I'm almost wondering if the FlowSpec didn't inadvertently target control plane or MP-BGP traffic inadvertently causing the undulating and unrecoverable flapping of their network.

Alfredo Rodriguez

PS: Good to see a PSC staffer here (from a former CMU student and 'Burgh resident).

-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of Randy Bush via Outages
Sent: Sunday, August 30, 2020 1:14 PM
To: outages@outages.org
Subject: Re: [outages] 3556 problems

Sabre Email Notification: This email is from an EXTERNAL source. Please use caution when clicking on links or opening attachments from an unknown or suspicious sender.To report a suspected phishing email, browse to: help.sabre.com > Risk & Security > Report a Security Issue

______________________________________________________________________
> initial research identified that an offending flowspec announcement
> prevented Border Gateway Protocol (BGP) from establishing

opsec paranoid here wonders about the content of the flowspec announcement and whence it originated.

randy
_______________________________________________
Outages mailing list
Outages@outages.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=FXJfUb1oWgygD0uNz-ujnA&r=YIif9c589xQ1-VT6Qqb1xgLuFPoYqoCsrVQEY29DkTM&m=i5c-xR5FjiDRhNkvsFY650mOQHWb0Tx3ST0PjtXHxFY&s=bjT6DVwICEkyj2ROw07oUzlc0eKV_Fqug5wxV5xeavc&e=
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages


________________________________

This e-mail and any files transmitted with it are the property of Computer Services, Inc. and/or its divisions, are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. The sender does not accept any responsibility for any loss, disruption or damage to your data or computer systems that may occur from content of any sort contained in, transmitted with, or added to this e-mail. If you are not a named recipient or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your computer. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited.
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Reminder: let's move all the after-action traffic to outages-discuss.

Thanks,
-- jra

----- Original Message -----
> From: "Randy Bush via Outages" <outages@outages.org>
> To: outages@outages.org
> Sent: Sunday, August 30, 2020 2:14:21 PM
> Subject: Re: [outages] 3556 problems

>> initial research identified that an offending flowspec announcement
>> prevented Border Gateway Protocol (BGP) from establishing
>
> opsec paranoid here wonders about the content of the flowspec
> announcement and whence it originated.
>
> randy
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

--
Jay R. Ashworth Baylink jra@baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages