Mailing List Archive

Level3 or Centurylink transport issue?
Hey has anyone noticed significant packet loss from level3 or centurylink?

We are having an issue with packet loss to BW media IPs 67.231.13.87 and
88. We rerouted all media traffic to out other DCs, and the poor MOS scores
have disappeared.



Rob Whitlatch

Director of DevOps

White Label Communications

215-987-4500
Re: Level3 or Centurylink transport issue? [ In reply to ]
Yes, I am currently experiencing an outage on Level3 on a wave between Ashburn and Raleigh.

Sounds as if their protect path is oversubscribed.

> On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages <outages@outages.org> wrote:
>
> Hey has anyone noticed significant packet loss from level3 or centurylink?
> We are having an issue with packet loss to BW media IPs 67.231.13.87 and 88. We rerouted all media traffic to out other DCs, and the poor MOS scores have disappeared.
>
> Rob Whitlatch
> Director of DevOps
> White Label Communications
> 215-987-4500
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org <mailto:Outages@outages.org>
> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
Re: Level3 or Centurylink transport issue? [ In reply to ]
There's a possible fibercut? between Ashburn and McLean VA on
Centurylink.? That may have something to do with it.

On 11/15/19 10:33 AM, Kate Gerry via Outages wrote:
> Yes, I am currently experiencing an outage on Level3 on a wave between
> Ashburn and Raleigh.
>
> Sounds as if their protect path is oversubscribed.
>
>> On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages
>> <outages@outages.org <mailto:outages@outages.org>> wrote:
>>
>> Hey has anyone noticed significant packet loss from level3 or
>> centurylink?
>> We are having an issue with packet loss to BW media IPs 67.231.13.87
>> and 88. We rerouted all media traffic to out other DCs, and the poor
>> MOS scores have disappeared.
>>
>> Rob Whitlatch
>> Director of DevOps
>> White Label Communications
>> 215-987-4500
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org <mailto:Outages@outages.org>
>> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
Re: Level3 or Centurylink transport issue? [ In reply to ]
Had a 20-second outage on a Level3 circuit in Arlington, VA 2 hours ago, but no connectivity/loss issues since.

--
Christopher Conley
Systems Administrator | Fors Marsh Group
1010 N. Glebe Rd, Suite 510
Arlington, VA 22201


From: Outages <outages-bounces@outages.org> On Behalf Of Kate Gerry via Outages
Sent: Friday, November 15, 2019 1:33 PM
To: Rob Whitlatch <rwhitlatch@whitelabelcomm.com>
Cc: outages@outages.org
Subject: Re: [outages] Level3 or Centurylink transport issue?

Yes, I am currently experiencing an outage on Level3 on a wave between Ashburn and Raleigh.

Sounds as if their protect path is oversubscribed.


On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:

Hey has anyone noticed significant packet loss from level3 or centurylink?
We are having an issue with packet loss to BW media IPs 67.231.13.87 and 88. We rerouted all media traffic to out other DCs, and the poor MOS scores have disappeared.

Rob Whitlatch
Director of DevOps
White Label Communications
215-987-4500

_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
Re: Level3 or Centurylink transport issue? [ In reply to ]
I lost my transit too.

On 15/11/2019 19:33, Kate Gerry via Outages wrote:
> Yes, I am currently experiencing an outage on Level3 on a wave between
> Ashburn and Raleigh.
>
> Sounds as if their protect path is oversubscribed.
>
>> On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages
>> <outages@outages.org <mailto:outages@outages.org>> wrote:
>>
>> Hey has anyone noticed significant packet loss from level3 or centurylink?
>> We are having an issue with packet loss to BW media IPs 67.231.13.87
>> and 88. We rerouted all media traffic to out other DCs, and the poor
>> MOS scores have disappeared.
>>
>> ?
>>
>> Rob Whitlatch
>> Director of DevOps
>> White Label Communications
>> 215-987-4500
>>
>> ?
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org <mailto:Outages@outages.org>
>> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


--
Alarig Le Lay
Grifon
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: Level3 or Centurylink transport issue? [ In reply to ]
864 count fiber got cut by a vertical boring operation.

>>>CenturyLink

This is to confirm that an update has been recorded on CenturyLink ticket x.


*** CASCADED EXTERNAL NOTES 15-Nov-2019 20:21:17 GMT From CASE: x - Event
Field Operations and the Outside Plant Engineer (OSP) have located the exact point of failure. A vertical boring operation has struck and cut an 864-strand fiber. The construction crew is onsite fully assessing the damage and are formulating a repair plan at this time. The fiber splicers are en route with an estimated time of arrival (ETA) of 21:00 GMT. The replacement fiber is also en route; however, the ETA for its arrival is currently unavailable.


?On 11/15/19, 3:21 PM, "Outages on behalf of Alarig Le Lay via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote:

I lost my transit too.

On 15/11/2019 19:33, Kate Gerry via Outages wrote:
> Yes, I am currently experiencing an outage on Level3 on a wave between
> Ashburn and Raleigh.
>
> Sounds as if their protect path is oversubscribed.
>
>> On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages
>> <outages@outages.org <mailto:outages@outages.org>> wrote:
>>
>> Hey has anyone noticed significant packet loss from level3 or centurylink?
>> We are having an issue with packet loss to BW media IPs 67.231.13.87
>> and 88. We rerouted all media traffic to out other DCs, and the poor
>> MOS scores have disappeared.
>>
>>
>>
>> Rob Whitlatch
>> Director of DevOps
>> White Label Communications
>> 215-987-4500
>>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org <mailto:Outages@outages.org>
>> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


--
Alarig Le Lay
Grifon
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages



_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: Level3 or Centurylink transport issue? [ In reply to ]
Update from CenturyLink:

The Outside Plant team on site has advised that the auger has been removed and the cable has been cut away from it. Efforts to manually hand dig down to the damage are currently ongoing. It has also been advised that cabling as well as some of the duct will need to be replaced between two handholes. The full extent of the damage to the duct and fiber will not be able to be confirmed until the excavation process is completed.



?On 11/15/19, 3:41 PM, "Outages on behalf of Wright, Chris via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote:

864 count fiber got cut by a vertical boring operation.

>>>CenturyLink

This is to confirm that an update has been recorded on CenturyLink ticket x.


*** CASCADED EXTERNAL NOTES 15-Nov-2019 20:21:17 GMT From CASE: x - Event
Field Operations and the Outside Plant Engineer (OSP) have located the exact point of failure. A vertical boring operation has struck and cut an 864-strand fiber. The construction crew is onsite fully assessing the damage and are formulating a repair plan at this time. The fiber splicers are en route with an estimated time of arrival (ETA) of 21:00 GMT. The replacement fiber is also en route; however, the ETA for its arrival is currently unavailable.


On 11/15/19, 3:21 PM, "Outages on behalf of Alarig Le Lay via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote:

I lost my transit too.

On 15/11/2019 19:33, Kate Gerry via Outages wrote:
> Yes, I am currently experiencing an outage on Level3 on a wave between
> Ashburn and Raleigh.
>
> Sounds as if their protect path is oversubscribed.
>
>> On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages
>> <outages@outages.org <mailto:outages@outages.org>> wrote:
>>
>> Hey has anyone noticed significant packet loss from level3 or centurylink?
>> We are having an issue with packet loss to BW media IPs 67.231.13.87
>> and 88. We rerouted all media traffic to out other DCs, and the poor
>> MOS scores have disappeared.
>>
>>
>>
>> Rob Whitlatch
>> Director of DevOps
>> White Label Communications
>> 215-987-4500
>>
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org <mailto:Outages@outages.org>
>> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


--
Alarig Le Lay
Grifon
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages



_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages


_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages