Mailing List Archive

Seeing high latency with packet loss from LA to Russia over level3



From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Total
Uptime Netops via Outages
Sent: Thursday, December 09, 2021 7:59 AM
To: outages@outages.org
Subject: [outages] [347-2944E457-001E] Lumen / Century Link / Legacy
Level3



Anyone seeing issues across the legacy Level3 network today? We've been
seeing a number of strange issues especially in the northeast and crossing
the Atlantic over the past few hours but haven't been able to nail
anything down.



Sincerely,



Charles Henry

Total Uptime Technologies, LLC
Support Portal: https://support.totaluptime.com

+1 800-584-1514 - TEL (US & Canada)
+44 330 808 0228 - TEL (United Kingdom)
+1 888-584-1194 - FAX

Privileged/Confidential Information may be contained in this message. If
you are not the addressee indicated in this message (or responsible for
delivery of the message to such person), you may not copy or deliver this
message to anyone. In such case, you should destroy this message and
kindly notify the sender by reply email. Please advise immediately if you
or your employer does not consent to email or messages of this kind.
Opinions, conclusions and other information in this message that do not
relate to the official business of the firm or its subsidiaries or
business units shall be understood as neither given nor endorsed by it.

We've had numerous users unable to VPN here in PA, all of them routing through Level3 handoffs from Verizon.

eApps is also reporting Level3 issues - https://twitter.com/eApps_Status/status/1468976793885491205

--Shane

From: Outages <outages-bounces@outages.org> On Behalf Of jd via Outages
Sent: Thursday, December 9, 2021 11:30
To: Total Uptime Netops <netops@totaluptime.com>
Cc: outages@outages.org
Subject: Re: [outages] [347-2944E457-001E] Lumen / Century Link / Legacy Level3

CAUTION! This email originated from outside Clearview. Do not reply, click links, or open attachments unless you are certain the sender is legitimate and the content is safe.

Seeing high latency with packet loss from LA to Russia over level3

From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Total Uptime Netops via Outages
Sent: Thursday, December 09, 2021 7:59 AM
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] [347-2944E457-001E] Lumen / Century Link / Legacy Level3

Anyone seeing issues across the legacy Level3 network today? We've been seeing a number of strange issues especially in the northeast and crossing the Atlantic over the past few hours but haven't been able to nail anything down.

Sincerely,

Charles Henry
Total Uptime Technologies, LLC
Support Portal: https://support.totaluptime.com

+1 800-584-1514 - TEL (US & Canada)
+44 330 808 0228 - TEL (United Kingdom)
+1 888-584-1194 - FAX

Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
NOTICE: This message may contain confidential or legally privileged information and is intended solely for the addressee(s). Any review, disclosure, re-transmission or copying of this message or its contents other than by the named recipient(s) is prohibited. If you received this message in error, please reply to inform us of the mistake then delete this communication from your computer system.

We have been receiving multiple reports from users with Verizon FIOS and
Verizon Wireless mobile hotspot unable to connect to VPN hosted on legacy
Level 3 in PA. Packet captures on both sides confirmed that the packet
loss was in the direction of Level 3 to Verizon. The problem was first
reported to us around 6:50am EST. At 2:02pm several Verizon VPN users
dropped for 5-10 seconds.

On Thu, Dec 9, 2021 at 12:02 PM Shane Simmons via Outages <
outages@outages.org> wrote:

> We’ve had numerous users unable to VPN here in PA, all of them routing
> through Level3 handoffs from Verizon.
>
>
>
> eApps is also reporting Level3 issues -
> https://twitter.com/eApps_Status/status/1468976793885491205
>
>
>
> --Shane
>
>
>
> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *jd via
> Outages
> *Sent:* Thursday, December 9, 2021 11:30
> *To:* Total Uptime Netops <netops@totaluptime.com>
> *Cc:* outages@outages.org
> *Subject:* Re: [outages] [347-2944E457-001E] Lumen / Century Link /
> Legacy Level3
>
>
>
> *CAUTION!* This email originated from outside Clearview. Do not reply,
> click links, or open attachments unless you are certain the sender is
> legitimate and the content is safe.
>
> Seeing high latency with packet loss from LA to Russia over level3
>
>
>
> *From:* Outages [mailto:outages-bounces@outages.org
> <outages-bounces@outages.org>] *On Behalf Of *Total Uptime Netops via
> Outages
> *Sent:* Thursday, December 09, 2021 7:59 AM
> *To:* outages@outages.org
> *Subject:* [outages] [347-2944E457-001E] Lumen / Century Link / Legacy
> Level3
>
>
>
> Anyone seeing issues across the legacy Level3 network today? We've been
> seeing a number of strange issues especially in the northeast and crossing
> the Atlantic over the past few hours but haven't been able to nail anything
> down.
>
>
>
> Sincerely,
>
>
>
> Charles Henry
>
> Total Uptime Technologies, LLC
> Support Portal: https://support.totaluptime.com
>
> +1 800-584-1514 - TEL (US & Canada)
> +44 330 808 0228 - TEL (United Kingdom)
> +1 888-584-1194 - FAX
>
> Privileged/Confidential Information may be contained in this message. If
> you are not the addressee indicated in this message (or responsible for
> delivery of the message to such person), you may not copy or deliver this
> message to anyone. In such case, you should destroy this message and kindly
> notify the sender by reply email. Please advise immediately if you or your
> employer does not consent to email or messages of this kind. Opinions,
> conclusions and other information in this message that do not relate to the
> official business of the firm or its subsidiaries or business units shall
> be understood as neither given nor endorsed by it.
> NOTICE: This message may contain confidential or legally privileged
> information and is intended solely for the addressee(s). Any review,
> disclosure, re-transmission or copying of this message or its contents
> other than by the named recipient(s) is prohibited. If you received this
> message in error, please reply to inform us of the mistake then delete this
> communication from your computer system.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

We have been having various issues passing different types of traffic from hosts outside the Verizon FIOS network to Verizon FIOS IPs through Level3 since about 0130 EST today. We’re located in the Washington, DC area. For a given host pair, sometimes ICMP was blackholed while other traffic types passed fine, and other times UDP was affected instead. Very inconsistent—seemed flow-by-flow. We just (1401 EST) saw a reroute of our level3 traffic from the DC Verizon handoff to Philly. Now everything seems back to normal. Would be nice to know what’s going on behind the scenes at Verizon today.

Patrick Shoemaker

From: Outages <outages-bounces@outages.org> on behalf of "outages@outages.org" <outages@outages.org>
Reply-To: Shane Simmons <SSimmons@clearviewfcu.org>
Date: Thursday, December 9, 2021 at 12:01 PM
To: "outages@outages.org" <outages@outages.org>
Subject: Re: [outages] [347-2944E457-001E] Lumen / Century Link / Legacy Level3

We’ve had numerous users unable to VPN here in PA, all of them routing through Level3 handoffs from Verizon.

eApps is also reporting Level3 issues - https://twitter.com/eApps_Status/status/1468976793885491205

--Shane

From: Outages <outages-bounces@outages.org> On Behalf Of jd via Outages
Sent: Thursday, December 9, 2021 11:30
To: Total Uptime Netops <netops@totaluptime.com>
Cc: outages@outages.org
Subject: Re: [outages] [347-2944E457-001E] Lumen / Century Link / Legacy Level3

CAUTION! This email originated from outside Clearview. Do not reply, click links, or open attachments unless you are certain the sender is legitimate and the content is safe.
Seeing high latency with packet loss from LA to Russia over level3

From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Total Uptime Netops via Outages
Sent: Thursday, December 09, 2021 7:59 AM
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] [347-2944E457-001E] Lumen / Century Link / Legacy Level3

Anyone seeing issues across the legacy Level3 network today? We've been seeing a number of strange issues especially in the northeast and crossing the Atlantic over the past few hours but haven't been able to nail anything down.

Sincerely,

Charles Henry
Total Uptime Technologies, LLC
Support Portal: https://support.totaluptime.com

+1 800-584-1514 - TEL (US & Canada)
+44 330 808 0228 - TEL (United Kingdom)
+1 888-584-1194 - FAX

Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
NOTICE: This message may contain confidential or legally privileged information and is intended solely for the addressee(s). Any review, disclosure, re-transmission or copying of this message or its contents other than by the named recipient(s) is prohibited. If you received this message in error, please reply to inform us of the mistake then delete this communication from your computer system.