Mailing List Archive

[18B-2709942B-0003] Telia issues in LAX
FYI, Telia has a major network outage in Los Angeles. Their case 01222734

We noticed some time ago when traffic between Los Angles and San Jose got there via Tokyo and Hong Kong. Lovey architecture that is probably affecting others.

For your amusement, I paste a trace:

Tracing route to te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 10.232.22.1
2 3 ms 3 ms 3 ms (redacted)
3 2 ms 2 ms 2 ms 10.220.204.1
4 3 ms 2 ms 2 ms eth.15.1.bb1.phx0.phoenixnap.com [108.170.0.10]
5 3 ms 2 ms 2 ms 10.140.2.42
6 12 ms 11 ms 12 ms unknown.telstraglobal.net [210.176.40.33]
7 12 ms 13 ms 12 ms i-93.1wlt-core02.telstraglobal.net [202.84.253.85]
8 11 ms 11 ms 13 ms i-95.eqla02.telstraglobal.net [202.84.143.30]
9 187 ms 187 ms 187 ms las-b21-link.telia.net [213.248.92.106]
10 * 191 ms 191 ms tky-b1-link.telia.net [62.115.142.129]
11 191 ms * * hnk-b2-link.telia.net [62.115.143.60]
12 * * 192 ms sjo-b21-link.telia.net [62.115.121.194]
13 191 ms * 191 ms ntt-ic-323771-sjo-b21.c.telia.net [62.115.12.53]
14 190 ms * * ae-10.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.26]
15 * 190 ms * ae-3.r25.lsanca07.us.bb.gin.ntt.net [129.250.4.151]
16 190 ms * * ae-2.r00.lsanca07.us.bb.gin.ntt.net [129.250.3.238]
17 189 ms * * ae-0.a00.lsanca07.us.bb.gin.ntt.net [129.250.2.104]
18 189 ms * 191 ms xe-0-0-15-0.a00.lsanca07.us.ce.gin.ntt.net [130.94.195.50]
19 * * * Request timed out.
20 * * * Request timed out.
21 189 ms 190 ms * te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
22 189 ms 189 ms 189 ms te10-1.lb14.lax1.totaluptime.net [199.38.148.127]

Sincerely,

Charles Henry
Network Operations Team
Total Uptime Technologies, LLC
Support Portal: https://support.totaluptime.com

+1 800-584-1514 - TEL (US & Canada)
+44 330 808 0228 - TEL (United Kingdom)
+1 888-584-1194 - FAX

Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
Re: [18B-2709942B-0003] Telia issues in LAX [ In reply to ]
Any idea if this is solved by now?

On Fri, Oct 2, 2020 at 3:41 AM Total Uptime Netops via Outages <
outages@outages.org> wrote:

> FYI, Telia has a major network outage in Los Angeles. Their case 01222734
>
> We noticed some time ago when traffic between Los Angles and San Jose got
> there via Tokyo and Hong Kong. Lovey architecture that is probably
> affecting others.
>
> For your amusement, I paste a trace:
>
> Tracing route to te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> over a maximum of 30 hops:
>
> 1 1 ms 1 ms 1 ms 10.232.22.1
> 2 3 ms 3 ms 3 ms (redacted)
> 3 2 ms 2 ms 2 ms 10.220.204.1
> 4 3 ms 2 ms 2 ms eth.15.1.bb1.phx0.phoenixnap.com
> [108.170.0.10]
> 5 3 ms 2 ms 2 ms 10.140.2.42
> 6 12 ms 11 ms 12 ms unknown.telstraglobal.net [210.176.40.33]
> 7 12 ms 13 ms 12 ms i-93.1wlt-core02.telstraglobal.net
> [202.84.253.85]
> 8 11 ms 11 ms 13 ms i-95.eqla02.telstraglobal.net
> [202.84.143.30]
> 9 187 ms 187 ms 187 ms las-b21-link.telia.net [213.248.92.106]
> 10 * 191 ms 191 ms tky-b1-link.telia.net [62.115.142.129]
> 11 191 ms * * hnk-b2-link.telia.net [62.115.143.60]
> 12 * * 192 ms sjo-b21-link.telia.net [62.115.121.194]
> 13 191 ms * 191 ms ntt-ic-323771-sjo-b21.c.telia.net
> [62.115.12.53]
> 14 190 ms * * ae-10.r24.snjsca04.us.bb.gin.ntt.net
> [129.250.3.26]
> 15 * 190 ms * ae-3.r25.lsanca07.us.bb.gin.ntt.net
> [129.250.4.151]
> 16 190 ms * * ae-2.r00.lsanca07.us.bb.gin.ntt.net
> [129.250.3.238]
> 17 189 ms * * ae-0.a00.lsanca07.us.bb.gin.ntt.net
> [129.250.2.104]
> 18 189 ms * 191 ms xe-0-0-15-0.a00.lsanca07.us.ce.gin.ntt.net
> [130.94.195.50]
> 19 * * * Request timed out.
> 20 * * * Request timed out.
> 21 189 ms 190 ms * te10-1.lb14.lax1.totaluptime.net
> [199.38.148.127]
> 22 189 ms 189 ms 189 ms te10-1.lb14.lax1.totaluptime.net
> [199.38.148.127]
>
> Sincerely,
>
> Charles Henry
> Network Operations Team
> Total Uptime Technologies, LLC
> Support Portal: https://support.totaluptime.com
>
> +1 800-584-1514 - TEL (US & Canada)
> +44 330 808 0228 - TEL (United Kingdom)
> +1 888-584-1194 - FAX
>
> Privileged/Confidential Information may be contained in this message. If
> you are not the addressee indicated in this message (or responsible for
> delivery of the message to such person), you may not copy or deliver this
> message to anyone. In such case, you should destroy this message and kindly
> notify the sender by reply email. Please advise immediately if you or your
> employer does not consent to email or messages of this kind. Opinions,
> conclusions and other information in this message that do not relate to the
> official business of the firm or its subsidiaries or business units shall
> be understood as neither given nor endorsed by it.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
Re: [18B-2709942B-0003] Telia issues in LAX [ In reply to ]
Apparently it was due to Verizon doing maintenance on their gateway
router. It might not have been a Telia problem exactly. It could have
been the handoff from Telia to Verizon.

It knocked some Verizon circuits completely offline for 1 1/2 hours.
From the Verizon side, it started at around 12:40 AM PST and was
resolved around 2:15 AM PST.

On Fri, Oct 2, 2020 at 12:02 PM Béla Törös via Outages
<outages@outages.org> wrote:
>
> Our ticket got updated that it is resolved and stable since 9:00 UTC.
>
> On Fri, 2 Oct 2020 at 18:17, Total Uptime Netops via Outages
> <outages@outages.org> wrote:
> >
> > We saw it clear at 4:30 AM ET / 8:30 AM UTC.
> >
> > No word yet from Telia.
> >
> > Sincerely,
> >
> > Charles Henry
> > Network Operations Team
> > Total Uptime Technologies, LLC
> > Support Portal: https://support.totaluptime.com
> >
> > +1 800-584-1514 - TEL (US & Canada)
> > +44 330 808 0228 - TEL (United Kingdom)
> > +1 888-584-1194 - FAX
> >
> > Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
> >
> > ________________________________
> > From: Tomas Lynch <tomas.lynch@gmail.com>
> > Sent: Fri, 02 Oct 2020 12:45:08 +0000
> > To: Total Uptime Netops <netops@totaluptime.com>
> > Subject: Re: [outages] [18B-2709942B-0003] Telia issues in LAX
> > Any idea if this is solved by now?
> >
> > On Fri, Oct 2, 2020 at 3:41 AM Total Uptime Netops via Outages <outages@outages.org> wrote:
> >>
> >> FYI, Telia has a major network outage in Los Angeles. Their case 01222734
> >>
> >> We noticed some time ago when traffic between Los Angles and San Jose got there via Tokyo and Hong Kong. Lovey architecture that is probably affecting others.
> >>
> >> For your amusement, I paste a trace:
> >>
> >> Tracing route to te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> >> over a maximum of 30 hops:
> >>
> >> 1 1 ms 1 ms 1 ms 10.232.22.1
> >> 2 3 ms 3 ms 3 ms (redacted)
> >> 3 2 ms 2 ms 2 ms 10.220.204.1
> >> 4 3 ms 2 ms 2 ms eth.15.1.bb1.phx0.phoenixnap.com [108.170.0.10]
> >> 5 3 ms 2 ms 2 ms 10.140.2.42
> >> 6 12 ms 11 ms 12 ms unknown.telstraglobal.net [210.176.40.33]
> >> 7 12 ms 13 ms 12 ms i-93.1wlt-core02.telstraglobal.net [202.84.253.85]
> >> 8 11 ms 11 ms 13 ms i-95.eqla02.telstraglobal.net [202.84.143.30]
> >> 9 187 ms 187 ms 187 ms las-b21-link.telia.net [213.248.92.106]
> >> 10 * 191 ms 191 ms tky-b1-link.telia.net [62.115.142.129]
> >> 11 191 ms * * hnk-b2-link.telia.net [62.115.143.60]
> >> 12 * * 192 ms sjo-b21-link.telia.net [62.115.121.194]
> >> 13 191 ms * 191 ms ntt-ic-323771-sjo-b21.c.telia.net [62.115.12.53]
> >> 14 190 ms * * ae-10.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.26]
> >> 15 * 190 ms * ae-3.r25.lsanca07.us.bb.gin.ntt.net [129.250.4.151]
> >> 16 190 ms * * ae-2.r00.lsanca07.us.bb.gin.ntt.net [129.250.3.238]
> >> 17 189 ms * * ae-0.a00.lsanca07.us.bb.gin.ntt.net [129.250.2.104]
> >> 18 189 ms * 191 ms xe-0-0-15-0.a00.lsanca07.us.ce.gin.ntt.net [130.94.195.50]
> >> 19 * * * Request timed out.
> >> 20 * * * Request timed out.
> >> 21 189 ms 190 ms * te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> >> 22 189 ms 189 ms 189 ms te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> >>
> >> Sincerely,
> >>
> >> Charles Henry
> >> Network Operations Team
> >> Total Uptime Technologies, LLC
> >> Support Portal: https://support.totaluptime.com
> >>
> >> +1 800-584-1514 - TEL (US & Canada)
> >> +44 330 808 0228 - TEL (United Kingdom)
> >> +1 888-584-1194 - FAX
> >>
> >> Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
> >> _______________________________________________
> >> Outages mailing list
> >> Outages@outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >
> > ________________________________
> > From: Total Uptime Netops <netops@totaluptime.com>
> > Sent: Fri, 02 Oct 2020 07:40:37 +0000
> > To: outages@outages.org
> > Subject: [outages] [18B-2709942B-0003] Telia issues in LAX
> > FYI, Telia has a major network outage in Los Angeles. Their case 01222734
> >
> > We noticed some time ago when traffic between Los Angles and San Jose got there via Tokyo and Hong Kong. Lovey architecture that is probably affecting others.
> >
> > For your amusement, I paste a trace:
> >
> > Tracing route to te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> > over a maximum of 30 hops:
> >
> > 1 1 ms 1 ms 1 ms 10.232.22.1
> > 2 3 ms 3 ms 3 ms (redacted)
> > 3 2 ms 2 ms 2 ms 10.220.204.1
> > 4 3 ms 2 ms 2 ms eth.15.1.bb1.phx0.phoenixnap.com [108.170.0.10]
> > 5 3 ms 2 ms 2 ms 10.140.2.42
> > 6 12 ms 11 ms 12 ms unknown.telstraglobal.net [210.176.40.33]
> > 7 12 ms 13 ms 12 ms i-93.1wlt-core02.telstraglobal.net [202.84.253.85]
> > 8 11 ms 11 ms 13 ms i-95.eqla02.telstraglobal.net [202.84.143.30]
> > 9 187 ms 187 ms 187 ms las-b21-link.telia.net [213.248.92.106]
> > 10 * 191 ms 191 ms tky-b1-link.telia.net [62.115.142.129]
> > 11 191 ms * * hnk-b2-link.telia.net [62.115.143.60]
> > 12 * * 192 ms sjo-b21-link.telia.net [62.115.121.194]
> > 13 191 ms * 191 ms ntt-ic-323771-sjo-b21.c.telia.net [62.115.12.53]
> > 14 190 ms * * ae-10.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.26]
> > 15 * 190 ms * ae-3.r25.lsanca07.us.bb.gin.ntt.net [129.250.4.151]
> > 16 190 ms * * ae-2.r00.lsanca07.us.bb.gin.ntt.net [129.250.3.238]
> > 17 189 ms * * ae-0.a00.lsanca07.us.bb.gin.ntt.net [129.250.2.104]
> > 18 189 ms * 191 ms xe-0-0-15-0.a00.lsanca07.us.ce.gin.ntt.net [130.94.195.50]
> > 19 * * * Request timed out.
> > 20 * * * Request timed out.
> > 21 189 ms 190 ms * te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> > 22 189 ms 189 ms 189 ms te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> >
> > Sincerely,
> >
> > Charles Henry
> > Network Operations Team
> > Total Uptime Technologies, LLC
> > Support Portal: https://support.totaluptime.com
> >
> > +1 800-584-1514 - TEL (US & Canada)
> > +44 330 808 0228 - TEL (United Kingdom)
> > +1 888-584-1194 - FAX
> >
> > Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
> > ________________________________
> > From: Netops
> > Sent: Fri, 02 Oct 2020 07:32:59 +0000
> > To: outages@outages.org
> > Subject: Telia issues in LAX
> > FYI, Telia has a major network outage in Los Angeles. Their case 01222734
> >
> > We noticed some time ago when traffic between Los Angles and San Jose got there via Tokyo and Hong Kong. Lovey architecture that is probably affecting others.
> >
> > For your amusement, I paste a trace:
> >
> > Tracing route to te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> > over a maximum of 30 hops:
> >
> > 1 1 ms 1 ms 1 ms 10.232.22.1
> > 2 3 ms 3 ms 3 ms (redacted)
> > 3 2 ms 2 ms 2 ms 10.220.204.1
> > 4 3 ms 2 ms 2 ms eth.15.1.bb1.phx0.phoenixnap.com [108.170.0.10]
> > 5 3 ms 2 ms 2 ms 10.140.2.42
> > 6 12 ms 11 ms 12 ms unknown.telstraglobal.net [210.176.40.33]
> > 7 12 ms 13 ms 12 ms i-93.1wlt-core02.telstraglobal.net [202.84.253.85]
> > 8 11 ms 11 ms 13 ms i-95.eqla02.telstraglobal.net [202.84.143.30]
> > 9 187 ms 187 ms 187 ms las-b21-link.telia.net [213.248.92.106]
> > 10 * 191 ms 191 ms tky-b1-link.telia.net [62.115.142.129]
> > 11 191 ms * * hnk-b2-link.telia.net [62.115.143.60]
> > 12 * * 192 ms sjo-b21-link.telia.net [62.115.121.194]
> > 13 191 ms * 191 ms ntt-ic-323771-sjo-b21.c.telia.net [62.115.12.53]
> > 14 190 ms * * ae-10.r24.snjsca04.us.bb.gin.ntt.net [129.250.3.26]
> > 15 * 190 ms * ae-3.r25.lsanca07.us.bb.gin.ntt.net [129.250.4.151]
> > 16 190 ms * * ae-2.r00.lsanca07.us.bb.gin.ntt.net [129.250.3.238]
> > 17 189 ms * * ae-0.a00.lsanca07.us.bb.gin.ntt.net [129.250.2.104]
> > 18 189 ms * 191 ms xe-0-0-15-0.a00.lsanca07.us.ce.gin.ntt.net [130.94.195.50]
> > 19 * * * Request timed out.
> > 20 * * * Request timed out.
> > 21 189 ms 190 ms * te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> > 22 189 ms 189 ms 189 ms te10-1.lb14.lax1.totaluptime.net [199.38.148.127]
> >
> > Sincerely,
> >
> > Charles Henry
> > Network Operations Team
> > Total Uptime Technologies, LLC
> > Support Portal: https://support.totaluptime.com
> >
> > +1 800-584-1514 - TEL (US & Canada)
> > +44 330 808 0228 - TEL (United Kingdom)
> > +1 888-584-1194 - FAX
> >
> > Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: [18B-2709942B-0003] Telia issues in LAX [ In reply to ]
More information below on the outage that occurred.

Verizon is doing emergency maintenance (10/14/2020 at 00:01 PDT) to
possibly fix the issue described below. Although the issue is pinned to
Telia, the traceroute at the time resolved to a Verizon/Telia hop.

Will post a separate thread on the scheduled emergency maintenance
notice received from Verizon.

On Fri, Oct 2, 2020 at 3:02 PM Brandon wrote:

The updates I am getting from Telia are essentially summarized as below:
>
> Power was lost to a device in the LA area. This was due to power
> maintenance and it seems they were not supposed to impact the A & B feeds
> at the same time. Additionally when this failed traffic re-routed and thus
> congested pacific crossing paths. They eventually got power to the device
> but at least one card was hanging the chassis boot. They got the chassis up
> but still one card fails to come online and some channels are still down as
> a result. They claim that if you’re currently up you shouldn’t be impacted
> any more by the pending work.
>
> Basically… power failed, then a card failed at boot.
>
> Thanks, Brandon
>
>
>