Mailing List Archive

[23E-270B4044-0002] TATA issues in Los Angeles
Starting at 1:36 PM UTC TATA Communications is having major issues in Los Angeles. We lost light on our ports and this appears to be affecting their trans-pacific traffic significantly as well. No update from them yet.

Sincerely,

Charles Henry
Network Operations Team
Total Uptime Technologies, LLC
Support Portal: https://support.totaluptime.com

+1 800-584-1514 - TEL (US & Canada)
+44 330 808 0228 - TEL (United Kingdom)
+1 888-584-1194 - FAX

Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
Re: [23E-270B4044-0002] TATA issues in Los Angeles [ In reply to ]
I'm seeing traceroutes from Ashburn to Toronto, Montreal, and LAX taking a
detour through Sydney Australia right now.

On Sat, Oct 3, 2020 at 7:07 AM Total Uptime Netops via Outages <
outages@outages.org> wrote:

> Starting at 1:36 PM UTC TATA Communications is having major issues in Los
> Angeles. We lost light on our ports and this appears to be affecting their
> trans-pacific traffic significantly as well. No update from them yet.
>
> Sincerely,
>
> Charles Henry
> Network Operations Team
> Total Uptime Technologies, LLC
> Support Portal: https://support.totaluptime.com
>
> +1 800-584-1514 - TEL (US & Canada)
> +44 330 808 0228 - TEL (United Kingdom)
> +1 888-584-1194 - FAX
>
> Privileged/Confidential Information may be contained in this message. If
> you are not the addressee indicated in this message (or responsible for
> delivery of the message to such person), you may not copy or deliver this
> message to anyone. In such case, you should destroy this message and kindly
> notify the sender by reply email. Please advise immediately if you or your
> employer does not consent to email or messages of this kind. Opinions,
> conclusions and other information in this message that do not relate to the
> official business of the firm or its subsidiaries or business units shall
> be understood as neither given nor endorsed by it.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
Re: [23E-270B4044-0002] TATA issues in Los Angeles [ In reply to ]
Here is are some traceroutes. Looks like some packet loss is starting to
happen.

Ashburn > LAX
[root@core0 ~]# mtr 209.58.46.11 --report
HOST: core0.vocaltransit.net Loss% Snt Last Avg Best Wrst StDev
1. rtr.rtc.vocaltransit.net 0.0% 10 0.1 0.2 0.1 0.4 0.1
2. ge-102-0-0-0.r05.asbnva02.us 0.0% 10 1.2 1.3 1.2 1.6 0.1
3. ae-7.r25.asbnva02.us.bb.gin. 90.0% 10 2.9 2.9 2.9 2.9 0.0
4. ae-2.r25.lsanca07.us.bb.gin. 20.0% 10 66.5 66.1 64.9 67.7 0.9
5. ae-7.r22.lsanca07.us.bb.gin. 0.0% 10 65.1 65.1 65.0 65.3 0.1
6. ae-5.r20.sydnau02.au.bb.gin. 0.0% 10 235.3 230.4 226.4 250.6 7.6
7. ix-ae-12-0.thar1.1mh-sydney. 0.0% 10 217.5 218.0 217.4 220.2 1.1
8. if-ae-2-2.tcore1.0pp-sydney. 0.0% 10 251.9 252.0 251.7 252.4 0.2
9. if-ae-5-2.thar1.pv4-piti.as6 0.0% 10 245.5 247.9 245.5 255.6 3.9
10. if-ae-21-2.tcore2.tv2-tokyo. 0.0% 10 250.1 247.5 245.6 250.3 2.0
11. if-et-24-2.hcore2.kv8-chiba. 20.0% 10 305.5 305.7 305.3 306.5 0.5
12. if-ae-20-2.tcore2.lvw-losang 80.0% 10 245.5 246.1 245.5 246.6 0.7
13. if-ae-2-2.tcore1.lvw-losange 10.0% 10 245.8 249.2 245.4 257.2 4.5
14. if-ae-14-2.vpe2.laa-losangel 10.0% 10 250.4 250.4 245.2 263.4 5.9
15. 209.58.46.11 10.0% 10 245.3 247.3 245.3 258.5 4.3

Ashburn > Toronto
[root@core0 ~]# mtr 209.58.46.109 --report
HOST: core0.vocaltransit.net Loss% Snt Last Avg Best Wrst StDev
1. rtr.rtc.vocaltransit.net 0.0% 10 0.2 0.2 0.1 0.3 0.1
2. ge-102-0-0-0.r05.asbnva02.us 0.0% 10 1.3 1.3 1.1 1.7 0.2
3. ae-7.r25.asbnva02.us.bb.gin. 0.0% 10 1.9 1.7 1.2 2.9 0.5
4. ae-2.r25.lsanca07.us.bb.gin. 0.0% 10 65.1 66.1 65.0 69.0 1.5
5. ae-7.r22.lsanca07.us.bb.gin. 0.0% 10 65.1 66.0 64.9 71.3 1.9
6. ae-5.r20.sydnau02.au.bb.gin. 0.0% 10 240.4 227.9 226.3 240.4 4.4
7. ix-ae-12-0.thar1.1mh-sydney. 0.0% 10 217.6 217.8 217.4 220.7 1.0
8. if-ae-2-2.tcore1.0pp-sydney. 0.0% 10 249.9 251.0 249.8 255.9 2.2
9. if-ae-5-2.thar1.pv4-piti.as6 0.0% 10 248.4 249.1 248.3 251.2 1.1
10. if-ae-3-2.tcore1.tv2-tokyo.a 0.0% 10 248.4 248.6 248.3 249.2 0.4
11. if-et-21-2.hcore1.kv8-chiba. 10.0% 10 307.1 312.3 307.0 342.1 11.5
12. if-ae-5-2.tcore2.sv1-santacl 80.0% 10 250.8 250.7 250.5 250.8 0.2
13. if-ae-18-4.tcore1.sqn-sanjos 0.0% 10 250.5 251.0 250.5 252.7 0.6
14. if-ae-1-2.tcore2.sqn-sanjose 0.0% 10 250.0 250.1 250.0 250.3 0.1
15. if-ae-51-2.tcore2.ct8-chicag 0.0% 10 252.6 251.2 250.6 252.6 0.8
16. if-ae-22-2.tcore1.ct8-chicag 0.0% 10 250.2 252.3 250.0 265.0 4.6
17. if-ae-8-2.tcore2.tnk-toronto 10.0% 10 249.9 250.0 249.8 251.4 0.5
18. if-ae-2-2.tcore1.tnk-toronto 10.0% 10 250.7 250.8 250.6 251.0 0.1
19. if-ae-25-2.tcore1.ttt-toront 10.0% 10 250.7 250.6 250.2 251.4 0.4
20. if-ae-1-2.vpe1.ttt-toronto.a 10.0% 10 249.7 249.6 249.5 249.7 0.1
21. 209.58.46.109 10.0% 10 249.7 249.7 249.6 249.8 0.1

Ashburn > Montreal
[root@core0 ~]# mtr 209.58.46.142 --report
HOST: core0.vocaltransit.net Loss% Snt Last Avg Best Wrst StDev
1. rtr.rtc.vocaltransit.net 0.0% 10 0.3 0.3 0.1 0.5 0.1
2. ge-102-0-0-0.r05.asbnva02.us 0.0% 10 1.2 1.2 1.1 1.3 0.1
3. ae-7.r25.asbnva02.us.bb.gin. 0.0% 10 3.8 2.3 1.1 7.6 2.1
4. ae-2.r25.lsanca07.us.bb.gin. 70.0% 10 61.1 61.3 60.9 61.9 0.5
5. ae-7.r22.lsanca07.us.bb.gin. 0.0% 10 61.0 62.5 60.8 75.5 4.5
6. ae-5.r20.sydnau02.au.bb.gin. 0.0% 10 222.6 222.6 222.3 224.0 0.5
7. ix-ae-12-0.thar1.1mh-sydney. 0.0% 10 213.6 213.4 213.3 213.6 0.1
8. if-ae-2-2.tcore1.0pp-sydney. 0.0% 10 251.7 249.9 249.0 251.7 1.1
9. if-ae-5-2.thar1.pv4-piti.as6 0.0% 10 248.8 249.0 248.5 251.5 0.9
10. if-ae-3-2.tcore1.tv2-tokyo.a 0.0% 10 249.1 248.6 248.2 249.1 0.3
11. if-et-21-2.hcore1.kv8-chiba. 10.0% 10 303.1 303.7 302.9 306.8 1.2
12. if-ae-5-2.tcore2.sv1-santacl 60.0% 10 249.6 249.7 249.5 250.1 0.3
13. if-ae-18-2.tcore1.sqn-sanjos 0.0% 10 249.4 249.3 249.2 249.4 0.1
14. if-ae-1-2.tcore2.sqn-sanjose 0.0% 10 249.2 249.2 249.0 249.5 0.2
15. if-ae-51-2.tcore2.ct8-chicag 0.0% 10 249.5 251.4 249.1 267.4 5.6
16. if-ae-30-2.tcore1.w6c-montre 10.0% 10 249.6 249.3 249.0 249.9 0.3
17. if-ae-12-2.tcore1.mtt-montre 10.0% 10 249.0 249.4 248.7 250.9 0.8
18. if-ae-1-2.vpe1.mtt-montreal. 10.0% 10 248.9 248.9 248.7 249.3 0.2
19. 209.58.46.142 10.0% 10 248.9 248.9 248.7 249.1 0.1

On Sat, Oct 3, 2020 at 9:52 AM Jared Geiger <jared@compuwizz.net> wrote:

> I'm seeing traceroutes from Ashburn to Toronto, Montreal, and LAX taking a
> detour through Sydney Australia right now.
>
> On Sat, Oct 3, 2020 at 7:07 AM Total Uptime Netops via Outages <
> outages@outages.org> wrote:
>
>> Starting at 1:36 PM UTC TATA Communications is having major issues in Los
>> Angeles. We lost light on our ports and this appears to be affecting their
>> trans-pacific traffic significantly as well. No update from them yet.
>>
>> Sincerely,
>>
>> Charles Henry
>> Network Operations Team
>> Total Uptime Technologies, LLC
>> Support Portal: https://support.totaluptime.com
>>
>> +1 800-584-1514 - TEL (US & Canada)
>> +44 330 808 0228 - TEL (United Kingdom)
>> +1 888-584-1194 - FAX
>>
>> Privileged/Confidential Information may be contained in this message. If
>> you are not the addressee indicated in this message (or responsible for
>> delivery of the message to such person), you may not copy or deliver this
>> message to anyone. In such case, you should destroy this message and kindly
>> notify the sender by reply email. Please advise immediately if you or your
>> employer does not consent to email or messages of this kind. Opinions,
>> conclusions and other information in this message that do not relate to the
>> official business of the firm or its subsidiaries or business units shall
>> be understood as neither given nor endorsed by it.
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
Re: [23E-270B4044-0002] TATA issues in Los Angeles [ In reply to ]
At least your packets are transiting down to Melbourne where they’d catch COVID-19 on ingress. ????

> On 4 Oct 2020, at 2:58 am, Jared Geiger via Outages <outages@outages.org> wrote:
>
> ?
> I'm seeing traceroutes from Ashburn to Toronto, Montreal, and LAX taking a detour through Sydney Australia right now.
>
>> On Sat, Oct 3, 2020 at 7:07 AM Total Uptime Netops via Outages <outages@outages.org> wrote:
>> Starting at 1:36 PM UTC TATA Communications is having major issues in Los Angeles. We lost light on our ports and this appears to be affecting their trans-pacific traffic significantly as well. No update from them yet.
>>
>> Sincerely,
>>
>> Charles Henry
>> Network Operations Team
>> Total Uptime Technologies, LLC
>> Support Portal: https://support.totaluptime.com
>>
>> +1 800-584-1514 - TEL (US & Canada)
>> +44 330 808 0228 - TEL (United Kingdom)
>> +1 888-584-1194 - FAX
>>
>> Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
Re: [23E-270B4044-0002] TATA issues in Los Angeles [ In reply to ]
If anyone at TATA 6453 is watching, you're announcing 209.58.46.0/24 to NTT
2914 in Sydney causing traffic to your North American VOIP network to go
through Australia. The rest of your announcements to your peers are for
209.58.0.0/17 which correctly keeps the traffic in North America. Filtering
the /24 out of NTT's BGP feed fixed my latency and packet loss problem. NTT
contacted the TATA NOC on my behalf but no one has fixed it yet.

On Sat, Oct 3, 2020 at 3:05 PM Dave Browning via Outages <
outages@outages.org> wrote:

> At least your packets are transiting down to Melbourne where they’d catch
> COVID-19 on ingress. ????
>
> On 4 Oct 2020, at 2:58 am, Jared Geiger via Outages <outages@outages.org>
> wrote:
>
> ?
> I'm seeing traceroutes from Ashburn to Toronto, Montreal, and LAX taking a
> detour through Sydney Australia right now.
>
> On Sat, Oct 3, 2020 at 7:07 AM Total Uptime Netops via Outages <
> outages@outages.org> wrote:
>
>> Starting at 1:36 PM UTC TATA Communications is having major issues in Los
>> Angeles. We lost light on our ports and this appears to be affecting their
>> trans-pacific traffic significantly as well. No update from them yet.
>>
>> Sincerely,
>>
>> Charles Henry
>> Network Operations Team
>> Total Uptime Technologies, LLC
>> Support Portal: https://support.totaluptime.com
>>
>> +1 800-584-1514 - TEL (US & Canada)
>> +44 330 808 0228 - TEL (United Kingdom)
>> +1 888-584-1194 - FAX
>>
>> Privileged/Confidential Information may be contained in this message. If
>> you are not the addressee indicated in this message (or responsible for
>> delivery of the message to such person), you may not copy or deliver this
>> message to anyone. In such case, you should destroy this message and kindly
>> notify the sender by reply email. Please advise immediately if you or your
>> employer does not consent to email or messages of this kind. Opinions,
>> conclusions and other information in this message that do not relate to the
>> official business of the firm or its subsidiaries or business units shall
>> be understood as neither given nor endorsed by it.
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>