Mailing List Archive: Spectrum/Charter IPv6 connectivity to CloudFlare down in Texas

Spectrum/Charter IPv6 connectivity to CloudFlare down in Texas

Apr 19, 2023, 4:25 PM

Post #1 of 5 (299 views)

Things have been rather slow the last couple of days, just figured out
why; No IPv6 connectivity to any CloudFlare services.

> mtr -6 --report-wide cloudflare.com
Start: 2023-04-19T23:20:42+0000
HOST: defiant Loss%   Snt   Last   Avg Best Wrst StDev
1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com
0.0%    10    3.6 19.1   1.5 93.2 36.1
2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com
0.0%    10   14.5 36.6 11.3 171.2 50.0
3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0%    10   18.8
37.3 12.0 143.9 45.3
4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0%    10   65.8 30.2
11.7 65.8 24.9
5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0%    10   26.5
25.5 24.5 26.5   1.4
6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0%    10   25.7
25.7 25.7 25.7   0.0
7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0%    10   25.4
26.3 18.2 44.4   9.5
8.|-- ??? 100.0    10    0.0   0.0   0.0   0.0   0.0

IPv4 works however:

> mtr -4 --report-wide cloudflare.com
Start: 2023-04-19T23:20:21+0000
HOST: defiant                                     Loss%   Snt Last
Avg Best Wrst StDev
1.|-- _gateway                                     0.0%    10 2.1
1.8   1.4   2.1   0.2
2.|-- cpe-70-114-160-1.austin.res.rr.com           0.0%    10 51.9
34.4 14.3 149.7 42.1
3.|-- lag-63.hcr01rdrktxaz.netops.charter.com      0.0%    10 13.1
38.5 13.1 159.1 44.6
4.|-- lag-19.ausutxla01r.netops.charter.com        0.0%    10 18.1
21.0 14.3 34.8   6.0
5.|-- lag-22.rcr01dllatx37.netops.charter.com      0.0%    10 23.4
23.3 18.3 33.3   4.3
6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0%    10 19.8
27.0 16.1 51.2 11.9
7.|-- lag-0.pr3.dfw10.netops.charter.com           0.0%    10 39.3
28.5 14.1 62.3 16.7
8.|-- 141.101.74.104                              40.0%    10 53.2
44.3 19.4 120.2 39.2
9.|-- 172.71.168.2                                 0.0%    10 28.3
28.8 10.8 79.4 19.5
10.|-- 104.16.133.229                               0.0%    10 122.0
42.8 16.9 122.0 34.4

It's kind of wild how many things don't fall back to IPv4 quickly (lots
of Android TV apps) or at all (.NET). I tried reaching out to Spectrum
support, but being a lowly consumer I was only told that "The modem
doesn't have any filters" and that "the problem may be on the CloudFlare
side", which was sadly what I predicted. So, I figured I would raise the
issue here in the hopes that actual network people would see it.

-Carl

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: Spectrum/Charter IPv6 connectivity to CloudFlare down in Texas [ In reply to ]

outages at outages

Apr 19, 2023, 8:37 PM

Post #2 of 5 (299 views)

Permalink

Have been dealing with this for 2 days, nobody at Spectrum can figure this out.

Daniel Marks
May Mobility, Inc

>
> On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages@outages.org> wrote:
>
> ?Things have been rather slow the last couple of days, just figured out why; No IPv6 connectivity to any CloudFlare services.
>
> > mtr -6 --report-wide cloudflare.com
> Start: 2023-04-19T23:20:42+0000
> HOST: defiant Loss% Snt Last Avg Best Wrst StDev
> 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0% 10 3.6 19.1 1.5 93.2 36.1
> 2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0% 10 14.5 36.6 11.3 171.2 50.0
> 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8 37.3 12.0 143.9 45.3
> 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2 11.7 65.8 24.9
> 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5 25.5 24.5 26.5 1.4
> 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7 25.7 25.7 25.7 0.0
> 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4 26.3 18.2 44.4 9.5
> 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
>
> IPv4 works however:
>
> > mtr -4 --report-wide cloudflare.com
> Start: 2023-04-19T23:20:21+0000
> HOST: defiant Loss% Snt Last Avg Best Wrst StDev
> 1.|-- _gateway 0.0% 10 2.1 1.8 1.4 2.1 0.2
> 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9 34.4 14.3 149.7 42.1
> 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1 38.5 13.1 159.1 44.6
> 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1 21.0 14.3 34.8 6.0
> 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4 23.3 18.3 33.3 4.3
> 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8 27.0 16.1 51.2 11.9
> 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3 28.5 14.1 62.3 16.7
> 8.|-- 141.101.74.104 40.0% 10 53.2 44.3 19.4 120.2 39.2
> 9.|-- 172.71.168.2 0.0% 10 28.3 28.8 10.8 79.4 19.5
> 10.|-- 104.16.133.229 0.0% 10 122.0 42.8 16.9 122.0 34.4
>
> It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it.
>
> -Carl
>
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outages&source=gmail-imap&ust=1682551587000000&usg=AOvVaw0omaxNXXnf3Ff4dHAQDhA8
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: Spectrum/Charter IPv6 connectivity to CloudFlare down in Texas [ In reply to ]

outages at outages

Apr 20, 2023, 6:05 AM

Post #3 of 5 (299 views)

Permalink

Hey folks,

Could either of you contact me off-list so I can investigate? A specific
start-of-problem would be super helpful.
--
Tom Strickx
Principal Network Engineer
AS13335 - Cloudflare

On Thu, Apr 20, 2023 at 4:38?AM Daniel Marks via Outages <
outages@outages.org> wrote:

> Have been dealing with this for 2 days, nobody at Spectrum can figure this
> out.
>
> Daniel Marks
> May Mobility, Inc
>
> >
> > On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages@outages.org>
> wrote:
> >
> > ?Things have been rather slow the last couple of days, just figured out
> why; No IPv6 connectivity to any CloudFlare services.
> >
> > > mtr -6 --report-wide cloudflare.com
> > Start: 2023-04-19T23:20:42+0000
> > HOST: defiant Loss% Snt Last Avg Best Wrst StDev
> > 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0%
> 10 3.6 19.1 1.5 93.2 36.1
> > 2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0%
> 10 14.5 36.6 11.3 171.2 50.0
> > 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8
> 37.3 12.0 143.9 45.3
> > 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2
> 11.7 65.8 24.9
> > 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5
> 25.5 24.5 26.5 1.4
> > 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7
> 25.7 25.7 25.7 0.0
> > 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4
> 26.3 18.2 44.4 9.5
> > 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
> >
> > IPv4 works however:
> >
> > > mtr -4 --report-wide cloudflare.com
> > Start: 2023-04-19T23:20:21+0000
> > HOST: defiant Loss% Snt Last
> Avg Best Wrst StDev
> > 1.|-- _gateway 0.0% 10 2.1
> 1.8 1.4 2.1 0.2
> > 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9
> 34.4 14.3 149.7 42.1
> > 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1
> 38.5 13.1 159.1 44.6
> > 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1
> 21.0 14.3 34.8 6.0
> > 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4
> 23.3 18.3 33.3 4.3
> > 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8
> 27.0 16.1 51.2 11.9
> > 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3
> 28.5 14.1 62.3 16.7
> > 8.|-- 141.101.74.104 40.0% 10 53.2
> 44.3 19.4 120.2 39.2
> > 9.|-- 172.71.168.2 0.0% 10 28.3
> 28.8 10.8 79.4 19.5
> > 10.|-- 104.16.133.229 0.0% 10 122.0
> 42.8 16.9 122.0 34.4
> >
> > It's kind of wild how many things don't fall back to IPv4 quickly (lots
> of Android TV apps) or at all (.NET). I tried reaching out to Spectrum
> support, but being a lowly consumer I was only told that "The modem doesn't
> have any filters" and that "the problem may be on the CloudFlare side",
> which was sadly what I predicted. So, I figured I would raise the issue
> here in the hopes that actual network people would see it.
> >
> > -Carl
> >
> >
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> >
> https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outages&source=gmail-imap&ust=1682551587000000&usg=AOvVaw0omaxNXXnf3Ff4dHAQDhA8
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

Re: Spectrum/Charter IPv6 connectivity to CloudFlare down in Texas [ In reply to ]

outages at outages

Apr 20, 2023, 8:18 AM

Post #4 of 5 (299 views)

Permalink

Charter has acknowledged the issue, and is tracking this internally
as INC000026440632. Appears to be an issue with a linecard in Houston.

We've rerouted IPv6 out of Dallas for now to mitigate impact. Sorry for the
trouble!
--
Tom Strickx
Principal Network Engineer
AS13335 - Cloudflare

On Thu, Apr 20, 2023 at 2:05?PM Tom Strickx <tstrickx@cloudflare.com> wrote:

> Hey folks,
>
> Could either of you contact me off-list so I can investigate? A specific
> start-of-problem would be super helpful.
> --
> Tom Strickx
> Principal Network Engineer
> AS13335 - Cloudflare
>
>
> On Thu, Apr 20, 2023 at 4:38?AM Daniel Marks via Outages <
> outages@outages.org> wrote:
>
>> Have been dealing with this for 2 days, nobody at Spectrum can figure
>> this out.
>>
>> Daniel Marks
>> May Mobility, Inc
>>
>> >
>> > On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages@outages.org>
>> wrote:
>> >
>> > ?Things have been rather slow the last couple of days, just figured out
>> why; No IPv6 connectivity to any CloudFlare services.
>> >
>> > > mtr -6 --report-wide cloudflare.com
>> > Start: 2023-04-19T23:20:42+0000
>> > HOST: defiant Loss% Snt Last Avg Best Wrst StDev
>> > 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com
>> 0.0% 10 3.6 19.1 1.5 93.2 36.1
>> > 2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com
>> 0.0% 10 14.5 36.6 11.3 171.2 50.0
>> > 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8
>> 37.3 12.0 143.9 45.3
>> > 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8
>> 30.2 11.7 65.8 24.9
>> > 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5
>> 25.5 24.5 26.5 1.4
>> > 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10
>> 25.7 25.7 25.7 25.7 0.0
>> > 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10
>> 25.4 26.3 18.2 44.4 9.5
>> > 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
>> >
>> > IPv4 works however:
>> >
>> > > mtr -4 --report-wide cloudflare.com
>> > Start: 2023-04-19T23:20:21+0000
>> > HOST: defiant Loss% Snt Last
>> Avg Best Wrst StDev
>> > 1.|-- _gateway 0.0% 10 2.1
>> 1.8 1.4 2.1 0.2
>> > 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9
>> 34.4 14.3 149.7 42.1
>> > 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1
>> 38.5 13.1 159.1 44.6
>> > 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1
>> 21.0 14.3 34.8 6.0
>> > 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4
>> 23.3 18.3 33.3 4.3
>> > 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8
>> 27.0 16.1 51.2 11.9
>> > 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3
>> 28.5 14.1 62.3 16.7
>> > 8.|-- 141.101.74.104 40.0% 10 53.2
>> 44.3 19.4 120.2 39.2
>> > 9.|-- 172.71.168.2 0.0% 10 28.3
>> 28.8 10.8 79.4 19.5
>> > 10.|-- 104.16.133.229 0.0% 10 122.0
>> 42.8 16.9 122.0 34.4
>> >
>> > It's kind of wild how many things don't fall back to IPv4 quickly (lots
>> of Android TV apps) or at all (.NET). I tried reaching out to Spectrum
>> support, but being a lowly consumer I was only told that "The modem doesn't
>> have any filters" and that "the problem may be on the CloudFlare side",
>> which was sadly what I predicted. So, I figured I would raise the issue
>> here in the hopes that actual network people would see it.
>> >
>> > -Carl
>> >
>> >
>> > _______________________________________________
>> > Outages mailing list
>> > Outages@outages.org
>> >
>> https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outages&source=gmail-imap&ust=1682551587000000&usg=AOvVaw0omaxNXXnf3Ff4dHAQDhA8
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>

Re: Spectrum/Charter IPv6 connectivity to CloudFlare down in Texas [ In reply to ]

outages at outages

Apr 20, 2023, 10:10 PM

Post #5 of 5 (299 views)

Permalink

As a follow up, we're still working with Spectrum engineers on a
workaround, IPv6 is still mostly down in the DFW area on Spectrum
outside of the large CDNs (i.e. Quad9 and Telia-bound traffic is still
offline) - their ETA is another 24-48 hours as a line card for Houston
was still on order, according to an engineer I spoke to. Not a great
time for us IPv6-only networks.

On Thu, Apr 20, 2023 at 11:18?AM Tom Strickx <tstrickx@cloudflare.com> wrote:
>
> Charter has acknowledged the issue, and is tracking this internally as INC000026440632. Appears to be an issue with a linecard in Houston.
>
> We've rerouted IPv6 out of Dallas for now to mitigate impact. Sorry for the trouble!
> --
> Tom Strickx
> Principal Network Engineer
> AS13335 - Cloudflare
>
>
> On Thu, Apr 20, 2023 at 2:05?PM Tom Strickx <tstrickx@cloudflare.com> wrote:
>>
>> Hey folks,
>>
>> Could either of you contact me off-list so I can investigate? A specific start-of-problem would be super helpful.
>> --
>> Tom Strickx
>> Principal Network Engineer
>> AS13335 - Cloudflare
>>
>>
>> On Thu, Apr 20, 2023 at 4:38?AM Daniel Marks via Outages <outages@outages.org> wrote:
>>>
>>> Have been dealing with this for 2 days, nobody at Spectrum can figure this out.
>>>
>>> Daniel Marks
>>> May Mobility, Inc
>>>
>>> >
>>> > On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages@outages.org> wrote:
>>> >
>>> > ?Things have been rather slow the last couple of days, just figured out why; No IPv6 connectivity to any CloudFlare services.
>>> >
>>> > > mtr -6 --report-wide cloudflare.com
>>> > Start: 2023-04-19T23:20:42+0000
>>> > HOST: defiant Loss% Snt Last Avg Best Wrst StDev
>>> > 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0% 10 3.6 19.1 1.5 93.2 36.1
>>> > 2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0% 10 14.5 36.6 11.3 171.2 50.0
>>> > 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8 37.3 12.0 143.9 45.3
>>> > 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2 11.7 65.8 24.9
>>> > 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5 25.5 24.5 26.5 1.4
>>> > 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7 25.7 25.7 25.7 0.0
>>> > 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4 26.3 18.2 44.4 9.5
>>> > 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
>>> >
>>> > IPv4 works however:
>>> >
>>> > > mtr -4 --report-wide cloudflare.com
>>> > Start: 2023-04-19T23:20:21+0000
>>> > HOST: defiant Loss% Snt Last Avg Best Wrst StDev
>>> > 1.|-- _gateway 0.0% 10 2.1 1.8 1.4 2.1 0.2
>>> > 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9 34.4 14.3 149.7 42.1
>>> > 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1 38.5 13.1 159.1 44.6
>>> > 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1 21.0 14.3 34.8 6.0
>>> > 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4 23.3 18.3 33.3 4.3
>>> > 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8 27.0 16.1 51.2 11.9
>>> > 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3 28.5 14.1 62.3 16.7
>>> > 8.|-- 141.101.74.104 40.0% 10 53.2 44.3 19.4 120.2 39.2
>>> > 9.|-- 172.71.168.2 0.0% 10 28.3 28.8 10.8 79.4 19.5
>>> > 10.|-- 104.16.133.229 0.0% 10 122.0 42.8 16.9 122.0 34.4
>>> >
>>> > It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it.
>>> >
>>> > -Carl
>>> >
>>> >
>>> > _______________________________________________
>>> > Outages mailing list
>>> > Outages@outages.org
>>> > https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outages&source=gmail-imap&ust=1682551587000000&usg=AOvVaw0omaxNXXnf3Ff4dHAQDhA8
>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages

--
-Dan
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages