Mailing List Archive

On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:

> Hi,
>
> I’ve discovered, several months ago already, that all the 1&1 web sites
> with IPv6 support enabled are broken, because they filter PMTUD, so any
> residential customer with has a reduced MTU because PPP or any other
> encapsulation/tunnel, etc., is not reaching them.

Do you have an example of a website they host that I can test against?

--
Mikael Abrahamsson email: swmike@swm.pp.se

Hi!

> > I've discovered, several months ago already, that all the 1&1 web sites
> > with IPv6 support enabled are broken, because they filter PMTUD, so any
> > residential customer with has a reduced MTU because PPP or any other
> > encapsulation/tunnel, etc., is not reaching them.
>
> Do you have an example of a website they host that I can test against?

www.corso-kino.de

--
pi@opsec.eu +49 171 3101372 4 years to go !

14 okt. 2016 kl. 12:38 skrev Kurt Jaeger <ipv6-ops@c0mplx.org<mailto:ipv6-ops@c0mplx.org>>:

Hi!

I've discovered, several months ago already, that all the 1&1 web sites
with IPv6 support enabled are broken, because they filter PMTUD, so any
residential customer with has a reduced MTU because PPP or any other
encapsulation/tunnel, etc., is not reaching them.

Do you have an example of a website they host that I can test against?

www.corso-kino.de<http://www.corso-kino.de>

Yes, it fails PTB test

https://ipv6alizer.se?address=http://www.corso-kino.de

/Tobbe



--
pi@opsec.eu<mailto:pi@opsec.eu> +49 171 3101372 4 years to go !

There’re tons of them !



Here are a couple of PMTUD test:


tbit from 2001:df0:4:4000::1:115 to 2001:8d8:1001:238f:3cf1:2223:88f2:c80a
server-mss 1440, result: pmtud-fail
app: http, url: http://diskmakerx.com/
[ 0.009] TX SYN 64 seq = 0:0
[ 0.288] RX SYN/ACK 64 seq = 0:1
[ 0.288] TX 60 seq = 1:1
[ 0.298] TX 233 seq = 1:1(173)
[ 0.577] RX 60 seq = 1:174
[ 0.812] RX 1500 seq = 1:174(1440)
[ 0.812] RX 1500 seq = 1441:174(1440)
[ 0.812] RX 1500 seq = 2881:174(1440)
[ 0.812] RX 69 seq = 4321:174(9)
[ 0.812] RX 1500 seq = 4330:174(1440)
[ 0.812] RX 1500 seq = 5770:174(1440)
[ 0.812] TX PTB 1280 mtu = 1280
[ 0.812] RX 1500 seq = 7210:174(1440)
[ 0.816] RX 1500 seq = 8650:174(1440)
[ 0.822] TX 60 seq = 174:1
[ 0.883] RX 1500 seq = 10090:174(1440)
[ 0.892] RX 1500 seq = 11530:174(1440)
[ 1.651] RX 1500 seq = 1:174(1440)
[ 1.651] TX PTB 1280 mtu = 1280
[ 3.335] RX 1500 seq = 1:174(1440)
[ 3.335] TX PTB 1280 mtu = 1280
[ 6.703] RX 1500 seq = 1:174(1440)
[ 6.703] TX PTB 1280 mtu = 1280
[ 13.439] RX 1500 seq = 1:174(1440)


tbit from 2001:df0:4:4000::1:115 to 2001:8d8:1000:d2ea:95d2:30d0:d4ad:9357
server-mss 1440, result: pmtud-fail
app: http, url: http://www.legalveritas.es/
[ 0.009] TX SYN 64 seq = 0:0
[ 0.285] RX SYN/ACK 64 seq = 0:1
[ 0.285] TX 60 seq = 1:1
[ 0.297] TX 238 seq = 1:1(178)
[ 0.572] RX 60 seq = 1:179
[ 0.810] RX 1492 seq = 1:179(1432)
[ 0.810] TX PTB 1280 mtu = 1280
[ 0.825] RX 1500 seq = 1433:179(1440)
[ 0.825] RX 1500 seq = 2873:179(1440)
[ 0.825] RX 1500 seq = 4313:179(1440)
[ 0.825] RX 1500 seq = 5753:179(1440)
[ 0.825] RX 1500 seq = 7193:179(1440)
[ 0.825] RX 1500 seq = 8633:179(1440)
[ 0.825] RX 1500 seq = 10073:179(1440)
[ 0.825] RX 1500 seq = 11513:179(1440)
[ 0.825] RX 1500 seq = 12953:179(1440)
[ 1.636] RX 1492 seq = 1:179(1432)
[ 1.636] TX PTB 1280 mtu = 1280
[ 3.296] RX 1492 seq = 1:179(1432)
[ 3.296] TX PTB 1280 mtu = 1280
[ 6.616] RX 1492 seq = 1:179(1432)
[ 6.616] TX PTB 1280 mtu = 1280
[ 13.248] RX 1492 seq = 1:179(1432)




Saludos,
Jordi


-----Mensaje original-----
De: <ipv6-ops-bounces+jordi.palet=consulintel.es@lists.cluenet.de> en nombre de Mikael Abrahamsson <swmike@swm.pp.se>
Organización: People's Front Against WWW
Responder a: <swmike@swm.pp.se>
Fecha: viernes, 14 de octubre de 2016, 12:32
Para: JORDI PALET MARTINEZ <jordi.palet@consulintel.es>
CC: <ipv6-ops@lists.cluenet.de>
Asunto: Re: contact with One & One ?

On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:

> Hi,
>
> I’ve discovered, several months ago already, that all the 1&1 web sites
> with IPv6 support enabled are broken, because they filter PMTUD, so any
> residential customer with has a reduced MTU because PPP or any other
> encapsulation/tunnel, etc., is not reaching them.

Do you have an example of a website they host that I can test against?

--
Mikael Abrahamsson email: swmike@swm.pp.se



**********************************************
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.

On Fri, 14 Oct 2016, Kurt Jaeger wrote:

> www.corso-kino.de

Thanks.

If it helps, point them to this website (still in development/beta):

https://ipv6alizer.se/

The result is (verifies what you said):

INFO: server-mss 1440, result: pmtud-fail
ERROR: http://www.corso-kino.de don't listen to PTB

--
Mikael Abrahamsson email: swmike@swm.pp.se

Hi!

> > www.corso-kino.de
>
> Thanks.
>
> If it helps, point them to this website (still in development/beta):
>
> https://ipv6alizer.se/
>
> The result is (verifies what you said):
>
> INFO: server-mss 1440, result: pmtud-fail
> ERROR: http://www.corso-kino.de don't listen to PTB

Thanks. It's just around the corner, and I think I can
get them to open a ticket with 1und1 8-}

--
pi@opsec.eu +49 171 3101372 4 years to go !

I don’t think it will help …

I’ve got several of their customers, several *months* ago, which opened a ticket, and they didn’t get a solution/response …

It may happen that the folks in the ticketing system don’t understand the problem or don’t scale it or whatever …

I think is time to retire happy-eye-balls, it is the only way the people will react to those issues!

That’s why, the ideal will be to have a direct contact with the team that is working on IPv6 …

Saludos,
Jordi


-----Mensaje original-----
De: <ipv6-ops-bounces+jordi.palet=consulintel.es@lists.cluenet.de> en nombre de Kurt Jaeger <ipv6-ops@c0mplx.org>
Responder a: <ipv6-ops@c0mplx.org>
Fecha: viernes, 14 de octubre de 2016, 12:58
Para: Mikael Abrahamsson <swmike@swm.pp.se>
CC: <ipv6-ops@lists.cluenet.de>
Asunto: Re: contact with One & One ?

Hi!

> > www.corso-kino.de
>
> Thanks.
>
> If it helps, point them to this website (still in development/beta):
>
> https://ipv6alizer.se/
>
> The result is (verifies what you said):
>
> INFO: server-mss 1440, result: pmtud-fail
> ERROR: http://www.corso-kino.de don't listen to PTB

Thanks. It's just around the corner, and I think I can
get them to open a ticket with 1und1 8-}

--
pi@opsec.eu +49 171 3101372 4 years to go !





**********************************************
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.

On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:

> I think is time to retire happy-eye-balls, it is the only way the people
> will react to those issues!

Happy eyeballs doesn't solve PMTU blackhole.

So this is actually customer breakage occuring, but I imagine lots of ISPs
are actually doing MSS re-write and/or announcing lower than 1500 MTU on
the customer LAN, so even if a customer has PPPoE with 1492 MTU, they
still won't see this problem.

I have seen swedish authorities websites with same "won't-respond-to-PTB",
no answer there either to fault reports.

--
Mikael Abrahamsson email: swmike@swm.pp.se

At $$$job we run quite a bit of dual stack towards customers as an ISP (mainly PPPoE) - our own public website fails the PTB test and quite honestly we’ve never fixed it. it works for lots of customer/visitors but breaks for others (and they fail back to IPv4) - we thought it was only external tunnel visitors but have found out otherwise… never fully understood what was going on and I keep meaning to look at it ..

NGINX front ends load balanced via anycast … pretty standard Ubuntu 16.04LTS setup on the server side. From what I’ve read it seems to be an ECMP related problem like what CloudFlare published a blog about …

Paul

> On Oct 14, 2016, at 7:45 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:
>
>> I think is time to retire happy-eye-balls, it is the only way the people will react to those issues!
>
> Happy eyeballs doesn't solve PMTU blackhole.
>
> So this is actually customer breakage occuring, but I imagine lots of ISPs are actually doing MSS re-write and/or announcing lower than 1500 MTU on the customer LAN, so even if a customer has PPPoE with 1492 MTU, they still won't see this problem.
>
> I have seen swedish authorities websites with same "won't-respond-to-PTB", no answer there either to fault reports.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se

The issue here is that customers (the ones that browse the broken web sites), don’t know about MTU, ICMP, etc.

So I guess is in your side as the “provider” of the content, who is the interested party in making sure it works for “all” your possible customers.

Up to now, every time I’ve seen this problem was just related to ICMPv6 being filtered, as many folks do in IPv4 …


By the way, interesting article, I didn’t read it before:
https://blog.cloudflare.com/path-mtu-discovery-in-practice/


Saludos,
Jordi


-----Mensaje original-----
De: <ipv6-ops-bounces+jordi.palet=consulintel.es@lists.cluenet.de> en nombre de Paul Stewart <paul@paulstewart.org>
Responder a: <paul@paulstewart.org>
Fecha: viernes, 14 de octubre de 2016, 13:52
Para: Mikael Abrahamsson <swmike@swm.pp.se>
CC: <ipv6-ops@lists.cluenet.de>, JORDI PALET MARTINEZ <jordi.palet@consulintel.es>
Asunto: Re: contact with One & One ?

At $$$job we run quite a bit of dual stack towards customers as an ISP (mainly PPPoE) - our own public website fails the PTB test and quite honestly we’ve never fixed it. it works for lots of customer/visitors but breaks for others (and they fail back to IPv4) - we thought it was only external tunnel visitors but have found out otherwise… never fully understood what was going on and I keep meaning to look at it ..

NGINX front ends load balanced via anycast … pretty standard Ubuntu 16.04LTS setup on the server side. From what I’ve read it seems to be an ECMP related problem like what CloudFlare published a blog about …

Paul

> On Oct 14, 2016, at 7:45 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:
>
>> I think is time to retire happy-eye-balls, it is the only way the people will react to those issues!
>
> Happy eyeballs doesn't solve PMTU blackhole.
>
> So this is actually customer breakage occuring, but I imagine lots of ISPs are actually doing MSS re-write and/or announcing lower than 1500 MTU on the customer LAN, so even if a customer has PPPoE with 1492 MTU, they still won't see this problem.
>
> I have seen swedish authorities websites with same "won't-respond-to-PTB", no answer there either to fault reports.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se






**********************************************
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.

On Fri, 14 Oct 2016, Paul Stewart wrote:

> honestly we’ve never fixed it. it works for lots of customer/visitors
> but breaks for others (and they fail back to IPv4) - we thought it was

Errr, how does this fallback work? I am not aware of any such mechanism.

Happy Eyeballs is done when the SYN+ACK gets back.

--
Mikael Abrahamsson email: swmike@swm.pp.se

Thanks .. I meant to include link to that article - appreciate you doing so :)

We don’t filter ICMPv6 on those servers and the problem we are pretty confident is ECMP related (as per what we learned from the Cloudflare blog) … need to set up some time to look deeper though as internally and on our own servers we’ve never been able to replicate the issue

Cheers,
Paul

> On Oct 14, 2016, at 8:02 AM, JORDI PALET MARTINEZ <jordi.palet@consulintel.es> wrote:
>
> The issue here is that customers (the ones that browse the broken web sites), don’t know about MTU, ICMP, etc.
>
> So I guess is in your side as the “provider” of the content, who is the interested party in making sure it works for “all” your possible customers.
>
> Up to now, every time I’ve seen this problem was just related to ICMPv6 being filtered, as many folks do in IPv4 …
>
>
> By the way, interesting article, I didn’t read it before:
> https://blog.cloudflare.com/path-mtu-discovery-in-practice/
>
>
> Saludos,
> Jordi
>
>
> -----Mensaje original-----
> De: <ipv6-ops-bounces+jordi.palet=consulintel.es@lists.cluenet.de> en nombre de Paul Stewart <paul@paulstewart.org>
> Responder a: <paul@paulstewart.org>
> Fecha: viernes, 14 de octubre de 2016, 13:52
> Para: Mikael Abrahamsson <swmike@swm.pp.se>
> CC: <ipv6-ops@lists.cluenet.de>, JORDI PALET MARTINEZ <jordi.palet@consulintel.es>
> Asunto: Re: contact with One & One ?
>
> At $$$job we run quite a bit of dual stack towards customers as an ISP (mainly PPPoE) - our own public website fails the PTB test and quite honestly we’ve never fixed it. it works for lots of customer/visitors but breaks for others (and they fail back to IPv4) - we thought it was only external tunnel visitors but have found out otherwise… never fully understood what was going on and I keep meaning to look at it ..
>
> NGINX front ends load balanced via anycast … pretty standard Ubuntu 16.04LTS setup on the server side. From what I’ve read it seems to be an ECMP related problem like what CloudFlare published a blog about …
>
> Paul
>
>> On Oct 14, 2016, at 7:45 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:
>>
>>> I think is time to retire happy-eye-balls, it is the only way the people will react to those issues!
>>
>> Happy eyeballs doesn't solve PMTU blackhole.
>>
>> So this is actually customer breakage occuring, but I imagine lots of ISPs are actually doing MSS re-write and/or announcing lower than 1500 MTU on the customer LAN, so even if a customer has PPPoE with 1492 MTU, they still won't see this problem.
>>
>> I have seen swedish authorities websites with same "won't-respond-to-PTB", no answer there either to fault reports.
>>
>> --
>> Mikael Abrahamsson email: swmike@swm.pp.se
>
>
>
>
>
>
> **********************************************
> IPv4 is over
> Are you ready for the new Internet ?
> http://www.consulintel.es
> The IPv6 Company
>
> This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.
>
>
>

You are correct - i misspoke on that … the reported issue from some visitors is site doesn’t load. Sorry for the confusion - need more caffeine this morning :)

> On Oct 14, 2016, at 8:05 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Fri, 14 Oct 2016, Paul Stewart wrote:
>
>> honestly we’ve never fixed it. it works for lots of customer/visitors but breaks for others (and they fail back to IPv4) - we thought it was
>
> Errr, how does this fallback work? I am not aware of any such mechanism.
>
> Happy Eyeballs is done when the SYN+ACK gets back.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se

Right I missed that too, and now reading the article instead of “quick review”, I think the solution is there:

https://github.com/cloudflare/pmtud


Saludos,
Jordi


-----Mensaje original-----
De: <ipv6-ops-bounces+jordi.palet=consulintel.es@lists.cluenet.de> en nombre de Paul Stewart <paul@paulstewart.org>
Responder a: <paul@paulstewart.org>
Fecha: viernes, 14 de octubre de 2016, 14:09
Para: Mikael Abrahamsson <swmike@swm.pp.se>
CC: <ipv6-ops@lists.cluenet.de>, JORDI PALET MARTINEZ <jordi.palet@consulintel.es>
Asunto: Re: contact with One & One ?

You are correct - i misspoke on that … the reported issue from some visitors is site doesn’t load. Sorry for the confusion - need more caffeine this morning :)

> On Oct 14, 2016, at 8:05 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Fri, 14 Oct 2016, Paul Stewart wrote:
>
>> honestly we’ve never fixed it. it works for lots of customer/visitors but breaks for others (and they fail back to IPv4) - we thought it was
>
> Errr, how does this fallback work? I am not aware of any such mechanism.
>
> Happy Eyeballs is done when the SYN+ACK gets back.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se






**********************************************
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.

On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:

> Up to now, every time I’ve seen this problem was just related to ICMPv6
> being filtered, as many folks do in IPv4 …

I know several cases where the problem was that the load balancer didn't
forward the ICMPv6 PTB to the correct host and didn't handle it itself. No
filtering, just bad vendor implementation or "oh, didn't think of that".

That's why I don't like people using the word "filtering", because this
not working isn't always intentional. "Filtering" implies intent.

--
Mikael Abrahamsson email: swmike@swm.pp.se

Yes and my understanding with ECMP on the network side is that this is exactly what’s happening … and that’s what Cloudflare blog entry is referring to as well …

I need to dig into this further - their code on Github for the fix I don’t believe will work in our network architecture… although we are thinking of a redesign on that area so now would be a great chance to fix this too :)

> On Oct 14, 2016, at 8:17 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:
>
>> Up to now, every time I’ve seen this problem was just related to ICMPv6 being filtered, as many folks do in IPv4 …
>
> I know several cases where the problem was that the load balancer didn't forward the ICMPv6 PTB to the correct host and didn't handle it itself. No filtering, just bad vendor implementation or "oh, didn't think of that".
>
> That's why I don't like people using the word "filtering", because this not working isn't always intentional. "Filtering" implies intent.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se