Mailing List Archive

Hoi,

I have seen this, and disabled ipv6 in my home network because of it.

Pim

On 19/01/2015 10:06, Ignatios Souvatzis wrote:
> Hi,
>
> there seems to be something that smells of a path mtu problem
> reaching google servers from here[1]... (first-hop of the University's
> external link is a short tunnel). Workaround is blocking google v6 using
> the 4or6 addon.
>
> Has anybody else seen this?

Related https://twitter.com/Prolixium/status/557018191919337474 ?

A

--
Alec Edworthy
A.Edworthy@lboro.ac.uk

On Mon, Jan 19, 2015 at 7:06 PM, Ignatios Souvatzis
<ignatios@cs.uni-bonn.de> wrote:
> Hi,
>
> there seems to be something that smells of a path mtu problem
> reaching google servers from here[1]... (first-hop of the University's
> external link is a short tunnel). Workaround is blocking google v6 using
> the 4or6 addon.
>
> Has anybody else seen this?
>
> -is
>
> [1]:
> inet6num: 2001:638:403::/48
> netname: UNI-BONN

Have you mentioned it to noc@google.com? Knowing which nodes you're
being served from would be helpful, too.

On Mon, Jan 19, 2015 at 10:13:25AM +0000, Alec Edworthy wrote:
>
>
> On 19/01/2015 10:06, Ignatios Souvatzis wrote:
> > Hi,
> >
> > there seems to be something that smells of a path mtu problem
> > reaching google servers from here[1]... (first-hop of the University's
> > external link is a short tunnel). Workaround is blocking google v6 using
> > the 4or6 addon.
> >
> > Has anybody else seen this?
>
> Related https://twitter.com/Prolixium/status/557018191919337474 ?

I had some problems at home, which is not tunneled, but a
less-than-1500-octet-mtu PPPoE DSL line, so - thinking it's tunnels
only ignores an increasing number of native private users.

-is

On Mon, 19 Jan 2015, Ignatios Souvatzis wrote:

> I had some problems at home, which is not tunneled, but a
> less-than-1500-octet-mtu PPPoE DSL line, so - thinking it's tunnels only
> ignores an increasing number of native private users.

Well, PPPoE is a tunnel, but I guess that's splitting hairs.

But regardless, I had trouble this morning with Youtube that might be
related to the same issue.

--
Mikael Abrahamsson email: swmike@swm.pp.se

19.01.2015 13:12, Mikael Abrahamsson написав(ла):
> On Mon, 19 Jan 2015, Ignatios Souvatzis wrote:
>
>> I had some problems at home, which is not tunneled, but a
>> less-than-1500-octet-mtu PPPoE DSL line, so - thinking it's tunnels
>> only ignores an increasing number of native private users.
>
> Well, PPPoE is a tunnel, but I guess that's splitting hairs.
>
> But regardless, I had trouble this morning with Youtube that might be
> related to the same issue.
>


One possible solution to force TCP MSS. For example on my FreeBSD
router at home i used PF's scrub rule for ipv6 tunnel to set
mss to 1220 bytes.

If you use PPPoE as underlying technology, decrease this vale little
bit more - 1212 or less.



--
Sincerely yours,
Artem Viklenko.

On 1/19/15 5:52 AM, Artem Viklenko wrote:
> 19.01.2015 13:12, Mikael Abrahamsson написав(ла):
>> On Mon, 19 Jan 2015, Ignatios Souvatzis wrote:
>>
>>> I had some problems at home, which is not tunneled, but a
>>> less-than-1500-octet-mtu PPPoE DSL line, so - thinking it's tunnels
>>> only ignores an increasing number of native private users.
>>
>> Well, PPPoE is a tunnel, but I guess that's splitting hairs.
>>
>> But regardless, I had trouble this morning with Youtube that might be
>> related to the same issue.
>>
>
>
> One possible solution to force TCP MSS. For example on my FreeBSD
> router at home i used PF's scrub rule for ipv6 tunnel to set
> mss to 1220 bytes.
>
> If you use PPPoE as underlying technology, decrease this vale little
> bit more - 1212 or less.
>
>
>

Min MTU size allowable for ipv6 is 1280 according to RFC. Wouldn't
setting the MSS size to 1212 or 1220 essentially break that RFC as well
(1220+40 < 1280) or does the RFC only explicitly state the MTU has to be
at least 1280?


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org / http://www.ahbl.org

On 19.01.2015 18:42, Brielle Bruns wrote:
> On 1/19/15 5:52 AM, Artem Viklenko wrote:
>> 19.01.2015 13:12, Mikael Abrahamsson написав(ла):
>>> On Mon, 19 Jan 2015, Ignatios Souvatzis wrote:
>>>
>>>> I had some problems at home, which is not tunneled, but a
>>>> less-than-1500-octet-mtu PPPoE DSL line, so - thinking it's tunnels
>>>> only ignores an increasing number of native private users.
>>>
>>> Well, PPPoE is a tunnel, but I guess that's splitting hairs.
>>>
>>> But regardless, I had trouble this morning with Youtube that might be
>>> related to the same issue.
>>>
>>
>>
>> One possible solution to force TCP MSS. For example on my FreeBSD
>> router at home i used PF's scrub rule for ipv6 tunnel to set
>> mss to 1220 bytes.
>>
>> If you use PPPoE as underlying technology, decrease this vale little
>> bit more - 1212 or less.
>>
>>
>>
>
> Min MTU size allowable for ipv6 is 1280 according to RFC. Wouldn't
> setting the MSS size to 1212 or 1220 essentially break that RFC as well
> (1220+40 < 1280) or does the RFC only explicitly state the MTU has to be
> at least 1280?
>
>

I'm not shure about RFC right now. But spending time trying to fugure
out why one or the other host can't be successfully reached using ipv6
finnally I stopped on this value. It may be not optimal, but it works
for me.

And indeed the main cause of this is the blocking ICPMv6 at the far end.


--
Sincerely yours,
Artem Viklenko.

On 20/01/2015 01:52, Artem Viklenko wrote:
> 19.01.2015 13:12, Mikael Abrahamsson написав(ла):
>> On Mon, 19 Jan 2015, Ignatios Souvatzis wrote:
>>
>>> I had some problems at home, which is not tunneled, but a
>>> less-than-1500-octet-mtu PPPoE DSL line, so - thinking it's tunnels
>>> only ignores an increasing number of native private users.
>>
>> Well, PPPoE is a tunnel, but I guess that's splitting hairs.
>>
>> But regardless, I had trouble this morning with Youtube that might be
>> related to the same issue.
>>
>
>
> One possible solution to force TCP MSS. For example on my FreeBSD
> router at home i used PF's scrub rule for ipv6 tunnel to set
> mss to 1220 bytes.
>
> If you use PPPoE as underlying technology, decrease this vale little
> bit more - 1212 or less.
>

netsh int ipv6 set int "12" mtu=1280
seems to help (or its equivalent for your preferred host o/s).
However, I've been seeing lots of problems with Cloudflare-mediated
sites recently and that does seem to be due to faulty MSS negotiation
at their end rather than plain PMTUD fails. Clues welcome...

Brian

No, the minimum MTU is a requirement on links, not hosts. Hosts are always
allowed to send smaller packets than the MTU if they want to. Also, the MSS
includes TCP options, so it might be 1208 on a 1280-byte link.

On Tue, Jan 20, 2015 at 1:42 AM, Brielle Bruns <bruns@2mbit.com> wrote:

> On 1/19/15 5:52 AM, Artem Viklenko wrote:
>
>> 19.01.2015 13:12, Mikael Abrahamsson написав(ла):
>>
>>> On Mon, 19 Jan 2015, Ignatios Souvatzis wrote:
>>>
>>> I had some problems at home, which is not tunneled, but a
>>>> less-than-1500-octet-mtu PPPoE DSL line, so - thinking it's tunnels
>>>> only ignores an increasing number of native private users.
>>>>
>>>
>>> Well, PPPoE is a tunnel, but I guess that's splitting hairs.
>>>
>>> But regardless, I had trouble this morning with Youtube that might be
>>> related to the same issue.
>>>
>>>
>>
>> One possible solution to force TCP MSS. For example on my FreeBSD
>> router at home i used PF's scrub rule for ipv6 tunnel to set
>> mss to 1220 bytes.
>>
>> If you use PPPoE as underlying technology, decrease this vale little
>> bit more - 1212 or less.
>>
>>
>>
>>
> Min MTU size allowable for ipv6 is 1280 according to RFC. Wouldn't
> setting the MSS size to 1212 or 1220 essentially break that RFC as well
> (1220+40 < 1280) or does the RFC only explicitly state the MTU has to be at
> least 1280?
>
>
> --
> Brielle Bruns
> The Summit Open Source Development Group
> http://www.sosdg.org / http://www.ahbl.org
>

On Mon, 19 Jan 2015, Mikael Abrahamsson wrote:

> But regardless, I had trouble this morning with Youtube that might be
> related to the same issue.

I turned off my IPv6 HE.net tunnel yesterday because family was
complaining about Youtube not working. I haven't enabled it again. Other
people who had problems, are things working again now?

It struck me that this should be possible to test on an host. Basically if
the host can just respond to a packet with PTB, then checking if the size
of the packets received changes or not would be interesting. Is anyone
aware of someone implementing this already?

Basic functionality:

Would sniff traffic, send PTB packets for select sessions (configurable),
and then I cold check using tcpdump or similar tool, if the packet sizes
go down after the PTB has been sent.

--
Mikael Abrahamsson email: swmike@swm.pp.se

On Jan 20, Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> I turned off my IPv6 HE.net tunnel yesterday because family was complaining
> about Youtube not working. I haven't enabled it again. Other people who had
> problems, are things working again now?
Yes for me (2a00:1450:400c:c04::93).

--
ciao,
Marco

> No, the minimum MTU is a requirement on links, not hosts. Hosts are
> always allowed to send smaller packets than the MTU if they want
> to. Also, the MSS includes TCP options, so it might be 1208 on a
> 1280-byte link.

There are conflicting RFCs, but the consensus seems to be the MSS is
computed using the minimum size IP and TCP headers (without options).
If a host sends packets with options, it is expected to count those
against the packet size it sends.

http://tools.ietf.org/html/rfc2460#section-8.3

> It struck me that this should be possible to test on an
> host. Basically if the host can just respond to a packet with PTB,
> then checking if the size of the packets received changes or not
> would be interesting. Is anyone aware of someone implementing this
> already?

Yes, the tbit tool does this.

http://icir.org/tbit/

We have implemented parts of tbit in scamper, and in particular
modified the PMTUD test algorithm, and to add IPv6 support. The main
limitation is that it requires IPFW (i.e. freebsd, or an older version
of MacOS X)

http://www.caida.org/~mjl/pubs/measuring-pmtud.pdf
http://www.caida.org/tools/measurement/scamper/

An example test from the past:

http://lists.cluenet.de/pipermail/ipv6-ops/2013-August/009213.html

Matthew

Hi,

On Tue, Jan 20, 2015 at 03:40:23PM +0100, Marco d'Itri wrote:
> On Jan 20, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> > I turned off my IPv6 HE.net tunnel yesterday because family was complaining
> > about Youtube not working. I haven't enabled it again. Other people who had
> > problems, are things working again now?
> Yes for me (2a00:1450:400c:c04::93).

For me (2001:638:403::), Google servers worked again yesterday,
but this was apparently implemented by our address ranges being
put onto Google's AAAA blacklist.

Today it seems to work with AAAA again.

Regards,
-is

On Wed, 21 Jan 2015, Ignatios Souvatzis wrote:

> Hi,
>
> On Tue, Jan 20, 2015 at 03:40:23PM +0100, Marco d'Itri wrote:
>> On Jan 20, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>>> I turned off my IPv6 HE.net tunnel yesterday because family was complaining
>>> about Youtube not working. I haven't enabled it again. Other people who had
>>> problems, are things working again now?
>> Yes for me (2a00:1450:400c:c04::93).
>
> For me (2001:638:403::), Google servers worked again yesterday,
> but this was apparently implemented by our address ranges being
> put onto Google's AAAA blacklist.
>
> Today it seems to work with AAAA again.

I switched my HE tunnel back on again today and everything seems to work
fine. As a data point, my "fullsize" packets from Youtube are IPv6 packets
with length 1408 (as reported by tcpdump in OSX).

Flags [S.], seq 825239677, ack 2182453102, win 28160, options [mss 1420,sackOK,TS val 1317228847 ecr 493983801,nop,wscale 7], length 0

The weird thing is that when I checked it again later, now both my machine
and googles machine are saying MSS 1220 and I'm seeing packets that are
1208. Some examples:

13:48:01.810082 IP6 2001:470:XXX.63869 > 2001:7f8:49:1::c.443: Flags [SEW], seq 4278620568, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 494398939 ecr 0,sackOK,eol], length 0
13:48:01.888071 IP6 2001:7f8:49:1::c.443 > 2001:470:XXXX.63869: Flags [S.], seq 1292334177, ack 4278620569, win 28160, options [mss 1420,sackOK,TS val 1317649207 ecr 494398939,nop,wscale 7], length 0

Then I get:

13:50:28.876356 IP6 2001:470:XX.63894 > 2001:7f8:49:1::c.443: Flags [SEW], seq 2121307480, win 65535, options [mss 1220,nop,wscale 5,nop,nop,TS val 494544092 ecr 0,sackOK,eol], length 0
13:50:28.970365 IP6 2001:7f8:49:1::c.443 > 2001:470:X.63894: Flags [S.], seq 2118432000, ack 2121307481, win 28560, options [mss 1440,sackOK,TS val 1317796289 ecr 494544092,nop,wscale 7], length 0

Notice the difference in MSS between those two machines that claim to have
the same IPv6 address?

$ dig +short AAAA r1.sn-gvopm-vu2e.c.youtube.com.
2001:7f8:49:1::c

--
Mikael Abrahamsson email: swmike@swm.pp.se

I had another problem with Youtube just now, with Chrome+OSX. I saw my
machine sending fragmented UDP-packets to google on 443, which it was not
getting any responses to. On a hunch, I disabled QUIC in Chrome which made
things start working again.

03:20:45.891286 IP6 2001:470:X > 2a02:808:1:100::c: frag (0|1232) 59430 > 443: UDP, length 1350
03:20:45.891286 IP6 2001:470:X > 2a02:808:1:100::c: frag (1232|126)

--
Mikael Abrahamsson email: swmike@swm.pp.se

Thanks for reporting this. What was the effect? Connections just getting
stuck? Was there fallback to IPv4?

Personally I don't understand why everyone behind a manually-configured
tunnel doesn't set the MTU in the RA to the MTU of the tunnel... but that's
not an excuse for things not working.

On Fri, Jan 23, 2015 at 11:45 AM, Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

>
> I had another problem with Youtube just now, with Chrome+OSX. I saw my
> machine sending fragmented UDP-packets to google on 443, which it was not
> getting any responses to. On a hunch, I disabled QUIC in Chrome which made
> things start working again.
>
> 03:20:45.891286 IP6 2001:470:X > 2a02:808:1:100::c: frag (0|1232) 59430 >
> 443: UDP, length 1350
> 03:20:45.891286 IP6 2001:470:X > 2a02:808:1:100::c: frag (1232|126)
>
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
>

The path MTU is available to applications. It seems like a bug in
QUIC or Chrome to not send smaller packets when it can know the packets
will be fragmented. Or just use TCP.

On Fri, Jan 23, 2015 at 11:55:57AM +0900, Lorenzo Colitti wrote:
> Thanks for reporting this. What was the effect? Connections just getting stuck?
> Was there fallback to IPv4?
>
> Personally I don't understand why everyone behind a manually-configured tunnel
> doesn't set the MTU in the RA to the MTU of the tunnel... but that's not an
> excuse for things not working.
>
> On Fri, Jan 23, 2015 at 11:45 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
>
> I had another problem with Youtube just now, with Chrome+OSX. I saw my
> machine sending fragmented UDP-packets to google on 443, which it was not
> getting any responses to. On a hunch, I disabled QUIC in Chrome which made
> things start working again.
>
> 03:20:45.891286 IP6 2001:470:X > 2a02:808:1:100::c: frag (0|1232) 59430 >
> 443: UDP, length 1350
> 03:20:45.891286 IP6 2001:470:X > 2a02:808:1:100::c: frag (1232|126)
>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
>
>

On Fri, 23 Jan 2015, Lorenzo Colitti wrote:

> Thanks for reporting this. What was the effect? Connections just getting
> stuck? Was there fallback to IPv4?

I would get all the CSS, comments etc, the player would start to try to
load the video stream, but there would be no video loaded at all (ie the
progress bar never moved).

> Personally I don't understand why everyone behind a manually-configured
> tunnel doesn't set the MTU in the RA to the MTU of the tunnel... but that's
> not an excuse for things not working.

I did that before, but I can't do it in my Airport Express. I have been
considering changing my IPv6 tunneling to another device.

However, I have no clue why my OSX box thought it couldn't send 1350 byte
sized UDP packets to the Internet. Could Google have sent me PTB=1280 from
this host, so my OSX box thinks it can't send them unfragmented?

Anyone know how I can check in OSX for what destinations it has received
PTB packets and what the PMTU it think it has for these destionations?

--
Mikael Abrahamsson email: swmike@swm.pp.se

On Fri, 23 Jan 2015, Mikael Abrahamsson wrote:

> Anyone know how I can check in OSX for what destinations it has received
> PTB packets and what the PMTU it think it has for these destionations?

Found it:

$ netstat -f inet6 -narlW
Internet6:
Destination Gateway Flags Refs Use Mtu Netif Expire
default fe80::9272:40ff:fe07:cc74%en0 UGc 14 0 1500 en0
...

$ netstat -f inet6 -narlW | grep -v 1500 | grep en0
2620:109:c007:102::5be1:f881 fe80::9272:40ff:fe07:cc74%en0 UGHWIi 2 237 1280 en0
2a00:1450:4005:801::1017 fe80::9272:40ff:fe07:cc74%en0 UGHWIi 1 454 1280 en0
2a00:1450:400f:804::2000 fe80::9272:40ff:fe07:cc74%en0 UGHWIi 2 73 1280 en0
2a00:1450:400f:804::200e fe80::9272:40ff:fe07:cc74%en0 UGHWIi 2 1476 1280 en0
2a00:1450:400f:805::2003 fe80::9272:40ff:fe07:cc74%en0 UGHWIi 1 20 1280 en0
2a00:1450:4010:c07::bd fe80::9272:40ff:fe07:cc74%en0 UGHWIi 1 820 1280 en0

So I guess the problem this time was some Google servers sending me
PTB=1280 and then Chrome not taking this into account when sending UDP
packets when using QUIC, resulting in fragmented IPv6 packets (which works
very badly in real life), and then not handling this situation by doing
fall-back to something else.

--
Mikael Abrahamsson email: swmike@swm.pp.se

* Mikael Abrahamsson <swmike@swm.pp.se>

> So I guess the problem this time was some Google servers sending me
> PTB=1280 and then Chrome not taking this into account when sending
> UDP packets when using QUIC, resulting in fragmented IPv6 packets
> (which works very badly in real life), and then not handling this
> situation by doing fall-back to something else.

I highly doubt that Google would be sending you PTBs. 10 SEK says it's
your tunnel ingress router (i.e., your Airport Express)...

But this could be found out, run "tcpdump -nvi eth0 'icmp6 and ip6[40]
== 2'" in a terminal while reproducing the problem and see what's the
source address of the PTBs?

Tore

On Fri, 23 Jan 2015, Tore Anderson wrote:

> I highly doubt that Google would be sending you PTBs. 10 SEK says it's
> your tunnel ingress router (i.e., your Airport Express)...

I just took for granted that this would be the case because I've been told
that google would be capping their communication at 1280. That's what I
get for taking things for granted.

> But this could be found out, run "tcpdump -nvi eth0 'icmp6 and ip6[40]
> == 2'" in a terminal while reproducing the problem and see what's the
> source address of the PTBs?

08:10:06.394230 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 1240) 2001:470:X:X::1 > 2001:470:28:35e:650d:12ec:4019:6a38: [icmp6 sum ok] ICMP6, packet too big, mtu 1280

You're correct. Wonder why I never noticed this before.

So I guess this explains why I sometimes saw SYN packets coming from my
machine with MSS 1440 and sometimes with 1220, the 1440 was for
destinations by which there was no PTB seen (yet), and 1220 for the ones
where PTB had been seen.

Ok, thanks for helping clearing that up!

--
Mikael Abrahamsson email: swmike@swm.pp.se

Airport Express is setting the IPv6 Tunnel MTU to 1280 in all cases and
it's not configurable, as far as I'm aware.

Andras


On Fri, Jan 23, 2015 at 6:56 AM, Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

> On Fri, 23 Jan 2015, Lorenzo Colitti wrote:
>
> Thanks for reporting this. What was the effect? Connections just getting
>> stuck? Was there fallback to IPv4?
>>
>
> I would get all the CSS, comments etc, the player would start to try to
> load the video stream, but there would be no video loaded at all (ie the
> progress bar never moved).
>
> Personally I don't understand why everyone behind a manually-configured
>> tunnel doesn't set the MTU in the RA to the MTU of the tunnel... but
>> that's
>> not an excuse for things not working.
>>
>
> I did that before, but I can't do it in my Airport Express. I have been
> considering changing my IPv6 tunneling to another device.
>
> However, I have no clue why my OSX box thought it couldn't send 1350 byte
> sized UDP packets to the Internet. Could Google have sent me PTB=1280 from
> this host, so my OSX box thinks it can't send them unfragmented?
>
> Anyone know how I can check in OSX for what destinations it has received
> PTB packets and what the PMTU it think it has for these destionations?
>
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
>