Mailing List Archive

Op 29-4-2016 om 8:30 schreef Mikael Abrahamsson:
> Hi,
>
> Site B which sends all data packets as fragments. This is most likely
> because they have some kind of AFTR where the IPv4 side has MTU1500 and
> the IPv6 side has MTU1320 or something like that.

The site cbs.nl does this as well. It's the statistics agency for the
Netherlands. They use a Juniper with IPv4 to IPv6 translation, however,
it sets the frag attribute for all packets including the ACK.

I've had extensive debugging to find out what was going wrong.
Eventually I found that our firewall was dropping IPv6 fragments, making
the website unreachable over IPv6.

The RFC for this translation mode was followed literally, so it could be
argued that this is "to spec".

Neither Juniper nor the website owner was willing to make any changes
(and make it reachable for anyone that dropped frags, it wasn't just
us). They could have just used a proxy or load balancer to terminate the
connections instead of relying on a passive NAT and not have any of
these problems.

Cheers,

Seth

See also RFC 6946 on this topic and the more controversial
draft-ietf-6man-deprecate-atomfrag-generation

-éric

On 29/04/16 08:43, "ipv6-ops-bounces+evyncke=cisco.com@lists.cluenet.de on
behalf of Seth Mos" <ipv6-ops-bounces+evyncke=cisco.com@lists.cluenet.de
on behalf of seth.mos@dds.nl> wrote:

>Op 29-4-2016 om 8:30 schreef Mikael Abrahamsson:
>> Hi,
>>
>> Site B which sends all data packets as fragments. This is most likely
>> because they have some kind of AFTR where the IPv4 side has MTU1500 and
>> the IPv6 side has MTU1320 or something like that.
>
>The site cbs.nl does this as well. It's the statistics agency for the
>Netherlands. They use a Juniper with IPv4 to IPv6 translation, however,
>it sets the frag attribute for all packets including the ACK.
>
>I've had extensive debugging to find out what was going wrong.
>Eventually I found that our firewall was dropping IPv6 fragments, making
>the website unreachable over IPv6.
>
>The RFC for this translation mode was followed literally, so it could be
>argued that this is "to spec".
>
>Neither Juniper nor the website owner was willing to make any changes
>(and make it reachable for anyone that dropped frags, it wasn't just
>us). They could have just used a proxy or load balancer to terminate the
>connections instead of relying on a passive NAT and not have any of
>these problems.
>
>Cheers,
>
>Seth

Seth,

At 2016-04-29 08:43:09 +0200
Seth Mos <seth.mos@dds.nl> wrote:

> Op 29-4-2016 om 8:30 schreef Mikael Abrahamsson:
> >
> > Site B which sends all data packets as fragments. This is most likely
> > because they have some kind of AFTR where the IPv4 side has MTU1500 and
> > the IPv6 side has MTU1320 or something like that.
>
> The site cbs.nl does this as well. It's the statistics agency for the
> Netherlands. They use a Juniper with IPv4 to IPv6 translation, however,
> it sets the frag attribute for all packets including the ACK.
>
> I've had extensive debugging to find out what was going wrong.
> Eventually I found that our firewall was dropping IPv6 fragments, making
> the website unreachable over IPv6.

Presumably this is because the firewall wants the capability to do DPI
(deep-packet inspection) or something like that, and the cost of
managing fragmented streams is too high?

This is how we end up with every end host setting MTU to 1280.
Otherwise their fragments will get dropped by broken middleboxes. :(

> The RFC for this translation mode was followed literally, so it could be
> argued that this is "to spec".
>
> Neither Juniper nor the website owner was willing to make any changes
> (and make it reachable for anyone that dropped frags, it wasn't just
> us). They could have just used a proxy or load balancer to terminate the
> connections instead of relying on a passive NAT and not have any of
> these problems.

True, true. It's odd behavior, but one that will hopefully become more
rare as native IPv6 becomes more normal.

OTOH, blocking all IPv6 fragments seems a bit too aggressive for
firewalls. I guess if you are only expecting TCP traffic it is probably
okay though, since TCP does PMTU discovery and shouldn't result in
fragmented traffic.

Cheers,

--
Shane

Hi,

On Fri, Apr 29, 2016 at 09:38:50AM +0200, Shane Kerr wrote:
> OTOH, blocking all IPv6 fragments seems a bit too aggressive for
> firewalls.

My guess is more along the lines of "this is on FreeBSD, using the pf(4)
packet filter, which is still not able to do anything reasonable with
IPv6 fragments" (you can permit-all or deny-all, but no reassembly and
no more educated filtering).

OpenBSD fixed that, but FreeBSD changed their networking stack enough
that they cannot just import new versions of pf(4) from OpenBSD anymore
(and it seems nobody really cares *sigh*).

We're using NAT64 techniques to access customer private networks (using
ULAs mapped to their private addresses on the outside) and these atomic
fragments + FreeBSD pf(4) has bitten us as well...

Gert Doering
-- NetMaster
--
have you enabled IPv6 on something today...?

SpaceNet AG Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279

As for testing tools, scamper is the only one I've found when I last looked
(in 2014?) it doesn't
support running on Linux out of the box. http://wand.net.nz/scamper/pmtud
For example, you can do:

sudo scamper -F ipfw -I "tbit -M 1280 -t pmtud -S $sourceip -u $url"
$targetip

Where $url has to be an HTTP URL that returns a large enough object and
$targetip is what you want
to test. On Linux, $sourceip is a dedicated non-primary address where you
have a firewall rule filter:

ip6tables -I INPUT -p tcp --sport 80 -d $sourceip -j DROP

The right way to do this on Linux would be to add a new backend to scamper
but for this to work you have to disable the existing check that BSD ipfw
is working.
For example:

==== CUT HERE ====
--- scamper-cvs-20140530/scamper/tbit/scamper_tbit_do.c 2013-08-07
17:55:29.000000000 -0400
+++ scamper-cvs-mod/scamper/tbit/scamper_tbit_do.c 2014-10-21
14:07:33.053470510 -0400
@@ -2634,11 +2636,26 @@
sfw.sfw_5tuple_sport = tbit->dport;
sfw.sfw_5tuple_dport = tbit->sport;

+
+
+
+
+
+#if 1 /* Hack to run on Linux... */
+ run with:
+ scamper -F ipfw -I "tbit -M 1280 -t pmtud -S $sourceip -u $url"
$targetip
+ after putting in a firewall rule first to drop inputs on a dedicated
address:
+ ip6tables -I INPUT -p tcp --sport 80 -d $sourceip -j DROP
+ */
+
+ state->fw = NULL;
+#else
if((state->fw = scamper_firewall_entry_get(&sfw)) == NULL)
{
scamper_debug(__func__, "could not get firewall entry");
goto err;
}
+#endif

state->mode = MODE_SYN;
}
==== CUT HERE ====















On Fri, Apr 29, 2016 at 2:30 AM, Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

>
> Hi,
>
> I've run into a scenario where a website doesn't seem to be listening to
> PTB. I can reach them just fine from an MTU1500 clean IPv6 connection, but
> if I reach from a MTU1500<->MTU1480<->MTU1500 connection, it doesn't work.
> I don't get the big packets after SYN handshake.
>
> I've been considering asking iis.se (the .SE ccTLD registry) who are
> already running multiple testing tools for web sites and domain name
> owners) to include these kinds of testing, and perhaps develop more of them.
>
> So I'd like to gather some information and feedback here.
>
> 1. Are there are already FOSS tools out there that could be used for this,
> or would be good to enhance to include capability for this kind of testing.
> I don't want to waste work, and if I can enhance FOSS tools already
> existing and also solve my problem, that's a double win.
>
> 2. Test cases? From my testing, I've seen two different behavior just in
> the last two days:
>
> Site A as described in top paragraph, probably doesn't listen to PTB. Can
> be either because they drop PTBs, or traffic traverses a load
> balancer/IPv4v6proxy that doesn't correctly handle PTB.
>
> Site B which sends all data packets as fragments. This is most likely
> because they have some kind of AFTR where the IPv4 side has MTU1500 and the
> IPv6 side has MTU1320 or something like that.
>
> Neither of this is of course optimal, and I'd like to be able to test for
> these and tell the site owner that their solution either is broken or
> suboptimal (the fragment case isn't strictly broken, it's just not a good
> way to do things).
>
> Opinions? Thoughts?
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
>

On Fri, Apr 29, 2016 at 08:30:50AM +0200, Mikael Abrahamsson wrote:
>
> Hi,
>
> I've run into a scenario where a website doesn't seem to be listening to
> PTB. I can reach them just fine from an MTU1500 clean IPv6 connection, but
> if I reach from a MTU1500<->MTU1480<->MTU1500 connection, it doesn't work. I
> don't get the big packets after SYN handshake.
>
> I've been considering asking iis.se (the .SE ccTLD registry) who are already
> running multiple testing tools for web sites and domain name owners) to
> include these kinds of testing, and perhaps develop more of them.

Yes please. Guess what I'm trying to debug right know, and having a
reliable testing point out there would be an immense help.

-is

On 26/05/2016 00:33, Ignatios Souvatzis wrote:
> On Fri, Apr 29, 2016 at 08:30:50AM +0200, Mikael Abrahamsson wrote:
>>
>> Hi,
>>
>> I've run into a scenario where a website doesn't seem to be listening to
>> PTB. I can reach them just fine from an MTU1500 clean IPv6 connection, but
>> if I reach from a MTU1500<->MTU1480<->MTU1500 connection, it doesn't work. I
>> don't get the big packets after SYN handshake.
>>
>> I've been considering asking iis.se (the .SE ccTLD registry) who are already
>> running multiple testing tools for web sites and domain name owners) to
>> include these kinds of testing, and perhaps develop more of them.
>
> Yes please. Guess what I'm trying to debug right know, and having a
> reliable testing point out there would be an immense help.

Make sure you cover MSS negotiation failure as well as PMTUD failure.
I've seen that cause things to stop after the SYN even if the MTU was
reduced correctly.

Brian

On 05/26/16 00:33, Ignatios Souvatzis wrote:
> On Fri, Apr 29, 2016 at 08:30:50AM +0200, Mikael Abrahamsson wrote:
>>
>> Hi,
>>
>> I've run into a scenario where a website doesn't seem to be listening to
>> PTB. I can reach them just fine from an MTU1500 clean IPv6 connection, but
>> if I reach from a MTU1500<->MTU1480<->MTU1500 connection, it doesn't work. I
>> don't get the big packets after SYN handshake.
>>
>> I've been considering asking iis.se (the .SE ccTLD registry) who are already
>> running multiple testing tools for web sites and domain name owners) to
>> include these kinds of testing, and perhaps develop more of them.
>
> Yes please. Guess what I'm trying to debug right know, and having a
> reliable testing point out there would be an immense help.

Try using

http://wand.net.nz/pmtud/webserver.php

one caveat: it assumes http, rather than https. That made sense in
2010. This limitation can be fixed if there is demand.

Matthew

On 05/26/16 00:33, Ignatios Souvatzis wrote:
> On Fri, Apr 29, 2016 at 08:30:50AM +0200, Mikael Abrahamsson wrote:
>>
>> Hi,
>>
>> I've run into a scenario where a website doesn't seem to be listening to
>> PTB. I can reach them just fine from an MTU1500 clean IPv6 connection, but
>> if I reach from a MTU1500<->MTU1480<->MTU1500 connection, it doesn't work. I
>> don't get the big packets after SYN handshake.
>>
>> I've been considering asking iis.se (the .SE ccTLD registry) who are already
>> running multiple testing tools for web sites and domain name owners) to
>> include these kinds of testing, and perhaps develop more of them.
>
> Yes please. Guess what I'm trying to debug right know, and having a
> reliable testing point out there would be an immense help.

We've revived and updated http://wand.net.nz/pmtud/webserver.php

Pass it an http or https URL of an object to test against.

Matthew

On Fri, 27 May 2016, Matthew Luckie wrote:

> Pass it an http or https URL of an object to test against.

This is great! Good work!

tbit from 2001:df0:4:4000::1:115 to 2001:67c:2b24:1000::30
server-mss 1380, result: pmtud-fail
app: http, url: http://www.tullverket.se/
[ 0.010] TX SYN 64 seq = 0:0
[ 0.331] RX SYN/ACK 64 seq = 0:1
[ 0.332] TX 60 seq = 1:1
[ 0.332] TX 236 seq = 1:1(176)
[ 0.652] RX 60 seq = 1:177
[ 0.664] RX 608 seq = 1:177(548) ECT
[ 0.664] RX 970 seq = 549:177(910) ECT
[ 0.664] TX 60 seq = 177:549
[ 0.664] TX 60 seq = 177:1459
[ 0.665] RX 1428 seq = 1459:177(1368) ECT
[ 0.665] RX 299 seq = 2827:177(239) ECT
[ 0.665] TX PTB 1280 mtu = 1280
[ 0.669] TX 60 seq = 177:1459
[ 0.985] RX 1428 seq = 3066:177(1368) ECT
[ 0.985] RX 1428 seq = 4434:177(1368) ECT
[ 0.985] RX 1428 seq = 5802:177(1368) ECT
[ 0.985] RX 1428 seq = 7170:177(1368) ECT
[ 0.990] RX 1428 seq = 8538:177(1368) ECT
[ 1.943] RX 1428 seq = 1459:177(1368)
[ 1.943] TX PTB 1280 mtu = 1280
[ 3.863] RX 1428 seq = 1459:177(1368)
[ 3.863] TX PTB 1280 mtu = 1280
[ 7.704] RX 1428 seq = 1459:177(1368)
[ 7.704] TX PTB 1280 mtu = 1280
[ 15.384] RX RST 60 seq = 9906:177

Here it's clear (to me) that they're not acting on the PTB message (just
keep resending large packtes).

Question is how could this be translated into a more easily understandable
test with some suggestions on what might be wrong? Because if I send the
above to someone who isn't an IPv6 expert, I'd have to include 10-20 lines
of text to explain why above behaviour is wrong.

--
Mikael Abrahamsson email: swmike@swm.pp.se

> 27 maj 2016 kl. 09:46 skrev Mikael Abrahamsson <swmike@swm.pp.se>:
>
> On Fri, 27 May 2016, Matthew Luckie wrote:
>
>> Pass it an http or https URL of an object to test against.
>
> This is great! Good work!


I’m new on the list and I have done some work for test with traceroute with mtu test, icmp6 and http at https://ipv6alizer.se/
Feel free to test and comment, It’s not even an Alfa version and It’s much work to do with the webinterface wuth rate limit and history.

/Tobbe