Mailing List Archive

Hi,

Is there anyway to intentionally and immediately get on Google's DNS
blacklist in order to avoid similar outages in the future affecting
only IPv6 traffic?
http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_aaaa.txt

Or maybe the smart thing to do is building another ISP controllable
blacklist of broken domains and tell BIND on the caches to return only
A records for blacklisted domains. Or the other way around: only AAAA
records for IPv4 broken/blacklisted domains...

Regards,
Manu

On Tue, Nov 11, 2014 at 3:18 AM, Lorenzo Colitti <lorenzo@google.com> wrote:
> On Sun, Nov 9, 2014 at 8:10 PM, Jeroen Massar <jeroen@massar.ch> wrote:
>>
>> > Another fun question is why folks are relying on PMTUD instead of
>> > adjusting their MTU settings (e.g., via RAs).
>>
>> Because why would anybody want to penalize their INTERNAL network?
>
>
> Lowering the MTU from 1500 to 1280 is only a 1% penalty in throughput. I'd
> argue that that 1% is way less important than the latency penalty.
>
>> Because you can't know if that is always the case.
>
>
> I'm not saying that PMTUD shouldn't work. I'm saying that if you know that
> your Internet connection has an MTU of 1280, setting an MTU of 1500 on your
> host is a bad idea, because you know for sure that you will experience a
> 1-RTT delay every time you talk to a new destination.
>
>> As you work at Google, ever heard of this QUIC protocol that does not
>>
>> use TCP?
>>
>> Maybe you want to ask your colleagues about that :)
>
>
> Does QUIC work from behind your tunnel? If so, maybe my colleagues have
> already solved that problem.
>
>>
>> > (Some parts of) Google infrastructure do not do
>> > PMTUD for the latency reasons above and for reasons similar to those
>> > listed
>> > in https://tools.ietf.org/html/draft-v6ops-jaeggli-pmtud-ecmp-problem-00
>> > .
>>
>> As such, you are ON PURPOSE breaking PMTUD, instead trying to fix it
>> with some other bandaid.
>
>
> The draft explains some of the reasons why infrastructure is often built
> this way.

On 2014-11-11 16:00, Emanuel Popa wrote:
> Hi,
>
> Is there anyway to intentionally and immediately get on Google's DNS
> blacklist in order to avoid similar outages in the future affecting
> only IPv6 traffic?
> http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_aaaa.txt
>
> Or maybe the smart thing to do is building another ISP controllable
> blacklist of broken domains and tell BIND on the caches to return only
> A records for blacklisted domains. Or the other way around: only AAAA
> records for IPv4 broken/blacklisted domains...

As most modern clients do Happy Eyeballs, you could just null route the
destination prefixes and see all clients fall back to IPv6.

But it is rather evil to do that especially at an ISP level. Could have
done that for SixXS and give people working stuff that way, but that
would not have actually resolved the problem, just hidden it.

If you expect that they have outages that they cannot quickly see or
not, then you should also expect a blacklist like to be broken or not
properly update. Hence, better to see the problems and to alert the
folks so that they can fix these issues properly (though Google is now
just hacking around with MSS clamping...).


They typically do not have these issues, they just did not notice it
this time around and thus it took a while for them to wake up (timezones
:) figure out what it is and fix the issue.

I am fairly confident though that Google is now monitoring their stuff
correctly. Lots of good folks there, stuff breaks, they fix it.

Greets,
Jeroen

On 11/11/2014 15:00, Emanuel Popa wrote:
> Is there anyway to intentionally and immediately get on Google's DNS
> blacklist in order to avoid similar outages in the future affecting
> only IPv6 traffic?
> http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_aaaa.txt
>
> Or maybe the smart thing to do is building another ISP controllable
> blacklist of broken domains and tell BIND on the caches to return only
> A records for blacklisted domains. Or the other way around: only AAAA
> records for IPv4 broken/blacklisted domains...

... or alternatively, depend on Google, Akamai and others not breaking.
This is what we do for ipv4 and it normally works well, but not always.

Bear it in mind that every time a hack is installed to work around a
potential future problem, that hack needs maintenance and if it breaks,
there's a chance that the resulting damage will be at least as bad as what
it was seeking to avoid in the first place. Unless there are persistent
reliability problems, hacks tend not to be worth it.

Nick

On Tue, Nov 11, 2014 at 3:36 PM, Jeroen Massar <jeroen@massar.ch> wrote:

> If you expect that they have outages that they cannot quickly see or
> not, then you should also expect a blacklist like to be broken or not
> properly update. Hence, better to see the problems and to alert the
> folks so that they can fix these issues properly (though Google is now
> just hacking around with MSS clamping...).
>

[de-cloak]

Google has been doing MSS clamping for a long time, I've seen this myself
in packet captures and Lorenzo also confirmed in his email:
"...some Google servers temporarily stopped doing MSS clamping."

They do it for a good reason: to prevent PMTUD as it introduces delay and
their customers (eyeballs) wouldn't like it. Lorenzo and others explained
this too several times.

The world is not spinning around sixxs and your design ideas. Please turn
off write-only.

Andras

[cloak]

Philipp Kern schreef op 10-11-2014 om 21:09:
> Depends on the RTT. In mobile networks it generally sucks.

A typical RTT om 3G mobile is about 100ms. Would a user really notice?

Google.com's TTL on DNS is 300s. That is also another RTT every 5
minutes. ;)

Dave

On Tue, Nov 11, 2014 at 8:30 AM, gboonie <gboonie@gmail.com> wrote:
>
> Philipp Kern schreef op 10-11-2014 om 21:09:
>>
>> Depends on the RTT. In mobile networks it generally sucks.
>
>
> A typical RTT om 3G mobile is about 100ms. Would a user really notice?

Empirically: yes, but not necessarily at a conscious level.

http://perspectives.mvdirona.com/2009/10/31/TheCostOfLatency.aspx

On 2014-11-11 19:09, Andras Toth wrote:
> On Tue, Nov 11, 2014 at 3:36 PM, Jeroen Massar <jeroen@massar.ch
> <mailto:jeroen@massar.ch>> wrote:
>
> If you expect that they have outages that they cannot quickly see or
> not, then you should also expect a blacklist like to be broken or not
> properly update. Hence, better to see the problems and to alert the
> folks so that they can fix these issues properly (though Google is now
> just hacking around with MSS clamping...).
>
>
> [de-cloak]
>
> Google has been doing MSS clamping for a long time, I've seen this
> myself in packet captures and Lorenzo also confirmed in his email:
> "...some Google servers temporarily stopped doing MSS clamping."
>
> They do it for a good reason: to prevent PMTUD as it introduces delay
> and their customers (eyeballs) wouldn't like it. Lorenzo and others
> explained this too several times.

That "explanation" was seen, at least for me, the first time in this thread.

As stated, the MSS clamping is just hiding the real problems. It does
not properly resolve anything.

> The world is not spinning around sixxs and your design ideas. Please
> turn off write-only.

Wow, here come the ad-hominem attacks, stay in lurk mode if you can't
handle people raising issues. If I had not commented about this problem,
it would never have come to light... maybe in several years when nothing
could have been done anymore. But today, we still can fix things.

Please realize that the world does have a lot more users than SixXS.

Noting problems and properly fixing them are important.

Greets,
Jeroen

Hoi,

2014-11-11 11:13 GMT-08:00 Jeroen Massar <jeroen@massar.ch>:
> As stated, the MSS clamping is just hiding the real problems. It does
> not properly resolve anything.
You are simply wrong about this statement. MSS clamping effectively
resolves issues with PMTUD by reducing its necessity in the first
place. I think I'm the ninth person to point that out in this thread?

The reason why operators resort to MSS clamping, is because they then
take end to end delivery reliability into their own hands, and have
more control over the flow of their data onto the internet. The "real
problems" you bring up, are almost impossible to address without
explicit cooperation from all affected parties - this is a method that
does not scale, and is not considered a winning strategy by operators
who wish to actually see their packets reach the intended recipient.

> If I had not commented about this problem,
> it would never have come to light... maybe in several years when nothing
> could have been done anymore. But today, we still can fix things.
While it's great that you noticed it, I think it's mendacious to claim
that if you had not commented on the problem, it would not have come
to light. Also, I'd like you to keep in mind that there can be some
significant wall clock time between noticing a problem and completing
its resolution, in a network or server deployment of sufficient size
this can take a while. So saying "they just did not notice it this
time around and thus it took a while for them to wake up (timezones :)
figure out what it is and fix the issue" carries little merit if you
don't actually know what happened, how it got noticed, or how it got
resolved.

> Noting problems and properly fixing them are important.
but but but ... the problem *was* fixed, and whether you like it or
not, it was fixed by restoring the intended behavior of MSS clamping
in the affected Google servers after they had a regression.

It's fine if you want to practice pedantry, and I applaud your
persistence. But you must understand that ipv6-ops being network
engineers by trade will in general resort to "doing that, which
actually gets the job done".

groet,
Pim
--
Pim van Pelt <pim@ipng.nl>
PBVP1-RIPE - http://www.ipng.nl/

On 2014-11-11 20:32, Pim van Pelt wrote:
> Hoi,
>
> 2014-11-11 11:13 GMT-08:00 Jeroen Massar <jeroen@massar.ch>:
>> As stated, the MSS clamping is just hiding the real problems. It does
>> not properly resolve anything.
> You are simply wrong about this statement.

There is nothing wrong with that statement even though you removed the
context.

Lets dissect my sentence then:

"the MSS clamping is just hiding the real problems"

Indeed with MSS clamping stuff "works" for TCP, but with a much lower
MTU. But when you start using anything non-TCP stuff is magically broken.

Hence, you are hiding the problem and do not resolve anything *PROPERLY*.

> MSS clamping effectively
> resolves issues with PMTUD by reducing its necessity in the first
> place. I think I'm the ninth person to point that out in this thread?
>
> The reason why operators resort to MSS clamping, is because they then
> take end to end delivery reliability into their own hands, and have
> more control over the flow of their data onto the internet.

I *FULLY* understand WHY big providers are resorting to it:
it fixes the problem on the short term. And that is great.

Still, it does not *PROPERLY* solve the problems that are in the
network. Which is exactly what I write above.

Next to reducing every link on the Internet to 1280 or whatever magic
MTU the one clamping the MSS thinks the link might be.

> The "real
> problems" you bring up, are almost impossible to address without
> explicit cooperation from all affected parties - this is a method that
> does not scale, and is not considered a winning strategy by operators
> who wish to actually see their packets reach the intended recipient.

But by hiding it, it won't be resolved ever. The IPv6 network is not
that large yet, eyeballs are not everywhere yet. But really, if the
"operators" want to give up already on resolving these kind of issues,
then well, just give up.


But maybe I should suggest another different approach to all the large
network operators:

Collect information on network sources where you do properly
receive ICMPv6 PTBs from and which sources you do not, but where
you can notice that

Google loves doing 1 in 10.000 or whatever connection experiments, thus
not breaking "too much". Such an experiment thus should easily be done
and give quite a lot of empirical data.

And as you have a good ear from all the operators, as well, everybody
wants proper connectivity to the Big G, maybe that could help fixing the
problem by opening everybodies eyes? (Instead of silently keeping them
closed...)


>> If I had not commented about this problem,
>> it would never have come to light... maybe in several years when nothing
>> could have been done anymore. But today, we still can fix things.
>
> While it's great that you noticed it, I think it's mendacious to claim
> that if you had not commented on the problem, it would not have come
> to light.

Of course. There will always be *somebody* who will discover it.

Note that I was not the one who noticed the issues, there where various
folks on both HE.net and SixXS forums complaining about it; I just
brought it to the attention here so that the people who are able to fix
it are able to fix it.


But as it is a hidden thing, it will have taken some time before
somebody was able to get somebody to state that this MSS clamping was
happening in the first place.


> Also, I'd like you to keep in mind that there can be some
> significant wall clock time between noticing a problem and completing
> its resolution, in a network or server deployment of sufficient size
> this can take a while.

I fully accept that and have also stated several times already that it
is great that Google folks have come forward with more information and
that they have resolved the issue.


> So saying "they just did not notice it this
> time around and thus it took a while for them to wake up (timezones :)
> figure out what it is and fix the issue" carries little merit if you
> don't actually know what happened, how it got noticed, or how it got
> resolved.

Unfortunately not everybody can see the insides of a big machine.

It would be great if those big machines would be a bit more open about
problems they are having and how they are attacking those so that other
folks don't have to repeat the same mistakes or are grasping in the dark
about things.

Note, I already wrote "timezones" as I assume that the folks who
actually working on it also sleep once in a while.


>> Noting problems and properly fixing them are important.
> but but but ... the problem *was* fixed, and whether you like it or
> not, it was fixed by restoring the intended behavior of MSS clamping
> in the affected Google servers after they had a regression.

I've already acked several times that the problem has been resolved.
AGAIN: THANK YOU FOR SOLVING THAT.

But: your version of "fix" works for TCP and it uses a suboptimal MTU.

Hence, it is not fully fixed. Yes, it is fixed for the services
affected, but that moment that you start globally using QUIC, QUIC is
not good enough anymore till they implement work-arounds again for the
same issue.

> It's fine if you want to practice pedantry, and I applaud your
> persistence. But you must understand that ipv6-ops being network
> engineers by trade will in general resort to "doing that, which
> actually gets the job done".

Patching up stuff is great, it makes things work indeed.

But lets also think about longer term effects and how to resolve this
problem on the longer term.

Or do you want to get a call again later on for the same issue and not
having a resolution ready?


And for everbody still reading, another suggested long term fix:
http://www.ietf.org/id/draft-massar-v6man-mtu-label-00.txt

But hey, I am just suggesting things so that your work does not call you
out of bed....

Please, lets fix this issue properly.

Greets,
Jeroen

On Sun, 9 Nov 2014 11:10:31 -1000, Lorenzo Colitti wrote:
> On Sat, Nov 8, 2014 at 11:48 PM, Jeroen Massar <jeroen@massar.ch>
> wrote:
>> >> The issue with IPv6 access to Google should now be resolved.
>> Please let
>> >> us know if you're still having problems.
>>
>> The fun question of course is: what/why/how the heck happened?
>>
>
> Another fun question is why folks are relying on PMTUD instead of
> adjusting
> their MTU settings (e.g., via RAs). But relying on PMTUD still
> imposes a
> 1-RTT penalty on every TCP connection to an IPv6 address you haven't
> talked
> to in the last few minutes. Why would you do that to your connection?
<snip>

hmmm almost like I feel for disabling IPv6 when using google, or just
don't use it at all if that's how you look at IPv6.
(with regard to PMTUD issue)


---

------------------------------
Roger Jorgensen | - ROJO9-RIPE
roger@jorgensen.no | - The Future is IPv6
-------------------------------------------------------

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?