Mailing List Archive

On 15/04/15 16:05, Brian Rak wrote:
> We noticed that we're no longer getting AAAA results back for google.com
> when we do queries from a few of our recursive servers (other ones are
> fine).
>
> A bit of searching revealed that a few of our servers are listed here
> http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_aaaa.txt
> (there are 4 listed for AS20473, which are the ones I'm referring to)
>
> However, I can't seem to find any information about why we'd be listed
> there, nor can I find anything telling me how to get delisted.

Yes. They don't provide that info, nor do they provide a way to request
a de-listing AFAIK.

You'll be removed automatically when "the problem" goes away.

There's a lot of discussion in the archives about this, but I believe
that, as far as is known publicly:

1. There's a web-bug in the google search page
2. They correlate IPv6 failures of this against the DNS lookup
3. When the DNS source IP hits a certain threshold of correlatd
failures, they stop serving AAAA records for a time (one week?).

Subjectively it's irritating that Google don't provide info to operators
as to the specifics of the cause, but look at it from their PoV -
there's a lot of info, some of it potentially personal / data-protected,
that they'd have to communicate securely to operators when they ask.

It would be a lot of work for them and I'm somewhat sympathetic on that
basis (although I wasn't when it was happening to us ;o)

My guess: you've got some form of broken IPv6 connectivity talking to
your resolvers; maybe rogue RAs, a tunnel, VPN, etc.

The customers with this problem aren't reporting it because Happy
Eyeballs, but the Google web-bug is detecting it.

We saw a reduction (and eventual end to) our experiences of this when we
finished our native dual-stack deployment *and* when we blacklisted
serving of AAAA to some of our more troublesome netblocks - mainly
remote access VPN users.

We monitor whether google are blacklisting us in our Nagios setup, so we
can see if problems come back.

An alternative might be to steer different classes of users to different
resolver query source IPs (either actual different resolvers, or using
views & multiple IPs). Then, you can see which source IP and thus class
of users is triggering it.

Good luck tracking it; it can be frustrating :o(

Cheers,
Phil

On 4/15/2015 11:28 AM, Phil Mayers wrote:
> On 15/04/15 16:05, Brian Rak wrote:
>> We noticed that we're no longer getting AAAA results back for google.com
>> when we do queries from a few of our recursive servers (other ones are
>> fine).
>>
>> A bit of searching revealed that a few of our servers are listed here
>> http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_aaaa.txt
>> (there are 4 listed for AS20473, which are the ones I'm referring to)
>>
>> However, I can't seem to find any information about why we'd be listed
>> there, nor can I find anything telling me how to get delisted.
>
> Yes. They don't provide that info, nor do they provide a way to
> request a de-listing AFAIK.
>
> You'll be removed automatically when "the problem" goes away.
>
> There's a lot of discussion in the archives about this, but I believe
> that, as far as is known publicly:
>
> 1. There's a web-bug in the google search page
> 2. They correlate IPv6 failures of this against the DNS lookup
> 3. When the DNS source IP hits a certain threshold of correlatd
> failures, they stop serving AAAA records for a time (one week?).
>
> Subjectively it's irritating that Google don't provide info to
> operators as to the specifics of the cause, but look at it from their
> PoV - there's a lot of info, some of it potentially personal /
> data-protected, that they'd have to communicate securely to operators
> when they ask.
>
> It would be a lot of work for them and I'm somewhat sympathetic on
> that basis (although I wasn't when it was happening to us ;o)
>
> My guess: you've got some form of broken IPv6 connectivity talking to
> your resolvers; maybe rogue RAs, a tunnel, VPN, etc.
>
> The customers with this problem aren't reporting it because Happy
> Eyeballs, but the Google web-bug is detecting it.
>
> We saw a reduction (and eventual end to) our experiences of this when
> we finished our native dual-stack deployment *and* when we blacklisted
> serving of AAAA to some of our more troublesome netblocks - mainly
> remote access VPN users.
>
> We monitor whether google are blacklisting us in our Nagios setup, so
> we can see if problems come back.
>
> An alternative might be to steer different classes of users to
> different resolver query source IPs (either actual different
> resolvers, or using views & multiple IPs). Then, you can see which
> source IP and thus class of users is triggering it.
>
> Good luck tracking it; it can be frustrating :o(
>
> Cheers,
> Phil

Well, we're a hosting provider and we have a very large number of users
doing all sorts of crazy things. These particular resolvers are used by
our low cost virtual server platform, so we see many users running a
wide variety of proxy software. It's pretty difficult to prevent people
from breaking their IPv6 connectivity when they have full control over
the machines.

There may be some things that we can do to reduce the number of broken
IPv6 setups.. I just wasn't sure if we'd ever drop off that list
automatically.

And you are not alone... While my employer has deployed a lot of IPv6
internally (still not 100% though), some internal DNS servers are
blacklisted by Google. Probably because a lot of our internal labs (which
are also IPv6-enabled of course) are managed by the engineers using the
lab, so, ending up in less than optimal IPv6 configuration in same case
(when the lab has nothing to do with the IP layer => mine is optimized for
IPv6 :-) )

-éric

On 15/04/15 21:16, "Brian Rak" <brak@choopa.com> wrote:

>On 4/15/2015 11:28 AM, Phil Mayers wrote:
>> On 15/04/15 16:05, Brian Rak wrote:
>>> We noticed that we're no longer getting AAAA results back for
>>>google.com
>>> when we do queries from a few of our recursive servers (other ones are
>>> fine).
>>>
>>> A bit of searching revealed that a few of our servers are listed here
>>> http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_aaaa.txt
>>> (there are 4 listed for AS20473, which are the ones I'm referring to)
>>>
>>> However, I can't seem to find any information about why we'd be listed
>>> there, nor can I find anything telling me how to get delisted.
>>
>> Yes. They don't provide that info, nor do they provide a way to
>> request a de-listing AFAIK.
>>
>> You'll be removed automatically when "the problem" goes away.
>>
>> There's a lot of discussion in the archives about this, but I believe
>> that, as far as is known publicly:
>>
>> 1. There's a web-bug in the google search page
>> 2. They correlate IPv6 failures of this against the DNS lookup
>> 3. When the DNS source IP hits a certain threshold of correlatd
>> failures, they stop serving AAAA records for a time (one week?).
>>
>> Subjectively it's irritating that Google don't provide info to
>> operators as to the specifics of the cause, but look at it from their
>> PoV - there's a lot of info, some of it potentially personal /
>> data-protected, that they'd have to communicate securely to operators
>> when they ask.
>>
>> It would be a lot of work for them and I'm somewhat sympathetic on
>> that basis (although I wasn't when it was happening to us ;o)
>>
>> My guess: you've got some form of broken IPv6 connectivity talking to
>> your resolvers; maybe rogue RAs, a tunnel, VPN, etc.
>>
>> The customers with this problem aren't reporting it because Happy
>> Eyeballs, but the Google web-bug is detecting it.
>>
>> We saw a reduction (and eventual end to) our experiences of this when
>> we finished our native dual-stack deployment *and* when we blacklisted
>> serving of AAAA to some of our more troublesome netblocks - mainly
>> remote access VPN users.
>>
>> We monitor whether google are blacklisting us in our Nagios setup, so
>> we can see if problems come back.
>>
>> An alternative might be to steer different classes of users to
>> different resolver query source IPs (either actual different
>> resolvers, or using views & multiple IPs). Then, you can see which
>> source IP and thus class of users is triggering it.
>>
>> Good luck tracking it; it can be frustrating :o(
>>
>> Cheers,
>> Phil
>
>Well, we're a hosting provider and we have a very large number of users
>doing all sorts of crazy things. These particular resolvers are used by
>our low cost virtual server platform, so we see many users running a
>wide variety of proxy software. It's pretty difficult to prevent people
>from breaking their IPv6 connectivity when they have full control over
>the machines.
>
>There may be some things that we can do to reduce the number of broken
>IPv6 setups.. I just wasn't sure if we'd ever drop off that list
>automatically.
>

I suggest checking if any of your affected users have broken 6to4 setups,
and that you are applying the relevant mitigations in RFC 6343.

MTU size issues and high latency have also both been mentioned as
possible reasons for the mysterious AAAA blacklist.

It has also been said that noc@google.com never replies.

Regards
Brian Carpenter

On 16/04/2015 07:16, Brian Rak wrote:
> On 4/15/2015 11:28 AM, Phil Mayers wrote:
>> On 15/04/15 16:05, Brian Rak wrote:
>>> We noticed that we're no longer getting AAAA results back for google.com
>>> when we do queries from a few of our recursive servers (other ones are
>>> fine).
>>>
>>> A bit of searching revealed that a few of our servers are listed here
>>> http://www.google.com/intl/en_ALL/ipv6/statistics/data/no_aaaa.txt
>>> (there are 4 listed for AS20473, which are the ones I'm referring to)
>>>
>>> However, I can't seem to find any information about why we'd be listed
>>> there, nor can I find anything telling me how to get delisted.
>>
>> Yes. They don't provide that info, nor do they provide a way to request a de-listing AFAIK.
>>
>> You'll be removed automatically when "the problem" goes away.
>>
>> There's a lot of discussion in the archives about this, but I believe that, as far as is known publicly:
>>
>> 1. There's a web-bug in the google search page
>> 2. They correlate IPv6 failures of this against the DNS lookup
>> 3. When the DNS source IP hits a certain threshold of correlatd failures, they stop serving AAAA records for a time (one week?).
>>
>> Subjectively it's irritating that Google don't provide info to operators as to the specifics of the cause, but look at it from
>> their PoV - there's a lot of info, some of it potentially personal / data-protected, that they'd have to communicate securely
>> to operators when they ask.
>>
>> It would be a lot of work for them and I'm somewhat sympathetic on that basis (although I wasn't when it was happening to us ;o)
>>
>> My guess: you've got some form of broken IPv6 connectivity talking to your resolvers; maybe rogue RAs, a tunnel, VPN, etc.
>>
>> The customers with this problem aren't reporting it because Happy Eyeballs, but the Google web-bug is detecting it.
>>
>> We saw a reduction (and eventual end to) our experiences of this when we finished our native dual-stack deployment *and* when
>> we blacklisted serving of AAAA to some of our more troublesome netblocks - mainly remote access VPN users.
>>
>> We monitor whether google are blacklisting us in our Nagios setup, so we can see if problems come back.
>>
>> An alternative might be to steer different classes of users to different resolver query source IPs (either actual different
>> resolvers, or using views & multiple IPs). Then, you can see which source IP and thus class of users is triggering it.
>>
>> Good luck tracking it; it can be frustrating :o(
>>
>> Cheers,
>> Phil
>
> Well, we're a hosting provider and we have a very large number of users doing all sorts of crazy things. These particular
> resolvers are used by our low cost virtual server platform, so we see many users running a wide variety of proxy software. It's
> pretty difficult to prevent people from breaking their IPv6 connectivity when they have full control over the machines.
>
> There may be some things that we can do to reduce the number of broken IPv6 setups.. I just wasn't sure if we'd ever drop off
> that list automatically.
>
>

On Thu, Apr 16, 2015 at 4:56 AM, Brian E Carpenter <
brian.e.carpenter@gmail.com> wrote:

> I suggest checking if any of your affected users have broken 6to4 setups,
> and that you are applying the relevant mitigations in RFC 6343.
>
> MTU size issues and high latency have also both been mentioned as
> possible reasons for the mysterious AAAA blacklist.
>

For the avoidance of mystery: Google performs measurements of IPv6
connectivity and latency on an ongoing basis. The Google DNS servers do not
return AAAA records to DNS resolvers if our measurements indicate that for
users of those resolvers, HTTP/HTTPS access to dual-stack Google services
is substantially worse than to equivalent IPv4-only services. "Worse"
covers both reliability (e.g., failure to load a URL) and latency (e.g.,
IPv6 is 100ms worse than IPv4 because it goes over an ocean). The resolvers
must also have a minimum query volume, which is fairly low.

On 16/04/15 01:57, Lorenzo Colitti wrote:

> For the avoidance of mystery: Google performs measurements of IPv6
> connectivity and latency on an ongoing basis. The Google DNS servers do
> not return AAAA records to DNS resolvers if our measurements indicate
> that for users of those resolvers, HTTP/HTTPS access to dual-stack
> Google services is substantially worse than to equivalent IPv4-only
> services. "Worse" covers both reliability (e.g., failure to load a URL)
> and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over an
> ocean). The resolvers must also have a minimum query volume, which is
> fairly low.

Lorenzo,

Thanks for the response.

Do you know if Google have given any thought as to how long they might
find it necessary to take these measures? Years, indefinitely?

Just curious.

On 4/15/2015 8:57 PM, Lorenzo Colitti wrote:
> On Thu, Apr 16, 2015 at 4:56 AM, Brian E Carpenter
> <brian.e.carpenter@gmail.com <mailto:brian.e.carpenter@gmail.com>> wrote:
>
> I suggest checking if any of your affected users have broken 6to4
> setups,
> and that you are applying the relevant mitigations in RFC 6343.
>
> MTU size issues and high latency have also both been mentioned as
> possible reasons for the mysterious AAAA blacklist.
>
>
> For the avoidance of mystery: Google performs measurements of IPv6
> connectivity and latency on an ongoing basis. The Google DNS servers do
> not return AAAA records to DNS resolvers if our measurements indicate
> that for users of those resolvers, HTTP/HTTPS access to dual-stack
> Google services is substantially worse than to equivalent IPv4-only
> services. "Worse" covers both reliability (e.g., failure to load a URL)
> and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over an
> ocean). The resolvers must also have a minimum query volume, which is
> fairly low.


As it turns out, we have a configuration error that's pushing out a a
default route via radvd to machines that don't have a publicly routable
IPv6 address assigned. I suspect this is at least partially responsible
here.

On Thu, Apr 16, 2015 at 7:41 PM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
> On 16/04/15 01:57, Lorenzo Colitti wrote:
>
>> For the avoidance of mystery: Google performs measurements of IPv6
>> connectivity and latency on an ongoing basis. The Google DNS servers do
>> not return AAAA records to DNS resolvers if our measurements indicate
>> that for users of those resolvers, HTTP/HTTPS access to dual-stack
>> Google services is substantially worse than to equivalent IPv4-only
>> services. "Worse" covers both reliability (e.g., failure to load a URL)
>> and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over an
>> ocean). The resolvers must also have a minimum query volume, which is
>> fairly low.
>
>
> Lorenzo,
>
> Thanks for the response.
>
> Do you know if Google have given any thought as to how long they might find
> it necessary to take these measures? Years, indefinitely?
>
> Just curious.

It seems to keep on finding things, so...

On 17/04/2015 15:17, Erik Kline wrote:
> On Thu, Apr 16, 2015 at 7:41 PM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
>> On 16/04/15 01:57, Lorenzo Colitti wrote:
>>
>>> For the avoidance of mystery: Google performs measurements of IPv6
>>> connectivity and latency on an ongoing basis. The Google DNS servers do
>>> not return AAAA records to DNS resolvers if our measurements indicate
>>> that for users of those resolvers, HTTP/HTTPS access to dual-stack
>>> Google services is substantially worse than to equivalent IPv4-only
>>> services. "Worse" covers both reliability (e.g., failure to load a URL)
>>> and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over an
>>> ocean). The resolvers must also have a minimum query volume, which is
>>> fairly low.
>>
>>
>> Lorenzo,
>>
>> Thanks for the response.
>>
>> Do you know if Google have given any thought as to how long they might find
>> it necessary to take these measures? Years, indefinitely?
>>
>> Just curious.
>
> It seems to keep on finding things, so...

But the incentive is wrong. Forcing users to drop back to IPv4 offers
no incentive to fix the IPv6 problem. The correct incentive would be to
tell an operator that they will be blacklisted unless they fix {X and Y}.

Brian

On Fri, Apr 17, 2015 at 12:28 PM, Brian E Carpenter
<brian.e.carpenter@gmail.com> wrote:
> On 17/04/2015 15:17, Erik Kline wrote:
>> On Thu, Apr 16, 2015 at 7:41 PM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
>>> On 16/04/15 01:57, Lorenzo Colitti wrote:
>>>
>>>> For the avoidance of mystery: Google performs measurements of IPv6
>>>> connectivity and latency on an ongoing basis. The Google DNS servers do
>>>> not return AAAA records to DNS resolvers if our measurements indicate
>>>> that for users of those resolvers, HTTP/HTTPS access to dual-stack
>>>> Google services is substantially worse than to equivalent IPv4-only
>>>> services. "Worse" covers both reliability (e.g., failure to load a URL)
>>>> and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over an
>>>> ocean). The resolvers must also have a minimum query volume, which is
>>>> fairly low.
>>>
>>>
>>> Lorenzo,
>>>
>>> Thanks for the response.
>>>
>>> Do you know if Google have given any thought as to how long they might find
>>> it necessary to take these measures? Years, indefinitely?
>>>
>>> Just curious.
>>
>> It seems to keep on finding things, so...
>
> But the incentive is wrong. Forcing users to drop back to IPv4 offers
> no incentive to fix the IPv6 problem. The correct incentive would be to
> tell an operator that they will be blacklisted unless they fix {X and Y}.

We almost never know what X or Y are. We only detect that there
appears to be a problem.

Hi,

On 4/17/2015 6:45 AM, Erik Kline wrote:
> On Fri, Apr 17, 2015 at 12:28 PM, Brian E Carpenter
>> But the incentive is wrong. Forcing users to drop back to IPv4 offers
>> no incentive to fix the IPv6 problem. The correct incentive would be to
>> tell an operator that they will be blacklisted unless they fix {X and Y}.
>
> We almost never know what X or Y are. We only detect that there
> appears to be a problem.

maybe just a short email to the whois contacts (some really read them)
"we have noticed something wrong with your IPv6, we're now not giving
AAAA to these resolvers: x.x.x.x/y" and maybe a link to a test site...

without that you're lowering chances of it being fixed. with that email
you increase them.

Frank

On 4/16/15 10:22 PM, Frank Habicht wrote:
> Hi,
>
> On 4/17/2015 6:45 AM, Erik Kline wrote:
>> On Fri, Apr 17, 2015 at 12:28 PM, Brian E Carpenter
>>> But the incentive is wrong. Forcing users to drop back to IPv4 offers
>>> no incentive to fix the IPv6 problem. The correct incentive would be to
>>> tell an operator that they will be blacklisted unless they fix {X and Y}.
>>
>> We almost never know what X or Y are. We only detect that there
>> appears to be a problem.
>
> maybe just a short email to the whois contacts (some really read them)
> "we have noticed something wrong with your IPv6, we're now not giving
> AAAA to these resolvers: x.x.x.x/y" and maybe a link to a test site...
>
> without that you're lowering chances of it being fixed. with that email
> you increase them.

They also increase chances of the operator trying drag Google into an
IPv6 helpdesk role.

I don't like Google's policy on this, and I'm not sure it's what I would
do, but I see their point.

Doug

--
I am conducting an experiment in the efficacy of PGP/MIME signatures.
This message should be signed. If it is not, or the signature does not
validate, please let me know how you received this message (direct, or
to a list) and the mail software you use. Thanks!

And in the meantime, accept that the users of that operator's network
cannot reliably reach our services?

If you were a user of that operator, I suspect you wouldn't like that. I
suspect you especially wouldn't like it if you called the operator and they
told you there were no problems, and most websites work fine.
Unfortunately, in our experience, both happen routinely. Often operators
will contact us and claim there is no problem in the network, and most of
the time it turns out that there was a problem they didn't know about. Once
the claim was made that "this is an IPv6-only network, so IPv6 must be
working". Unfortunately that wasn't true either.

If an operator is monitoring IPv6 traffic levels, it will be pretty clear
if Google stops serving AAAA records to their resolvers. If they're not
monitoring IPv6 traffic levels, then chances are they're not monitoring
reliability, because it's much easier to monitor traffic than to monitor
reliability.

There's also the question of how whether it's reasonable to expect websites
to to reduce the reliability of their services in order to fix problems in
other networks that they have no control over. Remember, IPv6 brokenness
was one of the main reasons it took so long for popular websites to enable
IPv6.

On Fri, Apr 17, 2015 at 12:28 PM, Brian E Carpenter <
brian.e.carpenter@gmail.com> wrote:

> On 17/04/2015 15:17, Erik Kline wrote:
> > On Thu, Apr 16, 2015 at 7:41 PM, Phil Mayers <p.mayers@imperial.ac.uk>
> wrote:
> >> On 16/04/15 01:57, Lorenzo Colitti wrote:
> >>
> >>> For the avoidance of mystery: Google performs measurements of IPv6
> >>> connectivity and latency on an ongoing basis. The Google DNS servers do
> >>> not return AAAA records to DNS resolvers if our measurements indicate
> >>> that for users of those resolvers, HTTP/HTTPS access to dual-stack
> >>> Google services is substantially worse than to equivalent IPv4-only
> >>> services. "Worse" covers both reliability (e.g., failure to load a URL)
> >>> and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over
> an
> >>> ocean). The resolvers must also have a minimum query volume, which is
> >>> fairly low.
> >>
> >>
> >> Lorenzo,
> >>
> >> Thanks for the response.
> >>
> >> Do you know if Google have given any thought as to how long they might
> find
> >> it necessary to take these measures? Years, indefinitely?
> >>
> >> Just curious.
> >
> > It seems to keep on finding things, so...
>
> But the incentive is wrong. Forcing users to drop back to IPv4 offers
> no incentive to fix the IPv6 problem. The correct incentive would be to
> tell an operator that they will be blacklisted unless they fix {X and Y}.
>
> Brian
>

You are right, it can not be Google's responsibility to fix other networks.

That being said, given initial and then monthly warnings, including a link
to a FAQ, from noreply@ to whois contacts, you would most likely help
increase IPv6 adoption.

Richard

Sent by mobile; excuse my brevity.

Hi,

On Thu, Apr 16, 2015 at 10:56:38PM -0700, Doug Barton wrote:
> On 4/16/15 10:22 PM, Frank Habicht wrote:
> >Hi,
> >
> >On 4/17/2015 6:45 AM, Erik Kline wrote:
> >>On Fri, Apr 17, 2015 at 12:28 PM, Brian E Carpenter
> >>>But the incentive is wrong. Forcing users to drop back to IPv4 offers
> >>>no incentive to fix the IPv6 problem. The correct incentive would be to
> >>>tell an operator that they will be blacklisted unless they fix {X and Y}.
> >>
> >>We almost never know what X or Y are. We only detect that there
> >>appears to be a problem.
> >
> >maybe just a short email to the whois contacts (some really read them)
> >"we have noticed something wrong with your IPv6, we're now not giving
> >AAAA to these resolvers: x.x.x.x/y" and maybe a link to a test site...
> >
> >without that you're lowering chances of it being fixed. with that email
> >you increase them.
>
> They also increase chances of the operator trying drag Google into an IPv6
> helpdesk role.
>
> I don't like Google's policy on this, and I'm not sure it's what I would do,
> but I see their point.

Well, as far as I (as an outsider) can detect: Google does whatever
can be formulated as algorithms or heuristics and given to their
machines, and nothing else (or at least they don't admit it unless
forced in court).

Google's business model wouldn't scale to their number of peers* if
they had individual human support.

Their AAAA serving policy is consistent with this rule. If they
would handle it through auto-generated e-mails, they'd land on some
blacklist sooner or later; if they'd try to phone the responsibles,
that'd consume too much manpower.

Not much that anybody can do about that. So if you're a knowledgable
peer detects "no AAAA", searching for IPv6 problems on your side
or your connectivity provider's is what you should do; and if you
don't, you'll get connectivity by v4.

I suspect Google will change their rule to be protocol independent
in the future... if v6 is better than v4 for you, they won't give
out A records to your resolver ;-)

-is

*) most don't pay them, so they're not customers.

On 17/04/15 07:24, Lorenzo Colitti wrote:
> Often operators will contact us and claim there is no problem in the
> network, and most of the time it turns out that there was a problem they
> didn't know about

I think it's the "know about" that's the problem. From experience, it
can be both baffling and frustrating. We sort of had to do a binary
search through guessed-at sets of networks before we found our
troublesome ones.

Does anyone know of tooling that last-mile providers can run, that can
detect this kind of v4/v6 differential breakage, e.g. by correlating
DNS/netflow or some other speculative mechanism? Presumably evil HTTP
injection isn't going to work as TLS becomes more widespread.

I might have asked this before, so forgive the repetition if so: have
Google considered making more detail available to operators of the
triggering events? I expect the answer is no, and I totally understand
the reasons - the data protection implications alone are tough - but
it's worth asking ;o)

Is it the case that "happy eyeballs" does not cover up these problems
sufficiently, justifying additional server-side measures? Or something
else I haven't considered?

Cheers,
Phil

On Thu, Apr 16, 2015 at 11:24 PM, Lorenzo Colitti <lorenzo@google.com>
wrote:

> And in the meantime, accept that the users of that operator's network
> cannot reliably reach our services?
>
> If you were a user of that operator, I suspect you wouldn't like that. I
> suspect you especially wouldn't like it if you called the operator and they
> told you there were no problems, and most websites work fine.
> Unfortunately, in our experience, both happen routinely. Often operators
> will contact us and claim there is no problem in the network, and most of
> the time it turns out that there was a problem they didn't know about. Once
> the claim was made that "this is an IPv6-only network, so IPv6 must be
> working". Unfortunately that wasn't true either.
>
> If an operator is monitoring IPv6 traffic levels, it will be pretty clear
> if Google stops serving AAAA records to their resolvers. If they're not
> monitoring IPv6 traffic levels, then chances are they're not monitoring
> reliability, because it's much easier to monitor traffic than to monitor
> reliability.
>
> There's also the question of how whether it's reasonable to expect
> websites to to reduce the reliability of their services in order to fix
> problems in other networks that they have no control over. Remember, IPv6
> brokenness was one of the main reasons it took so long for popular websites
> to enable IPv6.
>
>

I agree with Google's approach for now.

But eventually it will have to be re-visited since Google represents a huge
amount of traffic, pulling back AAAA and sending that huge amount of
traffic to a CGN that is not dimension for it.... you are going to have a
bad time.

And, AFAIK, these measurements and adjustments are not real-time... so they
blow up a CGN ... they wont automagically roll back for a while. So,
Google AAAA magic becomes a DDoS of sorts.


Maybe i am wrong.

CB



> On Fri, Apr 17, 2015 at 12:28 PM, Brian E Carpenter <
> brian.e.carpenter@gmail.com> wrote:
>
>> On 17/04/2015 15:17, Erik Kline wrote:
>> > On Thu, Apr 16, 2015 at 7:41 PM, Phil Mayers <p.mayers@imperial.ac.uk>
>> wrote:
>> >> On 16/04/15 01:57, Lorenzo Colitti wrote:
>> >>
>> >>> For the avoidance of mystery: Google performs measurements of IPv6
>> >>> connectivity and latency on an ongoing basis. The Google DNS servers
>> do
>> >>> not return AAAA records to DNS resolvers if our measurements indicate
>> >>> that for users of those resolvers, HTTP/HTTPS access to dual-stack
>> >>> Google services is substantially worse than to equivalent IPv4-only
>> >>> services. "Worse" covers both reliability (e.g., failure to load a
>> URL)
>> >>> and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over
>> an
>> >>> ocean). The resolvers must also have a minimum query volume, which is
>> >>> fairly low.
>> >>
>> >>
>> >> Lorenzo,
>> >>
>> >> Thanks for the response.
>> >>
>> >> Do you know if Google have given any thought as to how long they might
>> find
>> >> it necessary to take these measures? Years, indefinitely?
>> >>
>> >> Just curious.
>> >
>> > It seems to keep on finding things, so...
>>
>> But the incentive is wrong. Forcing users to drop back to IPv4 offers
>> no incentive to fix the IPv6 problem. The correct incentive would be to
>> tell an operator that they will be blacklisted unless they fix {X and Y}.
>>
>> Brian
>>
>
>

On 4/17/15 07:55, Ca By wrote:
>
>
> On Thu, Apr 16, 2015 at 11:24 PM, Lorenzo Colitti <lorenzo@google.com
> <mailto:lorenzo@google.com>> wrote:
>
> And in the meantime, accept that the users of that operator's
> network cannot reliably reach our services?
>
> If you were a user of that operator, I suspect you wouldn't like
> that. I suspect you especially wouldn't like it if you called the
> operator and they told you there were no problems, and most websites
> work fine. Unfortunately, in our experience, both happen routinely.
> Often operators will contact us and claim there is no problem in the
> network, and most of the time it turns out that there was a problem
> they didn't know about. Once the claim was made that "this is an
> IPv6-only network, so IPv6 must be working". Unfortunately that
> wasn't true either.
>
> If an operator is monitoring IPv6 traffic levels, it will be pretty
> clear if Google stops serving AAAA records to their resolvers. If
> they're not monitoring IPv6 traffic levels, then chances are they're
> not monitoring reliability, because it's much easier to monitor
> traffic than to monitor reliability.
>
> There's also the question of how whether it's reasonable to expect
> websites to to reduce the reliability of their services in order to
> fix problems in other networks that they have no control over.
> Remember, IPv6 brokenness was one of the main reasons it took so
> long for popular websites to enable IPv6.
>
>
>
> I agree with Google's approach for now.
>
> But eventually it will have to be re-visited since Google represents a
> huge amount of traffic, pulling back AAAA and sending that huge amount
> of traffic to a CGN that is not dimension for it.... you are going to
> have a bad time.
>
> And, AFAIK, these measurements and adjustments are not real-time... so
> they blow up a CGN ... they wont automagically roll back for a while.
> So, Google AAAA magic becomes a DDoS of sorts.

Speculation weakens your argument.

I'd like to point out that the microscope Lorenzo and Erik were under to
justify, implement, and instrument IPv6 inside Google was tremendous.
Very few people would've put up with all the roadblocks, and IPv6 would
be much less further along industry-wide, without those two dealing with
vast numbers of vendor bugs, poorly thought out protocol features, and
writing much of the code to get things going themselves. I'm proud to
say I helped Lorenzo get his IPv6 pilot going, but then I looked away
and when I looked back it was a train hurtling along. I doubt very much
the continued success of IPv6 will cause Google to start doing stupid
things.

Neither Lorenzo nor Erik should ever have to buy their own beer ever
again IMO.

-Scott


>
>
> Maybe i am wrong.
>
> CB
>
> On Fri, Apr 17, 2015 at 12:28 PM, Brian E Carpenter
> <brian.e.carpenter@gmail.com <mailto:brian.e.carpenter@gmail.com>>
> wrote:
>
> On 17/04/2015 15:17, Erik Kline wrote:
> > On Thu, Apr 16, 2015 at 7:41 PM, Phil Mayers
> <p.mayers@imperial.ac.uk <mailto:p.mayers@imperial.ac.uk>> wrote:
> >> On 16/04/15 01:57, Lorenzo Colitti wrote:
> >>
> >>> For the avoidance of mystery: Google performs measurements
> of IPv6
> >>> connectivity and latency on an ongoing basis. The Google
> DNS servers do
> >>> not return AAAA records to DNS resolvers if our
> measurements indicate
> >>> that for users of those resolvers, HTTP/HTTPS access to
> dual-stack
> >>> Google services is substantially worse than to equivalent
> IPv4-only
> >>> services. "Worse" covers both reliability (e.g., failure to
> load a URL)
> >>> and latency (e.g., IPv6 is 100ms worse than IPv4 because it
> goes over an
> >>> ocean). The resolvers must also have a minimum query
> volume, which is
> >>> fairly low.
> >>
> >>
> >> Lorenzo,
> >>
> >> Thanks for the response.
> >>
> >> Do you know if Google have given any thought as to how long
> they might find
> >> it necessary to take these measures? Years, indefinitely?
> >>
> >> Just curious.
> >
> > It seems to keep on finding things, so...
>
> But the incentive is wrong. Forcing users to drop back to IPv4
> offers
> no incentive to fix the IPv6 problem. The correct incentive
> would be to
> tell an operator that they will be blacklisted unless they fix
> {X and Y}.
>
> Brian
>
>
>

+1!

Some more info on metrics, cutoff points, etc would help reduce FUD, but I
can't see "your reaction to my problems on my network breaks my CGN setup"
being a very valid argument either way.

It _would_ be interesting to know if Google would do the inverse in case
IPv4 performs badly and IPv6 works well, though.

Richard

Sent by mobile; excuse my brevity.

On 17/04/2015 18:24, Lorenzo Colitti wrote:
> And in the meantime, accept that the users of that operator's network
> cannot reliably reach our services?

I understand the dilemma. But if you don't make it easy for the operator
in question to know that they have a problem, it will definitely not get
fixed. Please think about ways to do that.

Ignatios said:

> I suspect Google will change their rule to be protocol independent
> in the future... if v6 is better than v4 for you, they won't give
> out A records to your resolver ;-)

I like that thought.

Scott said:

> Neither Lorenzo nor Erik should ever have to buy their own beer ever again IMO.

In case there's any doubt: yes, I greatly appreciate your efforts, too. Thanks both.

Brian

>
> If you were a user of that operator, I suspect you wouldn't like that. I
> suspect you especially wouldn't like it if you called the operator and they
> told you there were no problems, and most websites work fine.
> Unfortunately, in our experience, both happen routinely. Often operators
> will contact us and claim there is no problem in the network, and most of
> the time it turns out that there was a problem they didn't know about. Once
> the claim was made that "this is an IPv6-only network, so IPv6 must be
> working". Unfortunately that wasn't true either.
>
> If an operator is monitoring IPv6 traffic levels, it will be pretty clear
> if Google stops serving AAAA records to their resolvers. If they're not
> monitoring IPv6 traffic levels, then chances are they're not monitoring
> reliability, because it's much easier to monitor traffic than to monitor
> reliability.
>
> There's also the question of how whether it's reasonable to expect websites
> to to reduce the reliability of their services in order to fix problems in
> other networks that they have no control over. Remember, IPv6 brokenness
> was one of the main reasons it took so long for popular websites to enable
> IPv6.
>
> On Fri, Apr 17, 2015 at 12:28 PM, Brian E Carpenter <
> brian.e.carpenter@gmail.com> wrote:
>
>> On 17/04/2015 15:17, Erik Kline wrote:
>>> On Thu, Apr 16, 2015 at 7:41 PM, Phil Mayers <p.mayers@imperial.ac.uk>
>> wrote:
>>>> On 16/04/15 01:57, Lorenzo Colitti wrote:
>>>>
>>>>> For the avoidance of mystery: Google performs measurements of IPv6
>>>>> connectivity and latency on an ongoing basis. The Google DNS servers do
>>>>> not return AAAA records to DNS resolvers if our measurements indicate
>>>>> that for users of those resolvers, HTTP/HTTPS access to dual-stack
>>>>> Google services is substantially worse than to equivalent IPv4-only
>>>>> services. "Worse" covers both reliability (e.g., failure to load a URL)
>>>>> and latency (e.g., IPv6 is 100ms worse than IPv4 because it goes over
>> an
>>>>> ocean). The resolvers must also have a minimum query volume, which is
>>>>> fairly low.
>>>>
>>>>
>>>> Lorenzo,
>>>>
>>>> Thanks for the response.
>>>>
>>>> Do you know if Google have given any thought as to how long they might
>> find
>>>> it necessary to take these measures? Years, indefinitely?
>>>>
>>>> Just curious.
>>>
>>> It seems to keep on finding things, so...
>>
>> But the incentive is wrong. Forcing users to drop back to IPv4 offers
>> no incentive to fix the IPv6 problem. The correct incentive would be to
>> tell an operator that they will be blacklisted unless they fix {X and Y}.
>>
>> Brian
>>
>

Lorenzo Colitti schreef op 16-4-2015 om 2:57:
>
> For the avoidance of mystery: Google performs measurements of IPv6
> connectivity and latency on an ongoing basis. The Google DNS servers
> do not return AAAA records to DNS resolvers if our measurements
> indicate that for users of those resolvers, HTTP/HTTPS access to
> dual-stack Google services is substantially worse than to equivalent
> IPv4-only services. "Worse" covers both reliability (e.g., failure to
> load a URL) and latency (e.g., IPv6 is 100ms worse than IPv4 because
> it goes over an ocean). The resolvers must also have a minimum query
> volume, which is fairly low.

A free hint to Google to help the industry:

If a network engineer could prove to have control of the DNS in a
network by adding some record (txt?) that Google would verify, a account
could be created at google which would provide a list of IPs causing the
trouble.

That would help the network team to find the users causing the AAAA
blacklisting and fix the issue. And that would help Google too. A win win?

Or did I misunderstand something?

Dave

What right do you have to know what your users are doing?

What right does Google have to hand over their information to a third party
(you)?

Richard

Sent by mobile; excuse my brevity.

Richard Hartmann schreef op 18-4-2015 om 11:04:
>
> What right do you have to know what your users are doing?
>
> What right does Google have to hand over their information to a third
> party (you)?
>
> Richard
>
> Sent by mobile; excuse my brevity.
>

Good point Richard. I think your argument is valid and probably is a
good reason for google not to share the info.

On the other side I do not think sharing IPs is a privacy concern. Not
with the general public, but with the owner/engineer maintaining a
network. If engineers could get the IPs they could find out what they
have in common without even looking at the traffic.

How is your view on that?

Regards,

Dave

Dave,

I have been to China for two weeks just now. my views on any sharing at all
have become a lot dimmer, recently.

You never know what into this might leak to whom. State, corporate or
private.

Richard

Sent by mobile; excuse my brevity.