Mailing List Archive

AS786 - not observed any issues from here in last 24h.

Best regards,
Josh Purcell
Me@JoshPurcell.com<mailto:Me@JoshPurcell.com>

From: Outages <outages-bounces@outages.org> On Behalf Of Justin Krejci via Outages
Sent: 08 February 2022 19:34
To: outages@outages.org
Subject: [outages] Ping to Google 8.8.8.8


Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.

This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these IPs as Internet reachability tests for determining failover to backup circuits and moved away from it after the first occurrence of this happening.

--
Christopher Conley
Systems Administrator | Fors Marsh Group
1010 N. Glebe Rd, Suite 510
Arlington, VA 22201

From: Outages <outages-bounces@outages.org> On Behalf Of Justin Krejci via Outages
Sent: Tuesday, February 8, 2022 2:34 PM
To: outages@outages.org
Subject: [EXTERNAL] [outages] Ping to Google 8.8.8.8


Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.

On Tue, Feb 08, 2022 at 07:33:52PM +0000,
Justin Krejci via Outages <outages@outages.org> wrote
a message of 52 lines which said:

> Seeing some hosts and not others get no reply to ICMP pings from the
> Minneapolis, MN area. I checked using HE's looking glass, 1 of their
> 2 Minneapolis routers gets no replies. Since many use and assume
> 8.8.8.8 is invincible, this has now started setting off monitoring
> alerts. DNS queries so far seem unaffected.

Replying to ICMP echoes is not part of the service Google Public DNS
officially does, so it is possible that they sometimes filter and/or
rate-limit ICMP echo. It's not an outage.
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On 2/8/22 11:33, Justin Krejci via Outages wrote:
> Seeing some hosts and not others get no reply to ICMP pings from the
> Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2
> Minneapolis routers gets no replies. Since many use and assume 8.8.8.8
> is invincible, this has now started setting off monitoring alerts. DNS
> queries so far seem unaffected.

Don't do that then.

Seriously, public DNS servers were never intended to be used as ping
targets used to measure Internet connectivity.

--
Jay Hennigan - jay@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

> On Feb 8, 2022, at 2:49 PM, Stephane Bortzmeyer via Outages <outages@outages.org> wrote:
>
> Replying to ICMP echoes is not part of the service Google Public DNS
> officially does, so it is possible that they sometimes filter and/or
> rate-limit ICMP echo. It's not an outage.

If this goes on much further, we should probably move to -discuss, but I digress. :)

I can confirm I’ve seen this behavior before with 8.8.8.8 - I’ve had times where ICMP echo indicates 100+ms latency to it (whereas 8.8.4.4 is fine), but both will reply to DNS queries with much, much less latency than ICMP echo was indicating. Definitely seems like they’re rate-limiting or de-prioritizing ICMP echo at least on occasion.

One should definitely not use ICMP echo against 8.8.8.8 / 8.8.4.4 in monitoring systems (although I know it happens.) This is a good reminder as to why. :)

Jeremy


_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

ICMPv6 to ipv6.google.com went down for me sometime before 9:44 am. Also
likely going through MICE (which is in the Minneapolis area).



root@nagios:~/bin# traceroute6 ipv6.google.com

traceroute to ipv6.google.com (2607:f8b0:4009:80a::200e), 30 hops max, 80
byte packets

1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.285 ms 0.328 ms 0.385
ms

2 2001:5f8:7f06:d::1 (2001:5f8:7f06:d::1) 0.200 ms 0.203 ms 0.197 ms

3 2001:5f8:7f06:7::1 (2001:5f8:7f06:7::1) 1.731 ms 1.892 ms 1.888 ms

4 AS15169.micemn.net (2001:504:27::3b41:0:1) 18.262 ms 18.284 ms 18.337
ms

5 2001:4860:0:1010::1 (2001:4860:0:1010::1) 17.176 ms * 17.204 ms

6 * 2001:4860:0:1::572b (2001:4860:0:1::572b) 17.191 ms *

7 ord37s34-in-x0e.1e100.net (2607:f8b0:4009:80a::200e) 17.158 ms 17.194
ms *

root@nagios:~/bin# ping6 ipv6.google.com

PING ipv6.google.com(ord30s21-in-x0e.1e100.net) 56 data bytes

^C

--- ipv6.google.com ping statistics ---

6 packets transmitted, 0 received, 100% packet loss, time 5000ms



root@nagios:~/bin#



Frank



From: Outages <outages-bounces@outages.org> On Behalf Of Justin Krejci via
Outages
Sent: Tuesday, February 8, 2022 1:34 PM
To: outages@outages.org
Subject: [outages] Ping to Google 8.8.8.8



Seeing some hosts and not others get no reply to ICMP pings from the
Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2
Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is
invincible, this has now started setting off monitoring alerts. DNS queries
so far seem unaffected.

Is it that atypical for these public-facing services to drop ICMP when they
get hot? I'd not consider it an indicator of service down unless you're
getting SRVFAILs

James Host
Architect - CDN Platform
E: jhost@vultr.com


On Tue, Feb 8, 2022 at 12:01 PM Christopher Conley via Outages <
outages@outages.org> wrote:

> This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these IPs
> as Internet reachability tests for determining failover to backup circuits
> and moved away from it after the first occurrence of this happening.
>
>
>
> --
>
> Christopher Conley
>
> Systems Administrator | Fors Marsh Group
>
> 1010 N. Glebe Rd, Suite 510
>
> Arlington, VA 22201
>
>
>
> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *Justin
> Krejci via Outages
> *Sent:* Tuesday, February 8, 2022 2:34 PM
> *To:* outages@outages.org
> *Subject:* [EXTERNAL] [outages] Ping to Google 8.8.8.8
>
>
>
> Seeing some hosts and not others get no reply to ICMP pings from the
> Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2
> Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is
> invincible, this has now started setting off monitoring alerts. DNS queries
> so far seem unaffected.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

On Tue, 8 Feb 2022 at 19:49, Stephane Bortzmeyer via Outages <
outages@outages.org> wrote:

> Replying to ICMP echoes is not part of the service Google Public DNS
> officially does, so it is possible that they sometimes filter and/or
> rate-limit ICMP echo. It's not an outage.
>

https://peering.google.com/#/learn-more/faq for more info.

M

I didn’t set it up, but I did tear it down.

From: Outages <outages-bounces@outages.org> On Behalf Of James Host via Outages
Sent: Tuesday, February 8, 2022 3:12 PM
To: outages@outages.org
Subject: Re: [outages] [EXTERNAL] Ping to Google 8.8.8.8

Is it that atypical for these public-facing services to drop ICMP when they get hot? I'd not consider it an indicator of service down unless you're getting SRVFAILs

James Host
Architect - CDN Platform
E: jhost@vultr.com<mailto:jhost@vultr.com>


On Tue, Feb 8, 2022 at 12:01 PM Christopher Conley via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these IPs as Internet reachability tests for determining failover to backup circuits and moved away from it after the first occurrence of this happening.

--
Christopher Conley
Systems Administrator | Fors Marsh Group
1010 N. Glebe Rd, Suite 510
Arlington, VA 22201

From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Justin Krejci via Outages
Sent: Tuesday, February 8, 2022 2:34 PM
To: outages@outages.org<mailto:outages@outages.org>
Subject: [EXTERNAL] [outages] Ping to Google 8.8.8.8


Seeing some hosts and not others get no reply to ICMP pings from the Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2 Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is invincible, this has now started setting off monitoring alerts. DNS queries so far seem unaffected.
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages

Yes, it has been this way for at least a decade. DNS are not polling
servers.

- Mike Bolitho

On Tue, Feb 8, 2022, 1:29 PM James Host via Outages <outages@outages.org>
wrote:

> Is it that atypical for these public-facing services to drop ICMP when
> they get hot? I'd not consider it an indicator of service down unless
> you're getting SRVFAILs
>
> James Host
> Architect - CDN Platform
> E: jhost@vultr.com
>
>
> On Tue, Feb 8, 2022 at 12:01 PM Christopher Conley via Outages <
> outages@outages.org> wrote:
>
>> This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these
>> IPs as Internet reachability tests for determining failover to backup
>> circuits and moved away from it after the first occurrence of this
>> happening.
>>
>>
>>
>> --
>>
>> Christopher Conley
>>
>> Systems Administrator | Fors Marsh Group
>>
>> 1010 N. Glebe Rd, Suite 510
>>
>> Arlington, VA 22201
>>
>>
>>
>> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *Justin
>> Krejci via Outages
>> *Sent:* Tuesday, February 8, 2022 2:34 PM
>> *To:* outages@outages.org
>> *Subject:* [EXTERNAL] [outages] Ping to Google 8.8.8.8
>>
>>
>>
>> Seeing some hosts and not others get no reply to ICMP pings from the
>> Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2
>> Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is
>> invincible, this has now started setting off monitoring alerts. DNS queries
>> so far seem unaffected.
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

I'm starting to get authentication failures when connecting to
IMAP.Gmail.com

Payment gateways and things have been really sluggish, jit emails are not
coming through in time..

On Tue., Feb. 8, 2022, 3:58 p.m. Mike Bolitho via Outages, <
outages@outages.org> wrote:

> Yes, it has been this way for at least a decade. DNS are not polling
> servers.
>
> - Mike Bolitho
>
> On Tue, Feb 8, 2022, 1:29 PM James Host via Outages <outages@outages.org>
> wrote:
>
>> Is it that atypical for these public-facing services to drop ICMP when
>> they get hot? I'd not consider it an indicator of service down unless
>> you're getting SRVFAILs
>>
>> James Host
>> Architect - CDN Platform
>> E: jhost@vultr.com
>>
>>
>> On Tue, Feb 8, 2022 at 12:01 PM Christopher Conley via Outages <
>> outages@outages.org> wrote:
>>
>>> This happens occasionally on 8.8.8.8 and 8.8.4.4. We used to use these
>>> IPs as Internet reachability tests for determining failover to backup
>>> circuits and moved away from it after the first occurrence of this
>>> happening.
>>>
>>>
>>>
>>> --
>>>
>>> Christopher Conley
>>>
>>> Systems Administrator | Fors Marsh Group
>>>
>>> 1010 N. Glebe Rd, Suite 510
>>>
>>> Arlington, VA 22201
>>>
>>>
>>>
>>> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *Justin
>>> Krejci via Outages
>>> *Sent:* Tuesday, February 8, 2022 2:34 PM
>>> *To:* outages@outages.org
>>> *Subject:* [EXTERNAL] [outages] Ping to Google 8.8.8.8
>>>
>>>
>>>
>>> Seeing some hosts and not others get no reply to ICMP pings from the
>>> Minneapolis, MN area. I checked using HE's looking glass, 1 of their 2
>>> Minneapolis routers gets no replies. Since many use and assume 8.8.8.8 is
>>> invincible, this has now started setting off monitoring alerts. DNS queries
>>> so far seem unaffected.
>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

On 2/8/22 21:51, Jay Hennigan via Outages wrote:

>
> Don't do that then.
>
> Seriously, public DNS servers were never intended to be used as ping
> targets used to measure Internet connectivity.

That ship long sailed.

If we want to stop this behaviour, we'll have to do something about it,
specifically, offering an alternative to Internet users that is regarded
as official, e.g., like we do with other public services such as NTP.

Mark.

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On Wed, 9 Feb 2022, 04:45 Mark Tinka via Outages, <outages@outages.org>
wrote:

> If we want to stop this behaviour, we'll have to do something about it,
> specifically, offering an alternative to Internet users that is regarded
> as official, e.g., like we do with other public services such as NTP.
>

Do a DNS query. You don't even have to randomise the id number, just query
for something that will have a small set of results (so, not the root) and
ensure checking is disabled. For 8.8.8.8, I'm guessing "dns.google" is
probably an excellent target.

If you wanted something generic, what about a PTR query for something in
10/8, directed at the AS112 project? That's pretty much the sinkhole that
expects that kind of unwanted traffic...

M

>

On 2/8/22 9:45 PM, Mark Tinka via Outages wrote:
> That ship long sailed.

So.

Many ships have sailed, as in left the dock, but never reached their
destination.

> If we want to stop this behaviour, we'll have to do something about it,

Yes.

> specifically, offering an alternative to Internet users that is regarded
> as official, e.g., like we do with other public services such as NTP.

That's /one/ option.

Another option is for Google, et al., to simply drop ICMP packets to
their DNS servers.

Is this heavy handed? Probably.

Is it a viable option to persuade people to not ping said DNS servers? yes.

Will it achieve the desired result of having people stop pinging
Google's DNS servers? Quite likely. Admittedly with a long tail.



--
Grant. . . .
unix || die

On 2/9/22 07:29, Matthew Walster wrote:

>
> Do a DNS query. You don't even have to randomise the id number, just
> query for something that will have a small set of results (so, not the
> root) and ensure checking is disabled. For 8.8.8.8, I'm guessing
> "dns.google" is probably an excellent target.
>
> If you wanted something generic, what about a PTR query for something
> in 10/8, directed at the AS112 project? That's pretty much the
> sinkhole that expects that kind of unwanted traffic...

Not a serious problem for you and me, who are operators that know how to
do this and understand it well.

For Jane, down the road, who doesn't care and just wants her MTV, she's
not interested in what a DNS query is. All she is know is the
tech-support on the other end of the horn asked her to "ping 8.8.8.8".

Mark.
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On 2/9/22 07:30, Grant Taylor via Outages wrote:

>
> Many ships have sailed, as in left the dock, but never reached their
> destination.

You mean like PMTUd, and such :-)?

We probably won't get that one back, and unless we do something,
inability to ping 8.8.8.8 will result in unnecessary NOC tickets
claiming "the Internet is down".


>
> That's /one/ option.
>
> Another option is for Google, et al., to simply drop ICMP packets to
> their DNS servers.
>
> Is this heavy handed?  Probably.
>
> Is it a viable option to persuade people to not ping said DNS servers?
> yes.
>
> Will it achieve the desired result of having people stop pinging
> Google's DNS servers?  Quite likely.  Admittedly with a long tail.

Yes, this helps Google not having to deal with the problem, but it
passes the burden both to the ISP who has to explain why the Internet is
now down, and to some other online service who now has to sink ping
traffic. Perhaps Yahoo will pick that priviledge up again, like they did
back in the day :-(...

Mark.
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On 2/8/22 11:46 PM, Mark Tinka via Outages wrote:
> You mean like PMTUd, and such :-)?

Not what I originally meant, but sure.

> We probably won't get that one back, and unless we do something,
> inability to ping 8.8.8.8 will result in unnecessary NOC tickets
> claiming "the Internet is down".

Probably some, for a while. (See more below.)

> Yes, this helps Google not having to deal with the problem, but it
> passes the burden both to the ISP who has to explain why the Internet is
> now down, and to some other online service who now has to sink ping
> traffic. Perhaps Yahoo will pick that priviledge up again, like they did
> back in the day :-(...

So ... an end user education issue.

- No, the Internet is not down.
- The specific test you are doing is (now) bad (for reasons).
- See how your $StreamingServiceVideo is still playing? -- Did you
receive the test email I just sent you?

The Internet is /up/.

Who should be responsible for an ISP's user base? I'd naively think
that the ISP should be responsible for their own user base. Why should
we foist this responsibility off onto another organization?



--
Grant. . . .
unix || die

This belongs on the discussion list and not on the Outages list.
On Feb 9, 2022, 11:38 AM -0700, Grant Taylor via Outages <outages@outages.org>, wrote:
> On 2/8/22 11:46 PM, Mark Tinka via Outages wrote:
> > You mean like PMTUd, and such :-)?
>
> Not what I originally meant, but sure.
>
> > We probably won't get that one back, and unless we do something,
> > inability to ping 8.8.8.8 will result in unnecessary NOC tickets
> > claiming "the Internet is down".
>
> Probably some, for a while. (See more below.)
>
> > Yes, this helps Google not having to deal with the problem, but it
> > passes the burden both to the ISP who has to explain why the Internet is
> > now down, and to some other online service who now has to sink ping
> > traffic. Perhaps Yahoo will pick that priviledge up again, like they did
> > back in the day :-(...
>
> So ... an end user education issue.
>
> - No, the Internet is not down.
> - The specific test you are doing is (now) bad (for reasons).
> - See how your $StreamingServiceVideo is still playing? -- Did you
> receive the test email I just sent you?
>
> The Internet is /up/.
>
> Who should be responsible for an ISP's user base? I'd naively think
> that the ISP should be responsible for their own user base. Why should
> we foist this responsibility off onto another organization?
>
>
>
> --
> Grant. . . .
> unix || die
>
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

AS15169 in Chicago did have issues Tuesday 2022-02-08, which would have
affected ICMP traffic to the DNS server IPs.
We believe the impact time was roughly 2022-02-08 09:30 - 22:30 CST.

I'd be interested in a unicast followup if anyone's still seeing issues
today.

While indeed we have no SLA for ICMP to Google public DNS, we're not
currently *intending* for it to suddenly stop working or otherwise have
dramatic behavior shifts.

Phil, for AS15169

On Wed, Feb 9, 2022 at 6:51 PM Carlos Alvarez via Outages <
outages@outages.org> wrote:

> This belongs on the discussion list and not on the Outages list.
> On Feb 9, 2022, 11:38 AM -0700, Grant Taylor via Outages <
> outages@outages.org>, wrote:
>
> On 2/8/22 11:46 PM, Mark Tinka via Outages wrote:
>
> You mean like PMTUd, and such :-)?
>
>
> Not what I originally meant, but sure.
>
> We probably won't get that one back, and unless we do something,
> inability to ping 8.8.8.8 will result in unnecessary NOC tickets
> claiming "the Internet is down".
>
>
> Probably some, for a while. (See more below.)
>
> Yes, this helps Google not having to deal with the problem, but it
> passes the burden both to the ISP who has to explain why the Internet is
> now down, and to some other online service who now has to sink ping
> traffic. Perhaps Yahoo will pick that priviledge up again, like they did
> back in the day :-(...
>
>
> So ... an end user education issue.
>
> - No, the Internet is not down.
> - The specific test you are doing is (now) bad (for reasons).
> - See how your $StreamingServiceVideo is still playing? -- Did you
> receive the test email I just sent you?
>
> The Internet is /up/.
>
> Who should be responsible for an ISP's user base? I'd naively think
> that the ISP should be responsible for their own user base. Why should
> we foist this responsibility off onto another organization?
>
>
>
> --
> Grant. . . .
> unix || die
>
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

On 2/9/22 20:29, Grant Taylor via Outages wrote:

>
> So ... an end user education issue.
>
>  - No, the Internet is not down.
>  - The specific test you are doing is (now) bad (for reasons).
>  - See how your $StreamingServiceVideo is still playing?  --  Did you
> receive the test email I just sent you?
>
> The Internet is /up/.
>
> Who should be responsible for an ISP's user base?  I'd naively think
> that the ISP should be responsible for their own user base. Why should
> we foist this responsibility off onto another organization?

I'd be fine with that, only if it were feasible. Most ISP's do not have
the capacity to educate the thousands, tens-of-thousands or millions of
customers they service, about why pinging 8.8.8.8 is a bad thing. In
some cases, it's because there are simply too many customers to educate
in a formal and structured way that sticks. In other cases, the ISP's
simply don't have the resources to execute this sort of thing.

This does not mean that Google should be responsible for providing a
ping target. It just means that the day they choose to stop doing so, we
have a massive problem, one we could have been prepared for, but seem to
be too lazy to.

In any event, providing an alternative that is globally standard, would
be far easier than trying to educate customers who only care about
getting their MTV.

Mark.
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On 10/02/2022 04:53, Mark Tinka via Outages wrote:

> In any event, providing an alternative that is globally standard,
> would be far easier than trying to educate customers who only care
> about getting their MTV.

I'm not sure an AS112-like alternative would be what we want here: for
the average user reaching Google/Facebook/Akamai/CloudFront is far more
important than reaching some random AS hosting the closest AS112 instance.

Overall, it's probably not fair expect to change the behaviour which
leads users to test common destinations for validating whether their
internet service is available. If we started pushing "ping 8.8.8.8" out
of people's mindset, they will likely move to "curl google.com".

Giorgio

--
www: grg.pw
email: me@grg.pw
mobile: +44 7716 604314 / +39 393 1049073

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages