Mailing List Archive

What about the zone not having a single point of failure? Both servers
are covered by the same /24.

% dig www.moi.gov.cy @212.31.118.19 +norec +dnssec

; <<>> DiG 9.19.11-dev <<>> www.moi.gov.cy @212.31.118.19 +norec +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17380
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 6387183a6031ef182fa6ade7641ad4ff2a078213f4e24fc9 (good)
;; QUESTION SECTION:
;www.moi.gov.cy. IN A

;; ANSWER SECTION:
www.moi.gov.cy. 3600 IN A 212.31.118.26

;; AUTHORITY SECTION:
moi.gov.cy. 3600 IN NS ns01.gov.cy.
moi.gov.cy. 3600 IN NS ns02.gov.cy.

;; ADDITIONAL SECTION:
ns02.gov.cy. 86400 IN A 212.31.118.20
ns01.gov.cy. 86400 IN A 212.31.118.19

;; Query time: 374 msec
;; SERVER: 212.31.118.19#53(212.31.118.19) (UDP)
;; WHEN: Wed Mar 22 21:14:23 AEDT 2023
;; MSG SIZE rcvd: 157

%

> On 22 Mar 2023, at 19:36, Saku Ytti <saku@ytti.fi> wrote:
>
> Am I correct to understand that 1.1.1.1 only does support via community forum?
>
> They had just enough interest in the service to collect user data to
> monetise, but 0 interest in trying to figure out how to detect and
> solve problems?
>
> Why not build a web form where they ask you to explain what is not
> working, in terms of automatically testable. Like no A record for X.
> Then after you submit this form, they test against all 1.1.1.1 and
> some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour,
> the ticket is accepted and sent to someone who understands DNS? If
> there is no difference in behaviour, direct people to community
> forums.
> This trivial, cheap and fast to produce support channel would ensure
> virtually 0 trash support cases, so you wouldn't even have to hire
> people to support your data collection enterprise.

The number of times that 8.8.8.8 “works” but there is an actual error
is enormous. 8.8.8.8 tolerates lots of protocol errors which ends up
causing support cases for others where the result is “the servers are
broken in this way”. You then try to report the issue but the report
is ignored because “It works with 8.8.8.8”.

> Very obviously they selfishly had no interest in ensuring 1.1.1.1
> actually works, as long as they are getting the data. I do not know
> how to characterise this as anything but unethical.
>
> https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lca-235m3/487469
> https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
>
> If you can't due to resources or competence support DNS, do not offer one.
>
> --
> ++ytti, cake having and cake eating user

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

If you wish to consult people on how to configure DNS, please reach
out to the responsible folk.

I am discussing a specific recursor in anycasted setup not resolving
domain and provider offering no remediation channel.

These are two entirely different classes of problem and collapsing
them into a single problem is not going to help in either case.

On Wed, 22 Mar 2023 at 12:25, Mark Andrews <marka@isc.org> wrote:
>
> What about the zone not having a single point of failure? Both servers
> are covered by the same /24.
>
> % dig www.moi.gov.cy @212.31.118.19 +norec +dnssec
>
> ; <<>> DiG 9.19.11-dev <<>> www.moi.gov.cy @212.31.118.19 +norec +dnssec
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17380
> ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ; COOKIE: 6387183a6031ef182fa6ade7641ad4ff2a078213f4e24fc9 (good)
> ;; QUESTION SECTION:
> ;www.moi.gov.cy. IN A
>
> ;; ANSWER SECTION:
> www.moi.gov.cy. 3600 IN A 212.31.118.26
>
> ;; AUTHORITY SECTION:
> moi.gov.cy. 3600 IN NS ns01.gov.cy.
> moi.gov.cy. 3600 IN NS ns02.gov.cy.
>
> ;; ADDITIONAL SECTION:
> ns02.gov.cy. 86400 IN A 212.31.118.20
> ns01.gov.cy. 86400 IN A 212.31.118.19
>
> ;; Query time: 374 msec
> ;; SERVER: 212.31.118.19#53(212.31.118.19) (UDP)
> ;; WHEN: Wed Mar 22 21:14:23 AEDT 2023
> ;; MSG SIZE rcvd: 157
>
> %
>
> > On 22 Mar 2023, at 19:36, Saku Ytti <saku@ytti.fi> wrote:
> >
> > Am I correct to understand that 1.1.1.1 only does support via community forum?
> >
> > They had just enough interest in the service to collect user data to
> > monetise, but 0 interest in trying to figure out how to detect and
> > solve problems?
> >
> > Why not build a web form where they ask you to explain what is not
> > working, in terms of automatically testable. Like no A record for X.
> > Then after you submit this form, they test against all 1.1.1.1 and
> > some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour,
> > the ticket is accepted and sent to someone who understands DNS? If
> > there is no difference in behaviour, direct people to community
> > forums.
> > This trivial, cheap and fast to produce support channel would ensure
> > virtually 0 trash support cases, so you wouldn't even have to hire
> > people to support your data collection enterprise.
>
> The number of times that 8.8.8.8 “works” but there is an actual error
> is enormous. 8.8.8.8 tolerates lots of protocol errors which ends up
> causing support cases for others where the result is “the servers are
> broken in this way”. You then try to report the issue but the report
> is ignored because “It works with 8.8.8.8”.
>
> > Very obviously they selfishly had no interest in ensuring 1.1.1.1
> > actually works, as long as they are getting the data. I do not know
> > how to characterise this as anything but unethical.
> >
> > https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lca-235m3/487469
> > https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
> >
> > If you can't due to resources or competence support DNS, do not offer one.
> >
> > --
> > ++ytti, cake having and cake eating user
>
> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
>


--
++ytti

Why would they need it, its free, they are not being paid to be your DNS servers. Assuming the provider is 1.1.1.1 itself. YOUR ISP SHOULD NOT USE 1.1.1.1 or 8.8.8.8, you should run your OWN DNS servers.

If its not within your circle of influence, don’t' risk your business on it!



Dennis Burgess, Mikrotik Certified Trainer
MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified
Author of "Learn RouterOS- Second Edition”
Link Technologies, Inc -- Mikrotik & WISP Support Services
Office: 314-735-0270  Website: http://www.linktechs.net
Need to Automate MikroTik Backups: https://cloud.linktechs.net
Create Wireless Coverage’s with www.towercoverage.com

-----Original Message-----
From: NANOG <nanog-bounces+dmburgess=linktechs.net@nanog.org> On Behalf Of Saku Ytti
Sent: Wednesday, March 22, 2023 6:53 AM
To: Mark Andrews <marka@isc.org>
Cc: nanog list <nanog@nanog.org>
Subject: Re: 1.1.1.1 support?

If you wish to consult people on how to configure DNS, please reach out to the responsible folk.

I am discussing a specific recursor in anycasted setup not resolving domain and provider offering no remediation channel.

These are two entirely different classes of problem and collapsing them into a single problem is not going to help in either case.

On Wed, 22 Mar 2023 at 12:25, Mark Andrews <marka@isc.org> wrote:
>
> What about the zone not having a single point of failure? Both
> servers are covered by the same /24.
>
> % dig www.moi.gov.cy @212.31.118.19 +norec +dnssec
>
> ; <<>> DiG 9.19.11-dev <<>> www.moi.gov.cy @212.31.118.19 +norec
> +dnssec ;; global options: +cmd ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17380 ;; flags: qr
> aa; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ; COOKIE: 6387183a6031ef182fa6ade7641ad4ff2a078213f4e24fc9 (good) ;;
> QUESTION SECTION:
> ;www.moi.gov.cy. IN A
>
> ;; ANSWER SECTION:
> www.moi.gov.cy. 3600 IN A 212.31.118.26
>
> ;; AUTHORITY SECTION:
> moi.gov.cy. 3600 IN NS ns01.gov.cy.
> moi.gov.cy. 3600 IN NS ns02.gov.cy.
>
> ;; ADDITIONAL SECTION:
> ns02.gov.cy. 86400 IN A 212.31.118.20
> ns01.gov.cy. 86400 IN A 212.31.118.19
>
> ;; Query time: 374 msec
> ;; SERVER: 212.31.118.19#53(212.31.118.19) (UDP) ;; WHEN: Wed Mar 22
> 21:14:23 AEDT 2023 ;; MSG SIZE rcvd: 157
>
> %
>
> > On 22 Mar 2023, at 19:36, Saku Ytti <saku@ytti.fi> wrote:
> >
> > Am I correct to understand that 1.1.1.1 only does support via community forum?
> >
> > They had just enough interest in the service to collect user data to
> > monetise, but 0 interest in trying to figure out how to detect and
> > solve problems?
> >
> > Why not build a web form where they ask you to explain what is not
> > working, in terms of automatically testable. Like no A record for X.
> > Then after you submit this form, they test against all 1.1.1.1 and
> > some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour,
> > the ticket is accepted and sent to someone who understands DNS? If
> > there is no difference in behaviour, direct people to community
> > forums.
> > This trivial, cheap and fast to produce support channel would ensure
> > virtually 0 trash support cases, so you wouldn't even have to hire
> > people to support your data collection enterprise.
>
> The number of times that 8.8.8.8 “works” but there is an actual error
> is enormous. 8.8.8.8 tolerates lots of protocol errors which ends up
> causing support cases for others where the result is “the servers are
> broken in this way”. You then try to report the issue but the report
> is ignored because “It works with 8.8.8.8”.
>
> > Very obviously they selfishly had no interest in ensuring 1.1.1.1
> > actually works, as long as they are getting the data. I do not know
> > how to characterise this as anything but unethical.
> >
> > https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-
> > cy-in-lca-235m3/487469
> > https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
> >
> > If you can't due to resources or competence support DNS, do not offer one.
> >
> > --
> > ++ytti, cake having and cake eating user
>
> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
>


--
++ytti

Matt Harris
VP OF INFRASTRUCTURE
Follow us on LinkedIn!
matt.harris@netfire.net
816-256-5446
www.netfire.com
On Wed, Mar 22, 2023 at 3:36?AM Saku Ytti <saku@ytti.fi> wrote:

> Am I correct to understand that 1.1.1.1 only does support via community
> forum?
>
> They had just enough interest in the service to collect user data to
> monetise, but 0 interest in trying to figure out how to detect and
> solve problems?
>
> Why not build a web form where they ask you to explain what is not
> working, in terms of automatically testable. Like no A record for X.
> Then after you submit this form, they test against all 1.1.1.1 and
> some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour,
> the ticket is accepted and sent to someone who understands DNS? If
> there is no difference in behaviour, direct people to community
> forums.
> This trivial, cheap and fast to produce support channel would ensure
> virtually 0 trash support cases, so you wouldn't even have to hire
> people to support your data collection enterprise.
>
> Very obviously they selfishly had no interest in ensuring 1.1.1.1
> actually works, as long as they are getting the data. I do not know
> how to characterise this as anything but unethical.
>
>
> https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lca-235m3/487469
> https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228
>
> If you can't due to resources or competence support DNS, do not offer one.
>

Saku,
When something is provided at no cost, I don't see how it can be unethical
unless they are explicitly lying about the ways in which they use the data
they gather.
Ultimately, you're asking them to provide a costly service (support for
end-users, the vast majority of whom will not ask informed, intelligent
questions like the members of this list would be able to, but would still
demand the same level of support) on top of a service they are already
providing at no cost. That's both unrealistic and unnecessary. There's an
exceedingly simple solution, here, after all: if you don't like their
service or it isn't working for you as an end-user, don't use it.

On the same token as network operators, it might be nice if
cloudflare's admins were accessible to address potential issues that may
actually be related to legitimate network misconfigurations or other
problems on their end that result in issues resolving some folks' resources
- and I suspect they may in fact be via this list or other similar ones, or
other open resources that are widely available to folks who are in the
know. That said, with regards to any specific case, we don't know whose end
the issue lies on. It's possible that the folks managing the Cypress
government resources have taken steps actively, or passively misconfigured,
their systems in such a way that causes the root problem that you're
pointing out. As I administer neither of the related networks, I can't
speak to this, but I think it's just as likely based on a coin flip that
they are responsible for the issue as it is that cloudflare is responsible
for the issue. On top of that, I suspect getting technology help from a
random government entity may be far less fruitful than even a public forum
would be.

Good luck getting a resolution to your resolution.

On Wed, 22 Mar 2023 at 15:26, Matt Harris <matt@netfire.net> wrote:


> When something is provided at no cost, I don't see how it can be unethical
> unless they are explicitly lying about the ways in which they use the data
> they gather.
> Ultimately, you're asking them to provide a costly service (support for
> end-users, the vast majority of whom will not ask informed, intelligent
> questions like the members of this list would be able to, but would still
> demand the same level of support) on top of a service they are already
> providing at no cost. That's both unrealistic and unnecessary. There's an
> exceedingly simple solution, here, after all: if you don't like their
> service or it isn't working for you as an end-user, don't use it.
>

Thank you for the philosophical perspective, but currently my interest is
not to debate merits or lack thereof in laissez-faire economics.

The problem is, a large number of people will use 1.1.1.1, 8.8.8.8 or
9.9.9.9 despite my or your position about it. There is incentive for
providers to provide it 'for free', as it adds value to their products as
users are compensating providers with the data.

Occasionally things don't work and when they do not, we need a way to
inform the provider 'hey you have a problem'. You could be anywhere in this
chain, with no ability to impact any of the decisions.

I know there is a real problem, I know real users are impacted, I know
almost none of them will have the ability to understand why there is a
problem or remediate it.

--
++ytti

Try asking dns-operations@lists.dns-oarc.net for someone at CloudFlare.

For what it's worth, it works for me. I'm in Troy, OH.

C:\Users\jluthman>dig www.moi.gov.cy @1.1.1.1 +short
212.31.118.26


On Wed, Mar 22, 2023 at 9:43?AM Saku Ytti <saku@ytti.fi> wrote:

>
>
> On Wed, 22 Mar 2023 at 15:26, Matt Harris <matt@netfire.net> wrote:
>
>
>> When something is provided at no cost, I don't see how it can be
>> unethical unless they are explicitly lying about the ways in which they use
>> the data they gather.
>> Ultimately, you're asking them to provide a costly service (support for
>> end-users, the vast majority of whom will not ask informed, intelligent
>> questions like the members of this list would be able to, but would still
>> demand the same level of support) on top of a service they are already
>> providing at no cost. That's both unrealistic and unnecessary. There's an
>> exceedingly simple solution, here, after all: if you don't like their
>> service or it isn't working for you as an end-user, don't use it.
>>
>
> Thank you for the philosophical perspective, but currently my interest is
> not to debate merits or lack thereof in laissez-faire economics.
>
> The problem is, a large number of people will use 1.1.1.1, 8.8.8.8 or
> 9.9.9.9 despite my or your position about it. There is incentive for
> providers to provide it 'for free', as it adds value to their products as
> users are compensating providers with the data.
>
> Occasionally things don't work and when they do not, we need a way to
> inform the provider 'hey you have a problem'. You could be anywhere in this
> chain, with no ability to impact any of the decisions.
>
> I know there is a real problem, I know real users are impacted, I know
> almost none of them will have the ability to understand why there is a
> problem or remediate it.
>
> --
> ++ytti
>

On 2023-03-22 10:36:03 +0200, Saku Ytti wrote:
>Am I correct to understand that 1.1.1.1 only does support via community forum?

The community forum is our preferred method of support, yes.

>Why not build a web form where they ask you to explain what is not
>working, in terms of automatically testable. Like no A record for X.
>Then after you submit this form, they test against all 1.1.1.1 and
>some 9.9.9.9 and 8.8.8.8 and if they find a difference in behaviour,
>the ticket is accepted and sent to someone who understands DNS? If
>there is no difference in behaviour, direct people to community
>forums.

I'll take this feedback to our developers.

>https://community.cloudflare.com/t/1-1-1-1-wont-resolve-www-moi-gov-cy-in-lca-235m3/487469
>https://community.cloudflare.com/t/1-1-1-1-failing-to-resolve/474228

I took a look at the above tickets, and it seems that one of the egress
ranges from that datacenter cannot connect to the authoritative
nameservers of `www.moi.gov.cy`: `ns01.gov.cy` and `ns02.gov.cy`.

Here's a redacted pcap for those who like details, showing no response:

IP a.b.c.d.56552 > 212.31.118.19.53: 51873+ [1au] A? www.moi.gov.cy. (55)
IP a.b.c.d.51718 > 212.31.118.20.53: 31021+ [1au] A? www.moi.gov.cy. (55)

TCP behaves similarly.

The source prefixes having issues connecting to 212.31.118.19 and
212.31.118.20 are: 172.68.130.0/24, while a neighbouring source prefix
172.68.171.0/24 seems to connect fine.

I'm filing an internal ticket right now to investigate, but I'd
appreciate if you could also help us on your end for any possible
solutions regarding this connectivity failure.

As a general note regarding the two community posts: the straight deep
dive into technical information makes it more difficult for others to
interpret the request. As you said in a later post here:

>I know almost none of them will have the ability to understand why
>there is a problem or remediate it.

Not everyone in the Community Forum (nor our company) can pull out the
specific datacenter used, the specific machine(s) used, and the source
ASN from the `my.ip.fi` curl.

An preamble will greatly help in context.

Thanks for reaching out and sorry that you had to escalate to another
medium,
--
alex [at] e [dot] sc
alexander [at] cloudflare [dot] com

Yes, it works in every other CF except LCA-CF. Thank you for the
additional data point.

You can use `dig CHAOS TXT id.server @1.1.1.1 +nsid` to get two
unicast identifiers for the server you got the response from.

On Wed, 22 Mar 2023 at 15:49, Josh Luthman <josh@imaginenetworksllc.com> wrote:
>
> Try asking dns-operations@lists.dns-oarc.net for someone at CloudFlare.
>
> For what it's worth, it works for me. I'm in Troy, OH.
>
> C:\Users\jluthman>dig www.moi.gov.cy @1.1.1.1 +short
> 212.31.118.26
>
>
> On Wed, Mar 22, 2023 at 9:43?AM Saku Ytti <saku@ytti.fi> wrote:
>>
>>
>>
>> On Wed, 22 Mar 2023 at 15:26, Matt Harris <matt@netfire.net> wrote:
>>
>>>
>>> When something is provided at no cost, I don't see how it can be unethical unless they are explicitly lying about the ways in which they use the data they gather.
>>> Ultimately, you're asking them to provide a costly service (support for end-users, the vast majority of whom will not ask informed, intelligent questions like the members of this list would be able to, but would still demand the same level of support) on top of a service they are already providing at no cost. That's both unrealistic and unnecessary. There's an exceedingly simple solution, here, after all: if you don't like their service or it isn't working for you as an end-user, don't use it.
>>
>>
>> Thank you for the philosophical perspective, but currently my interest is not to debate merits or lack thereof in laissez-faire economics.
>>
>> The problem is, a large number of people will use 1.1.1.1, 8.8.8.8 or 9.9.9.9 despite my or your position about it. There is incentive for providers to provide it 'for free', as it adds value to their products as users are compensating providers with the data.
>>
>> Occasionally things don't work and when they do not, we need a way to inform the provider 'hey you have a problem'. You could be anywhere in this chain, with no ability to impact any of the decisions.
>>
>> I know there is a real problem, I know real users are impacted, I know almost none of them will have the ability to understand why there is a problem or remediate it.
>>
>> --
>> ++ytti



--
++ytti

On Wed, 22 Mar 2023 at 16:04, Alexander Huynh via NANOG <nanog@nanog.org> wrote:

> I'll take this feedback to our developers.

Many thanks.

> I took a look at the above tickets, and it seems that one of the egress
> ranges from that datacenter cannot connect to the authoritative
> nameservers of `www.moi.gov.cy`: `ns01.gov.cy` and `ns02.gov.cy`.
>
> Here's a redacted pcap for those who like details, showing no response:
>
> IP a.b.c.d.56552 > 212.31.118.19.53: 51873+ [1au] A? www.moi.gov.cy. (55)
> IP a.b.c.d.51718 > 212.31.118.20.53: 31021+ [1au] A? www.moi.gov.cy. (55)
>
> TCP behaves similarly.

The recursor response suggests a loop, so network problem is highly likely.

> I'm filing an internal ticket right now to investigate, but I'd
> appreciate if you could also help us on your end for any possible
> solutions regarding this connectivity failure.

Sure, you might also want to look into nlnog ring, which allows a
broad perspective to issues.

> As a general note regarding the two community posts: the straight deep
> dive into technical information makes it more difficult for others to
> interpret the request. As you said in a later post here:

This is a very difficult subject. How to get help. If I had made it
more genetic, we could refute it as it doesn't contain needed
information. If I made it longer we could refute that it's not terse
enough. However we submit it, we can argue it wasn't the right way.
As seen in the original post, I fully appreciate almost every single
case about 1.1.1.1 is incorrect and user error. But I proposed a
mechanism to by-pass community forums and reach people who are able to
help and understand. If there is disagreement in 1.1.1.1, 8.8.8.8 and
9.9.9.9 then let humans analyse it. The ticket volume would be
trivial, if we look at community forums and see how many 1.1.1.1
complaints would bypass this filter.

> Not everyone in the Community Forum (nor our company) can pull out the
> specific datacenter used, the specific machine(s) used, and the source
> ASN from the `my.ip.fi` curl.

I gave the specific unicast ID for the DNS server in addition to my
IP. I cannot glean any other information.

I don't think we can fairly fault either of the cases in the community
forum. We must fault the process itself and look for ways to improve.
--
++ytti