Mailing List Archive

Help again please Fwd: please fix your broken DNS server
More help please.

Ken is now claiming other servers are broken, not his. He wants all
bind users to use allow-v6-synthesis to resolve the problem. So all
other DNS servers are broken, not his.

I am at a loss as to what to do and they throw the FUD back at the
political leadership.

Why the direct dig work and the indirect resolution not?

Ideas?
--
Joseph T. Klein

PSTN: +1 414 961 1690 VoIP: +1 414 431 4231 Mobile: +1 414 628 3380


Begin forwarded message:

> From: Ken Walker <kwalke@mpw.net>
> Date: July 8, 2005 10:27:28 AM CDT
> To: "Joseph T. Klein" <josephtklein@mac.com>
> Cc: Guyer Randolph <rguyer@milwaukee.gov>, Brousseau Dan
> <dbrous@mpw.net>, Froh Gerard J <gfroh@mpw.net>, Craig Hapanovich
> <CHAPAN@milwaukee.gov>
> Subject: Re: please fix your broken DNS server
>
> Joe,
> You missed the point of what I was trying to tell you. MY dns
> servers DO
> respond with the proper response. (Please see dig result below). What
> I was
> trying to offer was a possible reason why other sites are having
> trouble.
>
> I never said it was an AAAA v A6 problem. If you read the whole
> section on
> synthectic response, (see below) It also states it will try a lookup
> using A6
> records, if that fails then AAAA. The allow-v6-synthesis ALSO allows a
> server
> to return a correctly formated response when there are NO AAAA records
> to be
> found for a host. Your testing is flawed in the fact that there are
> either
> CNAME or AAAA records you are querying uwm for, in otherwords SOME
> sort of
> response, but not an unknown one. That's where this paramter would
> help a BIND
> server anyway. I can't help you fix other servers that do not belong
> to DPW.
> Mine work correctly.
>
> The security of our data and our network is of paramount
> importance to us
> and I WILL NOT discuss intiment details with anyone that is not a
> trusted city
> employee. There are over 500,000 people in milwaukee and they all have
> needs
> I have many many many concerns to deal with during the day and it
> is not
> fair for 1 person to monopolize my time for something that isn't our
> problem.
>
> I will not respond further to any future letters you write about
> this
> subject.
>
> Ken
>
> dig @charon.mpw.net AAAA gwise.ci.mil.wi.us
>
> ; <<>> DiG 9.2.3 <<>> @charon.mpw.net AAAA gwise.ci.mil.wi.us
> ;; global options: printcmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46769
> ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;gwise.ci.mil.wi.us. IN AAAA
>
> ;; Query time: 13 msec
> ;; SERVER: 216.54.131.130#53(charon.mpw.net)
> ;; WHEN: Fri Jul 8 10:10:29 2005
> ;; MSG SIZE rcvd: 36
>
> 6.2.14.13. Synthetic IPv6 responses
>
> Many existing stub resolvers support IPv6 DNS lookups as defined in
> RFC1886,
> using AAAA records for forward lookups and "nibble labels" in the
> IP6.INT
> domain for reverse lookups, but do not support RFC2874-style lookups
> (using A6
> records and binary labels in the IP6.ARPA domain).
>
> For those who wish to continue to use such stub resolvers rather than
> switching
> to the BIND 9 lightweight resolver, BIND 9 provides a way to
> automatically
> convert RFC1886-style lookups into RFC2874-style lookups and return
> the results
> as "synthetic" AAAA and PTR records.
>
> This feature is disabled by default and can be enabled on a per-client
> basis by
> adding a allow-v6-synthesis { address_match_list }; clause to the
> options or
> view statement. When it is enabled, recursive AAAA queries cause the
> server to
> first try an A6 lookup and if that fails, an AAAA lookups. No matter
> which one
> succeeds, the results are returned as a set of synthetic AAAA records.
> Similarly, recursive PTR queries in IP6.INT will cause a lookup in
> IP6.ARPA
> using binary labels, and if that fails, another lookup in IP6.INT. The
> results
> are returned as a synthetic PTR record in ip6.int.
>
>
Help again please Fwd: please fix your broken DNS server [ In reply to ]
On Fri, Jul 08, 2005 at 11:32:41AM -0500, Joseph T. Klein wrote:
> Why the direct dig work and the indirect resolution not?

The problem is a little bit subtle, because it is with an upstream
name server, and so the query never makes it to the name servers
that he mentions. The real problem queries are:

dig AAAA gwise.milwaukee.gov @itmddns1x.milwaukee.gov
dig AAAA gwise.milwaukee.gov @itmddns2x.milwaukee.gov
dig AAAA gwise.milwaukee.gov @itmddns3x.milwaukee.gov
dig AAAA gwise.milwaukee.gov @itmddns4x.milwaukee.gov

All these queries return SERVFAIL, and I think that that is where
the problem is comming from. I'm not sure who is responsible for
these name servers - it may or may not be the guy that you are
contacting. I would suggest that you ask him who to contact about
the itmddns*x.milwaukee.gov servers.

The reason these queries are important is because:

% dig NS milwaukee.gov
milwaukee.gov. 60 IN NS itmddns1x.milwaukee.gov.
milwaukee.gov. 60 IN NS itmddns2x.milwaukee.gov.
milwaukee.gov. 60 IN NS itmddns3x.milwaukee.gov.
milwaukee.gov. 60 IN NS itmddns4x.milwaukee.gov.
% dig MX milwaukee.gov
milwaukee.gov. 60 IN MX 10 gwise.milwaukee.gov.
milwaukee.gov. 60 IN MX 20 mhsgate.ci.mil.wi.us.
% dig AAAA gwise.milwaukee.gov @itmddns1x.milwaukee.gov
SERVFAIL
% dig AAAA gwise.milwaukee.gov @itmddns2x.milwaukee.gov
SERVFAIL
% dig AAAA gwise.milwaukee.gov @itmddns3x.milwaukee.gov
SERVFAIL
% dig AAAA gwise.milwaukee.gov @itmddns4x.milwaukee.gov
SERVFAIL

If you have the CNAME cached from some other query, then you
can get the correct result:

% dig cname gwise.milwaukee.gov
gwise.milwaukee.gov. 60 IN CNAME gwise.ci.mil.wi.us.
% dig ns gwise.ci.mil.wi.us
gwise.ci.mil.wi.us. 4 IN NS lpitmd-isp1.mpw.net.
gwise.ci.mil.wi.us. 4 IN NS lpitmd-isp2.mpw.net.
% dig AAAA gwise.ci.mil.wi.us @lpitmd-isp1.mpw.net
% dig AAAA gwise.ci.mil.wi.us @lpitmd-isp1.mpw.net

However, if you go straight after an AAAA for gwise.ci.mil.wi.us
you'll fail when you get to the ci.mil.wi.us level, because you run
into the same problem upstream nameservers again.

% dig AAAA gwise.ci.mil.wi.us @a.root-servers.net
us. 172800 IN NS A.GTLD.BIZ.
us. 172800 IN NS B.GTLD.BIZ.
us. 172800 IN NS C.GTLD.BIZ.
% dig AAAA gwise.ci.mil.wi.us @a.gtld.biz
mil.wi.us. 900 IN NS DNS1.SOL.NET.
mil.wi.us. 900 IN NS DNS2.SOL.NET.
mil.wi.us. 900 IN NS DNS3.SOL.NET.
mil.wi.us. 900 IN NS DNS4.SOL.NET.
mil.wi.us. 900 IN NS DNSY.SOL.NET.
% dig AAAA gwise.ci.mil.wi.us @dns1.sol.net
ci.mil.wi.us. 86400 IN NS itmddns1x.milwaukee.gov.
ci.mil.wi.us. 86400 IN NS itmddns2x.milwaukee.gov.
ci.mil.wi.us. 86400 IN NS itmddns3x.milwaukee.gov.
ci.mil.wi.us. 86400 IN NS itmddns4x.milwaukee.gov.
% dig AAAA gwise.ci.mil.wi.us @itmddns1x.milwaukee.gov
SERVFAIL
% dig AAAA gwise.ci.mil.wi.us @itmddns2x.milwaukee.gov
SERVFAIL
% dig AAAA gwise.ci.mil.wi.us @itmddns3x.milwaukee.gov
SERVFAIL
% dig AAAA gwise.ci.mil.wi.us @itmddns4x.milwaukee.gov
SERVFAIL

Hope this helps,

David.
Help again please Fwd: please fix your broken DNS server [ In reply to ]
David Malone wrote:
> On Fri, Jul 08, 2005 at 11:32:41AM -0500, Joseph T. Klein wrote:
>
>>Why the direct dig work and the indirect resolution not?
>
>
> The problem is a little bit subtle, because it is with an upstream
> name server, and so the query never makes it to the name servers
> that he mentions. The real problem queries are:
>
> dig AAAA gwise.milwaukee.gov @itmddns1x.milwaukee.gov
> dig AAAA gwise.milwaukee.gov @itmddns2x.milwaukee.gov
> dig AAAA gwise.milwaukee.gov @itmddns3x.milwaukee.gov
> dig AAAA gwise.milwaukee.gov @itmddns4x.milwaukee.gov

I actually suspect the problem lies with lpitmd-isp1.mpw.net and
lpitmd-isp2.mpw.net, which are in theory authoritative for
gwise.ci.mil.wi.us. notice:

$ dig a gwise.ci.mil.wi.us @lpitmd-isp1.mpw.net

; <<>> DiG 9.2.4 <<>> a gwise.ci.mil.wi.us @lpitmd-isp1.mpw.net
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13833
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;gwise.ci.mil.wi.us. IN A

;; ANSWER SECTION:
gwise.ci.mil.wi.us. 0 IN A 216.54.131.198
gwise.ci.mil.wi.us. 0 IN A 216.56.88.101

$ dig aaaa gwise.ci.mil.wi.us @lpitmd-isp1.mpw.net

; <<>> DiG 9.2.4 <<>> aaaa gwise.ci.mil.wi.us @lpitmd-isp1.mpw.net
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5071
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;gwise.ci.mil.wi.us. IN AAAA

;; AUTHORITY SECTION:
ci.mil.wi.us. 86400 IN SOA ci.mil.wi.us.
administrator.ci.mil.wi.us. 998545544 28800 7200 604800 86400

----

Thus we're getting an SOA for 'ci.mil.wi.us' when we asked for a AAAA of
gwise.ci.mil.wi.us. I suspect this is what's causing SERVFAIL's of every
server trying to track down the AAAAs, including the SERVFAILs from
itmddnsYx.milwaukee.gov.

Also note that lpitmd-isp*.mpw.net are actually in the (v4) path (at
least from my POV) to their other nameservers:

...
16 core-01-ge-3-1-2-1.chcg.twtelecom.net (66.192.244.32) 28.135 ms
28.394 ms 27.973 ms
17 dist-02-so-0-0-0-0.milw.twtelecom.net (66.192.244.103) 30.387 ms
29.867 ms 30.056 ms
18 hagg-01-ge-1-3-0-508.milw.twtelecom.net (66.192.244.115) 30.397 ms
30.440 ms 30.379 ms
19 207.250.123.18 (207.250.123.18) 30.448 ms 30.430 ms 30.418 ms
20 lp-isp1.mpw.net (216.54.131.251) 30.446 ms 30.621 ms 30.476 ms

(216.54.131.251 == lpitmd-isp1.mpw.net), so I suspect these nameservers
are some sort of router/firewall, and likely not handling AAAAs very well.

-Kevin
Help again please Fwd: please fix your broken DNS server [ In reply to ]
On 8-jul-2005, at 18:32, Joseph T. Klein wrote:

> Ken is now claiming other servers are broken, not his. He wants all
> bind users to use allow-v6-synthesis to resolve the problem. So all
> other DNS servers are broken, not his.

This guy is insane. (60 second TTL for NS records anyone?)

He says his server is working correctly, but I get:

% dig @charon.mpw.net AAAA gwise.ci.mil.wi.us

; <<>> DiG 9.2.2 <<>> @charon.mpw.net AAAA gwise.ci.mil.wi.us
;; global options: printcmd
;; connection timed out; no servers could be reached

Now if there were any substance to his claim that people should use
AAAA synthesis, I should be able to get an A6 record from him as
input for my synthesizing efforts, but guess what:

% dig @charon.mpw.net A6 gwise.ci.mil.wi.us

; <<>> DiG 9.2.2 <<>> @charon.mpw.net A6 gwise.ci.mil.wi.us
;; global options: printcmd
;; connection timed out; no servers could be reached

You may want to point him to RFC 3596, if you want to bother talking
to him again.

Lesson for the IETF: this is what happens when you keep changing
stuff on people.
Help again please Fwd: please fix your broken DNS server [ In reply to ]
On Fri, Jul 08, 2005 at 10:46:31PM +0200, Iljitsch van Beijnum wrote:
> On 8-jul-2005, at 18:32, Joseph T. Klein wrote:
>
> >Ken is now claiming other servers are broken, not his. He wants all
> >bind users to use allow-v6-synthesis to resolve the problem. So all
> >other DNS servers are broken, not his.
>
> This guy is insane. (60 second TTL for NS records anyone?)

It looks like those l*-isp[12] boxes are DNS load-balancers or
something like that. And this genre is as far as I know quite known for
having problems with stuff like IPv6 RRs. ;)

> Lesson for the IETF: this is what happens when you keep changing
> stuff on people.

One can blame many things on the IETF, but not the clue level and
attitude of DNS services operators out there. ;)

I think they are just running misbehaving software, probably even
closed-source shrinkwrap $BIGNUM $$$ stuff and don't bother to actually
read problem reports nor analyzing root causes and work with the vendor
to fix it.

Looking at the tone of the mails Joseph gets, they aren't willing to
admit they have a problem nor actually work on fixing it. I doubt that
this attitude is limited to DNS.


Best regards,
Daniel

--
CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
Help again please Fwd: please fix your broken DNS server [ In reply to ]
On Fri, Jul 08, 2005 at 04:40:55PM -0400, Kevin Miller wrote:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5071
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
> ;; QUESTION SECTION:
> ;gwise.ci.mil.wi.us. IN AAAA
> ;; AUTHORITY SECTION:
> ci.mil.wi.us. 86400 IN SOA ci.mil.wi.us.
> administrator.ci.mil.wi.us. 998545544 28800 7200 604800 86400

> Thus we're getting an SOA for 'ci.mil.wi.us' when we asked for a AAAA of
> gwise.ci.mil.wi.us. I suspect this is what's causing SERVFAIL's of every
> server trying to track down the AAAAs, including the SERVFAILs from
> itmddnsYx.milwaukee.gov.

The respones looks correct to me - we asked for AAAA records for
gwise.ci.mil.wi.us and the server said that there were 0 records
of that type and points us at ci.mil.wi.us as being authorititive.
AFAIK, that's a perfectly reasonable thing to do.

David.
Help again please Fwd: please fix your broken DNS server [ In reply to ]
On Fri, Jul 08, 2005 at 10:46:31PM +0200, Iljitsch van Beijnum wrote:

I think there's a routing problem to these machines that is
independent of the AAAA problem from one ASes I see:

% dig @charon.mpw.net AAAA gwise.ci.mil.wi.us
(ANSWER: 0)
% dig @charon.mpw.net A gwise.ci.mil.wi.us
gwise.ci.mil.wi.us. 0 IN A 216.54.131.198
gwise.ci.mil.wi.us. 0 IN A 216.56.88.101

and from two others:

% dig @charon.mpw.net AAAA gwise.ci.mil.wi.us
timed out
% dig @charon.mpw.net A gwise.ci.mil.wi.us
timed out

The AAAA problem seems to be elsewhere.

David.
Help again please Fwd: please fix your broken DNS server [ In reply to ]
On 9-jul-2005, at 0:01, David Malone wrote:

> I think there's a routing problem to these machines that is
> independent of the AAAA problem

Hm, I now don't get answers for A queries any more, but I did before:

% dig @charon.mpw.net A gwise.ci.mil.wi.us
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62014
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 2

Maybe someone should print these messages and send them to mil.wi by
regular mail...
Help again please Fwd: please fix your broken DNS server [ In reply to ]
>>Thus we're getting an SOA for 'ci.mil.wi.us' when we asked for a AAAA of
>>gwise.ci.mil.wi.us. I suspect this is what's causing SERVFAIL's of every
>>server trying to track down the AAAAs, including the SERVFAILs from
>>itmddnsYx.milwaukee.gov.
>>
>>
>
>The respones looks correct to me - we asked for AAAA records for
>gwise.ci.mil.wi.us and the server said that there were 0 records
>of that type and points us at ci.mil.wi.us as being authorititive.
>AFAIK, that's a perfectly reasonable thing to do.
>
>
Caching resolvers that query itmddnsYx.milwaukee.gov for A records of
gwise.ci.mil.wi.us will receive NS records pointing at
lpitmd-ispX.mpw.net, and will cache this. On subsequent queries for AAAA
requests for gwise.ci.mil.wi.us, they will query lpitmd-ispX directly,
and receive the SOA record with a label of ci.mil.wi.us. This is
inconsistent, as the NS records would indicate that gwise.ci.mil.wi.us
should be a zone apex (and lpitmd should have the SOA for gwise, not
giving us an SOA for ci.mil). I suspect this is what is causing the
SERVFAILs to be generated (by the resolvers).

$ dig gwise.ci.mil.wi.us a @itmddns1x.milwaukee.gov

; <<>> DiG 9.2.4 <<>> gwise.ci.mil.wi.us a @itmddns1x.milwaukee.gov
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31628
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;gwise.ci.mil.wi.us. IN A

;; ANSWER SECTION:
gwise.ci.mil.wi.us. 0 IN A 216.54.131.198
gwise.ci.mil.wi.us. 0 IN A 216.56.88.101

;; AUTHORITY SECTION:
gwise.ci.mil.wi.us. 60 IN NS lpitmd-isp1.mpw.net.
gwise.ci.mil.wi.us. 60 IN NS lpitmd-isp2.mpw.net.

It definitely seems like some sort of DNS load balancing is causing an
inconsistent presentation of the service.

A dig +trace aaaa gwise.ci.mil.wi.us demonstrates this nicely. Note that
+trace will fall back to an 'a' query when it doesn't get an answer for
AAAA, as it does when querying @itmddns1x.milwaukee.gov.

-Kevin